Anime Rating Estimation & Leakage Analysis
What I did
I explored anime metadata and community ratings, then extended the coursework with a reproducible, explicitly AI-assisted evaluation and leakage audit.
CS222 coursework: October–December 2025 · Later reproducibility extension
My contribution
Course-project author; substantial Codex assistance in the later refactor
- Developed the original questions, exploratory analysis, notebooks, and course report.
- Extended the project with Codex-assisted leakage checks and an auditable evaluation pipeline.
- Documented model comparisons, error patterns, provenance, and limits.
Outcome
Selected model: RMSE 0.4611 and R² 0.6233 on 1,943 held-out records; 38.6% lower RMSE than the mean baseline.
Starting from an interest in anime
My CS222 project began with questions about which anime attract attention and how metadata relates to community ratings. I analyzed a dataset of 10,000 titles, explored genre and theme patterns, constructed a weighted ranking, and compared regression models. The original project included notebooks, visualizations, a proposal, and a technical report.
A later review changed the emphasis. The dataset’s rating rank was nearly a direct ordering of the target score. Using it to estimate that same score made results appear much stronger than a legitimate prediction setting would support. Missing target values also needed to be excluded rather than filled in as if their ratings were known.
What changed in the evaluation
The revised workflow separates 286 records without observed labels, fits preprocessing inside training folds, and compares a training-mean baseline, Ridge, Random Forest, and CatBoost. Model-family settings and model selection use development cross-validation. The final internal holdout contains 1,943 records.
The revision was developed with substantial OpenAI Codex assistance, including the package refactor, evaluation protocol, tests, plots, and documentation. I distinguish that assistance from my original coursework. This project demonstrates an evaluation case study and reproducible workflow; it is not a new learning algorithm or entirely unaided implementation.
The result that matters
| Experiment | Held-out RMSE | Held-out R² |
|---|---|---|
| Training-mean baseline | 0.7513 | approximately 0 |
| Metadata-only CatBoost | 0.6086 | 0.3437 |
| Selected metadata + engagement CatBoost | 0.4611 | 0.6233 |
| Invalid rank-leakage control | 0.1833 | 0.9405 |
The last row is deliberately marked invalid for the intended prediction setting. It demonstrates leakage rather than the project’s best valid result. R² is a regression measure, not classification accuracy. The selected model reduces RMSE by 38.6% relative to the mean baseline on the same held-out rows.
Beyond one average number
The diagnostic plots show why an average error does not tell the whole story. Predictions tend to overestimate lower ratings and underestimate higher ratings. Titles with scores at or below 6 have larger errors in this snapshot. Feature importance describes model reliance, not a causal explanation of audience preferences.
Scope, reproducibility, and data access
The source is the Kaggle TOP ANIME DATA 2025 coursework dataset. The random holdout samples the same historical snapshot that was explored in the original coursework. There is no independent time-separated or franchise-separated benchmark. The workflow estimates observed ratings; it does not establish pre-release forecasting or personalized recommendations.
The release records a data hash, split rows, source hashes, configuration, and experiment outputs. Automated synthetic-data checks test mechanics, not the real-data findings. Raw records, individual predictions, serialized models, and the source CSV are not distributed on this site because data redistribution status remains unconfirmed.
Code and the technical report are available on request: contact me.