Yesterday TVLens got faster. Today we made it leaner.
The catalog in TVLens is smaller. For now, we set the vote_count floor to 100, which means the shows we ingest must have at least 100 votes. We apply it at the TMDb API, so those shows are never fetched at all. We also pruned the shows already in the catalog: 464 down to 248, and 487,000 episodes down to 164,176. Eventually we will lower the floor, but we will do that incrementally because TV show fans may enjoy less popular shows. The reason for the prune is that 124 of those 464 shows, over a quarter of the catalog, accounted for 82% of all the episodes. One thing we made sure of: a show the user has rated is never pruned.
We added a permutation test, let me explain why. TVLens connects shows through the people who make them. Some connections are cast and some are crew. If the shows I rate highly tend to be linked by shared actors rather than shared crew, maybe I prefer one kind of connection over the other. Preferring one over the other is what we call lean.
The problem is that every user has some lean by chance. Split any set of ratings by connection type and the two averages will differ by something. So the test takes my own ratings, deals them out to different shows at random, and recomputes the lean. It does that 200 times. If the real lean does not beat 95% of its own shuffles, it is arithmetic rather than an opinion, and TVLens says nothing.
We also made Show.id and Show.tmdb_id equal. Before today, Show.id was unique to one machine. That worked because I am only testing TVLens locally, but as we get to the end of version 1 we need it to be consistent for all users.
There were a few other technical changes:
- Dropped four unused TMDb endpoints
- Split runtime from dev dependencies
- Moved to Django 6.1