I am a big time weeb. Having watched lots of anime, it's hard to find new, good ones. Everybody on Reddit recommends the same old titles, the ones I've already watched. MyAnimeList has a collaborative recommendation system for each anime, but instead of looking at each anime's recommendations, I'd rather have it based on my existing animelist. Recommendation systems have always piqued my curiosity, so it was also time to quench it.
Architecture
Recommendation systems usually come in two varieties: collaborative and content-based. Collaborative recommenders take knowledge of user's past actions as well as the ones made by other users. Content based recommenders on the other hand, utilize the information and features of an item to filter other items.
Each kind has its tradeoffs: collaborative recommenders suffer from cold start issues, which cause new items (which haven't been used) to not occur in any recommendations, and content-based recommenders are limited in scope, since they can only make recommendations that are similar to the initial seed.
I chose to go forward with content based systems, since user-anime interactions dataset is hard to come by (as I figured later, anime dataset was also hard to come by). I chose a two step recomendation process, comprised of retrieval and ranking.
The retrieval is based on a two tower model. The anime tower will encode the anime's features, and the user tower will encode the animelist's features. The encoding will be performed by an embedding model (bge-small-en-v1.5) in a shared vector space. Cosine similarity will be used to measure the relevance between a userlist's average vector and an anime.
The ranking is heurestic-based. The candidate set fetched by the retriever will be scored according to domain informed heurestics like popularity, MAL score, relation and so on. The ranker's job is to boost items based on their metadata, which vector search often misses.
Implementation
Data Collection
I needed an anime dataset that:
- had all the features I wanted
- was reasonably new
- was reasonably large
I first searched Kaggle and GitHub for existing datasets, but they were either small, outdated or didn't have enough information that I needed for the recommendation engine. The largest dataset I found had 23k rows, but was cutoff at 2023 (some of my favorite anime have been more recent).
It didn't take me long to realize that I'd have to collect the data myself. So, I set off to write a scraper. The problem was, no service offered bulk anime data, nor did any service provide a way to get all anime IDs. And, MAL ID generation is totally inconsistent. Although the IDs are numeric, they aren't consecutive. So I decided to scan the entire range of 1-65000.
I initially tried the Jikan API because MyAnimeList API totally sucks (don't ask me how I know), but it was rate limited at 60 reqs/min. On a single machine, it would have taken minimum 18+ hours (65000/(60*60)) to scan the entire range. And that's considering all the requests succeed.
I provisioned 4 cloud VMs (4$/mo 512mb ram each) and implemented interval splitting for the scan range. That should theoretically cut the total time by 4x.
I setup the scraper on each VM and left the process running on them and came back to it 2 hours later. Turns out the process was not running anymore, and it stopped after processing barely 800 rows. I restarted it and checked again an hour later. Again, the scraper stopped at 1000 rows.
My assumption was correct, the OOM killer kept killing the process (verified using dmesg). I optimized the memory by not building entire pandas dataframes in memory, and writing batches in append mode to the CSV file. I also created a systemd service unit that would automatically restart the service on failure.
But it was still slow. I had used a library named animec earlier and also contributed to it. It utilized web scraping to fetch MAL data, but it was missing functionality. So I patched it up match the API response structure. Locally, it was 7x faster. I imagine it is easier on MAL side too cuz HTML is easy to serve via CDNs.
I deployed the update scraper. Got the VM IP blacklisted within minutes. All requests were failing. It was glorious. I suppose I should've used rotating proxies, user agents or some other anti-anti-scraping measures to avoid getting blacklisted. Reverted back to Jikan and let it run overnight.
Checked in 12 hours later -> everything finished. So I downloaded the data, merged the shards, and performed deduplication and sorting. Cost a grand total of 0.60$. Cloud computing yayy!
The resulting dataset is 30440 rows long. 23 features. The biggest anime dataset yet (that I was able to find). It exists on Kaggle with all its glory (aka null values). The cleaned dataset exists on GitHub.
Embedding Generation
Half the battle was over. The next task was to generate embeddings for the entire dataset, and store them in a vector database. I chose qdrant, primarly because it was easy to run locally with a docker image, and it was quite feature-rich. I chose the bge-small-en-v1.5 model because it was fast and small to run locally. I also chose the fastembed library to run the model, also developed my qdrant, because it uses a fast ONNX model runtime.
These are the techniques I employed to enhance the speed and throughput of embedding generation:
-
Using a small embedding model (output dimensions are just 384)
-
Using batched inference (with a
batch_sizeof 256) -
Using all the cores (
parallelset to 0) -
Using quantization (supported out of the box in qdrant), specifically Turboquant, a quantization technique recently developed by Google that compresses the embeddings upto 32x with near zero recall drop
-
Disabling the HNSW index creation (
mset to 0) during the ingestion process
Qdrant had a really good documentation, and the guides cover numerous real world tasks. All these techniques mentioned above are written in Qdrant docs. (link1 link2 link3)
The text features of the anime were chosen for the embedding content. Numeric features were skipped as they don't contribute much to the vector search.
def _row_to_doc(row: dict) -> str:
parts = [
("Title", row.get("title", "")),
("Alt title (EN)", row.get("alt_title_en", "")),
("Alt title (JP)", row.get("alt_title_jp", "")),
("Synopsis", row.get("synopsis", "")),
("Genres", row.get("genres", "")),
("Themes", row.get("themes", "")),
("Demographics", row.get("demographics", "")),
("Studios", row.get("studios", "")),
("Type", row.get("media_type", "")),
("Status", row.get("status", "")),
("Rating", row.get("rating", "")),
]
lines = [f"{label}: {value}" for label, value in parts if value]
return "\n".join(lines)Apart from the embeddings, the anime metadata is attached as payload, which helps in metadata filtering during searches.
After the seeding was finished, I re-enabled Qdrant's HNSW index by setting m to 16.
Retrieval
The retriever takes the user animelist as an input. It grabs all the anime IDs from the userlist, and embeds those anime records from the dataset.
Those embeddings are then averaged into one (the "user tower"). The average is weighted and influenced by the user-given score, progress (number of episodes watched), recency and the anime status (completed/on hold/dropped/watching).
The individual weights are multiplied to produce a final weight instead of adding them, so that only the titles that scored high across all dimensions influence the recommendations.
weights = (
userlist["score_weight"]
* userlist["progress_weight"]
* userlist["recency_weight"]
* userlist["status_weight"]
).to_numpy()Using the averaged vector, a similarity search is performed on the vector DB. The search query has filters for NSFW genres, titles not yet aired, promotional videos/commercial messages/etc. , and against the anime IDs user has already watched.
def search_similar_anime(
self,
vector: list[float],
userlist_ids: list[int],
limit: int,
disable_nsfw: bool,
):
must_not_conditions = [
models.FieldCondition(key="id", match=models.MatchAny(any=userlist_ids)),
models.FieldCondition(key="media_type", match=models.MatchAny(any=["Unknown", "Music", "CM", "PV"])),
models.FieldCondition(key="status", match=models.MatchValue(value="Not yet aired")),
]
if disable_nsfw:
for ng in NSFW_GENRES:
must_not_conditions.append(
models.FieldCondition(
key="genres", match=models.MatchPhrase(phrase=ng)
)
)
return self.client.query_points(
QDRANT_COLLECTION,
query=vector,
with_payload=True,
limit=limit,
query_filter=models.Filter(must_not=must_not_conditions),
)Ranking
The retriever fetches a large candidate set (300 items). The ranker scores the candidates based on certain domain-informed heurestics, and returns the 20 best ones.
The ranking takes in account the cosine similarity score, MAL score of the anime, popularity statistic, release date and related entries. Related entries (sequels, prequels, side stories, etc.) are boosted highly.
# each of these terms are multiplied with their respective terms beforehand
base_score = (
sim_term
+ quality_term
+ popularity_term
+ novelty_term
+ time_term
+ related_term
)After calculating the base scores, the ranker also diversifies the selection by not allowing genre overlap of more than 4 titles. It does so by penalizing the genres whose counts have exceeded the threshold.
Improvements
While the recommendation engine was working fine, I didn't like the vibes from recommendations. I would not watch the titles the recommendation system gave to me. This is how I improved the recommendations.
Latency Drop
While not related to the recommendation quality, I was re-embedding the userlist titles instead of fetching the already generated embeddings from the vector DB. I don't know why I did that, but I fixed it right away. The latency dropped by 95%. This also solved the concern I had regarding deployment of this app. It won't need a GPU to process userlist embeddings, I just need to take a snapshot of the vector DB and migrate it to the cloud VM. This also made testing iterations quick (~48s -> ~2s).
Capturing Distinct Tastes
The retriever finds similar anime by searching the vector space against an "averaged vector". While the average vector is weighted and is influenced by user-liked anime, it doesn't address the concerns of capturing the distinct taste profiles of the user.
Consider this graph (the vector space actually has 384 dimensions, not 2, but bear with me): the user watches both battle shounen and romcoms. No matter the weights, the average vector will lie somewhere between those two taste profiles. That part of the vector space might contain some other titles, which are not close to either battle shounen or romcoms. The average vector lies in a dead zone, or in a part where anime of some other genre exist (which the user might not watch).
The solution I implemented to address this issue is to perform a clustering over the userlist. The userlist will be divided into several clusters, each of which will capture a subset of the anime user watched. Then, that particular subset will be averaged (again, weighted) and a vector search will be performed. This way, we can recommend anime from multiple taste profiles the user might have.
I used the KMeans clustering technique.
k = max(1, min(4, len(normalized_embeddings) // 8))
kmeans = KMeans(k, n_init=10, init="k-means++")
cluster_labels = kmeans.fit_predict(normalized_embeddings)
cluster_centroids = []
for i in range(k):
cluster_mask = cluster_labels == i
cluster_embeds = clustering_embeddings[cluster_mask]
cluster_w = clustering_weights[cluster_mask]
# normalize internal weights for this specific cluster partition
w_sum = cluster_w.sum()
cluster_w = cluster_w / w_sum
weighted_centroid = np.average(cluster_embeds, weights=cluster_w, axis=0)
cluster_centroids.append(weighted_centroid)
for centroid in cluster_centroids:
search_response = self.qdrant_store.search_similar_anime(
centroid.tolist(), userlist_ids, limit_per_cluster, opts.disable_nsfw
)Reducing Franchise Spam
The clustering improved the recommendation quality a ton: I was finally getting recommendations similar to the ones I've watched, and the ones I might actually watch. But it introduced another problem: franchise spam.
I've watched all the non-filler content of Attack On Titan and rated them highly. That led to half of my recomendations (12 out of 20) filled with AoT filler content like movies and OVAs. This was undesirable.
The solution seemed obvious: the ranker needs to diversify the selection at franchise level just like it was doing already with genres. We need to group together titles of a particular franchise and assign them a franchise_id that uniquely identifies all the titles. We can then filter repetition by tracking the franchise_id. But another problem arises: how do you know that two particular anime belong to the same franchise? Text search is flaky, and might lead to high false negatives.
Enter related_entries. This field has already existed in the dataset. This is essentially a list of titles that are related to the current titles. The relation might be one of sequel, prequel, side_story, etc.
(the illustration only has single direct edges, the actual dataset has reverse directed edges as well (i.e., prequel-sequel edges both exist between nodes))
This is essentially a connected components problem. The relations are edges between two titles. We have a graph in our hand. We need to isolate the subgraphs (components) and assign all the nodes within that subgraph a particular franchise_id. I chose to go with a DSU (disjoint set union) approach.
for idx, relation_list in df["related_anime"].dropna().items():
for relation_str in relation_list.split(";"):
relation_str = relation_str.strip()
splitted = relation_str.split("|")
if len(splitted) != 3:
continue
id_, title, relation = splitted
try:
dsu.union(ids[idx], int(id_))
except KeyError:
continue
df["franchise_id"] = df["id"].apply(dsu.find)Then, I just needed the update the vector DB with this new payload and add penalty for franchise repetition in the ranker. It was fun implementing a leetcode solution in a real world problem.
Random trivia: There are 15743 unique franchise in my dataset (roughly half).
Conclusion
I am quite happy with the current implementation and satisfied with the current recomendation quality. I might train it on user-anime interactions later to improve the ranker. I also need to experiment with the weights for retriever to find the optimal balance.
I don't have enough cloud credits to deploy this project, but you can try it out by cloning the repo and running the server by yourself. All the code is present in this repo.
Now, back to watching anime.