[Feature] Use Notebook Context to Improve Clustering

[Harsh16gupta]

Right now we only use the note's body and title for clustering. The notebook (parent_id) is fetched but not used.

[Harsh16gupta]

I explored 3 ways to do this:

Option 1: One-Hot Encoding
Turn notebook IDs into small vectors, give them a weight (like 15%), and concatenate with the note embedding before clustering.

Problem: Adds extra sparse dimensions that can warp UMAP and mess up silhouette scores.

Option 2: Two-Stage Post-Processing
Cluster notes purely by text content first. Then check which notebooks the notes in each cluster came from.

Example: HDBSCAN creates a cluster with 8 "Recipes" notes + 1 "Work" note. That "Work" note is actually a recipe that was misplaced → we detect it and suggest moving it.

Option 3: Embed & Blend (same as title)
We already blend the title into the final vector using similarity-based weighting. Do the same with the notebook name - embed it and blend it in.

Problem: Notebook names are usually 1–2 words ("Food", "Work"). The embedding model works best on sentences, not single words. The similarity scores come out too low (~0.25–0.35) to have any real effect on clustering.

Why I think Option 2 is best
Option 1 adds noise to the vector space. Option 3 barely does anything because short notebook names produce weak embeddings.
Option 2 keeps the embedding space clean (384D, no extra dimensions). The clustering stays purely semantic.
It also unlocks a bonus feature: misplaced note detection. We can tell the user, "This note is in 'Work' but belongs with your recipes. Move it?"
I also checked how it scales - the post-processing is O(N) and adds <50ms even for 2000 notes.