Spotify has written publicly about the pain of vector search at scale. Their recommendation engine performs real-time similarity search across millions of tracks, podcasts, and user embeddings to power Discover Weekly, Daily Mix, search-as-you-type, and algorithmic playlists. With 600 million monthly active users generating billions of daily recommendation requests, every millisecond of vector search latency compounds into infrastructure costs, degraded user experience, and lost engagement. In-process HNSW at 0.0015ms per lookup changes the equation entirely.
The Scale of Spotify’s Recommendation Problem
Spotify’s catalog contains over 100 million tracks, 6 million podcast titles, and 600 million user profiles — each represented as high-dimensional embeddings. When a user opens the app, Spotify must assemble a personalized home screen in under 200ms. That home screen draws from multiple recommendation models: collaborative filtering for Discover Weekly, content-based similarity for radio stations, session-based models for real-time “what to play next” suggestions, and semantic search for the search bar. Each model needs embedding similarity lookups. A single home screen render can trigger 10–30 vector search queries across different embedding spaces.
Spotify has publicly discussed their use of approximate nearest neighbor (ANN) search through their open-source library Annoy and, more recently, Voyager. These libraries were born from the frustration of network-based vector search being too slow for their latency requirements. But even with optimized libraries, the architecture still involves network hops to centralized vector indices or distributed vector databases when the index does not fit in a single process.
Where the Latency Hides
A typical Spotify recommendation request follows this path: the recommendation service receives a user ID, fetches the user’s embedding from a feature store, then queries a vector index for the top-K most similar items (tracks, artists, or podcasts). The vector index query itself — the actual HNSW graph traversal — takes single-digit milliseconds on a well-tuned local index. But the network round-trip to a centralized vector service adds 2–10ms. Serialization and deserialization of high-dimensional vectors adds another 1–3ms. Connection pooling overhead, load balancer hops, and tail latency from shared infrastructure push p99 latency to 15–25ms per vector query.
At 10 vector queries per recommendation request, that is 100–250ms just for vector search — already exceeding the total latency budget before any ranking, filtering, or business logic runs.
Current: Network vector search per recommendation request
Multiply that by 10 models running in parallel — with shared infrastructure contention — and the real-world p99 for a complete home screen recommendation easily hits 80–150ms. That is before the client renders anything. Users feel it. Engagement suffers.
In-Process HNSW at 0.0015ms: The Architecture Shift
The insight is that Spotify’s embeddings follow an extreme power-law distribution. The top 500,000 tracks account for the vast majority of all listens. The top 100,000 artists represent nearly all discovery interactions. Active user profiles — the users who are actually online and making requests — are a fraction of the total user base at any given moment. Trending tracks, popular artists, and active user embeddings form a hot set that is surprisingly compact.
An in-process HNSW index loaded with this hot set eliminates the network entirely. Cachee’s L1 vector search delivers nearest-neighbor lookups in 0.0015ms — 1.5 microseconds. That is not a typo. For a recommendation request that needs 10 embedding lookups, the total vector search time drops from 50–100ms (network) to 0.015ms (in-process). The long tail of cold embeddings — obscure tracks, inactive users — falls through to the network vector database as an L2 fallback.
With in-process L1 HNSW (hot embeddings cached)
The Memory Math
The feasibility of in-process HNSW depends on whether the hot set fits in memory. Spotify uses 128–256 dimensional embeddings for most recommendation models. At 256 dimensions and 4 bytes per float, each embedding is 1KB. The top 500,000 tracks = 500MB. The top 100,000 artists = 100MB. Active user embeddings for 10 million concurrent users = 10GB. Total L1 footprint: approximately 11GB — easily within the memory of a modern recommendation server.
The HNSW graph structure adds approximately 30% overhead for neighbor lists and metadata. Even with this overhead, the entire hot index fits comfortably on a single node. Cachee’s predictive warming layer continuously updates the hot set based on time-of-day listening patterns, trending tracks, and user session starts, ensuring the L1 index always reflects current demand.
Recommendations Before the UI Animation Completes
A typical mobile app screen transition animation takes 300ms. With network vector search at 80–150ms per recommendation set, the data arrives after the animation starts — the user sees a loading skeleton, a spinner, or a stale cached page. With in-process HNSW, the complete recommendation pipeline runs in under 5ms. The data is ready before the animation’s first frame renders. The user sees a fully personalized home screen that appears to load instantly.
This is not a marginal improvement. It is the difference between a recommendation system that feels like it is reacting to you and one that feels like it is predicting you. Spotify’s own research has shown that recommendation relevance decays rapidly with latency — a recommendation served 200ms late is measurably less likely to be played than one served instantly. For Discover Weekly, Daily Mix, and search-as-you-type, the speed of the vector search layer directly impacts whether users engage with the recommendation or scroll past it.
At Spotify’s scale, even a 1% improvement in recommendation engagement translates into millions of additional streams per day, higher Premium conversion rates, and stronger retention. The path to that improvement runs directly through vector search latency.
Related Reading
Also Read
Serve Recommendations Before the Animation Completes.
In-process HNSW at 0.0015ms per lookup — 6,600x faster than network vector search. See it on your embeddings.
Start Free Trial Schedule Demo