How Glean Can Make Enterprise RAG Search 10x Faster With In-Process Vector Caching

Glean built its entire company around one promise: search everything in your enterprise instantly. Slack messages, Google Docs, Jira tickets, Confluence pages, emails, Notion wikis — one search bar, one answer. The product is enterprise RAG at scale. Every query that enters that search bar triggers an embedding, a vector search across millions of document chunks, context assembly, and LLM generation. The vector search step — typically 1–5ms per lookup, with 3–5 lookups per query — is the silent bottleneck standing between Glean’s promise and a user experience that feels truly instant.

The Bottleneck Hiding in Plain Sight

Glean indexes an enterprise’s entire knowledge graph. For a mid-size company, that means 50–200 million document chunks, each embedded as a high-dimensional vector and stored in a vector database. When a user types “What was the Q4 revenue decision from the board meeting?”, Glean’s pipeline does the following: embed the query into a vector, search the index for the top-k most relevant document chunks (typically 3–5 lookups across different collections — Slack, Docs, email), assemble those chunks as context, and pass them to an LLM for answer generation.

The LLM generation step takes 500ms–2s and is largely irreducible without model changes. But the vector search step — the retrieval in Retrieval-Augmented Generation — is where Glean controls its own destiny. Each vector lookup against an external database like Pinecone, Weaviate, or a managed Elasticsearch instance incurs 1–5ms of round-trip latency. Five lookups across different data source collections means 5–25ms before the context assembly even begins. That is 5–25ms of dead time where the user sees nothing — no streaming tokens, no partial results, just a spinner.

Why Search Speed IS the Product

For most SaaS companies, latency is a performance metric. For Glean, latency is the product itself. Enterprise search lives or dies on perceived speed. Google trained every knowledge worker on the planet to expect search results in under 200ms. When Glean’s answer takes 2.5 seconds instead of 1.8 seconds, users don’t analyze the pipeline breakdown — they feel the lag and start opening tabs manually. Every millisecond Glean shaves from the retrieval step compounds directly into user satisfaction, adoption rates, and renewal conversations with enterprise buyers.

The competitive landscape makes this existential. Microsoft Copilot is embedded in every Office 365 deployment. Google Vertex AI Search has native access to Workspace data. Both have the advantage of zero-hop access to their own document stores. Glean, as a third-party connector, must overcome the inherent latency penalty of indexing data it does not natively own. The vector search step is where that penalty materializes — and where in-process caching eliminates it.

The Hot Set Architecture

Not all 200 million document embeddings need to live in process. Enterprise search follows a steep power law: the top 10 million vectors cover approximately 95% of all queries. These are the recently edited documents, the frequently referenced Confluence pages, the Slack channels with daily activity, the Jira epics under active development. This is the hot set.

The architecture is a two-tier retrieval system. Tier 1 is an in-process HNSW index (via Cachee’s VADD and VSEARCH commands) holding the hot 10M vectors in the application’s own memory space. Tier 2 is the existing vector database holding the full index. Every query hits L1 first. On a hit — which happens 95% of the time — retrieval completes in 0.0015ms per lookup. On the rare miss (stale documents, archived content, rarely accessed pages), the query falls through to the vector DB at standard latency, and the result gets promoted into L1.

The impact is stark. Total pre-LLM latency drops from 5.8–25.8ms to 0.81ms. That is a 7x–32x reduction in the retrieval phase. The user perceives faster time-to-first-token because the LLM starts generating sooner. For streaming responses, this means tokens begin appearing 5–25ms earlier — a difference that feels immediate at human perception scale.

Memory Economics at Enterprise Scale

The natural objection: 10 million vectors in process sounds expensive. It is not. At 768 dimensions (standard for enterprise embedding models like text-embedding-3-large) with float32 precision, each vector consumes 3,072 bytes. Ten million vectors require 30.7 GB of RAM — well within the capacity of a single modern server instance. With quantized int8 vectors (which HNSW supports with minimal recall loss), that drops to 7.7 GB. Glean already runs substantial infrastructure per enterprise tenant. An additional 8–31 GB of RAM per deployment is a rounding error against the compute cost of the LLM inference they are already paying for.

The ROI calculation is straightforward. If Glean processes 500,000 queries per day for a large enterprise tenant, and each query saves 10ms of vector search latency, that is 5,000 seconds of cumulative latency eliminated daily. More importantly, every individual user experiences faster results on 95% of their searches. That is the difference between “Glean is fast” and “Glean is instant.”

Predictive Warming and Real-Time Sync

The hot set is not static. Document relevance shifts throughout the day. A Slack thread about a production incident becomes the hottest content in the index within minutes. Glean already tracks document access patterns, edit timestamps, and cross-reference frequency. This metadata feeds directly into a cache warming strategy: newly edited documents get their embeddings promoted to L1 within seconds. Documents that have not been accessed in 72 hours get evicted. The cache stays warm without manual intervention.

For Glean, this creates a feedback loop. The more a document is searched, the faster it is found. The faster it is found, the more users rely on Glean instead of navigating to the source application. The more users rely on Glean, the more search data feeds the warming model. In-process vector caching does not just accelerate Glean’s search — it deepens the product moat.

The Competitive Calculus

Microsoft Copilot has zero-hop access to SharePoint, OneDrive, and Exchange. Google Vertex AI Search has zero-hop access to Drive and Gmail. Glean connects to these systems over APIs, indexes asynchronously, and searches its own copy. That architectural tax is permanent. But the retrieval latency tax is not. With in-process HNSW, Glean can deliver search results on the hot set faster than Copilot or Vertex can query their own native stores through standard database paths. The connector disadvantage becomes a caching advantage — because Glean controls its own index format and can optimize the read path in ways that platform-native search cannot.

At 0.0075ms for 5 lookups, Glean could deliver contextual snippets before the user finishes typing. Autocomplete becomes answer-complete. That is not a performance optimization. That is a product category shift.

Related Reading

• AI Infrastructure Solutions
• Vector Search: In-Process HNSW
• Cachee Pricing
• Start Free Trial
• How Cachee Works

The Numbers That Matter

Cache performance discussions get philosophical fast. Here are the actual measured numbers from production deployments running on documented hardware, so you can compare against your own infrastructure instead of trusting marketing copy.

• L0 hot path GET: 28.9 nanoseconds on Apple M4 Max, single-threaded against pre-warmed in-memory cache. This is the floor — there's no faster way to read a key.
• L1 CacheeLFU GET: ~89 nanoseconds on AWS Graviton4 (c8g.metal-48xl). Sharded DashMap with admission filtering.
• Sustained throughput: 32 million ops/sec single-threaded on M4 Max, 7.41 million ops/sec at 16 workers on Graviton4 c8g.16xlarge.
• L2 fallback: Sub-millisecond hits against ElastiCache Redis 7.4 over same-AZ network when L1 misses cascade through.

The compounding effect matters more than any single number. A 28-nanosecond L0 hit means your application spends almost zero time on cache lookups in the hot path, leaving the CPU free for the actual business logic that generates revenue.

Average Latency Hides The Real Story

Average latency is the most misleading number in cache benchmarking. The percentile distribution is what actually breaks production systems. Tail latency — the slowest 0.1% of requests — is where users notice the lag and where SLAs get violated.

Percentile	Network Redis (same-AZ)	In-process L0
p50	~85 microseconds	28.9 nanoseconds
p95	~140 microseconds	~45 nanoseconds
p99	~280 microseconds	~80 nanoseconds
p99.9	~1.2 milliseconds	~150 nanoseconds

The p99.9 spike on networked Redis isn't a bug — it's the cost of running a single-threaded event loop that occasionally blocks on background tasks like RDB snapshots, AOF rewrites, and expired-key sweeps. Cachee's L0 stays inside a few hundred nanoseconds because the hot-path read is a lock-free shard lookup with no background work scheduled on the same thread.

If your application is sensitive to tail latency — payments, real-time bidding, fraud detection, trading — the p99.9 number is the one to optimize against. Average latency improvements that don't move the tail are vanity metrics.

Memory Efficiency Is The Hidden Cost Lever

Throughput numbers get the headlines but memory efficiency determines your monthly bill. A cache that stores the same hot data in less RAM lets you run a smaller instance class — and on AWS that's the difference between profitable and breakeven for a lot of services.

Redis stores each key as a Simple Dynamic String with 16 bytes of header overhead, plus dictEntry pointers in the main hashtable, plus embedded TTL metadata. For 1KB values, per-entry overhead lands around 1100-1200 bytes once you account for hashtable load factor and slab fragmentation. At a million keys, that's roughly 1.2 GB of resident memory just for the data.

Cachee's L1 layer uses sharded DashMap entries with compact packing — a 64-bit key hash, value bytes, an 8-byte expiry timestamp, and a small frequency counter for the CacheeLFU admission filter. Per-entry overhead lands at roughly 40 bytes of structural data on top of the value itself. For the same million-key workload, that's about 13% smaller resident memory. On AWS ElastiCache pricing, that gap is the difference between needing a cache.r7g.large versus a cache.r7g.xlarge for borderline workloads.

Observability And What To Measure

You can't tune what you can't measure. The four metrics that matter for any production cache deployment, in order of importance:

• Hit rate, broken down by key prefix or namespace. A global hit rate of 92% sounds great until you discover that one critical namespace is sitting at 40% and dragging your tail latency. Per-prefix hit rates expose which workloads are getting cache value and which aren't.
• Latency percentiles, not averages. p50, p95, p99, and p99.9 for both cache hits and cache misses. The cache miss latency is your fallback path performance — when the cache fails, this is what your users actually experience.
• Memory pressure and eviction rate. If your eviction rate is climbing while your hit rate stays flat, you're under-provisioned. If both are climbing, your access pattern shifted and you need to retune TTLs or rethink what you're caching.
• Stale-read rate. The percentage of cache hits that returned a value the application then discovered was stale. This is the canary for your invalidation strategy. If it's above 1%, your invalidation logic has a bug.

Cachee exposes all four out of the box via Prometheus metrics on the standard scrape endpoint, plus a real-time SSE stream for dashboards that need sub-second visibility. The right time to wire these into your monitoring stack is before the migration, not after the first incident.