You ever look up at the stars and marvel at being here as the current version of you rather than the near infinite alternate versions that could have been? Yeah me neither. And for good reason; it turns out that we’re so predictable that any other versions would probably be doing similar stuff. On a related note, if we look at the aggregate of questions we pass to the standard AI assistant interface, the distribution of those questions appears astoundingly similar to a rank-frequency log normal distribution. That means if you send each one of those questions to an LLM, you are leaving money on the table.
In 2024 Databricks published a semantic caching accelerator that stored cached questions in a
Vector Search index. This works great, but it was made before we had Lakebase, and a cache is naturally an OLTP workload: we want to write a row on every miss, delete rows on expiry, and filter by tenant and/or model before the ANN search. These features make pgvector on Lakebase a more compelling cache. Let’s see it in action.
Imagine we have a customer-facing AI assistant in our client application. In the cacheless paradigm, the question goes into the application, maybe gets processed, and then gets handed to the LLM to return an answer to the customer. You pay for the FM API call every time. Now imagine we add a semantic cache. The question gets embedded and looked up in the pgvector cache, and if a high quality match is found, the cached answer is returned without invoking the LLM. If the threshold is not met, the question goes to the LLM and its answer is written back to the cache for future questions. Our cache is self-populating.
The semantic match threshold serves as our main tuning parameter between high cost and latency savings on the one side and risking higher incorrect answers on the other. We have a few possible error paths here:
The threshold is met, but the wrong answer is returned from the cache
i.e. it should have picked a different cached answer
The threshold is met, but the correct answer wasn’t in the cache
i.e. it should have called the LLM instead
The threshold is met, but the answer saved was factually inaccurate
i.e. we need to better curate the cache
The threshold is not met, and the LLM returned the incorrect answer
i.e. we should have had more guardrails, better system context, etc.
It’s possible the correct answer was in the cache, but not triggered
From my experience in building this, the threshold sweet spot is somewhere between 0.7 and 0.8. There is an inflection point in this range where the repeats caught drop dramatically, but so do the percent of wrong answers from the cache, regardless of the specific cache failure path. Luckily for us, we don’t need the repeat questions caught to reach a very high percentage of total asked for this to be economically advantageous, because the embedding model and Lakebase instance are both so cheap that intercepting even low single digit percentages of questions makes it worth it at high request volumes.
A big question for enterprises considering a semantic cache is how to know what the tradeoff numbers look like before going to prod. That’s where a labeled evaluation set and an LLM judge come in. The judge takes a sample of hits the cache actually served, generates a fresh answer to each question, and decides whether the cached answer is correct. Before trusting it, check it against your labels. The verdicts land in a Delta table, and a Spark Declarative Pipeline turns them into a false-hit monitor compared against a limit you set. It reports OK, WATCH or ALERT and recommends an action, such as raising the threshold or not caching the categories causing wrong hits. You can run the judge from the app, or you could put it on a job and schedule it.
Usability is also important for real world applications, so we also create a Genie Space over these tables that we can ask questions over in plain text, in realtime.
FAQ
What if they ask user specific questions like account balance?
Partition with scope keys and send questions involving live user data around the cache before lookup
Include probes like these in your eval set so you’ll see this behavior in testing
Why pgvector? Why not Lakebase Search?
Our project’s pgvector backend uses HNSW, Lakebase Search uses IVF
If the cache is large and loaded in bulk, then you can afford to reindex on a schedule, but IVF clusters cannot be incrementally updated
What if the answer to a question changes, won’t I have to invalidate the cache to fix it?
The system prompt’s hash is part of every cache key, so editing the fact sheet means old answers are never served again, with nothing to invalidate
You can also set a TTL for cached items
How fast are hits?
About 10x faster than misses; factor in about 150-200ms for the embedding call and maybe 10ms for the Lakebase lookup
What does cache maintenance look like?
This is one of the most important tunable features here, because more cache entries means more neighbors, and probably higher rates of the wrong answer being returned
You could use an async, batch process to leverage an LLM judge to periodically cull closely related semantic items and keep only the best, most popular ones
What other tuning params are there?
Threshold is the most important by far
hnsw.ef_search is probably a good one to explore when sweeping your eval set during configuration
The embedding model itself is also a swappable component
Does this come with any guarantees or SLAs?
No, this is purely field originated project, so for now it comes with no performance, functionality, or version guarantees
If you make a PR or raise an issue in the repo I will get to it on a best effort basis
Repo here. Cheers and happy coding,
-Austin