I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.
This is so similar to human decision-making though. First I my innate experience to approximate to what I expect is right, then I map that to the truth.
It's the same for so many things:
- reading documentation (what do I expect this function to be called?)
- finding clothes in a shop (something long-sleeved and light)
- picking the fridge for dinner
- finding a book in the library...
so many analogues where I'm not coming cold to a choice.
I can't believe people spend their lives finding lazier ways to classify a bunch of objects that will end up heaped in dormitory dumpsters across the US next spring.
> On two occasions I have been asked, – "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
He clearly didn't know enough about vector embeddings.
I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic, an LLM running on a CPU with greedy decoding would be 100% deterministic and still have the intelligence we associate with LLMs
Valid point, it’s why I included the word “traditional”, to try to qualify that.
What I meant is something more like explicitly programmed vs. learned. Intelligence can result from learned behavior, but not from explicit programming of rules by humans.
An aspect of this is that “learning” is unpredictable - we can’t predict in advance exactly how the resulting model will behave, except broadly. It seems non-deterministic if only by virtue of its complexity, which is beyond anything we’re able to predictively model.
> I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic ...
Because LLMs are artificial neural networks[0] (ANN), which are statistical in nature, and thus intrinsically non-deterministic. Pretty much every AI algorithm has randomness involved in its definition and many (most?) incorporate probabilities.
Now that you mention it, I think "statistical" might be a good word choice as it probably closely matches what people mean when they say an LLM is nondeterministic. However, on a technical level, neural network inference truly is inherently deterministic. The only nondeterminism in LLMs is parallelism-dependency in the way floating point operations are carried out and in the decoding strategy
only if you bias your "random sampling of the distributions it gives"
Fixing the seed is still intentional bias. Or you could force it to always take the one token with the highest probability, but that is still biased sampling. Deterministic, sure, but intentionally wrong just to avoid a technically
There's nothing inherently biased or intentionally wrong with greedy decoding. Why would there be? Let's say you're trying to predict what an expert doctor would say to a patient with cancer. You think there's a 99% chance the doctor would say "start chemotherapy" and a 1% chance the doctor would say "don't worry about it". If you have to pick one to output, you'll pick "start chemotherapy". It would be crazy to say the best thing to do is to roll a 100-sided die to determine your answer. A language model is literally in that exact situation.
I've seen this debate several times and often there's a terminology issue, where the same word isn't being interpreted the same way by different sides.
Often it's a difference between repeatable versus predictable, or whether a system has chaotic aspects like the configurations of a double-pendulum or weather-forecasting.
Sometimes it's the difference between determinism in-theory versus in-practice, especially when various optimizations are being applied to save money.
I was hoping a lossy semantic reasoning machine would put to bed words such as "right" and "wrong" outside of formal logic, but apparently this fight has just begun
Yeah they solved the main issue I had with Google. "I remember this exact thing, but not exactly how it was phrased..."
Except not as much as I'd like... they often also don't know what the hell I'm talking about, and it still takes them twenty minutes of Googling to find the right page!
The entire problem of search is that the user has the wrong data and wants to use it to receive the correct data. That was the start, not the state we’ve ended up at - it is unironically how we got to LLMs.
If information is totally wrong then all you have to do is invert it to get the truth. What was it that Sherlock Holmes said? The problem ends up being that it often takes a tremendous number of counterexamples to eliminate everything that is impossible.
Worse is when you don't know whether the answers you have are totally wrong.
“Hallucinate” is misleading here. In the given example, a classification is being done very successfully - it’s just that it requires an extra step to map it to an arbitrary predefined list of classifications.
If you can articulate why you think this isn’t a good approach, I’d be interested to hear it.
A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LLM to take a random sample from each clustered group and create a classification for that group.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
The idea of distance thresholding models that are trained to satisfy an ordering constraint is a bit strange. The reason it's hard is that there isn't a threshold!
You can slice and dice it a ton of different ways, but the significance of groups is incidental.
It's a good starting point, but having done this a few times for a few companies it always seems like it needs substantial human review.
Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?
Even better is to search the corpus first with like naive BM25 / embedding search, aggregate over top N to get most representative categories, then have the LLM categorize in that set.
It's basically a variation of HyDE (Hypothetical Document Embeddings), and the rationale is that the embedding of the query is not necessarily close to the embedding of the answer. If you generate a hallucinated answer, it can line up with the actual document better (in the embedding space, via BM25, or hybrid).
But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.
I feel like querying for the smaller distance from the set of embeddings of both the query and a fake answer structure might solve that. If there's something very close to the query, it dominates, otherwise the fake answer is the guide. Plus, why not use an LLM to judge between them if we've got a token-burner all warmed up to start.
> In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
Yes what you're describing is a classic way of doing query understanding.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding search
Using a Nano model, a tad worse than shipping a vocabulary to a larger OpenAI model. (And it’s an huge improvement on not classifying the queries at all).
But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.
I did something similar 10 years ago, but instead of llms I used word2vec to calculate a embeddings of product descriptions and map those to existing categories. The LLM approach is very likely better, but I'm curious what the cost difference is.
Prompt expansion of input to extra categories makes sense if your embedding isn’t working well. But on its own, why use the LLM at all? I think you could have demonstrated the original step first and then shown that it’s useful.
New embedding models support queries, so you don’t need to hallucinate a document before finding the nearest neighbor. Curious how it compares to this approach since you’d get to skip the LLM altogether.
What does it mean that an "embedding model supports queries"? An embedding model maps text to embedding vectors. You can always perform queries with such embedding vectors against a stored set of embeddings.
I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON.
There you go, you just drastically reduced the output pricing.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
Nice. I heard something similar years ago. Instead of doing a RAG search based on the question, first hallucinate a plausible answer, and then use that as the query...
We have this running in production. Can get pretty expensive and slow. We are trying to replace this with cheaper and faster methods that don’t hammer our LLM and elastic search endpoints as much.
That's the whole problem. If you have tons (100s of thousands or more) of labels, then you have "structured" data, but how do you correctly classify that scraped item into the correct label?
Putting all the labels into the LLM is super expensive per call when you have millions of items to classify.
You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.
Smart trick, but assumes the “dumb” llm is smart enough not to derail into an article about the lives of South American red ants. Obvious exaggeration, the point being outcomes should stay strictly within topic, avoid unrelated bloat and hit the target.
I was in a project where we sent the whole taxonomy every request, 40k tokens + one article, ”plz classify”. This was before structured outputs. It was extremely expensive and still hallucinated. Good ol’ days.
Just this week I tried doing something similar with a nasty vibe-coded codebase I was trying to organize. I had Gemini Flash 3.6 classify each function/method in a similar way, giving a few plausible classifications for each (one agent per method).
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.
I’ve read a few different accounts, including OpenAI’s own admission, that Terra Medium or higher will likely produce better results than Luna xhigh and cost about the same or less.
Interesting technique, but even if you're getting rid of hallucinations it seems there's still no guarantee of consistent classifications. If you need to do a semantic (embedding) search anyways, then how does this really help?
They are already solving the problem with search engines, they're just using an LLM as a first pass to create better embeddings to run a similarity match on first. The difference in latency is likely made up for in accuracy.
Because that's structured data and structured data is usually hidden away from users _and_ machines. Product rarely want to be honest, unless it's B2B in a very competitive market (and even then!). So, yeah, it's not that they are bad, it's that there are few good sources of information.
(Lets ignore for now that no one seems to agree to what should be the spec sheets)
It can do that on run time, but it does not store data like that. The data is typically stored as embeddings in which it is hard to query data in a structured form. Example give me all products whose price is less than 200$ vs suggest me products for my spouse's birthday.
Instead: use an LLM to build a large (old-school) database of products with all their specifications. The LLM can also build the schema for that database as it finds more data.
Then use an LLM to query that database based on the user's specifications (+ add some intelligence to find nice suggestions for a birthday if wanted, but I'd consider that an extra).
With agentic commerce protocol / unified commerce protocol open ai and gemini are trying to solve this problem.
The idea is to make structured queries using these protocols which can be used to fetch top products matching the user needs instead of just relying on semantic search.
It gets worse: shopping agents are hostile adversaries to Amazon unless they're paying Amazon and they've agreed to be friendly agents. No agent that won't betray you to an Amazon pricing strategy is going to be allowed access to Amazon structured data. They might even be fed poisoned data to discredit them.
Because the data, in general, is not included in the LLM model and it needs to search/browse for external information. It cannot look indefinitely so it get the top results from lists, not "evrything".
Amazon Rufus has been mildly successful for me. I think the failures I've experienced with it are mostly because the product I'm looking for doesn't exist in the catalog.
It's been an absolute fucking disaster for me. It hallucinates endlessly and its searches are terrible. It even managed to confidently gaslight me about there being a VAT invoice available for a specific product.
I noticed yesterday when browsing on mobile that there used to be a box where I could search reviews and it got swapped with a Rufus box. I guess somebody needs to juice their engagement numbers for an investor briefing.
honestly, Amazon doesnt even need AI it just needs a better UI, more metadata for its products and to make reviews less scammy.
I asked a question once and now there is a effing alexa for shopping toolbar that takes a quarter of the screen that will not go away no matter how many times I close it, and the space remains taken even if I adblock it. Absolutely hostile implementation. I have words for this I cannot type out.
LLMs aren't architected to handle filter-style comprehensive search without setting them up with additional tools.
Asking an LLM for a list of every county in the USA for example, or every county with a population of more than 100,000 people.
Even if those county names and their populations are mixed up in their weights, the nature of next-token-prediction does not lend them to effectively answering comprehensive, detailed questions like that.
An agent system build on top of an LLM can do it, if it has access to tools which can help access eg a table of counties and then filter them with SQL or Pandas or similar.
I don't know the exact syntax any more, but I expect this could be solved by a single sql query that uses "inexact but close" queries and a bunch of indexes (and perhaps tags) on each category.
This is another riff on not embedding a full document, but doing a summarization of the document and embedding the summary for RAG. Nice usecase for high cardinality data!
Eh, maybe you should keep both paths. When LLMs eventually crawl the site to feed back to agentic shoppers, maybe they logically follow the more truncated less decorated path.
In the past, people would post advice on how to do something clever and useful yourself. Now, people post suggestions on how to talk out the side of their mouth to coax ther magic-8-ball slop generator to say something useful.
I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.
the trick is that it's not totally wrong to start with
If there is no truth, there is no wrong.
This is so similar to human decision-making though. First I my innate experience to approximate to what I expect is right, then I map that to the truth.
It's the same for so many things:
- reading documentation (what do I expect this function to be called?)
- finding clothes in a shop (something long-sleeved and light)
- picking the fridge for dinner
- finding a book in the library...
so many analogues where I'm not coming cold to a choice.
I can't believe people spend their lives finding lazier ways to classify a bunch of objects that will end up heaped in dormitory dumpsters across the US next spring.
> On two occasions I have been asked, – "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
He clearly didn't know enough about vector embeddings.
Unironically, LLM is absolutely amazing at giving you the right answers when you put in wrong input, compared to every other algorithm ever invented.
That’s one of the properties that distinguishes intelligence from traditional deterministic algorithms.
I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic, an LLM running on a CPU with greedy decoding would be 100% deterministic and still have the intelligence we associate with LLMs
Valid point, it’s why I included the word “traditional”, to try to qualify that.
What I meant is something more like explicitly programmed vs. learned. Intelligence can result from learned behavior, but not from explicit programming of rules by humans.
An aspect of this is that “learning” is unpredictable - we can’t predict in advance exactly how the resulting model will behave, except broadly. It seems non-deterministic if only by virtue of its complexity, which is beyond anything we’re able to predictively model.
> Intelligence can result from learned behavior, but not from explicit programming of rules by humans.
This is incorrect.
Simulated intelligence can and has been encoded explicitly by humans defining rules programmatically in the form of expert systems[0].
0 - https://en.wikipedia.org/wiki/Expert_system
> I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic ...
Because LLMs are artificial neural networks[0] (ANN), which are statistical in nature, and thus intrinsically non-deterministic. Pretty much every AI algorithm has randomness involved in its definition and many (most?) incorporate probabilities.
0 - https://en.wikipedia.org/wiki/Neural_network_(machine_learni...
Now that you mention it, I think "statistical" might be a good word choice as it probably closely matches what people mean when they say an LLM is nondeterministic. However, on a technical level, neural network inference truly is inherently deterministic. The only nondeterminism in LLMs is parallelism-dependency in the way floating point operations are carried out and in the decoding strategy
only if you bias your "random sampling of the distributions it gives"
Fixing the seed is still intentional bias. Or you could force it to always take the one token with the highest probability, but that is still biased sampling. Deterministic, sure, but intentionally wrong just to avoid a technically
There's nothing inherently biased or intentionally wrong with greedy decoding. Why would there be? Let's say you're trying to predict what an expert doctor would say to a patient with cancer. You think there's a 99% chance the doctor would say "start chemotherapy" and a 1% chance the doctor would say "don't worry about it". If you have to pick one to output, you'll pick "start chemotherapy". It would be crazy to say the best thing to do is to roll a 100-sided die to determine your answer. A language model is literally in that exact situation.
I've seen this debate several times and often there's a terminology issue, where the same word isn't being interpreted the same way by different sides.
Often it's a difference between repeatable versus predictable, or whether a system has chaotic aspects like the configurations of a double-pendulum or weather-forecasting.
Sometimes it's the difference between determinism in-theory versus in-practice, especially when various optimizations are being applied to save money.
deterministic but completely unpredictable is not really any more useful
I was hoping a lossy semantic reasoning machine would put to bed words such as "right" and "wrong" outside of formal logic, but apparently this fight has just begun
Yeah they solved the main issue I had with Google. "I remember this exact thing, but not exactly how it was phrased..."
Except not as much as I'd like... they often also don't know what the hell I'm talking about, and it still takes them twenty minutes of Googling to find the right page!
This been the case for a long time!
The entire problem of search is that the user has the wrong data and wants to use it to receive the correct data. That was the start, not the state we’ve ended up at - it is unironically how we got to LLMs.
If information is totally wrong then all you have to do is invert it to get the truth. What was it that Sherlock Holmes said? The problem ends up being that it often takes a tremendous number of counterexamples to eliminate everything that is impossible.
Worse is when you don't know whether the answers you have are totally wrong.
Eh? “Generate an approximation and refine it algorithmically” is a well-known technique.
The terminology is misleading you.
“Hallucinate” is misleading here. In the given example, a classification is being done very successfully - it’s just that it requires an extra step to map it to an arbitrary predefined list of classifications.
If you can articulate why you think this isn’t a good approach, I’d be interested to hear it.
I couldn't believe programming has become all strings either. In the olden days it was either a code smell or compiler tests.
A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LLM to take a random sample from each clustered group and create a classification for that group.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
I worked on spam classification for litigation targeting in the early days of CANSPAM [0] enforcement.
We had a similar problem where you can literally millions of email that we were pretty sure came from only a limited set of bad actors.
We first started classifying emails into buckets by From, mailserver relay chains etc as that's all we had to to go on.
Over time, those buckets got linked to spammer signatures and then we narrowed down from there.
Fascinating to see this happening nowadays with LLMs.
The idea of distance thresholding models that are trained to satisfy an ordering constraint is a bit strange. The reason it's hard is that there isn't a threshold!
You can slice and dice it a ton of different ways, but the significance of groups is incidental.
It's a good starting point, but having done this a few times for a few companies it always seems like it needs substantial human review.
Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?
Yeah I had the same question. What's the point of the intermediate step?
Agreed that you almost certainly can just embed the original with most modern embedding models.
Yes absolutely that's another good trick.
Even better is to search the corpus first with like naive BM25 / embedding search, aggregate over top N to get most representative categories, then have the LLM categorize in that set.
It's basically a variation of HyDE (Hypothetical Document Embeddings), and the rationale is that the embedding of the query is not necessarily close to the embedding of the answer. If you generate a hallucinated answer, it can line up with the actual document better (in the embedding space, via BM25, or hybrid).
But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.
I feel like querying for the smaller distance from the set of embeddings of both the query and a fake answer structure might solve that. If there's something very close to the query, it dominates, otherwise the fake answer is the guide. Plus, why not use an LLM to judge between them if we've got a token-burner all warmed up to start.
It might be a little worse, but it will definitely be way cheaper.
> In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
Yes what you're describing is a classic way of doing query understanding.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding search
Using a Nano model, a tad worse than shipping a vocabulary to a larger OpenAI model. (And it’s an huge improvement on not classifying the queries at all).
But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.
I did something similar 10 years ago, but instead of llms I used word2vec to calculate a embeddings of product descriptions and map those to existing categories. The LLM approach is very likely better, but I'm curious what the cost difference is.
Prompt expansion of input to extra categories makes sense if your embedding isn’t working well. But on its own, why use the LLM at all? I think you could have demonstrated the original step first and then shown that it’s useful.
New embedding models support queries, so you don’t need to hallucinate a document before finding the nearest neighbor. Curious how it compares to this approach since you’d get to skip the LLM altogether.
What does it mean that an "embedding model supports queries"? An embedding model maps text to embedding vectors. You can always perform queries with such embedding vectors against a stored set of embeddings.
I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON. There you go, you just drastically reduced the output pricing.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
good this yeah
Or press 9 to hear these options again
Your call is important to us. Please listen carefully, as our menu options have changed.
Nice. I heard something similar years ago. Instead of doing a RAG search based on the question, first hallucinate a plausible answer, and then use that as the query...
This is basically HyDE (Hypothetical Document Embeddings), no? I had tried this approach in the past, worked with limited success.
We have this running in production. Can get pretty expensive and slow. We are trying to replace this with cheaper and faster methods that don’t hammer our LLM and elastic search endpoints as much.
https://arxiv.org/abs/2212.10496 for others like me hearing about HyDE for the first time
I would try using structured outputs recursively.
```
Request 1: "brown coffee table: " + {Root Schema} => "Furniture"
Request 2: "brown coffee table: Furniture / " + {Furniture Schema} => "Living Room Furniture"
Request 3: "brown coffee table: Furniture / Living Room Furniture / " + {Living Room Furniture Schema} => "Coffee Tables"
```
Many more round trips, but classifying products is not a latency sensitive task.
No? This is just giving up and hoping.
No change from regular chatbot coding, then.
Is there a solution you are using to solve this that is more accurate and cost effective? I'm working through it now so would be curious
Is scraping and putting this in a structured format too inaccurate or expensive?
That's the whole problem. If you have tons (100s of thousands or more) of labels, then you have "structured" data, but how do you correctly classify that scraped item into the correct label?
Putting all the labels into the LLM is super expensive per call when you have millions of items to classify.
You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.
100s of thousand? In that case I would label about a 100 by hand and train a supervised learning model.
This problem has also been solved for 3 decades now.
Ahhh, but the LLM processing is the kind corporate leadership and investors will actually agree to buy!
Orginally I started writing that as sarcasm, and now I'm not quite so sure.
Smart trick, but assumes the “dumb” llm is smart enough not to derail into an article about the lives of South American red ants. Obvious exaggeration, the point being outcomes should stay strictly within topic, avoid unrelated bloat and hit the target.
If you use structured outputs they’ll usually stick to the program. Not to completely constrain the categories like TFA was saying, but something like
Where you don’t really care about the rationale but you’re using it as a pseudo thinking for models that don’t support it.
Luna is surprising capable and cheap, and I haven’t done this type of thing since before GPT 5 so might not be such a useful trick now
I was in a project where we sent the whole taxonomy every request, 40k tokens + one article, ”plz classify”. This was before structured outputs. It was extremely expensive and still hallucinated. Good ol’ days.
Just this week I tried doing something similar with a nasty vibe-coded codebase I was trying to organize. I had Gemini Flash 3.6 classify each function/method in a similar way, giving a few plausible classifications for each (one agent per method).
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.
I’ve read a few different accounts, including OpenAI’s own admission, that Terra Medium or higher will likely produce better results than Luna xhigh and cost about the same or less.
Smart! I've done the same trick for resolving extracted intents to selection.
But if accuracy matters, you can't rely on embedding sort to get a closet match. With a real test set they usually don't hold up under scrutiny.
Everything in AI is like this. You get an idea, try it once or twice, "LGTM" and you ship. Then it never survives contact reality.
Embedding sort gives you a better shortlist than the whole list, but you will probably want a heavier model to vet candidates.
Interesting technique, but even if you're getting rid of hallucinations it seems there's still no guarantee of consistent classifications. If you need to do a semantic (embedding) search anyways, then how does this really help?
Don't hallucinate, mask logits if possible?
TIL LLMs follow Cunningham's Law, or they claim to be.
Had stumbled on this library in the past
https://github.com/aurelio-labs/semantic-router
I guess it is based on the same fundamentals as well.
Isn't search engines quicker than calling a LLM ? It might have a huge impact between a 20ms search engine call and a 2s LLM call for the end user.
A 2s LLM call is pretty slow.
Try using Digital Ocean. Minutes spent on inference.
They are already solving the problem with search engines, they're just using an LLM as a first pass to create better embeddings to run a similarity match on first. The difference in latency is likely made up for in accuracy.
I wonder how more accurate this is compared to just doing embedding similarity of the query vector and the category labels
Can anyone explain why LLMs are so bad at finding products (their webpages) with given specifications?
You'd think they would have solved it by now.
Because that's structured data and structured data is usually hidden away from users _and_ machines. Product rarely want to be honest, unless it's B2B in a very competitive market (and even then!). So, yeah, it's not that they are bad, it's that there are few good sources of information.
(Lets ignore for now that no one seems to agree to what should be the spec sheets)
An LLM can read websites, right? And turn them into structured data.
It can do that on run time, but it does not store data like that. The data is typically stored as embeddings in which it is hard to query data in a structured form. Example give me all products whose price is less than 200$ vs suggest me products for my spouse's birthday.
Then they shouldn't store the data as embeddings.
Instead: use an LLM to build a large (old-school) database of products with all their specifications. The LLM can also build the schema for that database as it finds more data.
Then use an LLM to query that database based on the user's specifications (+ add some intelligence to find nice suggestions for a birthday if wanted, but I'd consider that an extra).
With agentic commerce protocol / unified commerce protocol open ai and gemini are trying to solve this problem.
The idea is to make structured queries using these protocols which can be used to fetch top products matching the user needs instead of just relying on semantic search.
https://developers.openai.com/commerce/specs/file-upload/pro...
It gets worse: shopping agents are hostile adversaries to Amazon unless they're paying Amazon and they've agreed to be friendly agents. No agent that won't betray you to an Amazon pricing strategy is going to be allowed access to Amazon structured data. They might even be fed poisoned data to discredit them.
But you'll be amazed by the abundance.
Because the data, in general, is not included in the LLM model and it needs to search/browse for external information. It cannot look indefinitely so it get the top results from lists, not "evrything".
Amazon Rufus has been mildly successful for me. I think the failures I've experienced with it are mostly because the product I'm looking for doesn't exist in the catalog.
It's been an absolute fucking disaster for me. It hallucinates endlessly and its searches are terrible. It even managed to confidently gaslight me about there being a VAT invoice available for a specific product.
I noticed yesterday when browsing on mobile that there used to be a box where I could search reviews and it got swapped with a Rufus box. I guess somebody needs to juice their engagement numbers for an investor briefing.
honestly, Amazon doesnt even need AI it just needs a better UI, more metadata for its products and to make reviews less scammy.
I asked a question once and now there is a effing alexa for shopping toolbar that takes a quarter of the screen that will not go away no matter how many times I close it, and the space remains taken even if I adblock it. Absolutely hostile implementation. I have words for this I cannot type out.
LLMs aren't architected to handle filter-style comprehensive search without setting them up with additional tools.
Asking an LLM for a list of every county in the USA for example, or every county with a population of more than 100,000 people.
Even if those county names and their populations are mixed up in their weights, the nature of next-token-prediction does not lend them to effectively answering comprehensive, detailed questions like that.
An agent system build on top of an LLM can do it, if it has access to tools which can help access eg a table of counties and then filter them with SQL or Pandas or similar.
Yes, I was assuming they'd use external tools. Using only the raw LLM doesn't sound like a good strategy.
Considering that agents are not a new concept, why isn't this a solved problem by now?
Agents are a very new concept.
We've got the early LLM-based AI agents in 2023, and it only became a popular, mainstream thing in 2025 - with Claude Code.
Since you map each breadcrumb of the path, how do you deal with differing lengths that would be more appropriate?
Jeopardy clustering
Pretty cool technique honestly. You could do it the other way as well right?
If you had a list of categories you have the model to generate a sample query and then do embedding on that?
Garbage in, gold out?
I don't know the exact syntax any more, but I expect this could be solved by a single sql query that uses "inexact but close" queries and a bunch of indexes (and perhaps tags) on each category.
Like some sort of Jaccard Index based on how many tags are shared?
https://en.wikipedia.org/wiki/Jaccard_index
This is another riff on not embedding a full document, but doing a summarization of the document and embedding the summary for RAG. Nice usecase for high cardinality data!
This is a really great trick, woah!
Eh, maybe you should keep both paths. When LLMs eventually crawl the site to feed back to agentic shoppers, maybe they logically follow the more truncated less decorated path.
In the past, people would post advice on how to do something clever and useful yourself. Now, people post suggestions on how to talk out the side of their mouth to coax ther magic-8-ball slop generator to say something useful.