This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept.
The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.
LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name.
Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.
But imagine Sqlite not supporting joins or window functions... sure they are useful but look how many LOC it adds! Who is the arbiter of what Lite actually means?
We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.
It may, but the latency it contributes to the end-to-end AI processing has been not noticeable in practice for our users.
That's not to say Bifrost wouldn't have been better, but the choice to use LiteLLM was arrived at after a fair bit of internal discussion (most of which predated my addition to the team), and so far we've seen nothing from LiteLLM that has been contradictory to the pros/cons they thought would be the case when LiteLLM was adopted.
Or in other words, the org will be happy indeed when they have solved so many of the rest of the problems we've had in AI uptake that the difference in latency between one AI gateway or the other becomes a problem to be solved.
And let's not forget that they regularly break things. For example they had merged a pull request that was supposed to fix issues related to the openrouter/free models but in the process broke openrouter for all models except the free ones.
The fix was deployed 2 weeks later (!!) to main (but you could downgrade of course).
Or that other time they broke model selection if you had selected "this key can used all models of their team" than the only model in the auto-selection for harnesses was an invalid "all-team-models" entry. Fixes this one in 1 week though.
Really? I mean, yeah, it's probably written by an LLM, but it's hardly the worst that I've seen. Looks way more straight forward than the modern README featuring a ton of badges, emojis, confusing out-of-context screenshots, "trust me bro" installation instructions, vague elevator pitches, "used by netflix, nasa, disney, good morning america, alex jones, the church of scientology", and other verbiage to create the illusion that the author won't immediately get bored and abandon their glorified dissertation piece. They all scream "give me your github stars" whereas this one doesn't. But I still get what you mean when it comes to the particular 'isms.
I have the same complaint about oh-my-pi's readme. The tone is obnoxious. It's somehow jaded and matter-of-fact at the same time. Like it was written by that one guy at work who never misses a chance to brag about how clever he is.
There is a band called The Protomen who do rock operas about the Mega Man storyline. They have a very earnest, gritty sound. Every time I read a project README written by AI, I hear the voice over from the first track of the first album, Act 1: Hope Rides Alone (https://youtu.be/VZ8jyGVioxg?is=gsXlgwa4Ubx6LmKE).
"Twelve years Light worked and on a cold night in the year 200X, Protoman was born.
A perfect man, an unbeatable machine, hell-bent on destroying every evil standing between man and freedom, built for one purpose, to destroy Wily's army of evil robots. Ready. Willing. Prepared to fight."
For the Protomen it makes sense. But for a project README it's so absurdly melodramatic.
One "AI-ism" I noticed is that LLMs often just put sentences behind one another without using any connecting adverbs - and just leave it to the reader's imagination how the sentences are related to each other.
E.g. in the readme: "LiteLLM routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC of proxy servers, caching layers [etc...]".
Those two sentences have opposite sentiment on LiteLLM, so a human author would at least put a "but" between them. In contrast, the LLM just strings them together.
This reads "blunt" and "matter-of-fact" at first glance, but I wonder if it's really just an artifact of allocating less space for text generation and more for code in LLMs.
Funny, everything you pruned away is the reason I’m deploying LiteLLM in our platform. Having a reliable way to track token spend per customer across different services is important to us, and LiteLLM handles this well
I like the API, and since this is open source I will copy the design with attribution: I have been organically hacking little bits of LLM client code for most providers in Common Lisp for about 3 1/2 years and it is time to clean up all my old code. I probably need to do the same sort of refactoring for my search API wrappers.
First off, cool project! It's always great to see derivatives that question the efficiency of the established product.
I think the main thing the readme is missing is the core benefits. Reducing LOC and dependencies is cool, but it would be great to understand if this provides some additional benefits like lower latency or memory requirements.
"litellm routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC"
Seriously anybody considering 100k+ LOC not a bloat? You made my day!
Let's just say the author's and my definition of bloat is not the same. Full disclosure, I'm the guy who reimplemented etcher (over 400Mb) in a mere 300Kb, Capstone (over 1Mb) in only 66Kb and who compressed LPC charactersheets (over 700Mb) into 4Mb. That's my interpretation of "non-bloated".
No, you forgot about the dependencies (others said it to be 700Mb). And 2,900 LOC is still way too much for handling a remote API call, especially in a high level language such as Python where the biggest part of the task is delegated to separate modules.
the smaller surface is nice. i'd still keep auth and spend caps outside the proxy though, because once every app shares one key the blast radius gets ugly fast.
Immediate first impression is that this tagline should go. If the project brings something valuable to the table it doesn't need to shit on other permissively licensed open source projects to make a case for itself
most software like this will be dematerialized, democratized, and demonetized - companies building in the infra band being increasingly disintermediated
This readme, when it talks about all the different AI endpoints it can use, reminds me of something.
I'm not an AI bro, but I've dabbled. It's kind of remarkable that all the different providers speak the same "openai compatibile" https endpoints. In other realms of software development, real interoperability like that can be kind of rare. Even if people support conceptually the same API, everybody always puts their unique incompatible spin on it. In the dabbling that I've done, big incompatibilities seem rare.
This is a 30 minute project with a frontier LLM. I don’t see why anyone would use anyone else’s router. Techniques are valuable today. Libraries are not.
I'm always confused by LLM proxies that claim to support tool calling. Even for Bifrost that claims to be doing it, at least when I was checking it out, I found out that while it injects the list of MCP tools that's available on the proxy-side, it doesn't actually make the call on client's behalf, and clients get confused by it (response returns MCP call request whose tool doesn't exist on the client-side).
I don't have specifics since it's been months, but LiteLLM on a hetzner server, deployed through Coolify was using 1-4 GB (cannot exactly, and dont know why either).
Bifrost feels very light, though.
Will try LiteLLM if it's in Rust.
Would also appreciate a slicker aesthetic. I got used to Bifrost shadcn with 0 border radius.
This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept.
The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.
LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name.
Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.
But imagine Sqlite not supporting joins or window functions... sure they are useful but look how many LOC it adds! Who is the arbiter of what Lite actually means?
LiteLLM's problem isn't really features, it's how bloated all the features are, and specifically how AI maximalist and janky their dev practices are.
thanks for the feedback. genuinely curious what you think we could be doing better, especially around our dev practices. Would love to hear specifics.
Didn't yall have a release of a package that stole user keys or such?
Don't worry about it. Claude did a thorough security review of their code.
Maybe this has improved since I last used it, but I was dismayed at not being able to do something like:
We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.
It may not be a huge resource hog, but it adds a ton of latency.
https://www.getmaxim.ai/bifrost/resources/benchmarks
Having ran both LiteLLM and Bifrost for months, I can largely confirm the numbers from those benchmarks for myself.
It may, but the latency it contributes to the end-to-end AI processing has been not noticeable in practice for our users.
That's not to say Bifrost wouldn't have been better, but the choice to use LiteLLM was arrived at after a fair bit of internal discussion (most of which predated my addition to the team), and so far we've seen nothing from LiteLLM that has been contradictory to the pros/cons they thought would be the case when LiteLLM was adopted.
Or in other words, the org will be happy indeed when they have solved so many of the rest of the problems we've had in AI uptake that the difference in latency between one AI gateway or the other becomes a problem to be solved.
LiteLLM is vibecoded trash.
IIRC it's 700mb with all deps included, to me that's clear bloat given you could accomplish it with a fraction of that
I guess they got that message so they're porting to rust!
https://docs.litellm.ai/blog/litellm-rust-launch
That blog post smells like Opus.
We run litellm at work and not once anybody cared about those 700mb.
And let's not forget that they regularly break things. For example they had merged a pull request that was supposed to fix issues related to the openrouter/free models but in the process broke openrouter for all models except the free ones.
The fix was deployed 2 weeks later (!!) to main (but you could downgrade of course).
Or that other time they broke model selection if you had selected "this key can used all models of their team" than the only model in the auto-selection for harnesses was an invalid "all-team-models" entry. Fixes this one in 1 week though.
all of them on the :latest docker tag btw
>> kind of the core value proposition of LiteLLM for many of their users.
THIS! And it's way cheaper than others like Kong =)
Where does it mention it selectively extract features from OSS projects?
I strongly recommend the authors rewrite the readme by hand. It’s kind of a snif test for how much care someone put into this project.
Agree. The LLM'isms are offputting.
Throwing my support for this. Do not use LLMs to write things humans should write.
Really? I mean, yeah, it's probably written by an LLM, but it's hardly the worst that I've seen. Looks way more straight forward than the modern README featuring a ton of badges, emojis, confusing out-of-context screenshots, "trust me bro" installation instructions, vague elevator pitches, "used by netflix, nasa, disney, good morning america, alex jones, the church of scientology", and other verbiage to create the illusion that the author won't immediately get bored and abandon their glorified dissertation piece. They all scream "give me your github stars" whereas this one doesn't. But I still get what you mean when it comes to the particular 'isms.
This readme is better than most readmes. However they came to making it, it's clearly working
I have the same complaint about oh-my-pi's readme. The tone is obnoxious. It's somehow jaded and matter-of-fact at the same time. Like it was written by that one guy at work who never misses a chance to brag about how clever he is.
There is a band called The Protomen who do rock operas about the Mega Man storyline. They have a very earnest, gritty sound. Every time I read a project README written by AI, I hear the voice over from the first track of the first album, Act 1: Hope Rides Alone (https://youtu.be/VZ8jyGVioxg?is=gsXlgwa4Ubx6LmKE).
"Twelve years Light worked and on a cold night in the year 200X, Protoman was born. A perfect man, an unbeatable machine, hell-bent on destroying every evil standing between man and freedom, built for one purpose, to destroy Wily's army of evil robots. Ready. Willing. Prepared to fight."
For the Protomen it makes sense. But for a project README it's so absurdly melodramatic.
One "AI-ism" I noticed is that LLMs often just put sentences behind one another without using any connecting adverbs - and just leave it to the reader's imagination how the sentences are related to each other.
E.g. in the readme: "LiteLLM routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC of proxy servers, caching layers [etc...]".
Those two sentences have opposite sentiment on LiteLLM, so a human author would at least put a "but" between them. In contrast, the LLM just strings them together.
This reads "blunt" and "matter-of-fact" at first glance, but I wonder if it's really just an artifact of allocating less space for text generation and more for code in LLMs.
omp is slopware through and through. And it shows, slow as hell to use
One of the 2 dependencies, httpx, isn't really maintained anymore. Pydantic picked it up as httpx2: https://pydantic.dev/docs/httpx2
Funny, everything you pruned away is the reason I’m deploying LiteLLM in our platform. Having a reliable way to track token spend per customer across different services is important to us, and LiteLLM handles this well
I like the API, and since this is open source I will copy the design with attribution: I have been organically hacking little bits of LLM client code for most providers in Common Lisp for about 3 1/2 years and it is time to clean up all my old code. I probably need to do the same sort of refactoring for my search API wrappers.
Thanks for the cool project.
Cool project, I had to implement the same for my experimental AI Agent: https://github.com/quantized-ai/luca-py
LiteLLM and LangChain are AWFUL pieces of software and should be avoided at ALL costs.
Btw, you should update httpx to httpx2, and I think it's not much effort to remove openai's SDK compatibility.
I'd love to have a provider-agnostic LLM router (almost) dependency free (aside from httpx2).
It would be great if there were a plugin/extension infrastructure. For example, to write the cache and costs however and whenever I see fit
I imagine many people code their own LLM client after getting fed up with the bad options out there. It’s very easy with ai coding tools.
I’m biased but I think mine is coded to a higher standard than litelm. https://github.com/s-banach/langchaint
First off, cool project! It's always great to see derivatives that question the efficiency of the established product.
I think the main thing the readme is missing is the core benefits. Reducing LOC and dependencies is cool, but it would be great to understand if this provides some additional benefits like lower latency or memory requirements.
ROTFL, I had to laugh so hard!
"Without the Bloat"
vs.
"litellm routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC"
Seriously anybody considering 100k+ LOC not a bloat? You made my day!
Let's just say the author's and my definition of bloat is not the same. Full disclosure, I'm the guy who reimplemented etcher (over 400Mb) in a mere 300Kb, Capstone (over 1Mb) in only 66Kb and who compressed LPC charactersheets (over 700Mb) into 4Mb. That's my interpretation of "non-bloated".
You misunderstood. This new project has 2,900 LOC. Maybe the spelling change is too subtle.
No, you forgot about the dependencies (others said it to be 700Mb). And 2,900 LOC is still way too much for handling a remote API call, especially in a high level language such as Python where the biggest part of the task is delegated to separate modules.
I wrote an entire https client from ground up in 117 LOC (and in a low level language, not Python): https://gitlab.com/bztsrc/skrellm/-/blob/main/src/https.c
Again, this guy and me disagree on what "not a bloat" means.
the smaller surface is nice. i'd still keep auth and spend caps outside the proxy though, because once every app shares one key the blast radius gets ugly fast.
Also Strongly recommend renaming to avoid confusion
Make a docker image rootless please, thanks
Immediate first impression is that this tagline should go. If the project brings something valuable to the table it doesn't need to shit on other permissively licensed open source projects to make a case for itself
most software like this will be dematerialized, democratized, and demonetized - companies building in the infra band being increasingly disintermediated
token counting is useful though. There's also other ways to reduce size, ie a plugin API.
This readme, when it talks about all the different AI endpoints it can use, reminds me of something.
I'm not an AI bro, but I've dabbled. It's kind of remarkable that all the different providers speak the same "openai compatibile" https endpoints. In other realms of software development, real interoperability like that can be kind of rare. Even if people support conceptually the same API, everybody always puts their unique incompatible spin on it. In the dabbling that I've done, big incompatibilities seem rare.
This is a 30 minute project with a frontier LLM. I don’t see why anyone would use anyone else’s router. Techniques are valuable today. Libraries are not.
Actually not. There are so many edge cases. Also these routers are only useful if they have a minimal layer of observability.
Yes, LLMs can do a great job at writing semi-working MVP. Turning it into a usable project still requires a team.
Yeah, maybe for your toy project you can use a LLM written tool.
Also, I am not saying LiteLLM is good either.
There are always people who need an entire team to produce something like OP repo. Enterprise FizzBuzz is real after all.
How does it compare to Bifrost?
I'm always confused by LLM proxies that claim to support tool calling. Even for Bifrost that claims to be doing it, at least when I was checking it out, I found out that while it injects the list of MCP tools that's available on the proxy-side, it doesn't actually make the call on client's behalf, and clients get confused by it (response returns MCP call request whose tool doesn't exist on the client-side).
LiteLLM is basically Bifrost in the Python ecosystem.
Yeah, but bifrost seems tighter and claims to use way fewer resources
hey - litellm will migrate its core to Rust soon. please follow this issue if you're interested! https://github.com/BerriAI/litellm/issues/31263
also if you had resource issues with your production deployment, keen to hear more details
I don't have specifics since it's been months, but LiteLLM on a hetzner server, deployed through Coolify was using 1-4 GB (cannot exactly, and dont know why either).
Bifrost feels very light, though.
Will try LiteLLM if it's in Rust.
Would also appreciate a slicker aesthetic. I got used to Bifrost shadcn with 0 border radius.
- We recently migrated to shadcn, which also comes with dark mode support. - You can subscribe to the issue or email me at yujong at berri.ai
I’m happy to provide any support if you’re willing to try out the initial version.
Drop the Lite, it's cleaner, just use the godamn LLM directly.