Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.
I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.
This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.
Well, if you only ever needed a $20 subscription's worth of tokens it would never make any sense to buy hardware since even at 2018 prices you could buy 20 years of subscriptions.
I mean, yes, it’s for sure too slow. If you want something faster you can run Qwen models locally at a very decent speed. Running Kimi K3 with 29GB of memory is pretty impressive in itself, but is more an experiment than something supposed to be viable
paying for the hardware is also too expensive for no reason, you're not querying the llm 24/7. Better use a service that hosts a lot of popular open weights LLMs, unless you have a very good reason not to.
It's about a million tokens a week. So you can do the usual calculations of rent Vs buy.
There are definitely some tasks, that if the system can run unsupervised (a largeish if), it doesn't matter as long as the result happens before a deadline.
In that respect it is easy to tell if this works for you or not. As a stepping stone to more efficiency in the future it has more value.
That’s a good way to think about it - do you want your expensive laptop grinding away constantly 24/7 consuming $5 of power to generate what amounts to $15 worth of tokens per week ?
You can get a frontier model subscription for about the same cost as the electricity (heavily subsidised by someone else’s money !) with instant results.
I know there are applications for this and it’s cool people are pushing the boundaries with local models. I run oMLX and Qwen myself, but it’s a toy really. It’s not anywhere near replacing Claude for my purposes.
For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.
They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...
I hope that reasoning is an after-the-fact justification by the LLM that wrote this.
It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
here at 0.5 tok/s, it's like getting 1/100 of a tomato for your lunch per hour, and you can't just let it grow, you have to look at it growing just as you're about to eat, and it doesn't accumulate between meals
It is worth thinking about, when you're buying something, how much of what you're paying is the supplier's margin. And whether they have efficiency advantages over you doing it yourself. But for large LLMs it does seem like there's a pretty big efficiency advantage to the rack-scale hardware in datacenters compared to hacks like this.
Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.
I got to the same conclusion another way. There's ~2.6million seconds a month and this is getting 0.5tok/s which is 1.3million tokens a month. Give some room for overhead and a reasonable rule of thumb;
The cost to run the machine per month is the cost per million tokens.
Shit, beats the hell out of GitHub CoPilot charging $.01 per "credit." For Opus that's one credit for 400 tokens genned, so $5 converts to 200k tokens down, plus the cost of whatever context you sent up.
It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.
You don't know if you are going to pay it anyway and depending on the retribution model you can loose them if you don't use them in time.
Cost is not the right mental model here, you can and should count them in the amortization plan of your installation (total cost / (projected generated kWh * lifetime)), so every kWh not used and injected is paying for it, while the kWh consumed should be discounted at the price you would pay for them at that specific moment.
> unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.
I know many many cases of people with overdimensioned PV installations that at some point want to leave their current energy company because prices are changing and have hundreds or more of euros in the "virtual battery".
I have no idea what you're trying to say. With PV, if you consume one additional kWh your bank balance will be 20c lower than it would otherwise have been. You can argue semantics about whether that is a "cost" all you want, but practically the end result is the same and any financial decisions about whether to consume that kWh should have an identical result to what you'd decide if you bought the energy in for the same price.
they're saying you might not use the extra energy if your PV runs year round. If you continuously had a year round surplus, you wouldn't have a need to buy any energy, so the credits would be monopoly money.
The grid doesn't give you money in these locations.
They give you credits for future consumption. You can only spend this on energy. If you always have a surplus, you will just rack up large amounts of credits.
I think this is the contentious point in this thread: in the UK you can get actual cash for exporting energy to the grid, at similar rates to import, with no strings attached.
Is the model you're describing (usage credits) in the European market somewhere ?
Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.
In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.
The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance.
The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.
I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.
> Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months.
Optanes are a good option here, right?
I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.
The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.
You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.
I have stopped using swap a quarter of century ago, and it was for the better.
I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.
Ever since I stopped using swap everywhere I haven't had to deal with that really annoying situation where a server is still online but limping and you're hoping ps and kill respond quickly enough before the server falls over
It is much better to not use any swap in Linux, so you will never have this probem, or any other problem caused by swap.
I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.
Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.
The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.
It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.
On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.
There was another project posted a few days ago that was quite similar and someone asked the same thing to the author. They said they tried that first and they got a 10x speed boost by doing it manually. It is the same reason why database engines write their own backing cache logic: the kernel's paging logic is generic and on-demand, while someone that's aware of the actual usage patterns can prefetch and pipeline the right data before it is used.
You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
Yes, I do this all the time, and also check in the co-authored project plans which drove the commits. For a project that is transparently only possible due to agentic coding, I don't see any reason to conceal the methods.
Honestly, Claude writes better commit messages than most people.
Personally I've mostly given in to letting it commit for me now, though I do occasionally take over and hand-write the messages if it's a particularly important concept and Claude's is too verbose.
Codex/GPT-x defaults to one-line commit messages, which are too short. Claude likes to write several paragraphs, which is usually too long.
If you tell it how to commit properly once per session it will stick with your standards for the rest of that session, and you can put that in AGENTS.md if you can be bothered to.
The agents write much better commit messages than me, at least.
They also write better PR descriptions. At work we have a PR authoring skill and we’ve included an instruction for it to write a reviewer guide that tells the most logical way to review the code and I’ve found that really helps, so much so that I’m creating a tool to have “literate” PR reviews, where it constructs a narrative interwoven with diffs.
That’s a needlessly antagonistic and insulting thing to say.
This person that you’ve never met and probably never will doesn’t owe any of us anything.
They’re out there doing what they want to do how they want to do it and if you don’t like it the correct response isn’t to insult them in front of a bunch of strangers on the internet for clout or whatever.
I doubt that you’d ever call them lazy to their face — why do it here?
That's a strange and arbitrary line to draw. There's plenty of times I make edits by hand and then tell an agent to kick it up to a PR, or on the other extreme, let an agent implement something autonomously, and review the diff myself once it's in PR. Both of those scenarios involve the agent running the git commands, neither scenario indicates "the chances they've reviewed the code is slim"
I often let Claude write my commit messages even when I'm the one who wrote the code. Claude is often damn good at writing commit messages, and they frequently end up much better than if I wrote it all by hand. I nearly always edit them somewhat, but it's like starting from 80% instead of 0%. Some might call it laziness, but I call it working smarter rather than harder.
A good commit message needs to explain why this change was needed/done rather than what is the content of the commit. Unless Claude also has access to the context via for example the associated issue, it simply cannot write a good commit message since it cannot generally guess why something was done from just the change. This is also the case for humans and the reason why you need to include this in the commit message.
I fully agree, but you'd be amazed at how good Claude is at figuring out the why based on the what. I would never have believed it if I hadn't tried it myself. If it doesn't get the why, then I will typically add it
If my harness wrote the code, I have it write the commit. I will hand author something at a higher level that expresses my intent for how I promoted the harness.
I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
> > I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.
>
> The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.
It's interesting that what I meant as a purely factual question with no prejudice either way has been taken by almost everyone to be explicitly negative & critical.
I need to be a tad less blunt I guess if I don't want to
Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.
Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere.
I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text?
I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it's got to be better than yet more "Here's the kicker" LinkedIn drivel.
Even worse than that, it's not the LLM, it's the operator. Mimicking your own style given a corpus is the lowest possible bar - you can also select among the great writers (technical or not) from history and have it give you eerily correct stylistic cues. Try Hemingway or Mark Twain next time you feel like generating a doc. Benjamin Franklin if you're feeling particularly Poor Richard.
There's no reason any more to read bad documentation. You can feed any docs in and get them in the precise style you want, so don't impose bad prose on others in your repos. Pick a solid technical writing base, jargon free, and go from there. I sometimes generate architecture proposals from the llm in RFC format and it's word perfect.
No idea why you're getting downvoted. Asking the AI to just copy my writing patterns dramatically reduced the amount of code comment and commit message rewriting I had to do. Ten years worth of HN comments finally became useful for something!
I just created a ~/social directory, launched Claude Code in it and told Claude I'd like to start tracking and studying my own social media presence. Before long there were scripts to dump all of my public posts into a SQLite database. I pulled HN, reddit, lobsters, stackoverflow and its sister sites, even mailing lists. Given a corpus inside a database, any sort of analysis can be done.
I started with linguistic analysis to reduce commit message rewriting. That one also used years worth of commit messages as input.
I immediately started doing more though. Sentiment analysis, interlocutors, negativity, positivity, recurring arguments, best and worst threads by voting patterns.
Discovered a lot of fun stuff. For example, it was hilarious and a little terrifying to realize I was randomly interacting with some industry heavy hitters without even realizing who they were. Claude on the other hand seems to know quite a bit about many people on HN.
Mapped out all of my free and open source software contributions. Counted my successes and failures, did post-mortems, saw what I did right and where I could improve. Got some very useful results out of that. For example, I will never attempt to contribute to GNU ever again.
Ask it to steelman every single argument I get into and every single worldview I have. Turned up some excellent literature too. This is great for sharpening my mind, and sometimes Claude even changes my mind.
The database also acts as a preservation archive in case any of these sites get wiped out.
It's also the best search engine I've ever used. Saying something like "I remember posting something along the X and Y lines" is enough to find pretty much anything.
Gets downright nostalgic at times. Claude found my GitHub projects dating to my teenage years.
> I assume you don't feed the whole corpus in every prompt
Claude has the local comment database and runs SQL against it to pull whatever it wants.
I speak English, I don't even know how to create one on my keyboard - is it a mac thing? I literally never saw one until chatgpt started to generate them
Not sure if it was Mac or phone, and I'm not sure if it was strictly emdash or one of the other seemingingly hundreds of different length dashes, but there was a time at least that -- would be rewritten to it. Also macs have this really really annoying habit of changing the dash when copying and pasting (but not always) so the nunver of times copying a bash command errors because it wasn't a dash, or I've copied a command to WhatsApp and the recipient got an error is enough to make me hate these clever dashes forever
This. Code written by LLMs using the new agentic SDLC is fine, can even be good code. But we shouldn't expect users to read sloppy slop, it's just plain lazy.
I disagree with the naysayers. The README was scannable and sectioned so I could read what I wanted. But some things I wish were clearer, and I’m not sure they would be if you wrote it by hand.
Don’t waste your time rewriting it by hand just because keyboard warriors are up in arms on HN.
The material feedback here reflects the questions who might be from someone who’s been running local LLMs, so ask the LLM to update it with that in mind.
Another poster pointed out it’s not clear what the quantization is. There’s that one paragraph but it’s confusing. What I want to know right away is: are you running the unquantized model or is it quantized? If so, how much? Use terms like Q3 or Q4 or 4-bit or 8-bit. Explain why it’s not practical to quantize less. How much precision loss do you think there is at the quantization selected? What if I had 128GB RAM and wanted to have better precision and not higher token speed—would it be a good idea to choose 4-bit instead of 3-bit for some of these layers?
Look I'll be honest in that I didn't read your code (or even open the link) so this totally might not apply to you, but:
> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.
Everyone says that. And most of them are wrong about the better part. Referring to your previous ability to write code is almost an appeal to authority at this point because it's coming with the assumption that you review the code with he same vigor, and that you are just as good as you were before you offloaded these tasks regularly to LLMs. Maybe you do and maybe you are, but I will say that a lot of the people I've worked with who I used to respect and whose code was good now produce absolute crap. The models are getting better so their code is getting a bit better, but I no longer believe they had anything to do with the quality, they no longer seem to care.
Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant for an outside reader looking at a "finished" product. So, we get sentences like this:
"Where the levers were is not where they are. Overlapping the expert reads with the arithmetic was worth ~1.6x and shipped; the two that looked bigger — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused, one because this family's router has no tail to demote and one because a cache the machine will not leave resident cannot be bought at any price."
What the fuck does that mean? Obviously some internal development decision, using the absolutely inscrutable internal terminology that Claude loves. If people would just read what they publish, I'm sure this would stick out immediately.
I'm not an LLM hater, I use them a ton and they work very well for writing complex code, it's undeniable. But they generate absolute dogshit first draft writing.
If that isn’t the perfect way to frame what I’ve seen and hated about LLM text, I don’t know what is. They certainly write for an audience with a historical context that almost no one has.
>They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development
The transformer does not yet understand the non-transformer.[0]
This is probably because all the data we trained it on was created by non-transformers, so it thinks it's a non-transformer, but it isn't.
I don't think we know how to train a transformer yet. All the training data is linear, but that's not how they think at all.
[0] It's a bit like the communication difficulties experienced between autistic people and neurotypicals. Each follow the Golden Rule, i.e. do unto others as you would have them do unto you -- and it fails in both directions. A Platinum Rule is necessary: do unto others as their API demands.
I think the models would need far far more introspection for the problem to be it understanding how it thinks but not how others think. I really doubt it understands how it thinks.
I heard a story about a guy who was trying to get Claude to implement some feature. It said it would be too hard, it would take weeks. Eventually convinced it to try, and it one shotted it in 30 seconds.
Yeah, I agree with you. The one time I tried to generate technical documentation for my project, I ended up rewriting almost all of the LLM output. They're extremely verbose, and needlessly so. Fable takes it up to eleven by having an obtuse sentence structure.
The thing about documentation though is humans won't actually read any of it. Maybe tailoring the documentation to the needs of LLMs isn't so bad since they're the ones who will actually consume all of those documents.
yes. its a quality test. pre LLMs you could easily judge if a project was a labor of love by attention to details, like docs and README. Nowadays if even readme is sloped, what else was slop vibecoded? everything? how much effort was put in there besides 3 prompts? 5? you can never tell
i thought, here on HN, we were past the "oooh it's written by a LLM it's bad!".
I care about the craft, well designed systems, good clean architecture and code, etc...
But i also care about reaching goals. Whether i do it working on my own, or with human coworkers or with AI coworkers doesn't matter that much to me. Yes, the result is sometimes the most important thing.
This is my take as well. I think the voice LLMs tend to use in writing sucks but I’m honestly not good enough at English as a discipline to tell you why. That said when it comes to code, did it do the thing and is it causing problems over time? If it did and doesn’t cause more issues than your coworker would, why would someone be so vehemently opposed?
Lately I've been landing on a couple of different reasons but I don’t think it’s one size fits all:
- Ego/identity - “I am the crafter of code. That is what I do.” If someone has their identity deeply wrapped up in the concept of being a “software engineer” or “programmer” then LLMs are a direct threat to that. People don’t tend to do well with this. Think about dogmatically religious people, pseudoscience followers etc who are confronted with evidence directly contradicting their beliefs. Their entire world view revolves around that identity and if you threaten that, you threaten the foundations of their self-perception.
- Career - you can take a lot of what I said above but also map it to threatening someone’s career. What if crafting software by hand becomes a niche, artisanal craft but most software is industrially generated? A lot of people will experience hardship if so and if they can’t find a way to be flexible into the future.
- Passion - the ones who love software purely for the craft see this change as robbing them of the one thing they enjoyed in their career. Work is a big part of our life and if you kill the joy for a large group of people, that’s tough to deal with.
- Lack of understanding their current purpose in role - I often see software engineers who don’t realize why they were hired. It was never to “write good clean code”, that was a means to an end. The end is generating business value for the company that hired you. That’s really it. It’s banal but it’s just a job like bagging groceries only it has required specialized knowledge so it pays well. Only very niche roles have actually hired for the craftsmanship. If someone is working for Groupon and they believe perfectly elegant systems are the value they provide, they are a bit deluded IMO. They build a platform that peddles coupons. That’s hardly comparable to building software that allows a surgeon to remotely operate a robot that does open heart surgery. Most of us do not work on truly mission critical software like that.
Most importantly though, I think this topic needs to be approached with empathy. This is truly a seismic shift in how we, as software developers, work. Change is not easy to cope with especially when it threatens physical safety and identity. Will this change be here to stay and are we in for the extinction of software creation as we have known it? I don’t know. I do know that the world’s financial and governmental systems do seem to be betting on that outcome, however.
are you using a code editor ? a compiler ? something that you didn't write yourself in pure assembly ? are you a real software engineer then ? where's the limit ?
I don't know why some people are so angry at AI/software writing with AI. It's just like being a team leader with junior(-ish) devs on the team. You don't write the code yourself, you give directions, you help/refactor/optimize where you can, that's the job.
Yes sometimes you need a team to do something, you can't code everything by yourself.
It looks like some people have a hard time accepting that software written with ai isn't a fad, it's here, it won't go away and it can be interesting and useful for the creator and for users.
Many people on the other hand have moved on and are now team leaders, except their team is mostly AIs instead of junior devs. Trade off: code is usually worse, but in the end AIs are more capable with vast knowledge and speed.
That doesn't mean we have to use AI everywhere, all the time though.
Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense parameters alone at native precision, plus ~25GB active sparse experts per token and some comparatively minor footprint for the KV cache). This is just not very helpful.
Absolutely. Anyone can remove flag and remove typical LLM slop at least. I wrote a simple tool for it as part of my CI: https://github.com/jv-k/deslopper
it is definitely the claudiest. it's weird, it's like there's a spot developing in my head next to all the other spots where i park mental models for how people write, but instead of being for a person, it's for the terse-staccato-prosodic-diarrhea that claude generates in readmes by default.
I have one setup that gets about 1.5 tok/s of a very large on prem LLM, on a system that lives under my desk. It's used for overnight project review runs and code review that it is fed at the end of each work day. When I look at it the next morning it has done quite a lot of useful work. Dealing with a big slow LLM as an effective tool is really about planning the workflow to feed it.
Once the tech catches up to the point that we can accurately select the right model for the task, then this ends up becoming a valuabke thing to have. You would spin it up sparingly as part of an automated discovery process maybe for 30 mins a day. And the rest of the time is spent using tiny models.
That’s not going to happen. We can’t even estimate how long it will take to complete a backlog item until after we complete it, there is no way to know how complex a task is without doing it.
Hi! I'm one of the deltafin devs. The biggest difference is that this is not actually a 100% "pure" uncut Kimi K3. This is requantized to 3-bit residual, whereas deltafin is the full real unaltered k3, through and through.
WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.
So if this Mac uses 30-50W, that's 40-60 tok/Wh...vs maybe 80k for a modern GPU cluster? So that's about 1000-2000x more power for the SSD streaming, unfortunately.
Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.
If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.
The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.
Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.
We are backed by the SQLite author, and we have the right to use the SQLite name. Please do your homework before writing such comments.
We used an Elastic license for some projects to protect our work from being used in SASS without a prior agreement. WASTE is and will always be available with a very permissive license.
I really don't understand why some people prefer to spread hate instead of just asking for clarification first.
Interesting project. The headline number (29 GB of RAM) is for 4k context.
From what I've read elsewhere, Kimi K3 is quite verbose in its thinking. At the quoted rate, it would generate only a total of 1.8k tokens in 1 hour. Is that enough for it to get any thinking done and produce output on more complicated prompts?
I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email.
Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
Sure, you cannot do 1M, but there are plenty of useful questions you could ask that are far more modest. Simple Q+A, look at this function, how would you design X? All of those could have few paragraphs of outputs that would finish within a day.
because claude ships these verbose READMEs with its 'honest' takes and justifications for the naming. Its goal is to prime the next Agent that reads it, not you, human
Ok, this one will take just 30h (compared to that other project that would take 6.25 days) to start writing output tokens after you say hi in Claude Code.
Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.
I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.
This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.
Yes it’s mostly unusable but ongoing iterations of projects like this will eventually lead to a usuable version so good to see
I pay $20 for codex, use it daily for coding, and still haven't dipped below 50% for weekly usage.
I wouldn't even be able to afford buying 29 gigs of ram, or a new video card with hardware prices the way they are now.
Maybe if it was 2016-2018 prices, I'd think about it.
Well, if you only ever needed a $20 subscription's worth of tokens it would never make any sense to buy hardware since even at 2018 prices you could buy 20 years of subscriptions.
I mean, yes, it’s for sure too slow. If you want something faster you can run Qwen models locally at a very decent speed. Running Kimi K3 with 29GB of memory is pretty impressive in itself, but is more an experiment than something supposed to be viable
paying for the hardware is also too expensive for no reason, you're not querying the llm 24/7. Better use a service that hosts a lot of popular open weights LLMs, unless you have a very good reason not to.
It's about a million tokens a week. So you can do the usual calculations of rent Vs buy.
There are definitely some tasks, that if the system can run unsupervised (a largeish if), it doesn't matter as long as the result happens before a deadline.
In that respect it is easy to tell if this works for you or not. As a stepping stone to more efficiency in the future it has more value.
That’s a good way to think about it - do you want your expensive laptop grinding away constantly 24/7 consuming $5 of power to generate what amounts to $15 worth of tokens per week ?
You can get a frontier model subscription for about the same cost as the electricity (heavily subsidised by someone else’s money !) with instant results.
I know there are applications for this and it’s cool people are pushing the boundaries with local models. I run oMLX and Qwen myself, but it’s a toy really. It’s not anywhere near replacing Claude for my purposes.
Really cool!!
For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.
They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...
I hope that reasoning is an after-the-fact justification by the LLM that wrote this.
It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
If you grow your own tomatoes, you'll have free tomatoes!
(Doesn't really get you a BLT but hey... at least you'll have saved the world a bit because they're not from the supermarket)
/s
here at 0.5 tok/s, it's like getting 1/100 of a tomato for your lunch per hour, and you can't just let it grow, you have to look at it growing just as you're about to eat, and it doesn't accumulate between meals
It is worth thinking about, when you're buying something, how much of what you're paying is the supplier's margin. And whether they have efficiency advantages over you doing it yourself. But for large LLMs it does seem like there's a pretty big efficiency advantage to the rack-scale hardware in datacenters compared to hacks like this.
Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.
I got to the same conclusion another way. There's ~2.6million seconds a month and this is getting 0.5tok/s which is 1.3million tokens a month. Give some room for overhead and a reasonable rule of thumb;
The cost to run the machine per month is the cost per million tokens.
Shit, beats the hell out of GitHub CoPilot charging $.01 per "credit." For Opus that's one credit for 400 tokens genned, so $5 converts to 200k tokens down, plus the cost of whatever context you sent up.
And what if I have PV?
In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.
It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.
Actually, saving $1 on something you would pay anyway is more valuable than earning $1 of taxable income.
I think that's just semantics -- unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.
You don't know if you are going to pay it anyway and depending on the retribution model you can loose them if you don't use them in time.
Cost is not the right mental model here, you can and should count them in the amortization plan of your installation (total cost / (projected generated kWh * lifetime)), so every kWh not used and injected is paying for it, while the kWh consumed should be discounted at the price you would pay for them at that specific moment.
> unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.
I know many many cases of people with overdimensioned PV installations that at some point want to leave their current energy company because prices are changing and have hundreds or more of euros in the "virtual battery".
I have no idea what you're trying to say. With PV, if you consume one additional kWh your bank balance will be 20c lower than it would otherwise have been. You can argue semantics about whether that is a "cost" all you want, but practically the end result is the same and any financial decisions about whether to consume that kWh should have an identical result to what you'd decide if you bought the energy in for the same price.
they're saying you might not use the extra energy if your PV runs year round. If you continuously had a year round surplus, you wouldn't have a need to buy any energy, so the credits would be monopoly money.
But unless you're off-grid you can always use extra PV energy by selling it to the grid.
The grid doesn't give you money in these locations.
They give you credits for future consumption. You can only spend this on energy. If you always have a surplus, you will just rack up large amounts of credits.
What locations? In the UK they give you actual money. You can withdraw it.
I think this is the contentious point in this thread: in the UK you can get actual cash for exporting energy to the grid, at similar rates to import, with no strings attached.
Is the model you're describing (usage credits) in the European market somewhere ?
Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.
In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.
The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance.
The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.
I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.
> Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months.
Optanes are a good option here, right?
I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.
> Letting the kernel use SSD based swap space for something this big
Why would it swap? If stuff gets evicted from page cache is just gets reread?
The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.
You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.
I have stopped using swap a quarter of century ago, and it was for the better.
I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.
Ever since I stopped using swap everywhere I haven't had to deal with that really annoying situation where a server is still online but limping and you're hoping ps and kill respond quickly enough before the server falls over
It is much better to not use any swap in Linux, so you will never have this probem, or any other problem caused by swap.
I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.
Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.
The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.
It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.
On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.
When you mmap a file read only, the OS fills it in on a page fault by reading. There's no writes.
There was another project posted a few days ago that was quite similar and someone asked the same thing to the author. They said they tried that first and they got a 10x speed boost by doing it manually. It is the same reason why database engines write their own backing cache logic: the kernel's paging logic is generic and on-demand, while someone that's aware of the actual usage patterns can prefetch and pipeline the right data before it is used.
Pretty soon we'll have 3T param models down to 1 bit. They'll be able to tell you whether they're off or on.
Thanks guys for all the comments, I am going to rewrite the README (without using an LLM)
That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
>Contributors
>...
>claude
You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
Why do you say lazy? maybe they are ok with people seeing it is claude?
To be fair, I appreciate when they are so upfront about who wrote the code without requiring further heuristics, so I encourage this behavior.
Yes, I do this all the time, and also check in the co-authored project plans which drove the commits. For a project that is transparently only possible due to agentic coding, I don't see any reason to conceal the methods.
Honestly, Claude writes better commit messages than most people.
Personally I've mostly given in to letting it commit for me now, though I do occasionally take over and hand-write the messages if it's a particularly important concept and Claude's is too verbose.
Codex/GPT-x defaults to one-line commit messages, which are too short. Claude likes to write several paragraphs, which is usually too long.
If you tell it how to commit properly once per session it will stick with your standards for the rest of that session, and you can put that in AGENTS.md if you can be bothered to.
The agents write much better commit messages than me, at least.
They also write better PR descriptions. At work we have a PR authoring skill and we’ve included an instruction for it to write a reviewer guide that tells the most logical way to review the code and I’ve found that really helps, so much so that I’m creating a tool to have “literate” PR reviews, where it constructs a narrative interwoven with diffs.
That’s a needlessly antagonistic and insulting thing to say.
This person that you’ve never met and probably never will doesn’t owe any of us anything.
They’re out there doing what they want to do how they want to do it and if you don’t like it the correct response isn’t to insult them in front of a bunch of strangers on the internet for clout or whatever.
I doubt that you’d ever call them lazy to their face — why do it here?
> I doubt that you’d ever call them lazy to their face — why do it here?
What makes you so sure about that?
Posting LLM generated obviously un-reviewed slop and wasting people's time deserves scorn.
That's a strange and arbitrary line to draw. There's plenty of times I make edits by hand and then tell an agent to kick it up to a PR, or on the other extreme, let an agent implement something autonomously, and review the diff myself once it's in PR. Both of those scenarios involve the agent running the git commands, neither scenario indicates "the chances they've reviewed the code is slim"
I often let Claude write my commit messages even when I'm the one who wrote the code. Claude is often damn good at writing commit messages, and they frequently end up much better than if I wrote it all by hand. I nearly always edit them somewhat, but it's like starting from 80% instead of 0%. Some might call it laziness, but I call it working smarter rather than harder.
A good commit message needs to explain why this change was needed/done rather than what is the content of the commit. Unless Claude also has access to the context via for example the associated issue, it simply cannot write a good commit message since it cannot generally guess why something was done from just the change. This is also the case for humans and the reason why you need to include this in the commit message.
I fully agree, but you'd be amazed at how good Claude is at figuring out the why based on the what. I would never have believed it if I hadn't tried it myself. If it doesn't get the why, then I will typically add it
If my harness wrote the code, I have it write the commit. I will hand author something at a higher level that expresses my intent for how I promoted the harness.
I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity.
I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.
The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.
Non sequitur
> > I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. > > The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.
It's interesting that what I meant as a purely factual question with no prejudice either way has been taken by almost everyone to be explicitly negative & critical.
I need to be a tad less blunt I guess if I don't want to
Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.
Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere.
I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text?
I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it's got to be better than yet more "Here's the kicker" LinkedIn drivel.
Even worse than that, it's not the LLM, it's the operator. Mimicking your own style given a corpus is the lowest possible bar - you can also select among the great writers (technical or not) from history and have it give you eerily correct stylistic cues. Try Hemingway or Mark Twain next time you feel like generating a doc. Benjamin Franklin if you're feeling particularly Poor Richard.
There's no reason any more to read bad documentation. You can feed any docs in and get them in the precise style you want, so don't impose bad prose on others in your repos. Pick a solid technical writing base, jargon free, and go from there. I sometimes generate architecture proposals from the llm in RFC format and it's word perfect.
Rewrite the readme in my voice, use all of my comments and responses in current context as source. Don't use em dashes or other LLM things.
Done, solved. Never had a problem with a readme or email since.
No idea why you're getting downvoted. Asking the AI to just copy my writing patterns dramatically reduced the amount of code comment and commit message rewriting I had to do. Ten years worth of HN comments finally became useful for something!
> Ten years worth of HN comments finally became useful for something!
This is a great idea! How do you do it practically? I assume you don't feed the whole corpus in every prompt - do you condense it somehow?
I just created a ~/social directory, launched Claude Code in it and told Claude I'd like to start tracking and studying my own social media presence. Before long there were scripts to dump all of my public posts into a SQLite database. I pulled HN, reddit, lobsters, stackoverflow and its sister sites, even mailing lists. Given a corpus inside a database, any sort of analysis can be done.
I started with linguistic analysis to reduce commit message rewriting. That one also used years worth of commit messages as input.
I immediately started doing more though. Sentiment analysis, interlocutors, negativity, positivity, recurring arguments, best and worst threads by voting patterns.
Discovered a lot of fun stuff. For example, it was hilarious and a little terrifying to realize I was randomly interacting with some industry heavy hitters without even realizing who they were. Claude on the other hand seems to know quite a bit about many people on HN.
Mapped out all of my free and open source software contributions. Counted my successes and failures, did post-mortems, saw what I did right and where I could improve. Got some very useful results out of that. For example, I will never attempt to contribute to GNU ever again.
Ask it to steelman every single argument I get into and every single worldview I have. Turned up some excellent literature too. This is great for sharpening my mind, and sometimes Claude even changes my mind.
The database also acts as a preservation archive in case any of these sites get wiped out.
It's also the best search engine I've ever used. Saying something like "I remember posting something along the X and Y lines" is enough to find pretty much anything.
Gets downright nostalgic at times. Claude found my GitHub projects dating to my teenage years.
> I assume you don't feed the whole corpus in every prompt
Claude has the local comment database and runs SQL against it to pull whatever it wants.
The emdash allergy is shortcut sheep mindset. Emdashes are commonly accepted and orthographically correct punctuation in many languages.
Also it looks like English might not be the author’s first language—might not lead to the best training corpus.
I speak English, I don't even know how to create one on my keyboard - is it a mac thing? I literally never saw one until chatgpt started to generate them
Not sure if it was Mac or phone, and I'm not sure if it was strictly emdash or one of the other seemingingly hundreds of different length dashes, but there was a time at least that -- would be rewritten to it. Also macs have this really really annoying habit of changing the dash when copying and pasting (but not always) so the nunver of times copying a bash command errors because it wasn't a dash, or I've copied a command to WhatsApp and the recipient got an error is enough to make me hate these clever dashes forever
This. Code written by LLMs using the new agentic SDLC is fine, can even be good code. But we shouldn't expect users to read sloppy slop, it's just plain lazy.
I disagree with the naysayers. The README was scannable and sectioned so I could read what I wanted. But some things I wish were clearer, and I’m not sure they would be if you wrote it by hand.
Don’t waste your time rewriting it by hand just because keyboard warriors are up in arms on HN.
The material feedback here reflects the questions who might be from someone who’s been running local LLMs, so ask the LLM to update it with that in mind.
Another poster pointed out it’s not clear what the quantization is. There’s that one paragraph but it’s confusing. What I want to know right away is: are you running the unquantized model or is it quantized? If so, how much? Use terms like Q3 or Q4 or 4-bit or 8-bit. Explain why it’s not practical to quantize less. How much precision loss do you think there is at the quantization selected? What if I had 128GB RAM and wanted to have better precision and not higher token speed—would it be a good idea to choose 4-bit instead of 3-bit for some of these layers?
Look I'll be honest in that I didn't read your code (or even open the link) so this totally might not apply to you, but:
> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.
Everyone says that. And most of them are wrong about the better part. Referring to your previous ability to write code is almost an appeal to authority at this point because it's coming with the assumption that you review the code with he same vigor, and that you are just as good as you were before you offloaded these tasks regularly to LLMs. Maybe you do and maybe you are, but I will say that a lot of the people I've worked with who I used to respect and whose code was good now produce absolute crap. The models are getting better so their code is getting a bit better, but I no longer believe they had anything to do with the quality, they no longer seem to care.
do people think these projects related to LLMs are ever going to be in anyway a pure human endevour?
How bout we make a new rule: only complain about LLM writing when the product as zero relevents to use with LLMs.
Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant for an outside reader looking at a "finished" product. So, we get sentences like this:
"Where the levers were is not where they are. Overlapping the expert reads with the arithmetic was worth ~1.6x and shipped; the two that looked bigger — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused, one because this family's router has no tail to demote and one because a cache the machine will not leave resident cannot be bought at any price."
What the fuck does that mean? Obviously some internal development decision, using the absolutely inscrutable internal terminology that Claude loves. If people would just read what they publish, I'm sure this would stick out immediately.
I'm not an LLM hater, I use them a ton and they work very well for writing complex code, it's undeniable. But they generate absolute dogshit first draft writing.
If that isn’t the perfect way to frame what I’ve seen and hated about LLM text, I don’t know what is. They certainly write for an audience with a historical context that almost no one has.
>They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development
The transformer does not yet understand the non-transformer.[0]
This is probably because all the data we trained it on was created by non-transformers, so it thinks it's a non-transformer, but it isn't.
I don't think we know how to train a transformer yet. All the training data is linear, but that's not how they think at all.
[0] It's a bit like the communication difficulties experienced between autistic people and neurotypicals. Each follow the Golden Rule, i.e. do unto others as you would have them do unto you -- and it fails in both directions. A Platinum Rule is necessary: do unto others as their API demands.
I think the models would need far far more introspection for the problem to be it understanding how it thinks but not how others think. I really doubt it understands how it thinks.
I heard a story about a guy who was trying to get Claude to implement some feature. It said it would be too hard, it would take weeks. Eventually convinced it to try, and it one shotted it in 30 seconds.
Yeah, I agree with you. The one time I tried to generate technical documentation for my project, I ended up rewriting almost all of the LLM output. They're extremely verbose, and needlessly so. Fable takes it up to eleven by having an obtuse sentence structure.
The thing about documentation though is humans won't actually read any of it. Maybe tailoring the documentation to the needs of LLMs isn't so bad since they're the ones who will actually consume all of those documents.
On the plus side, LLM slop like this in readmes are a great indicator that you can just close the tab and carry on with life - nothing of value lost
Does it matter?
I need an llm to block these comments.
yes. its a quality test. pre LLMs you could easily judge if a project was a labor of love by attention to details, like docs and README. Nowadays if even readme is sloped, what else was slop vibecoded? everything? how much effort was put in there besides 3 prompts? 5? you can never tell
i thought, here on HN, we were past the "oooh it's written by a LLM it's bad!".
I care about the craft, well designed systems, good clean architecture and code, etc...
But i also care about reaching goals. Whether i do it working on my own, or with human coworkers or with AI coworkers doesn't matter that much to me. Yes, the result is sometimes the most important thing.
This is my take as well. I think the voice LLMs tend to use in writing sucks but I’m honestly not good enough at English as a discipline to tell you why. That said when it comes to code, did it do the thing and is it causing problems over time? If it did and doesn’t cause more issues than your coworker would, why would someone be so vehemently opposed?
Lately I've been landing on a couple of different reasons but I don’t think it’s one size fits all:
- Ego/identity - “I am the crafter of code. That is what I do.” If someone has their identity deeply wrapped up in the concept of being a “software engineer” or “programmer” then LLMs are a direct threat to that. People don’t tend to do well with this. Think about dogmatically religious people, pseudoscience followers etc who are confronted with evidence directly contradicting their beliefs. Their entire world view revolves around that identity and if you threaten that, you threaten the foundations of their self-perception.
- Career - you can take a lot of what I said above but also map it to threatening someone’s career. What if crafting software by hand becomes a niche, artisanal craft but most software is industrially generated? A lot of people will experience hardship if so and if they can’t find a way to be flexible into the future.
- Passion - the ones who love software purely for the craft see this change as robbing them of the one thing they enjoyed in their career. Work is a big part of our life and if you kill the joy for a large group of people, that’s tough to deal with.
- Lack of understanding their current purpose in role - I often see software engineers who don’t realize why they were hired. It was never to “write good clean code”, that was a means to an end. The end is generating business value for the company that hired you. That’s really it. It’s banal but it’s just a job like bagging groceries only it has required specialized knowledge so it pays well. Only very niche roles have actually hired for the craftsmanship. If someone is working for Groupon and they believe perfectly elegant systems are the value they provide, they are a bit deluded IMO. They build a platform that peddles coupons. That’s hardly comparable to building software that allows a surgeon to remotely operate a robot that does open heart surgery. Most of us do not work on truly mission critical software like that.
Most importantly though, I think this topic needs to be approached with empathy. This is truly a seismic shift in how we, as software developers, work. Change is not easy to cope with especially when it threatens physical safety and identity. Will this change be here to stay and are we in for the extinction of software creation as we have known it? I don’t know. I do know that the world’s financial and governmental systems do seem to be betting on that outcome, however.
> I care about the craft
No you don't. Buying a table and sanding the edges off doesn't mean you're a carpenter.
are you using a code editor ? a compiler ? something that you didn't write yourself in pure assembly ? are you a real software engineer then ? where's the limit ?
I don't know why some people are so angry at AI/software writing with AI. It's just like being a team leader with junior(-ish) devs on the team. You don't write the code yourself, you give directions, you help/refactor/optimize where you can, that's the job.
Yes sometimes you need a team to do something, you can't code everything by yourself.
For reference, i'm not OP.
quite funny to see HN crowd downvoting this.
It looks like some people have a hard time accepting that software written with ai isn't a fad, it's here, it won't go away and it can be interesting and useful for the creator and for users.
Many people on the other hand have moved on and are now team leaders, except their team is mostly AIs instead of junior devs. Trade off: code is usually worse, but in the end AIs are more capable with vast knowledge and speed.
That doesn't mean we have to use AI everywhere, all the time though.
> It looks like some people have a hard time accepting that software written with ai isn't a fad, it's here, it won't go away
It looks like some people have a hard time accepting that a readme or a documentation is not software.
Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense parameters alone at native precision, plus ~25GB active sparse experts per token and some comparatively minor footprint for the KV cache). This is just not very helpful.
Yes, but very slowly.
Absolutely. Anyone can remove flag and remove typical LLM slop at least. I wrote a simple tool for it as part of my CI: https://github.com/jv-k/deslopper
it is definitely the claudiest. it's weird, it's like there's a spot developing in my head next to all the other spots where i park mental models for how people write, but instead of being for a person, it's for the terse-staccato-prosodic-diarrhea that claude generates in readmes by default.
I prefer to have such detailed readme's created by a LLM while iterating over no iteration documentation at all and usually the later is the standard.
are they allowed to use the "SQLite" name?
Yes. They are.
https://news.ycombinator.com/item?id=46389934
Claude might as well be .5 tok/s. I end up waiting several minutes and what it tells me could usually be summarized in under 100 words.
So I could potentially live with this if it was concise.
This is not how thinking works. Claude uses tens of thousands of thinking tokens to get to 100 words. Kimi is no different.
Additionally, GP can use the word ‘concise’ (or similar) in their prompt if they want more concise output from a model.
That doesn’t mean “use fewer thinking tokens”. It might use more as it mulls over how to make its response concise.
effort level is a separate parameter, and is probably what they want
I have one setup that gets about 1.5 tok/s of a very large on prem LLM, on a system that lives under my desk. It's used for overnight project review runs and code review that it is fed at the end of each work day. When I look at it the next morning it has done quite a lot of useful work. Dealing with a big slow LLM as an effective tool is really about planning the workflow to feed it.
In my experience Kimi k3 is even more verbose.
Once the tech catches up to the point that we can accurately select the right model for the task, then this ends up becoming a valuabke thing to have. You would spin it up sparingly as part of an automated discovery process maybe for 30 mins a day. And the rest of the time is spent using tiny models.
I could see a future like that.
That’s not going to happen. We can’t even estimate how long it will take to complete a backlog item until after we complete it, there is no way to know how complex a task is without doing it.
That's too high level. Even a basic regex for words leading to tool calls follows the principle. Claude does it right now.
We need something more sophisticated, but you could just escalate the model quality if if keeps failing the test. Not elegant but it will work.
wow that sounds miserable...
Love that it’s embeddable, I needed exactly this for the upcoming app I’m shipping!
How does this project compare to https://github.com/gavamedia/deltafin ?
Hi! I'm one of the deltafin devs. The biggest difference is that this is not actually a 100% "pure" uncut Kimi K3. This is requantized to 3-bit residual, whereas deltafin is the full real unaltered k3, through and through.
WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.
Why quantize to int8 when k3 is int4? I could be wrong, but that’s what I remember seeing.
For me at least, it makes way more sense on a DSv4 Flash size and ~10 tps and I would definitely try (or try to try) such a thing.
The concept and the proof of it is great, of course.
So if this Mac uses 30-50W, that's 40-60 tok/Wh...vs maybe 80k for a modern GPU cluster? So that's about 1000-2000x more power for the SSD streaming, unfortunately.
GPU sufficient to hold and run Kimi have a totally different up front capex cost, however.
Neat! But, what do you do with a 0.5tk/s LLM?
Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?
You could use it for long run tasks while you don’t use the laptop.
By the time the tokens start coming out 30h later you might need to use your laptop again...
Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.
Then have it summarise the meeting transcription over the entire week for the report next week.
ftfy.
If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.
Using half a kilowatt-hour, and if thinking is disabled during inference, yes.
You get 8 nvmes set them up in raid 0/1 across two full pcie5x16 ports and you could reach up to 4ish tokens per second, presumably.
The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.
> Neat! But, what do you do with a 0.5tk/s LLM?
Hopefully resolve incidents faster without people pasting slop into the incident thread.
I couldnt find anything explaining the name of this company on their website but is it okay that they’re riding on the name of an open source tool?
SQLite code itself is public domain but I’m not sure about the name.
Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.
Are these the sqlite developers..?
no, this is sqliteai aka SQLite Cloud, Inc
wow. shamelessly stealing the name of one of the most robust and effective libraries ever, to push some ai bullshit.
what an embarrassment
We are backed by the SQLite author, and we have the right to use the SQLite name. Please do your homework before writing such comments.
We used an Elastic license for some projects to protect our work from being used in SASS without a prior agreement. WASTE is and will always be available with a very permissive license.
I really don't understand why some people prefer to spread hate instead of just asking for clarification first.
Even so, sqliteai squandered cultivating its reputation by using a non-free license for various projects.
This sounds a lot like what the colibri project did for GLM-5.2. I'm a fan so keep at it!
justvugg.github.io/colibri
Interesting project. The headline number (29 GB of RAM) is for 4k context.
From what I've read elsewhere, Kimi K3 is quite verbose in its thinking. At the quoted rate, it would generate only a total of 1.8k tokens in 1 hour. Is that enough for it to get any thinking done and produce output on more complicated prompts?
I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email.
Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
I’m sure it’s possible, but I really struggle to think of an example that would result in a 1m token output.
it would be 1M tokens worth of work, with some small report for the end
Sure, you cannot do 1M, but there are plenty of useful questions you could ask that are far more modest. Simple Q+A, look at this function, how would you design X? All of those could have few paragraphs of outputs that would finish within a day.
Dear creator: you didn't ship the first draft of your code - why did you ship the first draft of your README??
because claude ships these verbose READMEs with its 'honest' takes and justifications for the naming. Its goal is to prime the next Agent that reads it, not you, human
we need a new HUMANS.md to replace the readme
It's too bad Optane PMem is dead
I don't get the fascination how is optane better than a modern nvme? I don't think it was faster.
It still has much better latency than modern NVMEs and much, much better write endurance (probably not critical for AI workloads)
28 Jul 2026 21:35:19 UTC
Running Kimi K3 on a M1 Max
https://github.com/gavamedia/deltafin
https://news.ycombinator.com/item?id=49090233
[ok]
29 Jul 2026 02:01:04 UTC
Kimi K3 running on any device locally
https://github.com/RightNow-AI/local-kimi
https://news.ycombinator.com/item?id=49092591
[dead]
29 Jul 2026 13:49:32 UTC
Show HN: Waste Run the full 2.78T-parameter Kimi K3 on a laptop
https://news.ycombinator.com/item?id=49097512
[flagged] [dead]
29 Jul 2026 14:38:35 UTC
Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution
https://aistack.imec-int.com/blog/gpu-self-hosting
https://news.ycombinator.com/item?id=49098130
[ok]
29 Jul 2026 14:57:14 UTC
Run Kimi K3 on a local computer
https://github.com/sqliteai/waste
https://news.ycombinator.com/item?id=49098395
[ok]
29 Jul 2026 15:41:50 UTC
Show HN: A new engine to run Kimi K3 on a laptop
https://news.ycombinator.com/item?id=49098966
[ok]
29 Jul 2026 19:54:13 UTC
Kimi K3 for local use (1.56TB > 594GB) compressed by Unsloth
https://huggingface.co/unsloth/Kimi-K3-GGUF
https://news.ycombinator.com/item?id=49102192
[ok]
29 Jul 2026 20:02:40 UTC
Kimi k3 now runs on one consumer GPU
https://twitter.com/Akashi203/status/2082555972380401852
https://news.ycombinator.com/item?id=49102291
[ok]
30 Jul 2026 06:12:55 UTC
Show HN: Run Full Kimi K3 with 29 GB of RAM
https://github.com/sqliteai/waste/
https://news.ycombinator.com/item?id=49106591
[ok]
30 Jul 2026 13:05:09 UTC
Kimi k3 run on RTX 5090
https://github.com/RightNow-AI/local-kimi
https://news.ycombinator.com/item?id=49109455
[ok]
30 Jul 2026 14:01:44 UTC
A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM
https://marcobambini.substack.com/p/the-waste-inference-engi...
https://news.ycombinator.com/item?id=49110183
[ok]
30 Jul 2026 16:52:58 UTC
Running Kimi K3 on a local computer
https://github.com/sqliteai/waste
https://news.ycombinator.com/item?id=49112587
[ok]
Does it not use Metal, on macOS? Would it be faster if it did?
We tried to use Metal, but for that specific project it was slower than just using NEON ARM optimizations. It is all documented in the docs.
Ok, this one will take just 30h (compared to that other project that would take 6.25 days) to start writing output tokens after you say hi in Claude Code.
People who bought CPU instead of Nvidia are coping with this reality.
I'm not sure how this happens. Reality distortion field?
Like, it is common knowledge at this point right?
Where can this 1tb k3.waste be downloaded?
It is not yet available, the only way is to download the official Kimi K3 model and then convert it:
# 1. preflight: reachable? how big? does it fit? tools/fetch_weights.sh --dest /Volumes/staging/k3 --dry-run
# 2. download — resumable, safe to kill, safe to re-run tools/fetch_weights.sh --dest /Volumes/staging/k3
# 3. convert into a container uv run --with torch --with safetensors python tools/convert.py \ --src /Volumes/staging/k3 \ --out ~/models/k3.waste --jobs 3
Yeah, saw this ... was hoping that there's a torrent of it somewhere already. Or something.
looks helpful
your readme is overly verbose
agents don't need that and its extremely low signal for humans too
tell your language model to get it to the point
Waste is/was a p2p client back in the 2000s
Fun. I see the novelty.
How does it compare to dsv4?