We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I use the Qwen models a lot in my private life and they're not quite on par with the SOTA models, but they've definitely gotten to the point where they're usable. Particularly if you invest a lot in harness engineering, they become very usable. It'll be even better when Qwen4 comes out, and less harness engineering will be needed for the same results, and an equivalent amount of harness engineering tuned for the updated models weaknesses will produce results much closer to current SOTA. I was also pretty stunned to see Unsloth's IQ1_M quants of Deepseek v4.1 Flash 1017 held up too. Even that has become runnable locally. It's truly a new era.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
Most of the look of the thing is about the built in tools and skills and harness settings, not the quality of the code. those things can be replicated locally as well.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
Those benchmarks are easily gamed in my opinion. I take them woth a huge grain of salt.
I am currently building an integrated CAD + Aero + CAM software package for rapid UAV design iteration, and the open models are all not very good at it. They make a ton of mistakes and I have to end up redoing a lot. And that's using very detailed specs and TDDs provided as context to the models. Opus/Astra/Argon/Gemini Flash 3.8/Sol does a significantly better job in my experience.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
Why 2 5090s instead of just 1 6000? That's 64GB VRAM vs 96GB VRAM. Perfomance is theoretically worse on the 6000 vs 2x 5090s, but given the overhead of tensor parallelism and how slow PCIE is, I imagine it's pretty close for less than half the power draw and heat.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
That's an 8GB card. For gaming (DLSS, etc.), you can still get newer 8GB cards like the 3070 which represent solid upgrades over the 1070 Ti on the used market for under $300. Still not great since similar cards were available at similar prices new a couple years ago, but at least not trading at multiples of MSRP like higher-end cards.
Even 8GB cards might start going up. I recently bought a cheap one so I can auxiliary models on it (think smaller LLMs for memory consolidation / decision models etc). Locally, multi-gpu is pretty cool. You can essentially have one GPU drive the others and do increasingly powerful things when you combine them.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
I see them at a little less than 400€ in the second hand market. So expensive for their age but not hard to replace if you were willing to pay 600-700 for it in 2020.
You should bite the bullet now and buy a decent card on the secondhand market, before more people figure out what they have and double, even triple, the price.
I replaced my 11 year old PC earlier this year, with a new 5070 on discount and only 16GB RAM. I thought it was expensive. Now prices surged like 40% since then and I regreted not getting more RAM.
I feel your pain. I run an RTX 2060 that I bought just before the Plague, and although it still works great on any game I own (I don't really play modern AAA games), I'm worried that I won't be able to afford a replacement if it randomly craps out on me one of these months... Especially since I make less money now than I did when I bought the components.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
Anyone has an idea when will it unclog even a little bit? When is the next NVIDIA lineup comming out? Are they going to release any middle of the road products like strong AMD Insights?
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
I think the discrete GPUs as a concept is coming to an end. Computer graphics is shifting more and more to using local AI-like things like DLSS that allows to hard-render simpler geometry and shadows and then apply AI-realism filters in real-time.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
DLSS still needs discrete hardware to do that (and keeps requiring new hardware on each generation). Not to mention the types of RAM are not the same and have entirely different tradeoffs - using GDDR for a CPU would tank performance due to an order of magnitude worse latency, using DDR for a GPU would make games a slideshow due to an order of magnitude worse bandwidth. And you'd have to redesign the entire concept of how CPUs and motherboards work to get there, while making AMD and NVIDIA agree to have their chips be behest to the bridges of shitty motherboard and consumer RAM manufacturers even more than they already are.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
> DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
How would DLSS, which runs on GPU hardware — and specifically, as I understand it, GPU hardware that accelerates AI inference — eliminate the GPU?
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
DLSS does not require any specific packaging configuration. There's no reason DLSS can't run on DGX Spark or GPU cores of the Apple's M-chips aside from specific Nvidia software optimizations that can surely be worked out if needed.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
>> given correct setup & harness
I am geninuely interested what stack you are using here. Could you please elaborate?
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I use the Qwen models a lot in my private life and they're not quite on par with the SOTA models, but they've definitely gotten to the point where they're usable. Particularly if you invest a lot in harness engineering, they become very usable. It'll be even better when Qwen4 comes out, and less harness engineering will be needed for the same results, and an equivalent amount of harness engineering tuned for the updated models weaknesses will produce results much closer to current SOTA. I was also pretty stunned to see Unsloth's IQ1_M quants of Deepseek v4.1 Flash 1017 held up too. Even that has become runnable locally. It's truly a new era.
https://unsloth.ai/docs/models/deepseek-v4.1-flash
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
Imagine thinking that the look of a website is the right metric to judge the performance of AI models.
It is one metric to judge them. And if you do frontend stuff, it's probably a fairly relevant one.
If your job is to produce websites that look good it's an excellent way to judge performance of AI models _for your workflow_.
What's the problem?
The problem is you can change the prompt a little and get correct results on a local AI too.
Frontend has been solved for years.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
Most of the look of the thing is about the built in tools and skills and harness settings, not the quality of the code. those things can be replicated locally as well.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
Have you tried basalt? I understand that it allows multiple concurrent requests
Those benchmarks are easily gamed in my opinion. I take them woth a huge grain of salt.
I am currently building an integrated CAD + Aero + CAM software package for rapid UAV design iteration, and the open models are all not very good at it. They make a ton of mistakes and I have to end up redoing a lot. And that's using very detailed specs and TDDs provided as context to the models. Opus/Astra/Argon/Gemini Flash 3.8/Sol does a significantly better job in my experience.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
> We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
Who is "we"?
Why 2 5090s instead of just 1 6000? That's 64GB VRAM vs 96GB VRAM. Perfomance is theoretically worse on the 6000 vs 2x 5090s, but given the overhead of tensor parallelism and how slow PCIE is, I imagine it's pretty close for less than half the power draw and heat.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
Same boat here. I'm stuck with my 1070ti.
That's an 8GB card. For gaming (DLSS, etc.), you can still get newer 8GB cards like the 3070 which represent solid upgrades over the 1070 Ti on the used market for under $300. Still not great since similar cards were available at similar prices new a couple years ago, but at least not trading at multiples of MSRP like higher-end cards.
Even 8GB cards might start going up. I recently bought a cheap one so I can auxiliary models on it (think smaller LLMs for memory consolidation / decision models etc). Locally, multi-gpu is pretty cool. You can essentially have one GPU drive the others and do increasingly powerful things when you combine them.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
My current GPU is a 10 GB RTX 3080. To replace it with something that isn't a downgrade, I would have to shell out at least $1,000 CAD.
I see them at a little less than 400€ in the second hand market. So expensive for their age but not hard to replace if you were willing to pay 600-700 for it in 2020.
You should bite the bullet now and buy a decent card on the secondhand market, before more people figure out what they have and double, even triple, the price.
I replaced my 11 year old PC earlier this year, with a new 5070 on discount and only 16GB RAM. I thought it was expensive. Now prices surged like 40% since then and I regreted not getting more RAM.
I feel your pain. I run an RTX 2060 that I bought just before the Plague, and although it still works great on any game I own (I don't really play modern AAA games), I'm worried that I won't be able to afford a replacement if it randomly craps out on me one of these months... Especially since I make less money now than I did when I bought the components.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
Anyone has an idea when will it unclog even a little bit? When is the next NVIDIA lineup comming out? Are they going to release any middle of the road products like strong AMD Insights?
When HBM starts to grow on trees or begins to yield like DRAM.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
Moore Threads Lushan GPU will be out by the end of this year with 64GB RAM.
Will this be available in the US? It looks interesting.
I think the discrete GPUs as a concept is coming to an end. Computer graphics is shifting more and more to using local AI-like things like DLSS that allows to hard-render simpler geometry and shadows and then apply AI-realism filters in real-time.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
DLSS still needs discrete hardware to do that (and keeps requiring new hardware on each generation). Not to mention the types of RAM are not the same and have entirely different tradeoffs - using GDDR for a CPU would tank performance due to an order of magnitude worse latency, using DDR for a GPU would make games a slideshow due to an order of magnitude worse bandwidth. And you'd have to redesign the entire concept of how CPUs and motherboards work to get there, while making AMD and NVIDIA agree to have their chips be behest to the bridges of shitty motherboard and consumer RAM manufacturers even more than they already are.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
> DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
I guess I understood "on its way out" to be a shorter timeframe than you intended, sorry!
How would DLSS, which runs on GPU hardware — and specifically, as I understand it, GPU hardware that accelerates AI inference — eliminate the GPU?
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
DLSS does not require any specific packaging configuration. There's no reason DLSS can't run on DGX Spark or GPU cores of the Apple's M-chips aside from specific Nvidia software optimizations that can surely be worked out if needed.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
So future PC go back to having GPU on motherboard ?
I think so, yeah. Clearly having separate RAM and GPU memory where memory becoming the main productivity bottle neck is not practical or sustainable.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.