Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.
One issue is that FPGAs are very fine grained and they need to gaurentee an image the tool confirms works for a chip must work across all produced chips. There is a level of binning for speed and binning for some disabled portions of the chip but it is a lot less flexible then disabling shader cores, memory controllers, and portions of the cache on a GPU.
With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
> With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
'FPGAs should only really be used for'
You're list is focused on using and FPGA as a processor but IME FPGAs are mostly used when you need massive precise IO along with precision timing . Or if you need any sort of reconfigurable glue logic or things like that.
Isn't it like FPGAs use more transistors per equivalent transistor it simulates?
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
The overhead of FPGA Vs ASIC is in the order of 10x to 100x, depending if you count logic gates, chip area, routing overhead or power.
This compares for the same technology (chip feature size) But for your chip you might not be able to pay for the smallest nm chip generation. That might make up for some of the gap
Their logic is based on lookup tables of the logic block, rather than the direct physical logic gates! So it's HUGELY inefficient for die area, like a few thousand transistors each rather than, say, 6 for a OR gate! But, the software won't waste them on a single gate, instead combining many connected logic elements into one "cell". For example, Xilinx has 6 inputs and one out, with 64 element LUT, which still a terrible, more than a 50x increase in die area, and therefore cost.
A 6 LUT is well below a thousand transistors. Roughly 800 transistors or less. You need 64 bit for the SRAM which is 360 transistors. Double that for the multiplexer and various other features like fracturing or shift register features.
And that's a big LUT, more economical FPGA designs have smaller LUTs. For example a 5 LUT would only need 32 bits of SRAM which means it is half the size.
You're confusing what they are. They're LUT based cells that emulate logic, meshed together with other LUT based cells, forming a "tile". They are not simple LUTs. Yourr probably right that the LUT portion of that more complex cell is somewhere around there, but then there's all the rest of the stuffs they do.
soon people like this won't be spending money on tokens because they will have synthesized their entire hardware, driver, software, inference engine and harness stack running at a large enough context window with high enough tokens/sec, with their chosen LLM modifying its drivers, software, inference engine, harness and its own weights for their use case, re-initializing itself and moving on
This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
Or... it may just be a lot less underpriced than subscriptions.
Both Anthropic and OpenAI are believed to be heavily subsidizing inference as they compete for market share. Once their shareholders demand profits over market share, the pricing model will rise (or mix in other revenue streams like ads).
Investors / debtors, same as Uber / Lyft before they went public.
But also "cost" and "API price" are different. The actual costs of inference will likely rise once Anthropic / OpenAI burn through their cash pile and investors want profits instead of competition for market share.
API price is high and fixed. API users have inference resource priority over subscribers.
Over the week or so I have vibe coded a Lean-like language with
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.
I used to run a medium sized adwords company that managed landing pages, adwords, and lead generation for about 50 companies. I moved into engineering at a bigger company after a while, but since LLMs I have essentially recreated all of the page builder, custom domain hosting, lead gen, CRM tools, and adwords integrations I used to subscribe to as my own platform and started taking on customers again on a home grown platform and its WAY better. I say all this just to say that I agree, and that even if you know the domain it still works and verifiably so.
> The Slurm Workload Manager, formerly known as Simple Linux Utility for Resource Management (SLURM), or simply Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.
But it helps, it’s shorter and you don’t have to know what exactly is making that thing slowl (slow owl).
In my Hobby project (cv generator) I don’t like the ui to be unresponsive during pdf generation and generally I want to have a fast system. So I set my cpu under heavy load (or put it into power saving mode where the cpu is running at 0.9ghz) and tell opus to make it faster.
This went from 45s under very heavy load to 5 sek
(Heavy load: almost 100 cpu constantly and 90% ram filled up, additional power saving mode on)
Claude often told me the system is busy and „this,that“-chat. But I could resist stopping saying „make it faster, don’t stop any processes. It’s your new reality. We’re now sharing out cpu with the neighbor university. Live with it“
And yes. Heavy load. Python. Running blizzfast now. It were like 10 millions of testing and code and git commits tokens and 5h to 10h to get it there. Somewhat 500 tests written for the speed optimization. With background knowledge I still would not know where to optimize and would need to tell it exactly what to test and to compose tests for
I checked and actually it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
> it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
Ok, let me know how that performs, especially for basically any bit manipulation task or true parallelism. How do you control the precise timing of execution?
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
I can't tell you many details, but a neural net upload into a tens-of-thousands-of-LUT FPGA can do thousands of hertz processing on all the device's (it's a communications device) status signals in less than 5W. Plus it's square, but about the area of a credit card. You're not doing that with anything else, not even an Nvidia Jetson Nano.
Word salad. Tens of thousands of LUTs are completely incomparable to any GPU, even ten year old GPUs lap every 100W+ FPGA in matmul and the vast majority of memory-bound tasks.
Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.
One issue is that FPGAs are very fine grained and they need to gaurentee an image the tool confirms works for a chip must work across all produced chips. There is a level of binning for speed and binning for some disabled portions of the chip but it is a lot less flexible then disabling shader cores, memory controllers, and portions of the cache on a GPU.
With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
> With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
Probably gathering dust somewhere along with HP's memristors ... :-D
'FPGAs should only really be used for' You're list is focused on using and FPGA as a processor but IME FPGAs are mostly used when you need massive precise IO along with precision timing . Or if you need any sort of reconfigurable glue logic or things like that.
Isn't it like FPGAs use more transistors per equivalent transistor it simulates?
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
Oh yeah.
In ye olde days, it was a universal gate an a few SRAM bits to configure each one.
"more" is a monumental understatement.
The overhead of FPGA Vs ASIC is in the order of 10x to 100x, depending if you count logic gates, chip area, routing overhead or power.
This compares for the same technology (chip feature size) But for your chip you might not be able to pay for the smallest nm chip generation. That might make up for some of the gap
Their logic is based on lookup tables of the logic block, rather than the direct physical logic gates! So it's HUGELY inefficient for die area, like a few thousand transistors each rather than, say, 6 for a OR gate! But, the software won't waste them on a single gate, instead combining many connected logic elements into one "cell". For example, Xilinx has 6 inputs and one out, with 64 element LUT, which still a terrible, more than a 50x increase in die area, and therefore cost.
A 6 LUT is well below a thousand transistors. Roughly 800 transistors or less. You need 64 bit for the SRAM which is 360 transistors. Double that for the multiplexer and various other features like fracturing or shift register features.
And that's a big LUT, more economical FPGA designs have smaller LUTs. For example a 5 LUT would only need 32 bits of SRAM which means it is half the size.
Also, a LUT is more powerful than an OR gate...
And how about all of the programmable interconnects and routing resources required to connect these LUTs?
You're confusing what they are. They're LUT based cells that emulate logic, meshed together with other LUT based cells, forming a "tile". They are not simple LUTs. Yourr probably right that the LUT portion of that more complex cell is somewhere around there, but then there's all the rest of the stuffs they do.
I did something like this with Opus 4.5: https://github.com/cpldcpu/smollm.c
It seems implementing transformers is the new "hello world" of AI assisted coding.
A fun and interesting project, but to spend $10k+ in tokens on a curiosity seems excessive! Outside of my budget for sure.
They mention it's from startup credits, so entirely free for them.
Also known as a singular 200$ subscription. Nobody sane is paying "real" api prices
Companies do and for grunt sw jobs even using opus is relatively cheap.
Which companies do that? It is way cheaper to get enterprise team plan.
soon people like this won't be spending money on tokens because they will have synthesized their entire hardware, driver, software, inference engine and harness stack running at a large enough context window with high enough tokens/sec, with their chosen LLM modifying its drivers, software, inference engine, harness and its own weights for their use case, re-initializing itself and moving on
I routinely spend the API-price equivalent of $14,000/mo on my $200/mo subscription.
Then use a local model. Qwen3.8-Flash-Next is very proficient at Verilog (and system verilog)
It shows the importance of automated indicators when giving AI huge tasks. Maybe more important than the intelligence of the model.
This is cool
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.
Good chance I'm not the only one thinking this: share the code
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
it only cost them $10,000 in Claude API usage. :)
You get that much usage in a monthly $200 subscription.
And a kitten.
So who's paying the cost?
Corporate customers
API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.
People believe in some wild stuff.
This comment is too vague to be useful to anyone.
If that’s true then the api is waaaaaay overpriced.
Or... it may just be a lot less underpriced than subscriptions.
Both Anthropic and OpenAI are believed to be heavily subsidizing inference as they compete for market share. Once their shareholders demand profits over market share, the pricing model will rise (or mix in other revenue streams like ads).
The claim was that both subs and API are profitable. If subs are profitable, then API at 100x the cost is way overpriced.
Widely believed != true
> As revenue has scaled over the last year, compute costs have fallen from USD 2.01 to just USD 0.48 per revenue dollar
https://global.morningstar.com/en-eu/stocks/where-anthropics...
Everyone else that doesn't use the entirety of the plan they pay for.
Retail and society as ANT shares get pulled into index and wealth funds
The IPO they just raised can subsidise it for a while.
Investors / debtors, same as Uber / Lyft before they went public.
But also "cost" and "API price" are different. The actual costs of inference will likely rise once Anthropic / OpenAI burn through their cash pile and investors want profits instead of competition for market share.
API price is high and fixed. API users have inference resource priority over subscribers.
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.
But that still not a breakthrough or anything for such a small active set of parameters that is quantized
Chat jimmy https://chatjimmy.ai/ runs around 15k tokens a second.
Over the week or so I have vibe coded a Lean-like language with
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.
I used to run a medium sized adwords company that managed landing pages, adwords, and lead generation for about 50 companies. I moved into engineering at a bigger company after a while, but since LLMs I have essentially recreated all of the page builder, custom domain hosting, lead gen, CRM tools, and adwords integrations I used to subscribe to as my own platform and started taking on customers again on a home grown platform and its WAY better. I say all this just to say that I agree, and that even if you know the domain it still works and verifiably so.
I've been doing similar things (turning spare computers into SLURM nodes) for the same reason for my pet projects too. It has been surprisingly fun.
What is "slurm node"?
> The Slurm Workload Manager, formerly known as Simple Linux Utility for Resource Management (SLURM), or simply Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.
https://en.wikipedia.org/wiki/Slurm_Workload_Manager
very nice. https://github.com/emberian/obend has been angling in a different direction, but maybe is quite compatible metatheoretically ...
Repetitive "make it faster" prompting is essentially what all these "RSI" claims are
As an engineer, I felt ashamed reading that.
Common man, prompt at the level of someone with at least a little of background knowledge.
But it helps, it’s shorter and you don’t have to know what exactly is making that thing slowl (slow owl).
In my Hobby project (cv generator) I don’t like the ui to be unresponsive during pdf generation and generally I want to have a fast system. So I set my cpu under heavy load (or put it into power saving mode where the cpu is running at 0.9ghz) and tell opus to make it faster.
This went from 45s under very heavy load to 5 sek
(Heavy load: almost 100 cpu constantly and 90% ram filled up, additional power saving mode on)
Claude often told me the system is busy and „this,that“-chat. But I could resist stopping saying „make it faster, don’t stop any processes. It’s your new reality. We’re now sharing out cpu with the neighbor university. Live with it“
And yes. Heavy load. Python. Running blizzfast now. It were like 10 millions of testing and code and git commits tokens and 5h to 10h to get it there. Somewhat 500 tests written for the speed optimization. With background knowledge I still would not know where to optimize and would need to tell it exactly what to test and to compose tests for
i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.
> we can emulate an FPGA on a rtx 3090?
Not with any degree of performance.
I checked and actually it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
> I checked
Checked what, an LLM?
> it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
Ok, let me know how that performs, especially for basically any bit manipulation task or true parallelism. How do you control the precise timing of execution?
crazyy
Now do silicon. ;)
Did you miss this?
https://architectlabs.com/blog/frostbite
They seem to target general-purpose non-homogeneous parallelism, not inference, right?
Sure, but that's the more difficult case. They already did the easier inference case:
https://architectlabs.com/blog/redwood
What lack of responsibility
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
Which FPGA can be "competitive with a big NVIDIA card" and draw just 5W?
Every big US+ and Versal chip in existence idles at more than 5W. Samsung Aquabolt HBM uses far more than 5W (even at 100GB/s.)
I can't tell you many details, but a neural net upload into a tens-of-thousands-of-LUT FPGA can do thousands of hertz processing on all the device's (it's a communications device) status signals in less than 5W. Plus it's square, but about the area of a credit card. You're not doing that with anything else, not even an Nvidia Jetson Nano.
Word salad. Tens of thousands of LUTs are completely incomparable to any GPU, even ten year old GPUs lap every 100W+ FPGA in matmul and the vast majority of memory-bound tasks.