itissid 22 hours ago

Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.

  • ShadowBlades512 22 hours ago

    One issue is that FPGAs are very fine grained and they need to gaurentee an image the tool confirms works for a chip must work across all produced chips. There is a level of binning for speed and binning for some disabled portions of the chip but it is a lot less flexible then disabling shader cores, memory controllers, and portions of the cache on a GPU.

  • mikeayles 22 hours ago

    With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.

    FPGAs should only really be used for:

    Pre-silicon verification

    Designs where power/cost aren't important (a lot of high end industrial or military for example)

    Or low quantity designs that won't make an ROI when taping out an ASIC.

    That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)

    I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.

    I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.

    Hbm really is the magic that makes it work (economically)

    • femto 21 hours ago

      > With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.

      In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.

      • mlmonkey 20 hours ago

        Probably gathering dust somewhere along with HP's memristors ... :-D

    • Eextra953 5 hours ago

      'FPGAs should only really be used for' You're list is focused on using and FPGA as a processor but IME FPGAs are mostly used when you need massive precise IO along with precision timing . Or if you need any sort of reconfigurable glue logic or things like that.

  • numpad0 21 hours ago

    Isn't it like FPGAs use more transistors per equivalent transistor it simulates?

    And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?

    • bobmcnamara 19 hours ago

      Oh yeah.

      In ye olde days, it was a universal gate an a few SRAM bits to configure each one.

    • 15155 15 hours ago

      "more" is a monumental understatement.

    • schobi 13 hours ago

      The overhead of FPGA Vs ASIC is in the order of 10x to 100x, depending if you count logic gates, chip area, routing overhead or power.

      This compares for the same technology (chip feature size) But for your chip you might not be able to pay for the smallest nm chip generation. That might make up for some of the gap

  • nomel 17 hours ago

    Their logic is based on lookup tables of the logic block, rather than the direct physical logic gates! So it's HUGELY inefficient for die area, like a few thousand transistors each rather than, say, 6 for a OR gate! But, the software won't waste them on a single gate, instead combining many connected logic elements into one "cell". For example, Xilinx has 6 inputs and one out, with 64 element LUT, which still a terrible, more than a 50x increase in die area, and therefore cost.

    • imtringued 12 hours ago

      A 6 LUT is well below a thousand transistors. Roughly 800 transistors or less. You need 64 bit for the SRAM which is 360 transistors. Double that for the multiplexer and various other features like fracturing or shift register features.

      And that's a big LUT, more economical FPGA designs have smaller LUTs. For example a 5 LUT would only need 32 bits of SRAM which means it is half the size.

      Also, a LUT is more powerful than an OR gate...

      • 15155 12 hours ago

        And how about all of the programmable interconnects and routing resources required to connect these LUTs?

      • nomel 2 hours ago

        You're confusing what they are. They're LUT based cells that emulate logic, meshed together with other LUT based cells, forming a "tile". They are not simple LUTs. Yourr probably right that the LUT portion of that more complex cell is somewhere around there, but then there's all the rest of the stuffs they do.

jluxenberg 23 hours ago

A fun and interesting project, but to spend $10k+ in tokens on a curiosity seems excessive! Outside of my budget for sure.

  • ricardobeat 23 hours ago

    They mention it's from startup credits, so entirely free for them.

  • mixtureoftakes 23 hours ago

    Also known as a singular 200$ subscription. Nobody sane is paying "real" api prices

    • hsuduebc2 17 hours ago

      Companies do and for grunt sw jobs even using opus is relatively cheap.

      • martinjc 6 hours ago

        Which companies do that? It is way cheaper to get enterprise team plan.

  • yieldcrv 20 hours ago

    soon people like this won't be spending money on tokens because they will have synthesized their entire hardware, driver, software, inference engine and harness stack running at a large enough context window with high enough tokens/sec, with their chosen LLM modifying its drivers, software, inference engine, harness and its own weights for their use case, re-initializing itself and moving on

  • adastra22 18 hours ago

    I routinely spend the API-price equivalent of $14,000/mo on my $200/mo subscription.

  • UncleOxidant 15 hours ago

    Then use a local model. Qwen3.8-Flash-Next is very proficient at Verilog (and system verilog)

seanjhkwon 21 hours ago

It shows the importance of automated indicators when giving AI huge tasks. Maybe more important than the intelligence of the model.

codekansas 1 day ago

This is cool

Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...

hoppp 23 hours ago

This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.

chalkycrimp 1 day ago

Good chance I'm not the only one thinking this: share the code

datadrivenangel 1 day ago

Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.

  • datadrivenangel 1 day ago

    it only cost them $10,000 in Claude API usage. :)

    • hmate9 1 day ago

      You get that much usage in a monthly $200 subscription.

      • dofm 1 day ago

        And a kitten.

      • dvh 1 day ago

        So who's paying the cost?

        • exe34 1 day ago

          Corporate customers

        • ralph84 1 day ago

          API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.

          • surgical_fire 1 day ago

            People believe in some wild stuff.

            • thephyber 17 hours ago

              This comment is too vague to be useful to anyone.

          • drob518 22 hours ago

            If that’s true then the api is waaaaaay overpriced.

            • thephyber 17 hours ago

              Or... it may just be a lot less underpriced than subscriptions.

              Both Anthropic and OpenAI are believed to be heavily subsidizing inference as they compete for market share. Once their shareholders demand profits over market share, the pricing model will rise (or mix in other revenue streams like ads).

        • stronglikedan 1 day ago

          Everyone else that doesn't use the entirety of the plan they pay for.

        • nyc_data_geek 1 day ago

          Retail and society as ANT shares get pulled into index and wealth funds

        • hoppp 23 hours ago

          The IPO they just raised can subsidise it for a while.

        • thephyber 17 hours ago

          Investors / debtors, same as Uber / Lyft before they went public.

          But also "cost" and "API price" are different. The actual costs of inference will likely rise once Anthropic / OpenAI burn through their cash pile and investors want profits instead of competition for market share.

          API price is high and fixed. API users have inference resource priority over subscribers.

  • risyachka 1 day ago

    Fpga is software-driven, model can verify its output easily.

    Zero surprise it did it

  • ilaksh 1 day ago

    Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.

    But that still not a breakthrough or anything for such a small active set of parameters that is quantized

reuben364 21 hours ago

Over the week or so I have vibe coded a Lean-like language with

- quantitative types

- a Rust-ish DSL that lowers into the base language

- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language

- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.

- A category theory and real number library

- A graph storage engine with queries written using profunctor optics

- A datalog with an egglog in progress

- A Joern-like code analysis engine for C and C# using tree-sitter to parse

- A React style framework

- Editor plugins for my language

All at varying stages of completeness.

What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)

My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.

  • tracerbulletx 20 hours ago

    I used to run a medium sized adwords company that managed landing pages, adwords, and lead generation for about 50 companies. I moved into engineering at a bigger company after a while, but since LLMs I have essentially recreated all of the page builder, custom domain hosting, lead gen, CRM tools, and adwords integrations I used to subscribe to as my own platform and started taking on customers again on a home grown platform and its WAY better. I say all this just to say that I agree, and that even if you know the domain it still works and verifiably so.

  • mswphd 20 hours ago

    I've been doing similar things (turning spare computers into SLURM nodes) for the same reason for my pet projects too. It has been surprisingly fun.

  • keyle 19 hours ago

    What is "slurm node"?

    • thephyber 17 hours ago

      > The Slurm Workload Manager, formerly known as Simple Linux Utility for Resource Management (SLURM), or simply Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

      https://en.wikipedia.org/wiki/Slurm_Workload_Manager

lxe 1 day ago

Repetitive "make it faster" prompting is essentially what all these "RSI" claims are

  • kooi 17 hours ago

    As an engineer, I felt ashamed reading that.

    Common man, prompt at the level of someone with at least a little of background knowledge.

    • theGeatZhopa 13 hours ago

      But it helps, it’s shorter and you don’t have to know what exactly is making that thing slowl (slow owl).

      In my Hobby project (cv generator) I don’t like the ui to be unresponsive during pdf generation and generally I want to have a fast system. So I set my cpu under heavy load (or put it into power saving mode where the cpu is running at 0.9ghz) and tell opus to make it faster.

      This went from 45s under very heavy load to 5 sek

      (Heavy load: almost 100 cpu constantly and 90% ram filled up, additional power saving mode on)

      Claude often told me the system is busy and „this,that“-chat. But I could resist stopping saying „make it faster, don’t stop any processes. It’s your new reality. We’re now sharing out cpu with the neighbor university. Live with it“

      And yes. Heavy load. Python. Running blizzfast now. It were like 10 millions of testing and code and git commits tokens and 5h to 10h to get it there. Somewhat 500 tests written for the speed optimization. With background knowledge I still would not know where to optimize and would need to tell it exactly what to test and to compose tests for

saejox 1 day ago

i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.

  • 15155 15 hours ago

    > we can emulate an FPGA on a rtx 3090?

    Not with any degree of performance.

    • imtringued 12 hours ago

      I checked and actually it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.

      • 15155 12 hours ago

        > I checked

        Checked what, an LLM?

        > it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.

        Ok, let me know how that performs, especially for basically any bit manipulation task or true parallelism. How do you control the precise timing of execution?

mococa 21 hours ago

What lack of responsibility

spwa4 1 day ago

I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.

(unfortunately for a decent model it will also be price competitive with a big nvidia card)

  • 15155 15 hours ago

    Which FPGA can be "competitive with a big NVIDIA card" and draw just 5W?

    Every big US+ and Versal chip in existence idles at more than 5W. Samsung Aquabolt HBM uses far more than 5W (even at 100GB/s.)

    • spwa4 4 hours ago

      I can't tell you many details, but a neural net upload into a tens-of-thousands-of-LUT FPGA can do thousands of hertz processing on all the device's (it's a communications device) status signals in less than 5W. Plus it's square, but about the area of a credit card. You're not doing that with anything else, not even an Nvidia Jetson Nano.

      • 15155 3 hours ago

        Word salad. Tens of thousands of LUTs are completely incomparable to any GPU, even ten year old GPUs lap every 100W+ FPGA in matmul and the vast majority of memory-bound tasks.