Ask HN: Can anyone run frontier AI models at usable speeds?
Eight months ago, EXO labs promised fast inference on local hardware. But it is not fast enough. It takes a minute and a half to get a response through hermes, using Kimi K2.6. Is there hardware that a dev can buy that can actually offset cloud bills?
The price of hardware to run a large model is around $500k at the low end (DGX workstation). Then you have to account for the electricity, each GB300 is 1k W.
To even rent this hardware requires connections and commitments right now. So I would largely say no, few can run frontier models in the current climate.
https://www.nvidia.com/en-us/products/workstations/dgx-stati...