We just launched Chip Benchmark, an open-source tool for hardware-centric benchmarking of open-weight LLMs across accelerators like NVIDIA A100/H100/L40S and AMD MI300X. It measures throughput, latency, and time-to-first-token with transparent scripts and an interactive web dashboard—making apples-to-apples comparisons easier.
We're actively welcoming contributions, new hardware support, and benchmark requests.
We just launched Chip Benchmark, an open-source tool for hardware-centric benchmarking of open-weight LLMs across accelerators like NVIDIA A100/H100/L40S and AMD MI300X. It measures throughput, latency, and time-to-first-token with transparent scripts and an interactive web dashboard—making apples-to-apples comparisons easier.
We're actively welcoming contributions, new hardware support, and benchmark requests.
Repo here: https://github.com/Herdora/chip-benchmark Dashboard: https://herdora.com/benchmark
Feedback and contributions welcome! We made it super easy to add other architectures by including the script we used for benchmarking.
^ We currently just have llama3.1-8b, so we'll be working on adding more models across more hardware options!