In order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions.
Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small.
I already started.
$ neofetch
-` x@decpti
.o+` -------
`ooo/ OS: Arch Linux ARM aarch64
`+oooo: Host: Apple Mac Studio (M1 Ultra, 2022)
`+oooooo: Kernel: 6.1.0-rc6-asahi-4-1-ARCH
-+oooooo+: Uptime: 4 hours, 23 mins
`/:-:++oooo+: Packages: 177 (pacman)
`/++++/+++++++: Shell: bash 5.1.16
`/++++++++++++++: Resolution: 1920x1080
`/+++ooooooooooooo/` Terminal: /dev/pts/0
./ooosssso++osssssso+` CPU: (20) @ 2.064GHz
.oossssso-````/ossssss+` Memory: 717MiB / 129540MiB
-osssssso. :ssssssso.
:osssssss/ osssso+++.
/ossssssss/ +ssssooo/-
Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!
tinygrad
https://github.com/geohot/tinygrad/tree/master/accel/ane
But I have not tested it on Linux since Asahi has not yet added support.
Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.
Please, this is not r/unixporn.
Is this a joke or an ad? Paying Apple hardware premiums (for hardware that isn't well supported by PyTorch, et al.) just to load Linux onto it and run AI models in a closet somewhere, all seems like an incredibly wasteful way to go about it.