Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression.
So they're both sourcing a bit broader zeitgeist.
It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.
Anybody working in the field will be very familiar with these concepts.
…for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it.
The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
common theory. see https://prize.hutter1.net/
And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.
In the article, she credits the 2023 DeepMind paper Language Modeling Is Compression as the source. She also links to this post from 2015: https://colah.github.io/posts/2015-09-Visual-Information/