points by simonw 23 hours ago Here's a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
nikcub 19 hours ago Real test here would be using tinker to fine tune a tinker model to generate pelicans
dozerly 20 hours ago I’m afraid you’re going to have to start randomizing your benchmarks somehow. I’m sure these models are trained on this problem by now. tyre 20 hours ago If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models. tstrimple 20 hours ago Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?! argee 11 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark. 0-_-0 13 hours ago It's not a benchmark, it's a meme OrangeMusic 10 hours ago To be fair, the "they're trained on this benchmark" response is also a meme.
tyre 20 hours ago If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models.
tstrimple 20 hours ago Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?! argee 11 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
argee 11 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
0-_-0 13 hours ago It's not a benchmark, it's a meme OrangeMusic 10 hours ago To be fair, the "they're trained on this benchmark" response is also a meme.
OrangeMusic 10 hours ago To be fair, the "they're trained on this benchmark" response is also a meme.
m3kw9 10 hours ago How is this still a valid test if there is a good chance they will now specifically train on it?
Real test here would be using tinker to fine tune a tinker model to generate pelicans
well it's good they didn't train on the test!
Gist is returning 403
I’m afraid you’re going to have to start randomizing your benchmarks somehow. I’m sure these models are trained on this problem by now.
If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models.
Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?!
That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
It's not a benchmark, it's a meme
To be fair, the "they're trained on this benchmark" response is also a meme.
How is this still a valid test if there is a good chance they will now specifically train on it?