Andrej Karpathy (@karpathy) “Anyone else find themselves estimating the "GPT grade" of things you hear/read?” — TopicDigg

Anyone else find themselves estimating the "GPT grade" of things you hear/read? When something is poorly written or generic, it's "GPT-2 grade" content. When something is lit, you can complement it as being "GPT-7 grade" etc. This reminds me of a fun side project I had saved for myself but will realistically never get around to, maybe someone can take a shot. Simply - train a classifier that predicts GPT-grade of any text. The training data would be samples from models of increasing strength. It might be that GPT models are too coarse and that too much changed between each one. Ideally you'd want a nice miniseries where everything is held constant except the model size, e.g. Llama 3 series, esp when they also release the smaller (and bigger!) models. Sample from the models over many prompts (or use base models?), classify the model size, then point it at various text on the internet, e.g. study the divergence between the comments section of WSJ and VC thought leadership :p. To be clear I have no idea if this would work, e.g. the classifier might very well latch on to the style a lot more than the content. Or it might measure not exactly an "intelligence" of text, but more just a "generic-ness", a proxy for frequency or so. It might also be an interesting way to study what is learned as you increase model size. But that's why it's an interesting project - it feels like it might kind of work, but it's not obvious and a number of details are tbd. Eye candy: ChatGPT attempts to visualize the above