OpenAI open sources Evals, its framework for automatically evaluating the performance of its AI models, letting users report shortcomings and guide improvements
Want early access to GPT-4? Do it now: https://github.com/... is an official framework for evaluating OpenAI models. They will grant GPT-4 access to those who submit high quality evals. Thanks to my friend Andrew Kondrich @kondrich2 who built this initiative at OpenAI!
If you see something GPT-4 can't do well, or think you can prove a fundamental deficiency, contribute evals! This is by far the best way to help close these skill gaps. Internally we use evals to guide enormous amounts of model development. https://github.com/...
as LLMs grow and grow and grow in capabilities, it is getting more impt to have good model evaluation/benchmarking frameworks. OpenAI is also releasing their eval framework, fully MIT licensed: https://github.com/... Used by Stripe and well documented. Runs MMLU in 189 LOC https:…
We are giving priority GPT-4 access to those who contribute evals to our new evals repo: https://github.com/... Here, you can write tests for the model so we can improve things over time.