This evaluation comes lower than 24 hours after the corporate launched an improve for the GPT-4o mannequin. “The mannequin’s artistic writing functionality has leveled up–additional pure, collaborating, and tailor-made writing to spice up relevance & readability,” OpenAI wrote on X. “It’s furthermore bigger at working with uploaded recordsdata, offering deeper insights & additional thorough responses.” Whether or not or not or not these claims proceed to carry up is now being strong unsure.
“We’ve accomplished working our unbiased evals on OpenAI’s GPT-4o launch yesterday and are persistently measuring materially decrease eval scores than the August launch of GPT-4o,” the Synthetic Evaluation launched by the use of an X submit on Thursday, noting that the mannequin’s Synthetic Evaluation High quality Index decreased from 77 to 71 (and is now equal to that of GPT-4o mini).
What’s additional, GPT-4o’s effectivity on the GPQA Diamond benchmark decreased from 51% to 39% whereas its MATH benchmarks decreased from 78% to 69%.
Concurrently, the researchers found larger than a doubling all through the speed enhance of the mannequin’s responses, accelerating from spherical 80 output tokens per second to roughly 180 tokens/s. “We’ve generally noticed considerably quicker speeds on launch day for OpenAI fashions (seemingly ensuing from OpenAI provisioning performance forward of adoption), nonetheless beforehand haven’t seen a 2x velocity distinction,” the researchers wrote.
Wait – is the mannequin new GPT-4o a smaller and fewer clever mannequin?
We’ve accomplished working our unbiased evals on OpenAI’s GPT-4o launch yesterday and are persistently measuring materially decrease eval scores than the August launch of GPT-4o.
GPT-4o (Nov) vs GPT-4o (Aug):
➤… pic.twitter.com/gjY2pBFuUv— Synthetic Evaluation (@ArtificialAnlys) November 21, 2024
“Based mostly completely on this knowledge, we conclude that it’s seemingly that OpenAI’s Nov twentieth GPT-4o mannequin is a smaller mannequin than the August launch,” they continued. “On condition that OpenAI has not reduce costs for the Nov twentieth model, we advocate that builders don’t shift workloads away from the August model with out cautious testing.”
GPT-4o was first launched in Might 2024 to surpass the prevailing GPT-3.5 and GPT-4 fashions. GPT-4o offers state-of-the-art benchmark ends in voice, multilingual, and imaginative and prescient duties, in accordance with OpenAI, making it good for superior features like real-time translation and conversational AI.