According to the Chatbot Arena leaderboard, a platform for evaluating AI determined by user votes, Gemini’s latest update--Gemini-Exp-1114--ranks best among large language models.
It is worth noting that leaders change somewhat frequently, with the top-five models presently all versions of OpenAI or Google models. Perhaps notably, Grok-2-08-13 ranks sixth.
source: Chatbot Arena
It might also be worth noting that OpenAI's models (such as GPT-4) and Anthropic's Claude models have consistently ranked near the top of the leaderboard.
And leadership seems to have changed since the spring of 2023. Consider the leaderboard published by LMsys in the spring of 2023. ChatGPT 3.5 launched in late 2022 and seems to have been in the top five of the Arena leaderboard since its inception in the spring of 2023.
source: lmsys.org
Eventually, business history would suggest leadership of the market will condense, as have other technology markets. So. the LLM market is likely to evolve into a structure characterized by oligopolistic competition among a few major players, complemented by a range of specialized providers catering to specific industries or use cases.
In 2023 five LLMs had more than 88 percent market share. That leadership group might condense further, eventually.
On the other hand, the room for specialized platforms might remain. How many of us would not see any way for OpenAI, Google, Microsoft, Meta, Amazon, Apple and IBM, for example, to continue as operators of domain-specific LLMs, no matter what happens ;with the broader market?
And who might doubt that specialized industry-specific platforms could number between 10 and 20 (catering to different sectors like healthcare, finance, legal)?
And of the leaders, might open-source initiatives include three to five significant contributors?
Might AI-as-a-Service providers number 10 to 15 “significant” players, even if the top five or so positions include AWS, Google Cloud, Azure, Meta and Amazon?
Also, if history is instructive, could there not exist five to 10 Integration and orchestration platforms as well?
The issue is what “winner takes all” will mean in the LLM ecosystem and platforms markets. Current examples include just one or perhaps two leaders in existing markets, which is more on the “operating system” model. On the other hand, most of us would have a hard time deciding on less than perhaps four leading LLMs for some time to come.
And some structural differences between existing technology market structures and LLMs come to mind. Unlike the operating system market, LLMs don't require the same level of user lock-in or hardware integration. So the "two-leaders” pattern might not emerge.
Roughly the same argument might be made about the e-commerce or search market structures, where one leader tends to emerge. The competitiveness of existing LLMs, with continual upgrades, tends to dispel the notion that any single provider will achieve technological superiority on a sustainable basis.
LLMs also lack the network effects and user-generated content central to social media platforms. So it is possible the one leader model might not develop. Right now, differences between leading platforms are relatively subtle.
So the likely direction is “winner take most” more than “winner take all.” Even if network effects are not so strong, high capital intensity, branding and trust issues and the ability to vertically integrate with existing ecosystems (Google, Apple, Microsoft, Meta) create enormous advantages for a few contenders.
At least for the moment, “winner take all” is hard to see. A still-oligopolistic, but “winner take most” structure with a handful of leaders might be more plausible.
