AI "Performance Plateau" is to be Expected

There is much talk now about generative artificial intelligence model improvement rates slowing. But such slowdowns are common for most--if not all--technologies. In fact, "hitting the performance plateau," is common. 

For generative AI, the “scaling” problem is at hand. The generative AI scaling problem refers to diminishing returns from increasing model size (number of parameters), the amount of training data, or computational resources.

In the context of generative AI, power laws describe how model performance scales with increases in resources such as model size, dataset size, or compute power. And power laws suggest performance gains will diminish as models grow larger or are trained on more data.

Power laws also mean that although model performance improves with larger training datasets, but the marginal utility of more data diminishes.

Likewise, the use of greater computational resources yields diminishing returns on performance gains.

But that is typical for virtually all technologies: performance gains diminish as additional inputs are increased. Eventually, however, workarounds are developed in other ways. Chipmakers facing a slowing of Moore’s Law rates of improvement got around those limits by creating multi-layer chips, using parallel processing or specialized architectures for example

Technology	Performance Plateau	Key Challenges	Breakthroughs or Workarounds
Steam Engines	Efficiency plateaued due to thermodynamic limits (Carnot cycle).	Material limitations and lack of advanced thermodynamics.	Development of internal combustion engines and electric motors.
Railroads	Speed and efficiency stagnated with steam locomotives.	Limited by steam engine performance and infrastructure capacity.	Introduction of diesel and electric trains.
Aviation	Propeller-driven planes hit speed and altitude limits (~400 mph).	Aerodynamic inefficiency and piston engine limitations.	Jet engines enabled supersonic and high-altitude flight.
Telecommunications	Copper wire networks reached data transmission capacity limits.	Signal attenuation and bandwidth limitations of copper cables.	Transition to fiber-optic technology and satellite communication.
Automotive Engines	Internal combustion engine efficiency (~30% thermal efficiency).	Heat losses and material constraints in engine design.	Adoption of hybrid and electric vehicle technologies.
Semiconductors (Moore's Law)	Scaling transistors beyond ~5 nm became increasingly difficult.	Quantum tunneling, heat dissipation, and fabrication costs.	Development of chiplets, 3D stacking, and quantum computing.
Renewable Energy (Solar)	Silicon solar cells plateaued at ~20–25% efficiency.	Shockley-Queisser limit and cost of advanced materials.	Emerging technologies like perovskite solar cells and tandem cells.
Battery Technology	Lithium-ion batteries plateaued at energy density (~300 Wh/kg).	Materials science constraints and safety issues.	Development of solid-state batteries and alternative chemistries.
Television Display Technology	LCD and OLED reached practical resolution and brightness limits.	Manufacturing cost and diminishing returns in visual quality.	Introduction of micro-LED and quantum dot technologies.

The limits of scaling laws for generative AI will eventually be overcome. But a plateau is not unexpected.