My MacBook Pro is three years old and for the first time in my life a three year primary PC does not feel like a crisis that needs to be resolved immediately. That's right, in part, because I'm waiting for Apple to fix their keypad debugger, and partly because I still can't accumulate the touchpad. But it's also because three years of productivity growth is not what it used to be.
It is no exaggeration to say that Moore's Law, the mind-blowing ruthless exponential growth in our global computing power, has been the most significant force in the world for the last fifty years. So slowing him down and / or dying is a big deal, not least because the consequences are now breaking into every home and every pocket.
We all live in the hope that some other field will come out exponentially, giving us another, similar era, of course. AI / machine learning was a great hope, especially the distant dream of a machine learning feedback cycle, AI has been improving AI at an exponential rate for decades. That seems terribly unlikely now.
In fact, this has always been the case. A few years ago, I spoke with an AI executive who claimed that AI progress was at the heart of the S-curve and we had already reached its peak for audio processing, approaching it for image and video, but were only half way up on the text curve. No reward for guessing which one his company specializes in ̵
Earlier this week, OpenAI released an update last year on its analysis of how the processing power used by AI 1 increased. The result? It "grows exponentially with a 3.4-month doubling (by comparison, Moore's Law has a 2-year doubling period). Since 2012, this figure has increased by more than 300,000x (a two-year doubling period will only result in a 7x increase). "
That's … a lot of computing power to improve the state of the art of artificial intelligence, and it's clear that this growth in computing cannot continue. No no; may not. Unfortunately, the exponential growth of the need for AI computing power has happened almost exactly at the same time as the exponential growth of Moore's law has decreased. Throwing more money at the problem will not help – again here we are talking about exponential growth rates, linear cost adjustments will not move the needle.
The elimination is that even if we take a major breakthrough in efficiency and productivity improvements to reduce the speed of doubling, AI progress seems to be increasingly limited in calculations at a time when our collective growth in computing power is beginning to decline . There may be some breakthrough, but in the absence of one, it sounds a lot like we're looking at the advancement of AI / machine learning, not for long, and in the foreseeable future.  1 Technically measures the "largest AI training courses," but that seems trendy.