Tamay Besiroglu

https://arxiv.org/abs/2202.05924

What do you need to develop advanced Machine Learning systems? Leading companies don’t know. But they are very interested in figuring it out. They dream of replacing all these pesky workers with reliable machines who take no leave and have no morale issues. 

So when they heard that throwing processing power at the problem might get you far along the way, they did not sit idly on their GPUs. But, how fast is their demand for compute growing? And is the progress regular?

Enter us. We have obsessively analyzed trends in the amount of compute spent training milestone Machine Learning models.

Our analysis shows that:

• Before the Deep Learning era, training compute approximately followed Moore’s law, doubling every ≈20 months.
• The Deep Learning era starts somewhere between 2010 and 2012. After that,