Jaime Sevilla

Announcing Epoch: A research organization investigating the road to Transformative AI

Summary * We are a new research organization working on investigating trends in Machine Learning and forecasting the development of Transformative Artificial Intelligence * This work is done in close collaboration with other organizations, like Rethink Priorities, Open Philanthropy, and MIT CSAIL * We will be hiring for 2-4 full-time roles this summer – more information here * You can find up-to-date information about Epoch on our website What is Epoch? Epoch is a new research organization that works to support AI strategy and improve forecasts around the development of Transformative Artificial Intelligence (TAI) – AI systems that have the potential to have an effect on society as large as that of the industrial revolution. Our founding team consists of seven members – Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Pablo Villalobos, Eduardo Infante-Roldán, Marius Hobbhahn, and Anson Ho. Collectively, we have backgrounds in Machine Learning, Statistics, Economics, Forecasting, Physics, Computer Engineering, and Software Engineering. Our work involves close collaboration with other organizations, such as MIT CSAIL, Open Philanthropy, and Rethink Priorities’ AI Governance and Strategy team. We are advised by Tom Davidson from Open Philanthropy and Neil Thompson from MIT CSAIL. Rethink Priorities is also our fiscal sponsor. Our mission Epoch seeks to clarify when and how TAI capabilities will be developed. We see these two problems as core questions for informing AI strategy decisions by grantmakers, policy-makers, and technical researchers. We believe that to make good progress on these questions we need to advance towards a field of AI forecasting. We are committed to developing tools, gathering data and creating a scientific ecosystem to make collective progress towards this goal. Epoch´s website Our research agenda Our work at Epoch encompasses two interconnected lines of research: * The analysis of trends in Machine Learning. We aim to gather data

97Jun 27, 2022

Jaime Sevilla

Message

1897

199

157

Announcing Epoch’s dashboard of key trends and figures in Machine Learning

Developments in Machine Learning have been happening extraordinarily fast, and as their impacts become increasingly visible, it becomes ever more important to develop a quantitative understanding of these changes. However, relevant data has thus far been scattered across multiple papers, has required expertise to gather accurately, or has been otherwise...

Apr 13, 202335

The longest training run

In short: Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms. [Edited 2022/09/22 to fix an error in the hardware improvements + rising...

Aug 17, 202271

Announcing Epoch: A research organization investigating the road to Transformative AI

Jun 27, 202297

Projecting compute trends in Machine Learning

Summary Using our dataset of milestone Machine Learning models, and our recent analysis of compute trends in ML, we project forward 70 years worth of trends in the amount of compute used to train Machine Learning models. Our simulations account for (a) uncertainty in estimates of the growth rates in...

Mar 7, 202259

Compute Trends Across Three eras of Machine Learning

https://arxiv.org/abs/2202.05924 What do you need to develop advanced Machine Learning systems? Leading companies don’t know. But they are very interested in figuring it out. They dream of replacing all these pesky workers with reliable machines who take no leave and have no morale issues. So when they heard that throwing...

Feb 16, 202294

Estimating training compute of Deep Learning models

by Jaime Sevilla, Lennart Heim, Marius Hobbhahn, Tamay Besiroglu, and Anson Ho You can find the complete article here. We provide a short summary below. In short: To estimate the compute used to train a Deep Learning model we can either: 1) directly count the number of operations needed or...

Jan 20, 202237

What’s the backward-forward FLOP ratio for Neural Networks?

Summary: 1. Classic settings, i.e. deep networks with convolutional layers and large batch sizes, almost always have backward-forward FLOP ratios close to 2:1. 2. Depending on the following criteria we can encounter ratios between 1:1 and 3:1 1. Type of layer: Passes through linear layers have as many FLOP as...

Dec 13, 202120

Load More (7/12)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Jaime Sevilla

Jaime Sevilla

Jaime Sevilla

Announcing Epoch: A research organization investigating the road to Transformative AI

Compute Trends Across Three eras of Machine Learning

The longest training run

Projecting compute trends in Machine Learning

Jaime Sevilla

Announcing Epoch’s dashboard of key trends and figures in Machine Learning

The longest training run

Announcing Epoch: A research organization investigating the road to Transformative AI

Projecting compute trends in Machine Learning

Compute Trends Across Three eras of Machine Learning

Estimating training compute of Deep Learning models

What’s the backward-forward FLOP ratio for Neural Networks?

Announcing Epoch: A research organization investigating the road to Transformative AI

Compute Trends Across Three eras of Machine Learning

The longest training run

Projecting compute trends in Machine Learning

Announcing Epoch’s dashboard of key trends and figures in Machine Learning

The longest training run

Announcing Epoch: A research organization investigating the road to Transformative AI

Projecting compute trends in Machine Learning

Compute Trends Across Three eras of Machine Learning

Estimating training compute of Deep Learning models

What’s the backward-forward FLOP ratio for Neural Networks?