We will perform dataset curation, synthetic data generation, and LLM training, fine-tuning, and evals to distinguish and quantify the effects of data improvements, separately from progress in algorithms and architectures, on increasing AI capabilities.
About the project
Figure 5 of Algorithmic Progress in Language Models illustrates that there was a 4_10^10x scale-up of LMs from 2012 to 2023, of which 1.7_10^7x was from physical compute scaling and 2.2*10^4x was from algorithmic progress (including algorithms, optimizers, architectures, and training data quality improvements in this second category). The goal is to isolate the effects of data quality improvements and estimate the data-only compute-equivalent gain, by doing experiments such as the following:
- Fixed recipe, swap only the data
- Hold constant: architecture, tokenizer, optimizer, hyperparameters, training code, numeric precision, evaluation procedure
- Vary dataset: raw web, deduplicated web, filtered web, curated or enriched datasets
- Test at several scales
- Data time-leap
- {old, new} data x {old, new} algorithms+architectures
- Then do a Shapley decomposition to find compute savings from each.
- Ablations on modern data pipeline
- Longitudinal experiment: same recipe; dataset from each year 2014-2026
- Generate datasets from models of different intelligence; measure student model quality
Theory of change
This tells us the rate of progress of AI capabilities due to data progress, which is relevant for the ability of AI pauses to prevent capability advancement by restricting algorithmic research or data operations.
Your role
Run the experiments proposed, as well as suggest and run follow-up experiments based on initial findings.
Prerequisites
Able to train and fine-tune a language model from scratch.
Location preference
Ideal availability is around 10:00-19:00 EST.
Application question(s)
What is the biggest open question in the field of algorithmic progress?
About the mentor
Robi's work tracks the inputs and development of advanced AI systems. Before joining MIRI, he worked at Epoch AI, building the database of machine learning hardware, GPU clusters, and AI models, and investigating their costs, algorithms, and development process. Robi is a contributor to the Stanford AI Index, and has a master's degree in data science from Harvard University.