A reading list is an argument about what mattered. This one makes a narrow claim: that a dozen-odd papers, read in sequence, are enough to explain how the field traveled from sequence-to-sequence models to systems that will grind on a hard problem for minutes before answering. It is not a canon, not a ranking, and not a syllabus you must finish. It is the shortest path we can defend through a literature that now adds thousands of papers a month.
The arc in four moves
The through-line is easier to hold if you group the work into four shifts, each of which reset what the next generation of researchers took for granted.
Architecture. The transformer replaced recurrence with attention in 2017 and, almost incidentally, handed the field an architecture that scales cleanly across accelerators. Nearly everything after it is a transformer, a lightly modified transformer, or a deliberate reaction to one.
Scale. The 2020 scaling-law paper and GPT-3 turned a qualitative hunch — bigger is better — into smooth, predictable curves, and reframed capability as an engineering variable you buy rather than a trick you stumble onto. DeepMind's Chinchilla then corrected the recipe: most large models of that era were badly undertrained for their size.
Alignment and usefulness. A raw pretrained model predicts text; it does not follow instructions or refuse harm. InstructGPT showed that reinforcement learning from human feedback could bend a base model toward what people actually wanted, and it is the reason the ChatGPT moment felt like a product rather than a demo. Constitutional AI later showed that much of that supervision could come from the model critiquing itself against a written set of principles.
Reasoning. Chain-of-thought prompting noticed that simply asking a model to work step by step unlocked latent capability. OpenAI's o1, in late 2024, turned that observation into a training objective — spend reinforcement learning and inference-time compute on the reasoning itself — and the 2025-2026 wave of reasoning models made deliberate, budgeted thinking a standard axis of the frontier alongside raw scale.
Read in that order, the individual results stop looking like a pile of tricks and start looking like a sequence of moves, each one opening the position for the next.
How to read an ML paper efficiently
You cannot read everything, and you should not read any single paper linearly on the first pass. The durable advice is Srinivasan Keshav's three-pass method: the first pass is title, abstract, figures, and conclusion — enough to decide whether to continue; the second reads the body for the argument while skipping proofs; the third, reserved for papers you need to trust or reproduce, reconstructs the work in your own head.
A few habits separate reading from skimming. Find the central claim and, separately, the evidence offered for it — they are not the same thing, and the gap between them is where most papers are weakest. Go to the ablations before the headline number; they tell you which component actually did the work. Treat a single leaderboard result with suspicion until you know the baseline, the compute budget, and whether the evaluation could have leaked into training. And read the limitations section as if the authors' competitors wrote it — because eventually they will.
Using the list below
The reading list that follows is deliberately short. Each entry carries a one-line note on why it earns its place and links to the paper and, where it exists, the reference code. Read them in order for the narrative above, or treat the list as a map and drop into whichever shift you are missing. If you only have an afternoon, read the transformer, Chinchilla, and InstructGPT; that triangle — architecture, scale, and alignment — is the smallest set that still tells the true story.
One caution worth stating plainly: a road-to-AGI reading list is a claim about direction, and directions are contested. Nothing here proves the current trajectory reaches general intelligence. These are the papers that got the argument to where it now sits — not evidence about where it ends.
ImageNet Classification with Deep Convolutional Neural Networks (2012)
Krizhevsky et al. · NeurIPS
A deep convolutional network trained on GPUs won the ImageNet competition by a large margin over prior methods. The result showed that deep neural networks could learn useful visual features directly from large labeled datasets, and it started the shift toward deep learning across the field.
Efficient Estimation of Word Representations in Vector Space (2013)
Mikolov et al. · ICLR Workshop
Introduced the word2vec models for learning dense word embeddings efficiently from large text corpora. The learned vectors captured syntactic and semantic relationships and became a standard input representation for later natural language processing systems.
Deep Residual Learning for Image Recognition (2015)
He et al. · CVPR
Introduced residual connections that let networks be trained with far greater depth than was previously practical. This addressed the degradation problem in very deep networks and the residual block became a common building component in later architectures.
Attention Is All You Need (2017)
Vaswani et al. · NeurIPS
Introduced the Transformer, an architecture built entirely on attention mechanisms rather than recurrence or convolution. It enabled more parallel training and became the basis for most subsequent large language models.
Proximal Policy Optimization Algorithms (2017)
Schulman et al. · arXiv
Proposed PPO, a reinforcement learning method that is simpler to implement and tune than earlier policy-gradient approaches while remaining stable. It became a widely used algorithm and is a common component of reinforcement learning from human feedback pipelines.
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (2017)
Silver et al. · arXiv
Described AlphaZero, a single reinforcement learning method that reached strong play in chess, shogi, and Go using only self-play and the game rules. It showed that a general algorithm without human game data could match or exceed specialized programs.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)
Devlin et al. · NAACL
Introduced a Transformer encoder pre-trained with masked language modeling to produce bidirectional text representations. Fine-tuning the pre-trained model improved results across many language understanding tasks and popularized the pre-train then fine-tune approach.
Scaling Laws for Neural Language Models (2020)
Kaplan et al. · arXiv
Measured how language model loss changes with model size, dataset size, and compute, finding smooth power-law relationships. These trends provided a basis for predicting performance and planning the training of larger models.
Language Models are Few-Shot Learners (2020)
Brown et al. · NeurIPS
Presented GPT-3, a 175-billion-parameter language model, and showed it could perform many tasks from instructions and a few examples in the prompt without gradient updates. This in-context learning behavior influenced how later models were used and evaluated.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)
Wei et al. · NeurIPS
Showed that prompting a model to produce intermediate reasoning steps improves accuracy on arithmetic, commonsense, and symbolic tasks. The effect appeared mainly in sufficiently large models and became a common prompting technique.
Training language models to follow instructions with human feedback (2022)
Ouyang et al. · NeurIPS
Described InstructGPT, which fine-tuned a language model using supervised examples and reinforcement learning from human feedback to better follow user instructions. Human raters preferred its outputs over the much larger base model, and the approach shaped later assistant-style systems.
Training Compute-Optimal Large Language Models (2022)
Hoffmann et al. · NeurIPS
Found that many large models were undertrained and that for a fixed compute budget model size and training tokens should scale in roughly equal proportion. The Chinchilla model trained under this guidance outperformed larger models, changing how training budgets were allocated.
ReAct: Synergizing Reasoning and Acting in Language Models (2022)
Yao et al. · ICLR
Combined step-by-step reasoning with actions such as querying external tools within a single prompting loop. This interleaving reduced fabricated answers on knowledge tasks and became a common pattern for building language model agents.
Constitutional AI: Harmlessness from AI Feedback (2022)
Bai et al. · arXiv
Proposed training a model to be less harmful using a written set of principles and AI-generated feedback, reducing reliance on human labels for harmful content. The method uses model self-critique and revision to guide behavior toward the stated principles.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
DeepSeek-AI · Nature
Showed that reasoning behavior such as self-checking and longer solution traces could be developed largely through reinforcement learning on verifiable tasks rather than supervised reasoning examples. The authors released the models and distilled smaller versions under an open license.