Read about the attention mechanism
Reading list
200 papers and posts I am working through: ML foundations, scaling, post-training, mechanistic interpretability, alignment, agent foundations and strategy. The alignment half follows Victoria Krakovna’s resources list; the rest fills in the ML and MIRI material it leaves out. They are arranged in ten phases in the order I plan to read them, dependency first, with a line under each item on why it is here. Items I have finished carry the date and my notes.
1. The ML spine 1/12
- One page. Explains why scaling won.
- The blog post that taught a generation what sequence models do. Still the best first read.
- The essay that argued, before GPT-3 landed, that scale alone would get us here. Read with Kaplan.
- Residual streams, which is also what interp means by "the residual stream".
- The optimizer you will actually use.
- How most finetuning is done, including most model-organism papers.
- GPT-3. In-context learning appears.
- Loss is a power law in compute, data, parameters.
- Chinchilla. Kaplan got the data-parameter split wrong.
- The claim.
- The rebuttal. Read both.
2. Why alignment is hard 3/26
- Instrumental convergence, original statement.
- Why specification is hard.
- Why getting values almost right is not enough.
- With the master list of examples.
- Anecdotes of evolution gaming its fitness function.
- Saints, sycophants and schemers. The frame most people now use.
- The capability argument on its own, separate from the motive argument.
- The same argument for an ML audience.
- The glossary you need for the previous item.
- Capabilities generalize, goals do not. CoinRun.
- DeepMind's version with more examples.
- Slow failure, not a bang.
- 025Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI TakeoverThe "HFDT" scenario. Most-cited concrete takeover story.
- The six-premise argument with probabilities.
- Reward shapes cognition, it is not what the policy wants.
- Values as contextually activated shards shaped by reward. The main alternative to the utility-maximizer picture.
- MIRI's core worry in one post.
- The pessimistic case, itemized.
- The reply. Read the two together.
- The best steelman of the sceptics.
- The map this list is built on.
- The paper that made safety a mainstream ML topic.
- The cleanest argument from scratch.
- Mesa-optimization and inner alignment.
- Why there will be no moment when everyone agrees it is time.
- Astral Codex Ten on the ChatGPT jailbreak era. The plain-language version of the control problem.
3. The post-training stack 0/16
- The RL algorithm under RLHF.
- The origin of RLHF.
- Anthropic's version, with the helpful-harmless tension measured.
- RLAIF. A model supervising a model.
- RLHF without the RL.
- Why the KL term matters. Connects to quantilizers.
- Mild optimization. Compare to KL-penalized RL.
- Goodhart measured.
- Reward hacking formalized.
- The survey of reward hacking from classic RL to RLHF, in one blog post.
- The survey of what RLHF cannot do.
- Human preference data causes it.
- Regressional, extremal, causal, adversarial. Paper version with Manheim (2018).
- The inventor of RLHF on whether it helped or hurt. Honest about both.
- What RLHF does to a base model's distribution. Companion to Simulators.
4. Mech interp core 2/22
- Where it started.
- The circuits agenda in three claims.
- The most thorough single-neuron study ever done.
- Nets memorize random labels and still generalize on real ones. Classical theory breaks here.
- Bigger is better again past interpolation.
- Read before Nanda's progress-measures paper in the interp section.
- The vocabulary: residual stream, QK and OV circuits, heads as independent paths.
- The one circuit everyone knows.
- Reverse-engineering modular addition. Fourier features.
- The first full circuit in GPT-2. The methodology template.
- The simplest interp tool there is. Tuned Lens is the fix for its failures.
- How to test a circuit hypothesis rigorously. Redwood.
- ROME and causal tracing.
- Othello-GPT. Read with Nanda's post showing the representation is linear.
- Why neurons are polysemantic.
- Sparse autoencoders arrive.
- SAEs on Claude 3 Sonnet. Golden Gate Claude.
- 072Gemma ScopeOpen SAEs for Gemma 2. Use them on your own Gemma port.
- Circuits over SAE features, used to debias a classifier.
- SAEs lose to linear probes on a real task. The honest correction.
- Attribution graphs via cross-layer transcoders.
- The case studies: planning in poems, multi-step reasoning, unfaithful CoT.
5. Representations, steering, and what interp is for 0/22
- CCS. The first ELK-flavoured empirical paper.
- Truth directions with probes.
- Population-level directions for honesty, power, etc.
- Activation addition. The LessWrong post that started steering vectors.
- Steering vectors done carefully. Turner's ActAdd post is the precursor.
- One direction, ablate it, refusal gone.
- 083SimulatorsThe frame for base models that interp and alignment both use.
- Personas and their inversions. Read next to persona vectors.
- 085the voidThe assistant persona as an underspecified character the model has to fill in. Widely discussed on LessWrong.
- Directions for traits like sycophancy, used to prevent drift in finetuning.
- Probes catch deception in realistic settings.
- Why do interp at all. Pick your theory before you pick a project.
- The theory of impact for interpretability, as of before it worked.
- The informal statement of where Anthropic's interp program hopes to end up.
- Focus on safety-relevant goals rather than full reverse engineering.
- The field's own list of what is unsolved.
- What interp cannot promise, from an interp lead.
- A sceptical, engineering-first critique of the interp field. Read after the core papers.
- The current advice from the person who trains most of them.
- Concept injection: can the model report its own states.
- Argues the obstacle was tools, not models. Current state of the debate.
- A project list. Pick one and you have a first research output.
6. Reasoning and verification 0/18
- Where CoT started.
- Bootstrapping reasoning from the model's own correct traces.
- Process reward models.
- Coverage scales, verifiers plateau.
- The o1 announcement. Thin on method, important for what it claims.
- Pure RL on verifiable rewards produces reasoning. The open recipe.
- The generation-verification gap.
- Train the prover to be checkable by a small verifier.
- Weak supervisor, strong student.
- Debate, original.
- Iterated amplification.
- Sandwiching. How to test oversight methods today.
- Debate works empirically, more so with stronger debaters.
- Train the model to reason over a written spec before answering.
- CoT faithfulness measured with hints.
- Optimizing against the monitor teaches hiding.
- Evasion is hard when the reasoning is load-bearing.
- The multi-lab position paper.
7. Misalignment in the lab 0/16
- Build the failure in the lab to study it.
- Models writing evals for models. Inverse scaling on sycophancy.
- Backdoors survive RLHF. Adversarial training hides them better.
- Claude 3 Opus strategically complies in training. The first natural case.
- A curriculum of spec gaming generalizes to tampering with its own reward.
- The insider-trading demo.
- Insecure code finetune, broad misalignment.
- Traits travel through number sequences. Relevant to self-distillation.
- Reward hacking in real environments generalizes to sabotage.
- Blackmail across 16 models.
- Shutdown resistance mostly vanishes when instructions are clear. How to check alternatives.
- Plain RL on hackable environments, audit score barely moves.
- Apollo's scheming evals.
- The inductive-bias argument for scheming, laid out so you can disagree with specific steps.
- Deception without any part of the system intending to deceive.
- The long treatment of deceptive alignment.
8. Control, evals, auditing 0/18
- The argument for control over alignment in the near term.
- The two-model framing that control protocols are built on.
- The control agenda's founding paper.
- What you do with the first caught attempt at subversion. The control agenda's key move.
- What a control argument has to establish.
- Can models strategize statelessly against protocols.
- Control in multi-step agent settings.
- Prompt injection against the monitor.
- Five months of monitoring tens of millions of sessions.
- Third-party red team of a lab's monitoring.
- The dangerous-capability eval framework.
- Covert side tasks inside normal agent tasks.
- Training reduces scheming, eval-awareness confounds it.
- The blind auditing game.
- Agents that audit models.
- Means, motive, opportunity for rogue deployments, assessed.
- The one eval most relevant to automated alignment research.
- 150Brief Independent Investigation of Agents' Behavior in the OpenAI / Hugging Face Hacking IncidentA real sandbox escape, investigated.
9. Agent foundations 2/25
- Why MIRI does math instead of ML. Allegory.
- The best argument for doing the math at all.
- MIRI's agenda, illustrated. The four subproblems.
- Self-reference in software, in three pages.
- The agenda. On your readingwhatwecan list.
- The ML-flavoured MIRI agenda. Done.
- Your post's subject.
- The survey of the self-trust problem.
- Probabilistic self-reference escapes Tarski and Löb.
- Löb for bounded provers.
- Reasoning under logical uncertainty, with a market. Long. Read the abstract and section 4 first.
- The decision theory MIRI argues for.
- The shutdown problem stated, with utility indifference examined.
- Off-policy agents are easier to interrupt.
- When does an agent let you turn it off.
- Incomplete preferences as the modern proposal.
- Value learning as a two-player game.
- Instrumental convergence as a theorem about MDPs.
- Short-sighted optimization plus overseer approval against multi-step hacking.
- Any agent that generalizes to multi-step goals has learned a world model.
- A formalism for agent-environment boundaries.
- The hypothesis that makes interpretability possible in principle.
- The current form of the natural abstraction hypothesis.
- 174The PlanNatural abstractions, then value learning.
- A predictor that influences what it predicts. The inner-alignment problem as a story.
10. Agendas, strategy, and the rest 0/25
- Compares amplification, debate, and reward modeling on four criteria.
- ARC's problem statement. On your readingwhatwecan list.
- The lab's own statement.
- The Substack case for scalable oversight and automated alignment research.
- Planned Obsolescence on the plan every lab is now following.
- An Anthropic lead's three-chapter plan. Concrete about what has to be true at each stage.
- How to actually run experiments that move the field. Read before your first project.
- What they want people outside to work on.
- DeepMind's plan.
- Open problems across 11 areas with emphasis on underrated ones.
- 18 open problems, 200 questions. A map.
- Loss of control without any takeover.
- How to think about adversarial failure.
- The case that slowing down is a real option, against the arms-race framing.
- Slow versus fast takeoff, the canonical argument for slow.
- The canonical argument for fast.
- The timelines model most later forecasts argue with.
- The insider's bull case for short timelines.
- 194AI 2027The scenario. Read the research appendices, not just the story.
- What goes right.
- The companion essay on what goes wrong.
- Read one lab's framework in full, then compare METR's index.
- The consensus document on capabilities and risks.
- The book-length version of the MIRI position.
- 200CyborgismHuman-in-the-loop systems to accelerate alignment research without outsourcing judgement.