Reading

Papers, posts, and books that have shaped how I think about AI in production engineering. The bottleneck isn't capability - it's trust. We can build AI systems that perform; we struggle to build systems we can verify, understand, and safely operate.

Reading trails

Three paths through the material, each ending at one of my essays:

• The Bainbridge trail: Bainbridge → Cook → Klein → Simkute et al. → Ironies of Automation. Why automation creates the problems it was meant to solve.
• The trust trail: Lamport → Charity Majors → Rudin → Huyen → Observable, Reversible, Enforceable. Building verifiable systems and operational trust.
• The people trail: Klein (both) → Allspaw → Larson → Building Engineers in the Age of AI. How AI reshapes expertise development.

The foundational problem

• Lisanne Bainbridge - Ironies of Automation (1983)
  Advanced automation increases human criticality while reducing their capability to provide it.
• Gary Klein - Sources of Power: How People Make Decisions (1998)
  Experts decide via pattern recognition from experience, not analysis.
• Gary Klein - Seeing What Others Don't (2013)
  How expert insight and pattern recognition function.
• Erik Hollnagel & David Woods - Joint Cognitive Systems (2005)
  Human-machine combinations form integrated cognitive systems.
• Auste Simkute, Lev Tankelevitch et al. - Ironies of Generative AI (2024)
  Applies Bainbridge's ironies to LLM coding assistants. Four mechanisms for productivity loss: role shift, workflow disruption, interruptions, hard tasks made harder.

Safety and resilience

• Richard Cook - How Complex Systems Fail (1998)
  Failure is normal, and safety comes from how systems handle it.
• Sidney Dekker - The Field Guide to Understanding Human Error (2014)
  Reframes errors as symptoms reflecting why actions seemed reasonable at the time.
• Nancy Leveson - Engineering a Safer World (2011)
  Accidents stem from control failures, not component failures.
• David Woods - Four Concepts for Resilience (2015)
  Distinguishes rebound, robustness, graceful extensibility, and sustained adaptability.

Trust and verification

• Leslie Lamport - Time, Clocks, and the Ordering of Events in a Distributed System (1978)
  Foundational work on understanding and auditing automated behaviour sequences.
• Charity Majors - Observability Engineering (2022)
  Practical guidance for understanding production systems.
• Cynthia Rudin - Stop Explaining Black Box Machine Learning Models (2019)
  Post-hoc explanations prove unreliable; build interpretable models for high-stakes decisions.
• Chip Huyen - AI Engineering (2025)
  Foundation model lifecycle: prompt engineering, RAG, fine-tuning, agents, evaluation, and the latency-cost trade-off in production.

AI in practice

• Simon Willison - simonwillison.net
  Thorough documentation of practical AI usage through TILs and link blogging.
• Andrej Karpathy - Software 2.0 (2017)
  Code replaced by data, and what that means for programmer roles.
• Dan McKinley - Choose Boring Technology (2015)
  Weighing novelty costs against stability benefits.
• METR - Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025)
  RCT with 16 experienced developers. They predicted AI would speed them up by 24%. It slowed them down by 19%. They still believed it helped.

Building engineers

• John Allspaw - The Infinite Hows (2014)
  How engineers develop judgment through incident learning.
• Will Larson - Staff Engineer (2021)
  Senior engineers create impact beyond individual coding contributions.

Financial services context

• Andrew Haldane - The Dog and the Frisbee (2012)
  Simple rules often outperform complex models in regulation.
• Man Group Research - Machine Learning Papers
  Oxford-Man Institute publications on finance ML.
• Marcos Lopez de Prado - Advances in Financial Machine Learning (2018)
  Rigorous ML treatment emphasising backtesting pitfalls.
• Emanuel Derman - Models.Behaving.Badly (2011)
  Theories describe what something is. Models describe what something is like.