If you would like to participate, please drop an email to mechanistic-interpretability-seminar@googlegroups.com until October 17 (23:59). In your email, please:

• Give your name, semester, study program
• Write some words on why you want to take part in this course
• List some of your previous experience:
  • your background in deep learning or machine learning
  • your background in natural language processing in general

Prerequisites: This seminar is primarily targeted at Master students, but is also open to advanced Bachelor students. We expect you to have a curious mind and advanced familiarity with large language models. At the very least, we expect all students to have read (and understood :-)) the BERT paper and the Transformer paper.

Seminar Content

The rise of deep learning in AI has dramatically increased the performance of models across many sub-fields such as natural language processing or computer vision. In the last 5 years, large pretrained language models (LLMs) and their variants (BERT, ChatGPT etc.) have changed the NLP landscape drastically. Such models got larger and larger over the last years, reaching increasingly impressive performance peeks, sometimes even surpassing humans.

A central issue with deep learning models with millions or billions of parameters is that they are essentially black boxes: From the model's parameters, it is not inherently clear why a model exhibits a certain behavior or makes certain classification decision. The rapidly growing field of interpretable and explainable AI (XAI) develops methods to peek into the black box that LLMs are, trying to understand the inner workings of such large models.

In this seminar we will investiagte a subfield in XAI, namely Mechanistic Interpretability (MI). MI aims to understand and explain the internal workings of complex machine learning models, in particular deep neural networks, by "reverse-engineering" internal mechanisms and computations inside a network's parameters. This involves analysing how certain mechanisms in the model contribute to decisions and results. The aim is to break down the ‘black box’ nature of these models and reveal the underlying calculations and internal representations that lead to predictions or behaviours.

We will concentrate on a range of basic methods in MI and then focus on their applications within natural language processing.

---

List of relevant Papers and Topics (subject to changes)

MI Basics

• Zoom In: An Introduction to Circuits
• A Mathematical Framework for Transformer Circuits
• Transformer Feed-Forward Layers Are Key-Value Memories
• Talking Heads: Understanding Inter-layer Communication in Transformer Language Models

Model Editing Editing, Knowledge Location and Extraction

• Crawling The Internal Knowledge-Base of Language Models
• Locating and Editing Factual Associations in GPT
• Mass-Editing Memory in a Transformer
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
• Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model
• Evaluating ripple effects 
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors

Sparse Autoencoders and Monosemanticity

• Knowledge Neurons in Pretrained Transformers
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
• Transcoders Find Interpretable LLM Feature Circuits

Some words on grading: This seminar is meant to be as interactive as possible. Final grades will be based on students' presentations, term papers (optional), but also on participation and discussion in class.

The participants are expected to prepare for classes accordingly, by reading the relevant papers and also doing background reading, if necessary. Based on this preparation, the participants should be able to discuss the presented papers in depth and to understand relevant context during the discussion.