Interpreting concept typicality and asymmetric similarity in LLMs

Understanding the properties of concepts learned by AI systems is essential for interpretability. Interpretability tools that accurately reflect the structure of learned concepts may be better able to decompose the activations of neural networks into meaningful features.

One of the most robust findings in the study of human concepts in cognitive psychology is typicality. Concepts are not homogeneous, but contain some members that are typical and some that are atypical. One empirical phenomenon associated with typicality is asymmetrical similarity: people judge an atypical item to be more similar to a typical item, than the typical item is to the atypical item.

Typicality can be naturally understood in the context of categories defined by a family resemblance: categories aren’t defined by necessary features, but loosely united by common features with exceptions. Typical category members have many common features, and atypical category members have few rare features. Empirically, typical items are rated by humans to be more similar to other typical items than atypical items are to other atypical items.

In this project, we will apply the lens of mechanistic interpretability to empirically investigate concept typicality and asymmetric similarity in the internal representations of LLMs. Prior works have demonstrated typicality effects in strong LLMs using behavioral experiments, but the intersection of typicality with mechanistic interpretability has so far been limited. This project will employ standard mechanistic interpretability tools such as linear probes and sparse autoencoders (SAEs) to investigate the fundamental properties of concepts learned by LLMs, as opposed to developing new tools. Careful construction of high-quality datasets is essential for this work.

Potential research directions:

1. Linear representations of typicality: How do LLMs internally represent the extent to which an input is typical? In particular, is there a linear direction that globally represents the concept-independent property of typicality, or is the representation of typicality concept-dependent (e.g., magnitude of the projection onto a concept vector)? We will address this question by constructing multi-category datasets of prompts stratified by typicality and training linear probes on the activations of LLMs. The project’s findings could be extended by testing the causal efficacy of the recovered representations using steering experiments.

2. Quantifying asymmetric similarity using SAE features: Most similarity measures (e.g. cosine similarity) are symmetric, making them unable to capture the asymmetric judgments sometimes expressed by humans. One model that can express asymmetric similarity is Tversky’s contrast model, which describes inputs in terms of sets of discrete latent features. We will quantify asymmetric similarity in LLMs by constructing datasets of typical and atypical examples (as in the above project), getting the set of active SAE features for each example using pretrained SAEs, and fitting Tversky’s contrast model to those features.

3. Feature absorption and typicality: SAEs display several known pathologies, such as feature absorption, in which larger SAEs can split off low-level special cases from an otherwise interpretable high-level feature. Are SAE features activating on atypical examples more likely to undergo absorption? Do category-level features activate more strongly on typical examples? We will investigate these questions by identifying category-level features in pretrained SAEs, measuring their activations on typical and atypical examples, and testing whether absorption is correlated with atypical examples.

References

• Features of Similarity, Tversky 1977 (https://pages.ucsd.edu/~scoulson/203/tversky-features.pdf)
• Category-based induction, Osherson et al. 1990 (https://psycnet.apa.org/record/1990-18953-001)
• Inductive reasoning in humans and large language models, Han et al. 2024 (https://arxiv.org/abs/2306.06548)
• Characterizing the Role of Similarity in the Property Inferences of Language Models, Rodriguez et al., 2024 (https://arxiv.org/abs/2410.22590)

This research aims to advance the basic science of mechanistic interpretability and representations of concepts in language models. The nature of learned concepts and the processes by which AI systems ground, represent, combine, and translate them into downstream performance all remain poorly understood. An improved understanding of how features are organized in language models can underpin and inform mechanistic interpretability research directed at sophisticated AI systems.

Mentees will conduct their research projects semi-independently with feedback and guidance from me, including constructing datasets, performing experiments, and writing up the results. We will have weekly one-on-one meetings as well as a regular group meeting, and can communicate asynchronously.

• Proficiency in Python
• Practical experience working with PyTorch and transformers
• Familiarity with mechanistic interpretability concepts (e.g. linear probes, sparse autoencoders)
• Bonus: familiarity with empirical studies & theoretical models of concepts in cognitive psychology

Mentees must be available for meetings during the workday US Eastern time.