Results.

Setup

PLACE is evaluated on DINOv2^{4DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab et al.
TMLR, 2024}, ViT-MAE^{5Masked Autoencoders Are Scalable Vision Learners
Kaiming He et al.
CVPR, 2022}, and a supervised ViT-B/16^{3An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy et al.
ICLR, 2021} for parity, on matched subsets of ImageNet^{7ImageNet: A Large-Scale Hierarchical Image Database
Jia Deng et al.
CVPR, 2009,8ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky et al.
IJCV, 2015} (10 classes, at least 800 images each) and COCO^{9Microsoft COCO: Common Objects in Context
Tsung-Yi Lin et al.
ECCV, 2014} (5 classes), following the FACE^{2FACE: Faithful Automatic Concept Extraction
Dipkamal Bhusal et al.
NeurIPS, 2025} protocol. Concepts are extracted at layer $\ell = 9$ with rank $r = 25$. Results are the mean and standard deviation over 5 seeds. Significance against the SAE baselines uses two-sided Welch t-tests with Holm-Bonferroni correction.

Two metrics need no classifier: MSE for reconstruction quality and C-Gini for concept sparsity, a proxy for monosemanticity. $D_{\mathrm{KL}}$ measures predictive consistency through a surrogate linear probe, used only for evaluation. C-Insertion and C-Deletion are reported for comparison with prior work (Figure 4), though masking creates artifacts and the sequential order confounds individual concept contributions.

Benchmark comparison

Comparing PLACE with SAEs^{10Sparse Autoencoders Find Highly Interpretable Features in Language Models
Robert Huben et al.
ICLR, 2024,11Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models
Mateusz Pach et al.
NeurIPS, 2025} and Gated SAEs^{12Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders
Senthooran Rajamanoharan et al.
NeurIPS, 2024} at the same dictionary rank, after Holm-Bonferroni correction:

1. Sparsity. PLACE has higher C-Gini than both SAE variants in four of six configurations: all three backbones on ImageNet, and ViT-B/16 on COCO. On DINOv2 and ImageNet it reaches 0.933, against 0.865 for SAEs and 0.724 for Gated SAEs. On ViT-MAE and ImageNet, PLACE (+Gram) reaches 0.678, against 0.238 and 0.276.
2. Reconstruction and predictive consistency. On DINOv2 and ImageNet, Gated SAEs reach lower MSE and $D_{\mathrm{KL}}$. In the other five configurations the differences are not significant. We observe a tradeoff between sparsity and reconstruction.
3. Causal metrics. No C-Ins difference is significant. PLACE improves C-Del on COCO with ViT-B/16 (0.979 against 0.115 and 0.033).

Table 1 also lists ICE^{14Invertible Concept-Based Explanations for CNN Models with Non-Negative Concept Activation Vectors
Ruihan Zhang et al.
AAAI, 2021}, CRAFT^{1CRAFT: Concept Recursive Activation FacTorization for Explainability
Thomas Fel et al.
CVPR, 2023}, and FACE^{2FACE: Faithful Automatic Concept Extraction
Dipkamal Bhusal et al.
NeurIPS, 2025} on a ResNet-34^{13Deep Residual Learning for Image Recognition
Kaiming He et al.
CVPR, 2016}, and the sliding-window adaptations of CRAFT and FACE on ViTs, as references for magnitude.

Runtime

The Gram term adds 0.3 s over plain reconstruction (5.6 s against 5.3 s), since it requires no forward or backward pass through $g_{>\ell}$ or the classifier. KL supervision backpropagates through the full suffix at every step, and SAEs train a separate encoder and decoder.

Patch tokens vs. crop-and-resize

The sliding-window adaptations of CRAFT^{1CRAFT: Concept Recursive Activation FacTorization for Explainability
Thomas Fel et al.
CVPR, 2023} and FACE^{2FACE: Faithful Automatic Concept Extraction
Dipkamal Bhusal et al.
NeurIPS, 2025} for ViTs, introduced with RSVC^{15Representational Similarity via Interpretable Visual Concepts
Neehar Kondapaneni, Oisin Mac Aodha, Pietro Perona
ICLR, 2025}, factorize the final CLS tokens of 64 × 64 crops resized to 224 × 224. Two effects inflate their C-Ins scores. The CLS token sits right before the classifier head, so the concepts shortcut the predictive distribution. Each resized crop also aggregates context as if it were a complete image. The representation error gap is consistent with this mismatch: on ViT-B/16 the sliding-window baseline reaches an MSE of 3.489, against 0.292 for PLACE, although the two values are computed on activation spaces of different scale.

The crop-and-resize page explains why this pipeline distorts ViT representations.

Agreement with supervised concepts

To check the semantics of the unsupervised concepts, we compare their concept banks with the supervised (+KL) bank using RSVC^{15Representational Similarity via Interpretable Visual Concepts
Neehar Kondapaneni, Oisin Mac Aodha, Pietro Perona
ICLR, 2025}: maximum cosine similarity (MCS) for one-to-one alignment, and the Pearson correlation of Lasso predictions (CMCS) for agreement of the whole subspace.

Both unsupervised variants align closely with the supervised reference (MCS at least 0.840, CMCS at least 0.722), and the differences between them are within one standard deviation. In this class-conditioned setting the Gram loss preserves alignment and mainly improves dictionary stability, raising W-Stab on 8 of 10 ImageNet classes (church from 0.897 to 0.953, tench from 0.959 to 0.995), at about 20× lower cost than the KL reference.

Task-agnostic concept banks

A harder test learns one concept bank from unlabeled multi-class data: 200 images sampled at random from the union of the evaluated classes, rank $r = 100$, 20 seeds. Neither data selection nor the objective uses labels or classifier outputs, and the +KL variant is computed only as a reference.

Here the Gram loss improves matched concept alignment on both backbones and both datasets, with MCS gains of 0.124 to 0.552. It also raises dictionary stability in all four configurations, close to the +KL reference. Against SAEs, PLACE (+Gram) reaches higher C-Gini by 0.06 to 0.07 on DINOv2 and 0.22 to 0.25 on ViT-MAE, while SAEs reach lower MSE in all four configurations.

Limitations

• The KL bound is loose because of worst-case Lipschitz compounding. It serves as a mechanistic motivation for the Gram loss.
• PLACE assumes concepts are linearly factorizable and needs the rank $r$ fixed in advance. Token grouping hides variation inside each pooled block.
• NMF-based factorization degrades on very small sample sets (under about 50 images), which explains the lower COCO scores.
• Architectures with other tokenization schemes, such as Swin^{16Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  Ze Liu et al.
  ICCV, 2021,17Swin Transformer V2: Scaling Up Capacity and Resolution
  Ze Liu et al.
  CVPR, 2022}, DeiT^{18Training Data-Efficient Image Transformers & Distillation through Attention
  Hugo Touvron et al.
  ICML, 2021}, JEPA^{19Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
  Mahmoud Assran et al.
  CVPR, 2023,20Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning
  Shentong Mo, Shengbang Tong
  NeurIPS, 2024}, and registers^{21Vision Transformers Need Registers
  Timothée Darcet et al.
  ICLR, 2024,22Vision Transformers Don't Need Trained Registers
  Nicholas Jiang et al.
  NeurIPS, 2025}, are left for future work. Heatmaps are evaluated qualitatively, without human studies, and concepts get no automatic text labels.

Ablations, per-class results, sanity checks, and the proof of the KL bound are in the appendix of the paper. Back to the overview.

References

• [1]Thomas Fel et al. CRAFT: Concept Recursive Activation FacTorization for Explainability. CVPR, 2023.
• [2]Dipkamal Bhusal et al. FACE: Faithful Automatic Concept Extraction. NeurIPS, 2025.
• [3]Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
• [4]Maxime Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. TMLR, 2024.
• [5]Kaiming He et al. Masked Autoencoders Are Scalable Vision Learners. CVPR, 2022.
• [7]Jia Deng et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
• [8]Olga Russakovsky et al. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
• [9]Tsung-Yi Lin et al. Microsoft COCO: Common Objects in Context. ECCV, 2014.
• [10]Robert Huben et al. Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR, 2024.
• [11]Mateusz Pach et al. Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models. NeurIPS, 2025.
• [12]Senthooran Rajamanoharan et al. Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders. NeurIPS, 2024.
• [13]Kaiming He et al. Deep Residual Learning for Image Recognition. CVPR, 2016.
• [14]Ruihan Zhang et al. Invertible Concept-Based Explanations for CNN Models with Non-Negative Concept Activation Vectors. AAAI, 2021.
• [15]Neehar Kondapaneni, Oisin Mac Aodha, Pietro Perona. Representational Similarity via Interpretable Visual Concepts. ICLR, 2025.
• [16]Ze Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV, 2021.
• [17]Ze Liu et al. Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR, 2022.
• [18]Hugo Touvron et al. Training Data-Efficient Image Transformers & Distillation through Attention. ICML, 2021.
• [19]Mahmoud Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR, 2023.
• [20]Shentong Mo, Shengbang Tong. Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning. NeurIPS, 2024.
• [21]Timothée Darcet et al. Vision Transformers Need Registers. ICLR, 2024.
• [22]Nicholas Jiang et al. Vision Transformers Don't Need Trained Registers. NeurIPS, 2025.