Crop and resize

Why the concept proposals of CNN-era methods distort what a vision transformer sees, and how factorizing patch tokens in place avoids it.

On this page

Why it works for CNNs
Why ViTs break it
Four failure modes
How PLACE avoids them
The evidence
References

CRAFT^{1CRAFT: Concept Recursive Activation FacTorization for Explainability
Thomas Fel et al.
CVPR, 2023}, FACE^{2FACE: Faithful Automatic Concept Extraction
Dipkamal Bhusal et al.
NeurIPS, 2025}, and their ViT adaptations in RSVC^{15Representational Similarity via Interpretable Visual Concepts
Neehar Kondapaneni, Oisin Mac Aodha, Pietro Perona
ICLR, 2025} propose concepts the same way: crop a region of the image, resize the crop to the model's full input resolution, pass it through the network on its own, and factorize the resulting activation vectors. This pipeline relies on translation equivariance, a property of CNNs that ViTs^{3An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy et al.
ICLR, 2021} do not have.

Left: a parachute and a church at full resolution, as PLACE sees them. Right: pixelated, upscaled crops of the same images, as crop-and-resize methods feed them to the model. — Figure 2.The Crop-and-Resize Problem in ViTs. While PLACE operates on the full-resolution image seen at training time (left), current crop-and-resize methods feed sub-resolution patches (right). We show in Section 4.2 this shift from the training distribution biases concept extraction.

Why it works for CNNs

A convolution applies the same kernel at every position, so shifting the input shifts the output by the same amount, and pointwise nonlinearities and pooling preserve this. Each position of a CNN feature map therefore depends only on a bounded receptive field of the input, and the activation of a resized crop approximates the matching slice of the full-image feature map (Appendix A.1 and A.2):

f^{(\ell)}_{\text{cnn}}\big(\texttt{resize}(\mathbf{x}_p)\big) \approx \mathbf{A}[:,\ i'_0 : i'_1,\ j'_0 : j'_1].

(10)

The activation of each crop therefore describes the local visual content of that region.

Why ViTs break it

A ViT has no such guarantee (Appendix A.3). Positional embeddings stay tied to grid positions while the content moves, so translating the image changes the input to the first block:

\mathbf{e}^{(0)}_i(T_{\boldsymbol\tau}\mathbf{x}) = \mathbf{W}_{\text{proj}}\, \mathbf{x}_{\sigma(i)} + \mathbf{p}_i \;\neq\; \mathbf{W}_{\text{proj}}\, \mathbf{x}_{\sigma(i)} + \mathbf{p}_{\sigma(i)}.

(11)

Self-attention is also global and content-dependent. After the first layer every token depends on every patch, so Eq. (10) has no ViT counterpart.

Four failure modes

Feeding a ViT a crop $\mathbf{x}_p$ resized to $H \times W$ causes four failure modes (Appendix A.4):

Patches tile content at the wrong scale. In a 224 × 224 image with patch size 16, each patch covers 16 × 16 pixels of the scene. In a 64 × 64 crop resized to 224 × 224, each patch covers about 4.6 × 4.6 pixels of the original. The patch projection was never trained at that magnification.
Positional embeddings encode the wrong locations. A patch of sky from the top-left corner of the image can land at the center of the resized crop, and receive the embedding the network associates with a different place and context.
The attention context is different. In the full image a token attends to the whole scene, while in the crop it attends only to tokens from the crop region, so the CLS token summarizes the crop as if it were the entire image:
$a^{\text{crop}}_{ij} \neq a^{\text{full}}_{ij} \qquad \forall\, l \ge 1,\ \forall\, i, j.$
(13)
No spatial correspondence. The CLS token of a resized crop does not approximate any slice or projection of the full-image representation.

How PLACE avoids them

PLACE factorizes the patch-token activations of the full image at layer $\ell$ (Appendix A.6), which avoids all four failure modes:

No re-tokenization. Every patch is seen at the scale and resolution the model was trained on.
Correct positions. Every token carries the positional embedding of its actual location.
One attention context. All tokens of an image share the full scene as context, so differences between their concept coefficients reflect spatial variation across the scene.
Spatial anchoring. Each row of $\mathbf{U}$ belongs to one grid position, so a concept's heatmap is obtained by reshaping its coefficients onto the token grid.

After the first layer a patch token is no longer purely local, so the discovered concepts mix local visual evidence with global context, following how the ViT represents the image.

The evidence

The reconstruction error of a low-rank factorization is consistent with this. On ViT-B/16 and ImageNet, the sliding-window baseline reaches an MSE of 3.489, against 0.292 for PLACE (Section 4.2 and Table 1), although the two values are computed on activation spaces of different scale. RSVC reports Semi-NMF reconstruction errors 5 to 8× higher for ViTs than for ResNets^{13Deep Residual Learning for Image Recognition
Kaiming He et al.
CVPR, 2016} on the same ImageNet classes (Appendix A.5): two crops with the same local content but different surroundings give different CLS vectors, which inflates the effective rank of the activation matrix.

Section 3, Section 4.2, and Appendix A in the paper Back to the overview

References

[1]Thomas Fel et al. CRAFT: Concept Recursive Activation FacTorization for Explainability. CVPR, 2023.
[2]Dipkamal Bhusal et al. FACE: Faithful Automatic Concept Extraction. NeurIPS, 2025.
[3]Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
[13]Kaiming He et al. Deep Residual Learning for Image Recognition. CVPR, 2016.
[15]Neehar Kondapaneni, Oisin Mac Aodha, Pietro Perona. Representational Similarity via Interpretable Visual Concepts. ICLR, 2025.