49 lines
5.2 KiB
TeX
49 lines
5.2 KiB
TeX
|
|
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
|
|
\begin{figure}
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{img/fig-1.pdf}
|
|
\caption{Comparison of traditional image classification training and training when using \schemename. \schemename recombines foreground objects with different backgrounds each epoch, thus creating a more diverse training set. We still apply strong traditional data augmentation afterwards.}
|
|
\label{fig:fig-1}
|
|
\end{figure}
|
|
|
|
Image classification, a fundamental task in computer vision (CV), involves assigning labels to images from a set of categories.
|
|
It underpins a wide range of applications, like medical diagnosis~\cite{Sanderson2022,Vezakis2024}, autonomous driving~\cite{Wang2022b}, and object recognition~\cite{Carion2020,He2017,Girshick2013} and facilitates large-scale pretraining~\cite{Dosovitskiy2021,Liu2021,Touvron2021b}, and progress evaluation in CV~\cite{Khan2022, Rangel2024}.
|
|
The advent of large-scale datasets, particularly ImageNet~\cite{Deng2009}, served as a catalyst for the rise of large-scale CV models~\cite{Krizhevsky2012, He2016} and remains the most important CV benchmark for more than a decade \cite{Krizhevsky2012,Touvron2022, Wortsman2022, He2016}.
|
|
While traditionally, convolutional neural networks (CNNs) have been the go-to architecture in CV, Transformers \cite{Vaswani2017}, particularly the Vision Transformer (ViT) \cite{Dosovitskiy2021}, have emerged as a powerful alternative and go-to architecture, demonstrating
|
|
superior performance in various vision tasks, including image classification \cite{Wortsman2022,Yu2022,Carion2020,Zong2022,Wang2022a}.
|
|
|
|
|
|
|
|
Data augmentation is a key technique for training image classification models.
|
|
Traditional augmentation methods, such as cropping, flipping, or color shifts, are commonly employed to increase data diversity~\cite{Xu2023d, Shorten2019}, but remain bound to existing image compositions.
|
|
While these preserve the images' semantic meaning, their ability to teach spatial invariances is limited.
|
|
While combinations of these data augmentations are still used today, they originally were proposed to benefit CNNs.
|
|
However, the architectural differences of CNNs and Transformers suggest that the latter might benefit from different data augmentation strategies.
|
|
In particular, the self-attention mechanism, unlike a CNN, is not translation equivariant~\cite{RojasGomez2023,Ding2023a}, meaning that the model is not designed to understand the spatial relationships between pixels.
|
|
|
|
Recognizing that Transformers need to learn spatial relationships directly from data,
|
|
we propose \schemename, a data augmentation method that makes these relationships explicit by recombining foreground objects with diverse backgrounds.
|
|
Thus, \schemename goes beyond existing image compositions and encodes desired invariances directly into the training data (see \Cref{fig:fig-1}).
|
|
Applying \schemename to a dataset like ImageNet is a two-step process:
|
|
(1)~We separate the foreground objects in ImageNet from their backgrounds, using an open-world object detector~\cite{Ren2024} and fill in the background in a neutral way using an object removal model~\cite{Sun2024,Suvorov2021}.
|
|
(2)~This allows us to then recombine any foreground object with any background on the fly, creating a highly diverse training set.
|
|
By exploiting the control over foreground size and position during recombination, \schemename explicitly teaches spatial invariances that image classification models typically must learn implicitly.
|
|
We show that using \schemename additionally to strong traditional data augmentation increases the model accuracy of Transformers by up to 4.5 p.p. on ImageNet and reduces the error rate by up to $7.3$ p.p. in downstream tasks.
|
|
|
|
Beyond training, \schemename becomes a diagnostic tool for analyzing model behavior and biases, when used during evaluation.
|
|
We utilize our control over the image distribution to measure a model's background robustness (by varying the choice of background), foreground focus (by leveraging our knowledge about the placement of the foreground object), center bias (by controlling position), and size bias (by controlling size).
|
|
These analyses provide valuable insights into model behavior and biases, which is crucial for model deployment and future robustness optimizations.
|
|
We show that training using \schemename significantly reduces all of these biases.
|
|
We make our code for \schemename and the output of \schemename's segmentation phase on ImageNet publicly available\footnote{Link will go here.} to facilitate further research.
|
|
|
|
\subsection*{Contributions}
|
|
\begin{itemize}
|
|
\item We propose \schemename, a novel data augmentation scheme, that recombines objects and backgrounds. \schemename allows us to move beyond the (possibly biased) image compositions in the dataset while preserving label integrity.
|
|
\item We show that training a standard ViT using \schemename leads to up to 4.5 p.p. improved accuracy on ImageNet-1k and 7.3 p.p. on downstream tasks.
|
|
\item We propose novel \schemename-based metrics to analyze and quantify fine-grained biases of trained models: Background Robustness, Foreground Focus, Center Bias, and Size Bias. We show that \schemename significantly reduces these biases by encoding invariance that benefits ViT into the training data.
|
|
\end{itemize}
|