412 lines
43 KiB
TeX
412 lines
43 KiB
TeX
|
|
\section{Experiments}
|
|
\label{sec:experiments}
|
|
|
|
|
|
We conduct a comprehensive suit of experiments to validate the effectiveness of our approach,
|
|
comparing ImageNet-training with and without \schemename for 10 different models.
|
|
Furthermore, we assess the impact of using \schemename for pretraining on multiple fine-grained downstream datasets.
|
|
Finally, we exploit \schemename's control over the image distribution to quantify model behaviors and biases.
|
|
We always report the mean and standard deviation of three independent training runs.
|
|
|
|
\subsection{Design Choices of ForAug}
|
|
\label{sec:ablation}
|
|
|
|
We start by ablating the design choices of \schemename on TinyImageNet~\cite{Le2015}, a subset of ImageNet containing 200 categories with 500 images each. %
|
|
\Cref{tab:ablation-segment} presents ablations for segmentation and \Cref{tab:ablation-recombine} for recombination.
|
|
|
|
\begin{table}
|
|
\caption{Ablation of the design decisions in the segmentation phase of \schemename on TinyImageNet.
|
|
The first line is our baseline, while the other lines are using \schemename.
|
|
We use basic settings with the \emph{same} background strategy during recombination for this experiment.
|
|
}
|
|
\label{tab:ablation-segment}
|
|
\centering
|
|
\small
|
|
\resizebox{.9\columnwidth}{!}{
|
|
\begin{tabular}{cccc}
|
|
\toprule
|
|
\multirow{2.5}{*}{\makecell{Detect. \\Prompt}} & \multirow{2.5}{*}{\makecell{Infill \\ Model}} & \multicolumn{2}{c}{TinyImageNet Accuracy [\%]} \\
|
|
\cmidrule{3-4}
|
|
& & ViT-Ti & ViT-S \\
|
|
\midrule
|
|
\multicolumn{2}{l}{\textbf{TinyImageNet}} & $66.1 \pm 0.5$ & $68.3 \pm 0.7$ \\
|
|
specific & LaMa \cite{Suvorov2021} & $65.5 \pm 0.4$ & $71.2 \pm 0.5$ \\
|
|
general & \gtxt{LaMa \cite{Suvorov2021}} & $66.4 \pm 0.6$ & $72.9 \pm 0.6$ \\
|
|
\gtxt{general} & Att. Eraser \cite{Sun2024} & $67.5 \pm 1.2$ & $72.4 \pm 0.5$ \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{Ablation of the recombination phase of \schemename on TinyImageNet (top) and ImageNet (bottom). The first experiments use the initial segmentation settings with LaMa \cite{Suvorov2021}.}
|
|
\label{tab:ablation-recombine}
|
|
\centering
|
|
\resizebox{\columnwidth}{!}{
|
|
\begin{tabular}{ccccccccccc}
|
|
\toprule
|
|
\multirow{2.5}{*}{\makecell{FG. \\size}} & \multirow{2.5}{*}{\makecell{Augment.\\Order}} & \multirow{2.5}{*}{\makecell{BG\\Strat.}} & \multirow{2.5}{*}{\makecell{BG.\\Prune}} & \multirow{2.5}{*}{\makecell{Original\\Mixing}} & \multirow{2.5}{*}{\makecell{Edge\\Smooth.}} & \multicolumn{2}{c}{Accuracy [\%]} \\
|
|
\cmidrule{7-8}
|
|
& & & & & & ViT-Ti & ViT-S \\
|
|
\midrule
|
|
\multicolumn{6}{l}{\textbf{TinyImageNet}} & \gtxt{$66.1\pm0.5$} & \gtxt{$68.3\pm0.7$} \\
|
|
mean & crop$\to$paste & same & - & - & \gtxt{-} & $64.6\pm0.5$ & $70.0\pm0.6$ \\
|
|
range & \gtxt{crop$\to$paste} & \gtxt{same} & \gtxt{-} & \gtxt{-} & \gtxt{-} & $65.5\pm0.4$ & $71.2\pm0.5$ \\
|
|
\midrule
|
|
{range} & {crop$\to$paste} & {same} & {-} & {-} & {-} & $67.5\pm1.2$ & $72.4\pm0.5$ \\
|
|
\gtxt{range} & paste$\to$crop & \gtxt{same} & \gtxt{-} & \gtxt{-} & \gtxt{-} & $67.1\pm1.2$ & $72.9\pm0.5$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 1.0 & \gtxt{-} & \gtxt{-} & $67.0\pm1.2$ & $73.0\pm0.3$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 0.8 & \gtxt{-} & \gtxt{-} & $67.2\pm1.2$ & $72.9\pm0.8$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 0.6 & \gtxt{-} & \gtxt{-} & $67.5\pm1.0$ & $72.8\pm0.7$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.2$ & \gtxt{-} & $69.8\pm0.5$ & $75.0\pm0.3$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.33$ & \gtxt{-} & $69.5\pm0.4$ & $75.2\pm1.0$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.5$ & \gtxt{-} & $70.3\pm1.0$ & $74.2\pm0.2$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & linear & \gtxt{-} & $70.1\pm0.7$ & $74.9\pm0.8$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & reverse lin. & \gtxt{-} & $67.6\pm0.2$ & $73.2\pm0.3$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & cos & \gtxt{-} & $71.3\pm1.0$ & $75.7\pm0.8$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & $\sigma_\text{max} = 4.0$ & $70.0\pm0.8$ & $75.5\pm0.7$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & orig. & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & $67.2\pm0.9$ & $69.9\pm1.0$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & all & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & $70.1\pm0.7$ & $77.5\pm0.6$ \\
|
|
\midrule
|
|
\multicolumn{6}{l}{\textbf{ImageNet}} & \gtxt{-} & \gtxt{$79.1\pm0.1$} \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & \gtxt{-} & - & $80.5\pm0.1$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & $\sigma_\text{max} = 4.0$ & - & $80.7\pm0.1$ \\
|
|
\gtxt{range} & \gtxt{paste$\to$crop} & all & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & - & $81.4\pm0.1$ \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
|
|
\textbf{Prompt.}
|
|
First, we evaluate the type of prompt used to detect the foreground object.
|
|
Here, the \emph{general} prompt, which contains the class and the more general object category, outperforms only having the class name (\emph{specific}).
|
|
|
|
\textbf{Inpainting.} Among inpainting models, Attentive Eraser~\cite{Sun2024} produces slightly better results compared to LaMa~\cite{Suvorov2021} ($+0.5$ p.p. on average).
|
|
For inpainting examples, see the supplementary material.
|
|
|
|
\textbf{Foreground size}
|
|
significantly impacts performance.
|
|
Employing a \emph{range} of sizes during recombination, rather than a fixed \emph{mean} size, boosts accuracy by approximately 1 p.p.
|
|
This suggests that the added variability is beneficial.
|
|
|
|
\textbf{Order of data augmentation.}
|
|
Applying all augmentations after foreground-background recombination (\emph{paste$\to$crop$\to$color}) improves ViT-S's performance compared to applying crop-related augmentations before pasting (\emph{crop$\to$paste$\to$color}).
|
|
ViT-Ti results are ambiguous.
|
|
|
|
\textbf{Background pruning.}
|
|
When it comes to the backgrounds to use, we test different pruning thresholds ($t_\text{prune}$) to exclude backgrounds with large inpainting.
|
|
A threshold of $t_\text{prune}=1.0$ means that we use all backgrounds that are not fully infilled.
|
|
Varying $t_\text{prune}$ has minimal impact.
|
|
We choose $t_\text{prune} = 0.8$ to exclude predominantly artificial backgrounds.
|
|
|
|
\textbf{Mixing} \schemename-augmented samples with the original ImageNet data proves crucial.
|
|
While constant and linear mixing schedules improve performance over no mixing by $2-3$ p.p. compared to only augmented samples, the cosine annealing schedule proves optimal, boosting accuracy by $3-4$ p.p.
|
|
|
|
\textbf{Edge smoothing.}
|
|
We evaluate the impact of using Gaussian blurring to smooth the edges of the foreground masks.
|
|
For larger models, this gives us a slight performance boost on the full ImageNet (second to last line in \Cref{tab:ablation-recombine}).
|
|
|
|
\textbf{Background strategy.}
|
|
Another point is the allowed choice of background image for each foreground object.
|
|
We compare using the original background, a background from the same class, and any background.
|
|
These strategies go from low diversity and high shared information content between the foreground and background to high diversity and low shared information content.
|
|
For \emph{ViT-Ti}, the latter two strategies perform comparably, while \emph{ViT-S} benefits from the added diversity of using any background.
|
|
The same is true when training on the full ImageNet.
|
|
|
|
|
|
\begin{table}
|
|
\caption{Accuracy of ViT-S on TinyImageNet (TIN) in percent using \schemename with different foreground position distributions by varying the Bates parameter $\eta$.
|
|
The best performance is achieved when using the uniform distribution ($\eta=1$) for training.}
|
|
\label{tbl:foreground-eta}
|
|
\centering
|
|
\small
|
|
\resizebox{.9\columnwidth}{!}{
|
|
\begin{tabular}{ccccccc}
|
|
\toprule
|
|
\multirow{2.5}{*}{\makecell{Bates Parameter \\during training}} & \multirow{2.5}{*}{\makecell{TIN \\w/o \schemename}} & \multicolumn{5}{c}{TIN w/ \schemename} \\
|
|
\cmidrule(l){3-7}
|
|
& & $\eta=-3$ & $-2$ & $1/-1$ & $2$ & $3$ \\
|
|
\midrule
|
|
Baseline & 68.9 & 60.5 & 60.2 & 60.8 & 62.6 & 63.1 \\
|
|
$\eta=-3$ & 71.3 & 79.3 & 79.5 & 79.1 & 79.3 & 79.1 \\
|
|
$\eta=-2$ & 71.5 & 80.0 & 78.7 & 79.3 & 79.1 & 78.8 \\
|
|
$\eta=1/-1$ & 72.3 & 79.5 & 78.9 & 80.2 & 79.7 & 80.4 \\
|
|
$\eta=2$ & 71.3 & 78.2 & 77.8 & 79.1 & 79.6 & 79.9 \\
|
|
$\eta=3$ & 71.4 & 77.2 & 76.9 & 78.6 & 79.6 & 79.7 \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\textbf{Foreground position.}
|
|
Finally, we analyze the foreground object's positioning in the image, using a
|
|
generalization of the Bates distribution~\cite{Bates1955} with parameter $\eta \in \Z$.
|
|
The Bates distribution presents an easy way to sample from a bounded domain with just one hyperparameter that controls its concentration.
|
|
$\eta = 1/-1$ corresponds to the uniform distribution; $\eta > 1$ concentrates the distribution around the center; and for $\eta < -1$, the distribution is concentrated at the borders (see supplementary material for details).
|
|
When sampling more towards the center of the image, the difficulty of the task is reduced, which reduces performance on TinyImageNet (\Cref{tbl:foreground-eta}).
|
|
This is reflected in the performance when evaluating using \schemename with $\eta=2$ and $\eta=3$ compared to $\eta=-1/1$.
|
|
We observe a similar reduction for $\eta < -1$.
|
|
|
|
\begin{table}
|
|
\caption{Dataset statistics for TinyImageNet and ImageNet with and without \schemename. For \schemename we report the number of foreground/background pairs.}
|
|
\label{tab:dataset-stats}
|
|
\centering
|
|
\resizebox{.9\columnwidth}{!}{
|
|
\begin{tabular}{l S[table-format=4.0] S[table-format=7.0] S[table-format=5.0]}
|
|
\toprule
|
|
Dataset & {Classes} & {\makecell{Training \\ Images}} & {\makecell{Validation \\ Images}} \\
|
|
\midrule
|
|
TinyImageNet & 200 & 100000 & 10000 \\
|
|
TinyImageNet + \schemename & 200 & 99404 & 9915 \\
|
|
ImageNet & 1000 & 1281167 & 50000 \\
|
|
ImageNet + \schemename & 1000 & 1274557 & 49751 \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
After fixing the optimal design parameters in \Cref{tab:ablation-segment,tab:ablation-recombine} (last rows), we run \schemename's segmentation step on the entire ImageNet dataset.
|
|
\Cref{tab:dataset-stats} shows the resulting dataset statistics.
|
|
The slightly reduced image count for \schemename is due to instances where Grounded SAM fails to produce valid segmentation masks.
|
|
|
|
|
|
\subsection{Image Classification Results}
|
|
|
|
\begin{table}
|
|
\caption{ImageNet results of models trained on ImageNet with and without \schemename. \schemename improves the performance of most models, with a larger gain for larger models.}
|
|
\label{tab:imagenet-results}
|
|
\centering
|
|
\small
|
|
\resizebox{.8\columnwidth}{!}{\begin{tabular}{lccc}
|
|
\toprule
|
|
\multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{ImageNet Accuracy [\%]}} & \multirow{2.5}{*}{Delta} \\
|
|
\cmidrule(lr){2-3}
|
|
& w/o \schemename & w/ \schemename & \\
|
|
\midrule
|
|
ViT-S & $79.1\pm0.1$ & $81.4\pm0.1$ & \grntxt{$+2.3$} \\
|
|
ViT-B & $77.6\pm0.2$ & $81.1\pm0.4$ & \grntxt{$+3.5$} \\
|
|
ViT-L & $75.3\pm0.4$ & $79.8\pm0.1$ & \grntxt{$+4.5$} \\
|
|
\midrule
|
|
DeiT-S & $80.1 \pm 0.1$ & $80.0\pm0.3$ & \gtxt{$-0.1$} \\
|
|
DeiT-B & $81.9 \pm 0.3$ & $81.9\pm0.2$ & \gtxt{$\pm0.0$} \\
|
|
DeiT-L & $79.3\pm2.3$ & $82.4\pm0.1$ & \grntxt{$+3.1$} \\
|
|
\midrule
|
|
Swin-Ti & $77.9\pm0.2$ & $79.7\pm0.1$ & \grntxt{$+1.8$} \\
|
|
Swin-S & $79.4\pm0.1$ & $80.6\pm0.1$ & \grntxt{$+1.2$} \\
|
|
\midrule
|
|
ResNet-50 & $78.3\pm0.1$ & $78.8\pm0.1$ & \grntxt{$+0.5$} \\
|
|
ResNet-101 & $79.4\pm0.1$ & $80.4\pm0.1$ & \grntxt{$+1.0$} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\Cref{tab:imagenet-results} compares the ImageNet performance of models trained with and without \schemename.
|
|
We adopt the training setup of \cite{Nauen2025} and \cite{Touvron2022} for training ViT \cite{Dosovitskiy2021}, Swin \cite{Liu2021} and ResNet \cite{He2016} (representing CNNs) models as well as the setup of DeiT \cite{Touvron2021b} for that model.
|
|
Both setups are using strong data augmentations like RandAugment, CutMix, and Mixup optimized for Transformers (details in supplementary material).
|
|
Notably, \schemename improves performance across all tested architectures, including the ResNet models, %
|
|
demonstrating benefits beyond Transformers.
|
|
For DeiT we only observe benefits on ImageNet for the larger models.
|
|
For other transformers, we observe improvements from $1.2$ p.p. to $4.5$ p.p. with increasing gains for larger models.
|
|
\schemename's improvements counteract the drop in performance for increasing model sizes.
|
|
Without \schemename this drop is $3.8$ p.p. (ViT-S to L), while with \schemename it is reduced to $1.6$ p.p.
|
|
For DeiT there is a drop of $0.8$ p.p. from small to large while when using \schemename there is a \emph{gain} of $2.4$ p.p.
|
|
|
|
\begin{table}
|
|
\caption{Comparison of \schemename and simple Copy-Paste methods. We train ViT-S on ImageNet using the same 3-augment data augmentation on top of the copy-paste augmentation.}
|
|
\label{tab:copy-paste-comparison}
|
|
\centering
|
|
\resizebox{\columnwidth}{!}{
|
|
\begin{tabular}{lcc S[table-format=+2.1,retain-explicit-plus,detect-inline-weight=math,detect-weight=true]}
|
|
\toprule
|
|
Augmentation & labels & \makecell{ Accuracy [\%]} & {\makecell{Delta \\to Prev.}} \\
|
|
\midrule
|
|
Baseline + \textbf{Simple Copy-Paste} & bg & $31.3 \pm 0.6$ & \\
|
|
+ mixed labels & fg + bg & $32.0 \pm 0.8$ & +0.7 \\
|
|
+ fg labels & fg & $31.6 \pm 0.9$ & -0.4 \\
|
|
+ \emph{range} foreground size variation & \gtxt{fg} & $43.0 \pm 1.2$ & \bfseries +11.4 \\
|
|
+ infilled backgrounds & \gtxt{fg} & $68.7 \pm 0.2$ & \bfseries +25.7 \\
|
|
+ \emph{cos} mixing strategy & \gtxt{fg} & $81.2 \pm 0.1$ & \bfseries +12.5 \\
|
|
+ edge smoothing & \gtxt{fg} & $81.3 \pm 0.1$ & +0.1 \\
|
|
+ background pruning$=$ \textbf{\schemename} & \gtxt{fg} & $81.4 \pm 0.1$ & +0.1 \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
\textbf{Comparison to Simple Copy-Paste.}
|
|
We compare \schemename to a simple adaption of the Copy-Paste augmentation inspired by \cite{Ge2023,Ghiasi2020,Shermaine2025} in \Cref{tab:copy-paste-comparison}.
|
|
Contrary to semantic segmentation we do not have foreground masks available.
|
|
Thus, we paste the extracted foreground objects from \emph{\schemename's segmentation stage} onto normal ImageNet images.
|
|
We observe 3 large jumps in accuracy: (\textbf{1}) From our \emph{range} foreground size variation (+11.4\%), (\textbf{2}) from using our infilled backgrounds instead of images from the dataset (+25.7\%), and (\textbf{3}) from our \emph{cos} mixing strategy with non-augmented images (+12.5\%).
|
|
\schemename's changes to the naive copy-paste augmentation are thus imperative for good classification performance.
|
|
|
|
\begin{table}[t]
|
|
\caption{Downstream accuracy in percent when finetuning on other datasets. Models are pretrained on ImageNet with and without \schemename. Pretraining using \schemename increases transformer downstream accuracy.
|
|
}
|
|
\label{tab:downstream-results}
|
|
\centering
|
|
\resizebox{\columnwidth}{!}{\begin{tabular}{lcccccc}
|
|
\toprule
|
|
Model & \schemename & Aircraft & Cars & Flowers & Food & Pets \\
|
|
\midrule
|
|
ViT-S & \xmark & $72.4\pm1.0$ & $89.8\pm0.3$ & $94.5\pm0.2$ & $89.1\pm0.1$ & $93.8\pm0.2$ \\
|
|
ViT-S & \cmark & $78.6\pm0.5$ & $92.2\pm0.2$ & $95.5\pm0.2$ & $89.6\pm0.1$ & $94.5\pm0.2$ \\
|
|
& & \grntxt{$+6.2$} & \grntxt{$+2.4$} & \grntxt{$+1.0$} & \grntxt{$+0.5$} & \grntxt{$+0.7$} \\
|
|
\cmidrule(r){1-1}
|
|
ViT-B & \xmark & $71.7\pm0.5$ & $90.0\pm0.2$ & $94.8\pm0.4$ & $89.8\pm0.2$ & $94.1\pm0.4$ \\
|
|
ViT-B & \cmark & $79.0\pm2.2$ & $93.3\pm0.1$ & $ 96.5\pm0.1$ & $90.9\pm0.1$ & $95.1\pm0.4$ \\
|
|
& & \grntxt{$+7.3$} & \grntxt{$+3.3$} & \grntxt{$+1.7$} & \grntxt{$+1.1$} & \grntxt{$+1.0$} \\
|
|
\cmidrule(r){1-1}
|
|
ViT-L & \xmark & $72.1\pm1.0$ & $88.8\pm0.3$ & $94.4\pm0.3$ & $90.1\pm0.2$ & $94.2\pm0.4$ \\
|
|
ViT-L & \cmark & $77.6\pm1.2$ & $89.1\pm0.2$ & $96.6\pm0.1$ & $91.3\pm0.1$ & $95.1\pm0.1$ \\
|
|
& & \grntxt{$+5.5$} & \grntxt{$+0.3$} & \grntxt{$+2.2$} & \grntxt{$+1.2$} & \grntxt{$+0.9$} \\
|
|
\midrule
|
|
DeiT-S & \xmark & $75.3\pm0.4$ & $91.1\pm0.2$ & $94.8\pm0.4$ & $89.2\pm0.2$ & $92.4\pm0.2$ \\
|
|
DeiT-S & \cmark & $76.8\pm0.8$ & $91.9\pm0.2$ & $95.2\pm0.3$ & $89.1\pm0.2$ & $92.3\pm0.4$ \\
|
|
& & \grntxt{$+1.5$} & \grntxt{$+0.8$} & \grntxt{$+0.4$} & \gtxt{$-0.1$} & \gtxt{$-0.1$} \\
|
|
\cmidrule(r){1-1}
|
|
DeiT-B & \xmark & $77.0\pm1.2$ & $92.9\pm0.2$ & $96.1\pm0.2$ & $91.2\pm0.1$ & $93.3\pm0.4$ \\
|
|
DeiT-B & \cmark & $79.3\pm0.3$ & $93.1\pm0.1$ & $96.4\pm0.2$ & $91.3\pm0.1$ & $93.3\pm0.1$ \\
|
|
& & \grntxt{$+2.3$} & \gtxt{$+0.2$} & \grntxt{$+0.3$} & \gtxt{$+0.1$} & \gtxt{$\pm0.0$} \\
|
|
\cmidrule(r){1-1}
|
|
DeiT-L & \xmark & $72.8\pm5.5$ & $92.8\pm1.0$ & $95.8\pm1.5$ & $90.5\pm2.6$ & $92.4\pm2.0$ \\
|
|
DeiT-L & \cmark & $78.8\pm0.8$ & $93.8\pm0.2$ & $97.0\pm0.2$ & $92.0\pm0.2$ & $93.5\pm0.2$ \\
|
|
& & \grntxt{$+6.0$} & \grntxt{$+1.0$} & \grntxt{$+1.2$} & \grntxt{$+1.5$} & \grntxt{$+1.1$} \\
|
|
\midrule
|
|
Swin-Ti & \xmark & $77.0\pm0.1$ & $91.3\pm0.6$ & $95.9\pm0.1$ & $90.0\pm0.2$ & $94.2\pm0.1$ \\
|
|
Swin-Ti & \cmark & $81.1\pm0.8$ & $92.8\pm0.4$ & $96.2\pm0.1$ & $90.4\pm0.3$ & $94.8\pm0.5$ \\
|
|
& & \grntxt{$+4.1$} & \grntxt{$+2.5$} & \grntxt{$+0.3$} & \grntxt{$+0.4$} & \grntxt{$+0.6$} \\
|
|
\cmidrule(r){1-1}
|
|
Swin-S & \xmark & $75.7\pm1.4$ & $91.0\pm0.3$ & $95.9\pm0.5$ & $91.1\pm0.2$ & $94.4\pm0.1$ \\
|
|
Swin-S & \cmark & $81.4\pm0.2$ & $93.1\pm0.2$ & $96.3\pm0.3$ & $91.2\pm0.2$ & $94.9\pm0.3$ \\
|
|
& & \grntxt{$+5.7$} & \grntxt{$+2.1$} & \grntxt{$+1.4$} & \gtxt{$+0.1$} & \grntxt{$+0.5$} \\
|
|
\midrule
|
|
ResNet-50 & \xmark & $78.2\pm0.5$ & $89.8\pm0.2$ & $91.7\pm0.4$ & $84.4\pm0.2$ & $93.7\pm0.3$ \\
|
|
ResNet-50 & \cmark & $80.3\pm0.4$ & $90.4\pm0.2$ & $91.7\pm0.2$ & $84.5\pm0.2$ & $93.7\pm0.3$ \\
|
|
& & \grntxt{$+2.1$} & \grntxt{$+0.6$} & \gtxt{$\pm0.0$} & \gtxt{$+0.1$} & \gtxt{$\pm0.0$} \\
|
|
\cmidrule(r){1-1}
|
|
ResNet-101 & \xmark & $78.4\pm0.6$ & $90.3\pm0.1$ & $91.2\pm0.5$ & $86.0\pm0.2$ & $94.3\pm0.2$ \\
|
|
ResNet-101 & \cmark & $81.4\pm0.5$ & $91.3\pm0.1$ & $92.9\pm0.2$ & $86.3\pm0.1$ & $94.0\pm0.3$ \\
|
|
& & \grntxt{$+3.0$} & \grntxt{$+1.3$} & \grntxt{$+1.7$} & \grntxt{$+0.3$} & \textcolor{red}{$-0.3$} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\textbf{Downstream tasks.} To assess the transferability of \schemename-trained models, we finetune models pretrained on ImageNet with and without \schemename on five fine-grained datasets:
|
|
FGVC-Aircraft \cite{Maji2013}, Stanford Cars~\cite{Dehghan2017}, Oxford Flowers \cite{Nilsback2008}, Food-101 \cite{Kaur2017}, and Oxford-IIIT Pets \cite{Parkhi2012}.
|
|
In \Cref{tab:downstream-results} we see transformer accuracies improve on all these datasets by up to 7.3 p.p.
|
|
Notably, training with \schemename boosts the downstream performance of DeiT-S and DeiT-B, despite similar ImageNet results.
|
|
This shows the improved representations from training with \schemename translate to gains beyond better ImageNet scores.
|
|
|
|
|
|
\subsection{Bias and Robustness Evaluation}
|
|
Beyond its use for training, \schemename's unique properties and controlled data generation capabilities make it a powerful tool for analyzing behavior and biases of black-box models.
|
|
|
|
\begin{figure*}
|
|
\centering
|
|
\includegraphics[width=.95\textwidth]{img/bg_robustness.pdf}
|
|
\caption{Evaluation of background robustness on ImageNet + \schemename, ImageNet9 and CounterAnimal.
|
|
We plot the in-distribution (top of arrow) and the out-of-distribution (bottom of arrow) accuracy when training with and without \schemename.
|
|
We annotate each arrow with its length $\Delta$.
|
|
Training with \schemename improves the background robustness of all transformers by mostly boosting the out-of-distribution accuracy.
|
|
}
|
|
\label{fig:background-robustness}
|
|
\end{figure*}
|
|
|
|
\textbf{Background Robustness.}
|
|
We assess the robustness of models to shifts in the background distribution from a class-related background to any background.
|
|
\Cref{fig:background-robustness} presents the background robustness results for three datasets: ImageNet with \schemename (all backgrounds vs. backgrounds of same class), ImageNet9 \cite{Xiao2020} (random backgrounds vs. original backgrounds), and CounterAnimal \cite{Wang2024f} (counter vs. common background).
|
|
The top triangle of each arrow represents the in-distribution backgrounds and the bottom triangle represents the out-of-distribution ones.
|
|
We follow ImageNet9 and CounterAnimal and assess the background robustness in terms of the accuracy gap when evaluating a model on images of normal background distribution compared to out-of-distribution backgrounds (length of each arrow; $\Delta$).
|
|
Crucially, \schemename improves the background robustness of all models and across datasets, reducing the background-gap by boosting the performance on the out-of-background-distribution samples more than the in-distribution ones.
|
|
These findings highlight the generalization benefits of \schemename to unusual image compositions.
|
|
|
|
\begin{figure*}
|
|
\centering
|
|
\includegraphics[width=.95\textwidth]{img/fg_focus.pdf}
|
|
\caption{Evaluation of the foreground focus (\Cref{eq:fg-focus}) using GradCam, GradCam++ and IntegratedGradients (IG) of models trained on ImageNet. Training with \schemename improves the foreground focus of almost all models.}
|
|
\label{fig:foreground-focus}
|
|
\end{figure*}
|
|
|
|
\textbf{Foreground Focus.}
|
|
Leveraging our inherent knowledge of the foreground masks when using \schemename, as well as common XAI techniques~\cite{Selvaraju2016,Chattopadhay2018,Sundararajan2017}, we can evaluate a model's focus on the foreground object.
|
|
We can directly evaluate ImageNet-trained models, but this technique can also be extended to other datasets without relying on manually annotated foreground masks.
|
|
To evaluate the foreground focus, we employ Grad-CAM \cite{Selvaraju2016}, Grad-CAM++ \cite{Chattopadhay2018} and IntegratedGradients (IG) \cite{Sundararajan2017} to compute the per-pixel importance of an image for the model's prediction.
|
|
The foreground focus is defined to be the ratio of the foreground's relative importance to its relative size in the image:
|
|
\begin{align} \label{eq:fg-focus}
|
|
\text{FG Focus}(\text{img}) = \frac{\text{Area}(\text{img}) \hspace{3pt} \text{Importance}(\text{fg})}{\text{Area}(\text{fg}) \hspace{3pt} \text{Importance}(\text{img})}
|
|
\end{align}
|
|
If all pixels uniformly receive the same importance value, the foreground focus is one.
|
|
The foreground focus of a model is its average focus over all test images.
|
|
\Cref{fig:foreground-focus} presents our findings.
|
|
Using \schemename significantly increases the foreground focus of ViT, DeiT and ResNet across all XAI metrics.
|
|
We hypothesize Swin's below-uniform foreground focus with GradCam is due to its specific implementation.
|
|
|
|
\begin{table}[t]
|
|
\caption{
|
|
Accuracy relative to the center accuracy of multiple instantiations of the models when the foreground objects is in different cells of a $3 \times 3$ grid.
|
|
We calculate center bias according to \Cref{eq:center-bias}.
|
|
Using \schemename significantly reduces models' center bias.}
|
|
\label{tab:center-bias}
|
|
\centering
|
|
\resizebox{.78\columnwidth}{!}{
|
|
\begin{tabular}{lccc}
|
|
\toprule
|
|
\multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{Center Bias [\%] when trained}} & \multirow{2.5}{*}{Delta} \\
|
|
\cmidrule(lr){2-3}
|
|
& w/o \schemename & w/ \schemename \\
|
|
\midrule
|
|
ViT-S & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-S_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-S_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-S_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-S_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-S_RecombNet_all_v3.pdf}} \\
|
|
& $25.5\pm0.8$ & $22.0\pm0.3$ & \grntxt{$-3.5$} \\
|
|
ViT-B & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-B_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-B_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-B_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-B_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-B_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-B_RecombNet_all_v3.pdf}} \\
|
|
& $25.4\pm0.4$ & $19.0\pm0.2$ & \grntxt{$-6.4$} \\
|
|
ViT-L & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-L_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-L_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-L_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ViT-L_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-L_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ViT-L_RecombNet_all_v3.pdf}} \\
|
|
& $24.3\pm1.1$ & $11.7\pm0.7$ & \grntxt{$-12.6$} \\
|
|
\midrule
|
|
DeiT-S & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/DeiT-S_ImageNet_vNone.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-S_ImageNet_v3.pdf} } & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/DeiT-S_fornet_all_linear_v1.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-S_fornet_all_linear_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-S_fornet_all_linear_v3.pdf}} \\
|
|
& $20.4 \pm 0.2$ & $21.2 \pm 0.1$ & \gtxt{$+0.8$} \\
|
|
DeiT-B & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/DeiT-B_ImageNet_vNone.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-B_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-B_ImageNet_v3.pdf} } & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/DeiT-B_fornet_all_cos_v1.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-B_fornet_all_cos_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-B_fornet_all_cos_v3.pdf}} \\
|
|
& $19.0 \pm 0.7$ & $19.0 \pm 0.2$ & \gtxt{$\pm0.0$} \\
|
|
DeiT-L & \raisebox{-6pt}{ \includegraphics[width=.08\columnwidth]{img/DeiT-L_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-L_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-L_ImageNet_v3.pdf} } & \raisebox{-6pt}{ \includegraphics[width=.08\columnwidth]{img/DeiT-L_fornet_all_cos_v1.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-L_fornet_all_cos_v2.pdf} \includegraphics[width=.08\columnwidth]{img/DeiT-L_fornet_all_cos_v3.pdf} } \\
|
|
& $21.2 \pm 0.2$ & $18.0 \pm 0.2$ & \grntxt{$-3.2$} \\
|
|
\midrule
|
|
Swin-Ti & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/Swin-Ti_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-Ti_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-Ti_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/Swin-Ti_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-Ti_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-Ti_RecombNet_all_v3.pdf}} \\
|
|
& $25.0\pm0.7$ & $16.5\pm0.2$ & \grntxt{$-8.5$} \\
|
|
Swin-S & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/Swin-S_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-S_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/Swin-S_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-S_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/Swin-S_RecombNet_all_v3.pdf}} \\
|
|
& $23.2\pm0.1$ & $15.6\pm0.2$ & \grntxt{$-7.6$} \\
|
|
\midrule
|
|
ResNet50 & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ResNet50_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet50_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet50_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ResNet50_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet50_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet50_RecombNet_all_v3.pdf}} \\
|
|
& $26.3\pm0.3$ & $19.7\pm0.3$ & \grntxt{$-6.6$} \\
|
|
ResNet101 & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ResNet101_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet101_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet101_ImageNet_v3.pdf}} & \raisebox{-6pt}{\includegraphics[width=.08\columnwidth]{img/ResNet101_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet101_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth]{img/ResNet101_RecombNet_all_v3.pdf}} \\
|
|
& $23.0\pm0.3$ & $19.9\pm0.2$ & \grntxt{$-3.1$} \\
|
|
\bottomrule
|
|
\end{tabular} }
|
|
\includegraphics[width=.8\columnwidth]{img/colorbar_horizontal.pdf}
|
|
\end{table}
|
|
|
|
\textbf{Center Bias.}
|
|
With \schemename we have unique control over the position of the foreground object in the image.
|
|
This lets us quantify the center bias of models trained with and without \schemename.
|
|
We divide the image into a $3 \times 3$ grid and evaluate model accuracy when the (scaled-down) foreground object is in each of the $9$ grid cells.
|
|
Each cell's accuracy is divided by the accuracy in the center cell for normalization, which gives us the relative performance drop when the foreground is in each part of the image.
|
|
The center bias is calculated as one minus the average of the minimum performance of a corner cell and the minimum performance of a side cell:
|
|
\begin{align} \label{eq:center-bias}
|
|
\text{Center Bias} = 1 - \frac{\min\limits_{c \in \text{sides}} \text{Acc}(c) + \min\limits_{c \in \text{corners}} \text{Acc}(c)}{2 \text{Acc}(c_\text{center})}
|
|
\end{align}
|
|
\Cref{tab:center-bias} visualizes the center bias of three instantiations of each model.
|
|
Performance is generally highest in the center and lowest in the four corners.
|
|
Interestingly, ImageNet-trained models perform slightly better when the foreground object is on the right side of the image, compared to the left side, despite our use of random flipping with a probability of $0.5$ during training.
|
|
Using \schemename significantly reduces center bias across models, with a more uniform performance especially across the middle row.
|
|
Thus, \schemename makes the model recognize objects across a wider spatial distribution, counteracting the center-bias of ImageNet.
|
|
|
|
\begin{figure}[t!]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{img/size_bias_grid.pdf}
|
|
\caption{Evaluation of the size bias of models trained on ImageNet. We plot the accuracy relative to the accuracy when using the default size ($f_\text{size} = 1.0$).}
|
|
\label{fig:size-bias}
|
|
\end{figure}
|
|
|
|
\textbf{Size Bias.}
|
|
Finally, we evaluate the impact of different sized foreground objects on the accuracy.
|
|
For this evaluation, we use the \emph{mean} foreground size strategy.
|
|
We introduce a size factor $f_\text{size}$ by which we additionally scale the foreground object before pasting it onto the background.
|
|
Results are normalized by the accuracy when using $f_\text{size} = 1.0$.
|
|
\Cref{fig:size-bias} shows the size bias curves of models trained with and without \schemename.
|
|
Models trained using \schemename maintain perform better, especially with smaller foreground objects.
|
|
Therefore, \schemename-training improves robustness to variations in object scale, especially for larger models.
|