first commit of eccv data
This commit is contained in:
Tobias Nauen
27
sec/abstract.tex
Normal file
27
sec/abstract.tex
Normal file
@@ -0,0 +1,27 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\begin{abstract}
|
||||
% Transformers, particularly Vision Transformers (ViTs), have achieved state-of-the-art performance in large-scale image classification.
|
||||
% However, they often require large amounts of data and can exhibit biases, such as center or size bias, that limit their robustness and generalizability.
|
||||
% This paper introduces \schemename, a novel data augmentation operation that addresses these challenges by explicitly imposing invariances into the training data, which are otherwise part of the neural network architecture.
|
||||
% \schemename is constructed by using pretrained foundation models to separate and recombine foreground objects with different backgrounds.
|
||||
% This recombination step enables us to take fine-grained control over object position and size, as well as background selection.
|
||||
% We demonstrate that using \schemename significantly improves the accuracy of ViTs and other architectures by up to 4.5 percentage points (p.p.) on ImageNet, which translates to 7.3 p.p. on downstream tasks.
|
||||
% Importantly, \schemename not only improves accuracy but also opens new ways to analyze model behavior and quantify biases.
|
||||
% Namely, we introduce metrics for background robustness, foreground focus, center bias, and size bias and show that using \schemename during training substantially reduces these biases.
|
||||
% In summary, \schemename provides a valuable tool for analyzing and mitigating biases, enabling the development of more robust and reliable computer vision models.
|
||||
% Our code and dataset are publicly available at \code{<url>}.
|
||||
|
||||
Large-scale image classification datasets exhibit strong compositional biases: objects tend to be centered, appear at characteristic scales, and co-occur with class-specific context.
|
||||
% Models can exploit these biases to achieve high in-distribution accuracy, yet remain brittle under distribution shifts.
|
||||
By exploiting such biases, models attain high in-distribution accuracy but remain fragile under distribution shifts.
|
||||
To address this issue, we introduce \schemename, a controlled composition augmentation scheme that factorizes each training image into a \emph{foreground object} and a \emph{background} and recombines them to explicitly manipulate object position, object scale, and background identity.
|
||||
\schemename uses off-the-shelf segmentation and inpainting models to (i) extract the foreground and synthesize a neutral background, and (ii) paste the foreground onto diverse neutral backgrounds before applying standard strong augmentation policies.
|
||||
Compared to conventional augmentations and content-mixing methods, our factorization provides direct control knobs that break foreground-background correlations. % while preserving the label.
|
||||
Across 10 architectures, \schemename improves ImageNet top-1 accuracy by up to 6 percentage points (p.p.) and yields gains of up to 7.3 p.p. on fine-grained downstream datasets.
|
||||
Moreover, the same control knobs enable targeted diagnostic tests: we quantify background reliance, foreground focus, center bias, and size bias via controlled background swaps and position/scale sweeps, and show that training with \schemename substantially reduces these shortcut behaviors and significantly increases accuracy on standard distribution-shift benchmarks by up to $19$ p.p.
|
||||
% Moreover, the same control knobs enable targeted diagnostic tests: we quantify background reliance, foreground focus, center bias, and size bias via controlled background swaps and position/scale sweeps, and show that training with \schemename substantially reduces these shortcut behaviors and significantly increases accuracy on standard distribution-shift benchmarks like ImageNet-A/-C/-R by up to $19$ p.p.
|
||||
Our code and dataset are publicly available at \code{<url>}.
|
||||
|
||||
\keywords{Data Augmentation \and Vision Transformer \and Robustness}
|
||||
\end{abstract}
|
||||
8
sec/acks.tex
Normal file
8
sec/acks.tex
Normal file
@@ -0,0 +1,8 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\subsection*{Acknowledgements}
|
||||
\label{sec:acknowledgements}
|
||||
Will be in the final paper.
|
||||
|
||||
% This work was funded by the Carl-Zeiss Foundation under the Sustainable Embedded AI project (P2021-02-009) and by the EU project SustainML (Horizon Europe grant agreement No 101070408).
|
||||
% All compute was done thanks to the Pegasus cluster at DFKI Kaiserslautern.
|
||||
530
sec/appendix.tex
Normal file
530
sec/appendix.tex
Normal file
@@ -0,0 +1,530 @@
|
||||
% !TeX root = ../supplementary.tex
|
||||
|
||||
|
||||
|
||||
\section{Training Setup}
|
||||
\label{sec:training_setup}
|
||||
|
||||
\begin{table*}[h!]
|
||||
\centering
|
||||
\caption{Training setup and hyperparameters for our ImageNet training.}
|
||||
\label{tab:in-setup}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Augmentation Pipeline: & Basic & 3-Augment~\cite{Touvron2022} & RandAugment~\cite{Touvron2021b} \\
|
||||
\midrule
|
||||
Image Resolution & \multicolumn{3}{c}{$224 \times 224$} \\
|
||||
Epochs & \multicolumn{3}{c}{300} \\
|
||||
Learning Rate & S/B: 1e-3, L: 5e-4 & 3e-3 & S/B: 1e-3, L: 5e-4 \\
|
||||
Learning Rate Schedule & \multicolumn{3}{c}{cosine decay} \\
|
||||
Batch Size & 1024 & 2048 & 1024 \\
|
||||
GPUs & \multicolumn{3}{c}{$4\times$ NVIDIA A100/H100/H200} \\
|
||||
Warmup Schedule & \multicolumn{3}{c}{linear} \\
|
||||
Warmup Epochs & \multicolumn{3}{c}{3} \\
|
||||
Weight Decay & 0.05 & 0.02 & 0.05 \\
|
||||
Label Smoothing & \multicolumn{3}{c}{0.1} \\
|
||||
Optimizer & AdamW & Lamb \cite{You2020} & AdamW \\
|
||||
\midrule
|
||||
Augmentations & \makecell{RandomResizedCrop \\ Horizontal Flip \\ ColorJitter} & \makecell{Resize \\ RandomCrop \\ Horizontal Flip \\ Grayscale \\ Solarize \\ Gaussian-Blur \\ Color Jitter} & \makecell{RandomResizedCrop \\ Horizontal Flip \\ RandomErase \cite{Zhong2020} \\ RandAugment \cite{Cubuk2020} \\ Color Jitter} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table*}
|
||||
|
||||
\begin{table}[h!]
|
||||
\centering
|
||||
\caption{Training setup for finetuning on different downstream datasets. Other settings are the same as in \Cref{tab:in-setup}. For finetuning, we always utilize 3-Augment and the related parameters from the \emph{ViT, Swin, ResNet} column of \Cref{tab:in-setup}}
|
||||
\label{tab:downstream-setup}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
Dataset & Batch Size & Epochs & Learning Rate & Num. GPUs \\
|
||||
\midrule
|
||||
Aircraft & 512 & 500 & 3e-4 & 2 \\
|
||||
Cars & 1024 & 500 & 3e-4 & 4 \\
|
||||
Flowers & 256 & 500 & 3e-4 & 1 \\
|
||||
Food & 2048 & 100 & 3e-4 & 4 \\
|
||||
Pets & 512 & 500 & 3e-4 & 2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
On ImageNet, we test three different data augmentation pipelines and hyperparameter settings as shown in \Cref{tab:in-setup}: A basic pipeline, a pipeline using RandAugment based on the DeiT~\cite{Touvron2021b} setup and 3-Augment, as used in \cite{Touvron2022,Nauen2025}.
|
||||
When comparing different architectures, ViT, Swin, and ResNet are trained with the 3-Augment pipeline and DeiT is trained with the RandAugment pipeline.
|
||||
% On ImageNet we use the same training setup as \cite{Nauen2025} and \cite{Touvron2022} without pretraining for ViT, Swin, and ResNet.
|
||||
% For DeiT, we train the same ViT architecture but using the data augmentation scheme and hyperparameters from \cite{Touvron2021b}.
|
||||
As our focus is on evaluating the changes in accuracy due to \schemename, like \cite{Nauen2025}, we stick to one set of hyperparameters for all models.
|
||||
We list the settings used for training on ImageNet in \Cref{tab:in-setup} and the ones used for finetuning those weights on the downstream datasets in \Cref{tab:downstream-setup}.
|
||||
Our implementation is using PyTorch \cite{Paszke2019} and the \emph{timm} library \cite{Wightman2019} for model architectures and basic functions.
|
||||
|
||||
\begin{table*}[ht!]
|
||||
\centering
|
||||
\caption{Hardware and Software specifics used for both training and evaluation.}
|
||||
\label{tab:hw-sw-versions}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
Parameter & Value \\
|
||||
\midrule
|
||||
GPU & $4 \times$ NVIDIA A100/H100/H200 \\
|
||||
CPU & 24 CPU cores (Intel Xenon) per GPU \\
|
||||
Memory & up to 120 GB per GPU \\
|
||||
Operating System & Enroot container for SLURM based on Ubuntu 24.04 LTS \\
|
||||
Python & 3.12.3 \\
|
||||
PyTorch & 2.7.0 \\
|
||||
TorchVision & 0.22.0 \\
|
||||
Timm & 1.0.15 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
\Cref{tab:hw-sw-versions} lists the specific hardware we use, as well as versions of the relevant software packages.
|
||||
|
||||
\section{Resource Usage of \schemename}
|
||||
To utilize the proposed \schemename, specific computational resources are necessary, particularly for computing and storing for the output of the segmentation stage and for on-the-fly processing of the recombination stage.
|
||||
|
||||
\paragraph{Segmentation.}
|
||||
% While calculating the segmentations and infills takes a lot of compute, this is effort that has to be spent only once per dataset.
|
||||
\schemename involves a computationally expensive segmentation and infill stage, which is a one-time calculation per dataset.
|
||||
Once computed, the segmentation and infill results can be perpetually reused, amortizing the initial cost over all subsequent experiments and applications.
|
||||
On NVIDIA H100 GPUs, the segmentation stage will compute at a rate of $374.3 \frac{\text{img}}{\text{GPU} \times \text{h}}$ when using Attentive Eraser or $5 338.6 \frac{\text{img}}{\text{GPU} \times \text{h}}$ for LaMa.
|
||||
For ImageNet this comes down to just under 9 days (Attentive Eraser) or 16 hours (LaMa) on two 8 GPU nodes.
|
||||
To facilitate immediate use and reproduction of results, we publicly provide the precalculated segmentation stage output for the ImageNet dataset for download\footnote{Link will go here.}.
|
||||
The output of \schemename's segmentation step on ImageNet dataset requires 73 GB of additional disk space for the segmentation output, which is separate from the base 147 GB ImageNet size.
|
||||
|
||||
\paragraph{Recombination.}
|
||||
The recombination step of \schemename is implemented as a based data loader operation.
|
||||
It's thus offloaded to the CPU, where it can be heavily parallelized and thus only results in a very minor increase in the training step-time.
|
||||
For example, using a ViT-B model on an NVIDIA A100 GPU, the average update step-time increased by $1\%$, from $528 \pm 2$ ms to $534 \pm 1$ ms.
|
||||
|
||||
\section{Extended Bates Distribution}
|
||||
\label{apdx:bates-distribution}
|
||||
\begin{figure}[h!]
|
||||
\centering
|
||||
\includegraphics[width=.5\columnwidth]{img/bates.pdf}
|
||||
\caption{Plot of the probability distribution function (PDF) of the extended Bates distribution for different parameters $\eta$. Higher values of $\eta$ concentrate the distribution around the center.}
|
||||
\label{fig:bates-pdf}
|
||||
\end{figure}
|
||||
|
||||
We introduce an extension of the Bates distribution~\cite{Bates1955} to include negative parameters, enabling sampling of foreground object positions away from the image center.
|
||||
The standard Bates distribution, for $\eta \in \N$, is defined as the mean of $\eta$ independent random variables drawn from a uniform distribution \cite{Jonhson1995}.
|
||||
A larger $\eta$ value increases the concentration of samples around the distribution's mean, which in this case is the image center.
|
||||
|
||||
To achieve an opposite effect--concentrating samples at the image borders--we extend the distribution to $\eta \leq 1$.
|
||||
\begin{align*}
|
||||
X \sim \text{Bates}(\eta) :\Leftrightarrow s(X) \sim \text{Bates}(-\eta)
|
||||
\end{align*}
|
||||
This is accomplished by sampling from a standard Bates distribution with parameter $-\eta \geq 1$ and then applying a sawtooth function.
|
||||
The sawtooth function on the interval $[0,1]$ is defined as
|
||||
\begin{align}
|
||||
s(x) = \begin{cases}
|
||||
x + 0.5 & \text{if } 0 < x < 0.5 \\
|
||||
x - 0.5 & \text{if } 0.5 \leq x \leq 1
|
||||
\end{cases}
|
||||
\end{align}
|
||||
This function effectively maps the central portion of the interval to the edges and the edge portions to the center.
|
||||
For example, a value of 0.3 (central-left) is mapped to 0.8 (edge-right), while 0.8 (edge-right) is mapped to 0.3 (central-left).
|
||||
This transformation inverts the distribution's concentration, shifting the probability mass from the center to the borders.
|
||||
We visualize the distribution function of the extended Bates distribution in \Cref{fig:bates-pdf}.
|
||||
Both $\eta = 1$ and $\eta = -1$ result in a uniform distribution across the image.
|
||||
|
||||
\section{Design Choices of \schemename}
|
||||
\label{sec:ablation}
|
||||
|
||||
We start by ablating the design choices of \schemename on TinyImageNet~\cite{Le2015}, a subset of ImageNet containing 200 categories with 500 images each. %, and Tiny\name, the application of \schemename to TinyImageNet.
|
||||
% \Cref{tab:ablation} presents the results of these ablations.
|
||||
\Cref{tab:ablation-segment} presents ablations for segmentation and \Cref{tab:ablation-recombine} for recombination.
|
||||
|
||||
\begin{table}
|
||||
\caption{Ablation of the design decisions in the segmentation phase of \schemename on TinyImageNet.
|
||||
The first line is our baseline, while the other lines are using \schemename.
|
||||
We use basic settings with the \emph{same} background strategy during recombination for this experiment.
|
||||
}
|
||||
\label{tab:ablation-segment}
|
||||
\centering
|
||||
\small
|
||||
% \resizebox{.9\columnwidth}{!}{
|
||||
\begin{tabular}{llcc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{\makecell{Detect. \\Prompt}} & \multirow{2.5}{*}{\makecell{Infill \\ Model}} & \multicolumn{2}{c}{TinyImageNet Accuracy [\%]} \\
|
||||
\cmidrule{3-4}
|
||||
& & ViT-Ti & ViT-S \\
|
||||
\midrule
|
||||
\multicolumn{2}{l}{\textbf{TinyImageNet}} & $66.1 \pm 0.5$ & $68.3 \pm 0.7$ \\
|
||||
specific & LaMa \cite{Suvorov2022} & $65.5 \pm 0.4$ & $71.2 \pm 0.5$ \\
|
||||
general & \gtxt{LaMa \cite{Suvorov2022}} & $66.4 \pm 0.6$ & $72.9 \pm 0.6$ \\
|
||||
\gtxt{general} & Att. Eraser \cite{Sun2025} & $67.5 \pm 1.2$ & $72.4 \pm 0.5$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
% }
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Ablation of the recombination phase of \schemename on TinyImageNet (top) and ImageNet (bottom). The first experiments use the initial segmentation settings with LaMa \cite{Suvorov2022}.}
|
||||
\label{tab:ablation-recombine}
|
||||
\centering
|
||||
% \resizebox{.9\columnwidth}{!}{
|
||||
\begin{tabular}{ccccccccccc}
|
||||
\toprule
|
||||
% FG. & Augment. & BG. & BG. & Edge & Original & \multicolumn{2}{c}{Accuracy [\%]} \\
|
||||
% Size & Order & Strat. & Prune & Smoothing & Mixing & ViT-Ti & ViT-S \\
|
||||
\multirow{2.5}{*}{\makecell{FG. \\size}} & \multirow{2.5}{*}{\makecell{Augment.\\Order}} & \multirow{2.5}{*}{\makecell{BG\\Strat.}} & \multirow{2.5}{*}{\makecell{BG.\\Prune}} & \multirow{2.5}{*}{\makecell{Original\\Mixing}} & \multirow{2.5}{*}{\makecell{Edge\\Smooth.}} & \multicolumn{2}{c}{Accuracy [\%]} \\
|
||||
\cmidrule{7-8}
|
||||
& & & & & & ViT-Ti & ViT-S \\
|
||||
\midrule
|
||||
% TinyImageNet & & & & & & & $66.1\pm0.5$ & $68.3\pm0.7$ \\
|
||||
\multicolumn{6}{l}{\textbf{TinyImageNet}} & \gtxt{$66.1\pm0.5$} & \gtxt{$68.3\pm0.7$} \\
|
||||
mean & crop$\to$paste & same & - & - & \gtxt{-} & $64.6\pm0.5$ & $70.0\pm0.6$ \\
|
||||
range & \gtxt{crop$\to$paste} & \gtxt{same} & \gtxt{-} & \gtxt{-} & \gtxt{-} & $65.5\pm0.4$ & $71.2\pm0.5$ \\
|
||||
\midrule
|
||||
% \gtxt{range} & \gtxt{crop$\to$paste} & \gtxt{same} & \gtxt{-} & \gtxt{-} & \gtxt{-} & $66.4\pm0.6$ & $72.9\pm0.6$ \\
|
||||
{range} & {crop$\to$paste} & {same} & {-} & {-} & {-} & $67.5\pm1.2$ & $72.4\pm0.5$ \\
|
||||
\gtxt{range} & paste$\to$crop & \gtxt{same} & \gtxt{-} & \gtxt{-} & \gtxt{-} & $67.1\pm1.2$ & $72.9\pm0.5$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 1.0 & \gtxt{-} & \gtxt{-} & $67.0\pm1.2$ & $73.0\pm0.3$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 0.8 & \gtxt{-} & \gtxt{-} & $67.2\pm1.2$ & $72.9\pm0.8$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & 0.6 & \gtxt{-} & \gtxt{-} & $67.5\pm1.0$ & $72.8\pm0.7$ \\
|
||||
% \gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $\sigma_\text{max} = 2.0$ & \gtxt{-} & $67.2\pm0.4$ & $72.9\pm0.5$ \\
|
||||
% \gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $\sigma_\text{max} = 4.0$ & \gtxt{-} & $65.9\pm0.5$ & $72.4\pm0.6$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.2$ & \gtxt{-} & $69.8\pm0.5$ & $75.0\pm0.3$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.33$ & \gtxt{-} & $69.5\pm0.4$ & $75.2\pm1.0$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & $p=0.5$ & \gtxt{-} & $70.3\pm1.0$ & $74.2\pm0.2$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & linear & \gtxt{-} & $70.1\pm0.7$ & $74.9\pm0.8$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & reverse lin. & \gtxt{-} & $67.6\pm0.2$ & $73.2\pm0.3$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & cos & \gtxt{-} & $71.3\pm1.0$ & $75.7\pm0.8$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & $\sigma_\text{max} = 4.0$ & $70.0\pm0.8$ & $75.5\pm0.7$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & orig. & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & $67.2\pm0.9$ & $69.9\pm1.0$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & all & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & $70.1\pm0.7$ & $77.5\pm0.6$ \\
|
||||
\midrule
|
||||
\multicolumn{6}{l}{\textbf{ImageNet}} & \gtxt{-} & \gtxt{$79.1\pm0.1$} \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & \gtxt{-} & - & $80.5\pm0.1$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & \gtxt{same} & \gtxt{0.8} & \gtxt{cos} & $\sigma_\text{max} = 4.0$ & - & $80.7\pm0.1$ \\
|
||||
\gtxt{range} & \gtxt{paste$\to$crop} & all & \gtxt{0.8} & \gtxt{cos} & \gtxt{$\sigma_\text{max} = 4.0$} & - & $81.4\pm0.1$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
% }
|
||||
\end{table}
|
||||
|
||||
|
||||
\textbf{Prompt.}
|
||||
% We present the ablation of our main design decisions in \Cref{tab:ablation}.
|
||||
First, we evaluate the type of prompt used to detect the foreground object.
|
||||
Here, the \emph{general} prompt, which contains the class and the more general object category, outperforms only having the class name (\emph{specific}).
|
||||
|
||||
\textbf{Inpainting.} Among inpainting models, Attentive Eraser~\cite{Sun2025} produces slightly better results compared to LaMa~\cite{Suvorov2022} ($+0.5$ p.p. on average).
|
||||
For inpainting examples, see the supplementary material.
|
||||
% (see the supplementary material for examples).
|
||||
% When comparing the infill models, the GAN-based LaMa \cite{Suvorov2022} gets outperformed by the Attentive Eraser \cite{Sun2025}.
|
||||
|
||||
\textbf{Foreground size}
|
||||
% We observe that LaMa's often infills unnatural textures compared to Attentive Eraser.
|
||||
% The size of foreground objects during training has a significant impact on the performance.
|
||||
% Here, using the greater variability of the \emph{range} strategy increases the performance by $\approx 1\%$ compared to the \emph{mean} strategy.
|
||||
significantly impacts performance.
|
||||
Employing a \emph{range} of sizes during recombination, rather than a fixed \emph{mean} size, boosts accuracy by approximately 1 p.p.
|
||||
This suggests that the added variability is beneficial.
|
||||
|
||||
\textbf{Order of data augmentation.}
|
||||
% (1) Applying the image crop related augmentations \emph{before} pasting the foreground object and the color-based ones \emph{after} pasting or (2) applying all data augmentations after pasting the foreground object.
|
||||
% While results are ambiguous, we choose the second strategy, as it improves the performance of ViT-S, although not the one of ViT-Ti.
|
||||
Applying all augmentations after foreground-background recombination (\emph{paste$\to$crop$\to$color}) improves ViT-S's performance compared to applying crop-related augmentations before pasting (\emph{crop$\to$paste$\to$color}).
|
||||
ViT-Ti results are ambiguous.
|
||||
|
||||
\textbf{Background pruning.}
|
||||
When it comes to the backgrounds to use, we test different pruning thresholds ($t_\text{prune}$) to exclude backgrounds with large inpainting.
|
||||
% and only use backgrounds with an relative size of the infilled region of at most $t_\text{prune}$ (exclusive).
|
||||
A threshold of $t_\text{prune}=1.0$ means that we use all backgrounds that are not fully infilled.
|
||||
% We find that the background pruning does not significantly impact the models' performance.
|
||||
% We choose $t_\text{prune}=0.8$ for the following experiments to exclude backgrounds that are mostly artificial.
|
||||
Varying $t_\text{prune}$ has minimal impact.
|
||||
We choose $t_\text{prune} = 0.8$ to exclude predominantly artificial backgrounds.
|
||||
|
||||
% One of the most important design decisions is the mixing of the original dataset with \name.
|
||||
\textbf{Mixing} \schemename-augmented samples with the original ImageNet data proves crucial.
|
||||
While constant and linear mixing schedules improve performance over no mixing by $2-3$ p.p. compared to only augmented samples, the cosine annealing schedule proves optimal, boosting accuracy by $3-4$ p.p.
|
||||
|
||||
\textbf{Edge smoothing.}
|
||||
We evaluate the impact of using Gaussian blurring to smooth the edges of the foreground masks.
|
||||
% Similarly, applying edge smoothing to foreground masks with Gaussian blurring actually hurts performance on Tiny\name, but slightly improves it on \name.
|
||||
For larger models, this gives us a slight performance boost on the full ImageNet (second to last line in \Cref{tab:ablation-recombine}).
|
||||
|
||||
\textbf{Background strategy.}
|
||||
Another point is the allowed choice of background image for each foreground object.
|
||||
% We evaluate three different strategies.
|
||||
% (1) Picking the background from which that specific foreground was originally extracted.
|
||||
% The major difference to ImageNet when using this setup is the variability in size and position of the foreground object.
|
||||
% (2) Picking a background that originally had a foreground object of the same class in it.
|
||||
% Here, we have backgrounds where objects of this type can typically appear while also creating a wider variety of samples due to pairing each foreground object with different backgrounds each time.
|
||||
% (3) Picking any background.
|
||||
% This choice has the largest variety of backgrounds, but the backgrounds are not semantically related to the foreground object anymore.
|
||||
% We find in \Cref{fig:bg-strategy} that choosing only a foreground's original background is the worst choice.
|
||||
We compare using the original background, a background from the same class, and any background.
|
||||
These strategies go from low diversity and high shared information content between the foreground and background to high diversity and low shared information content.
|
||||
For \emph{ViT-Ti}, the latter two strategies perform comparably, while \emph{ViT-S} benefits from the added diversity of using any background.
|
||||
The same is true when training on the full ImageNet.
|
||||
|
||||
|
||||
\begin{table}
|
||||
\caption{Accuracy of ViT-S on TinyImageNet (TIN) in percent using \schemename with different foreground position distributions by varying the Bates parameter $\eta$.
|
||||
The best performance is achieved when using the uniform distribution ($\eta=1$) for training.}
|
||||
\label{tbl:foreground-eta}
|
||||
\centering
|
||||
\small
|
||||
% \resizebox{.9\columnwidth}{!}{
|
||||
\begin{tabular}{ccccccc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{\makecell{Bates Parameter \\during training}} & \multirow{2.5}{*}{\makecell{TIN \\w/o \schemename}} & \multicolumn{5}{c}{TIN w/ \schemename} \\
|
||||
\cmidrule(l){3-7}
|
||||
& & $\eta=-3$ & $-2$ & $1/-1$ & $2$ & $3$ \\
|
||||
\midrule
|
||||
Baseline & 68.9 & 60.5 & 60.2 & 60.8 & 62.6 & 63.1 \\
|
||||
$\eta=-3$ & 71.3 & 79.3 & 79.5 & 79.1 & 79.3 & 79.1 \\
|
||||
$\eta=-2$ & 71.5 & 80.0 & 78.7 & 79.3 & 79.1 & 78.8 \\
|
||||
$\eta=1/-1$ & 72.3 & 79.5 & 78.9 & 80.2 & 79.7 & 80.4 \\
|
||||
$\eta=2$ & 71.3 & 78.2 & 77.8 & 79.1 & 79.6 & 79.9 \\
|
||||
$\eta=3$ & 71.4 & 77.2 & 76.9 & 78.6 & 79.6 & 79.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
% }
|
||||
\end{table}
|
||||
|
||||
\textbf{Foreground position.}
|
||||
Finally, we analyze the foreground object's positioning in the image, using a
|
||||
generalization of the Bates distribution~\cite{Bates1955} with parameter $\eta \in \Z$ (see \Cref{apdx:bates-distribution}).
|
||||
The Bates distribution presents an easy way to sample from a bounded domain with just one hyperparameter that controls its concentration.
|
||||
$\eta = 1/-1$ corresponds to the uniform distribution; $\eta > 1$ concentrates the distribution around the center; and for $\eta < -1$, the distribution is concentrated at the borders (see supplementary material for details).
|
||||
% We utilize an extended Bates distribution to sample the position of the foreground object.
|
||||
% The Bates distribution with parameter $\eta \geq 1$ is the mean of $\eta$ independent uniformly distributed random variables \cite{Jonhson1995}.
|
||||
% The larger $\eta$, the more concentrated the distribution is at the center, $\eta < -1$ concentrates the distribution at the edges.
|
||||
% We extend this concept to $\eta \leq -1$, shifting the distribution away from the center and towards the edges.
|
||||
When sampling more towards the center of the image, the difficulty of the task is reduced, which reduces performance on TinyImageNet (\Cref{tbl:foreground-eta}).
|
||||
This is reflected in the performance when evaluating using \schemename with $\eta=2$ and $\eta=3$ compared to $\eta=-1/1$.
|
||||
We observe a similar reduction for $\eta < -1$.
|
||||
% This experiment is conducted using the LaMa infill model.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Dataset statistics for TinyImageNet and ImageNet with and without \schemename. For \schemename we report the number of foreground/background pairs.}
|
||||
\label{tab:dataset-stats}
|
||||
\centering
|
||||
% \resizebox{.5\columnwidth}{!}{
|
||||
\begin{tabular}{l S[table-format=4.0] S[table-format=7.0] S[table-format=5.0]}
|
||||
\toprule
|
||||
Dataset & {Classes} & {\makecell{Training \\ Images}} & {\makecell{Validation \\ Images}} \\
|
||||
\midrule
|
||||
TinyImageNet & 200 & 100000 & 10000 \\
|
||||
TinyImageNet + \schemename & 200 & 99404 & 9915 \\
|
||||
ImageNet & 1000 & 1281167 & 50000 \\
|
||||
ImageNet + \schemename & 1000 & 1274557 & 49751 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
% }
|
||||
\end{table}
|
||||
After fixing the optimal design parameters in \Cref{tab:ablation-segment,tab:ablation-recombine} (last rows), we run \schemename's segmentation step on the entire ImageNet dataset.
|
||||
\Cref{tab:dataset-stats} shows the resulting dataset statistics.
|
||||
% The slightly lower number of images in \name is due to \emph{Grounded SAM} returning no or invalid detections for some images.
|
||||
The slightly reduced image count for \schemename is due to instances where Grounded SAM fails to produce valid segmentation masks.
|
||||
|
||||
\section{Robustness Evaluation on Corner-Cases}
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Evaluation on the Corner-Cases dataset. Objects cut from ImageNet evaluation bounding boxes are pasted onto infilled backgrounds. Objects have three sizes: $56$px, $84$px, and $112$px. Objects are places in the center (CeX) or corner (CoX) of an image its original background (XxO) or a random background (XxR).}
|
||||
\label{tab:corner-cases}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lcccccccccccccc}
|
||||
\toprule
|
||||
\multirow{4}{*}{Model} & \multirow{4}{*}{w/ \schemename} & \multicolumn{12}{c}{Corner Cases Accuracy [\%]} \\
|
||||
\cmidrule(l){3-14}
|
||||
& & \multicolumn{4}{c}{56} & \multicolumn{4}{c}{84} & \multicolumn{4}{c}{112} \\
|
||||
\cmidrule(lr){3-6} \cmidrule(lr){7-10} \cmidrule(l){11-14}
|
||||
& & CeO & CoO & CeR & CoR & CeO & CoO & CeR & CoR & CeO & CoO & CeR & CoR \\
|
||||
\midrule
|
||||
ViT-S & \xmark & $40.5 \pm 2.0$ & $28.6 \pm 0.8$ & $10.3 \pm 0.9$ & $6.4 \pm 0.2$ & $56.8 \pm 1.2$ & $47.6 \pm 1.0$ & $31.3 \pm 0.7$ & $25.5 \pm 0.5$ & $70.9 \pm 0.1$ & $66.9 \pm 1.6$ & $55.2 \pm 0.2$ & $51.1 \pm 0.8$ \\
|
||||
ViT-S & \cmark & $49.4 \pm 0.6$ & $39.9 \pm 0.5$ & $22.7 \pm 0.4$ & $17.6 \pm 0.3$ & $66.3 \pm 0.3$ & $60.0 \pm 0.3$ & $47.7 \pm 0.7$ & $43.2 \pm 0.2$ & $76.5 \pm 0.2$ & $74.9 \pm 0.4$ & $66.8 \pm 0.6$ & $64.9 \pm 0.1$ \\
|
||||
& & \grntxt{$+8.9$} & \grntxt{$+11.3$} & \grntxt{$+12.4$} & \grntxt{$+11.2$} & \grntxt{$+9.4$} & \grntxt{$+12.4$} & \grntxt{$+16.4$} & \grntxt{$+17.7$} & \grntxt{$+5.6$} & \grntxt{$+8.0$} & \grntxt{$+11.6$} & \grntxt{$+13.7$} \\
|
||||
\cmidrule(r){1-2}
|
||||
ViT-B & \xmark & $37.9 \pm 1.4$ & $29.3 \pm 0.7$ & $14.0 \pm 1.7$ & $11.9 \pm 1.1$ & $51.5 \pm 0.7$ & $45.0 \pm 0.8$ & $27.3 \pm 0.8$ & $26.3 \pm 0.8$ & $64.7 \pm 0.3$ & $61.8 \pm 0.6$ & $46.3 \pm 0.3$ & $45.5 \pm 0.5$ \\
|
||||
ViT-B & \cmark & $50.4 \pm 0.8$ & $42.4 \pm 0.6$ & $26.5 \pm 0.6$ & $22.8 \pm 0.8$ & $65.3 \pm 0.9$ & $60.9 \pm 0.6$ & $47.6 \pm 0.3$ & $45.6 \pm 0.1$ & $75.7 \pm 0.6$ & $74.0 \pm 0.6$ & $65.7 \pm 0.7$ & $64.3 \pm 0.5$ \\
|
||||
& & \grntxt{$+12.5$} & \grntxt{$+13.1$} & \grntxt{$+12.4$} & \grntxt{$+10.9$} & \grntxt{$+13.8$} & \grntxt{$+15.9$} & \grntxt{$+20.2$} & \grntxt{$+19.3$} & \grntxt{$+11.0$} & \grntxt{$+12.2$} & \grntxt{$+19.3$} & \grntxt{$+18.8$} \\
|
||||
\cmidrule(r){1-2}
|
||||
ViT-L & \xmark & $32.8 \pm 1.6$ & $24.8 \pm 1.1$ & $14.8 \pm 2.2$ & $9.7 \pm 1.2$ & $42.7 \pm 0.9$ & $33.8 \pm 0.7$ & $21.3 \pm 1.5$ & $16.3 \pm 1.0$ & $55.7 \pm 0.7$ & $49.7 \pm 0.7$ & $36.0 \pm 1.3$ & $32.5 \pm 0.9$ \\
|
||||
ViT-L & \cmark & $45.7 \pm 0.6$ & $39.0 \pm 0.5$ & $25.6 \pm 0.6$ & $24.1 \pm 0.8$ & $59.1 \pm 0.3$ & $55.2 \pm 0.4$ & $41.9 \pm 1.0$ & $42.7 \pm 0.6$ & $71.4 \pm 0.3$ & $69.0 \pm 0.4$ & $60.7 \pm 1.0$ & $60.3 \pm 0.8$ \\
|
||||
& & \grntxt{$+12.9$} & \grntxt{$+14.2$} & \grntxt{$+10.8$} & \grntxt{$+14.4$} & \grntxt{$+16.3$} & \grntxt{$+21.5$} & \grntxt{$+20.5$} & \grntxt{$+26.4$} & \grntxt{$+15.7$} & \grntxt{$+19.3$} & \grntxt{$+24.7$} & \grntxt{$+27.8$} \\
|
||||
\cmidrule(r){1-2}
|
||||
DeiT-S & \xmark & $46.3 \pm 0.7$ & $38.1 \pm 0.3$ & $13.1 \pm 0.5$ & $9.9 \pm 0.1$ & $62.8 \pm 0.4$ & $58.2 \pm 0.2$ & $37.1 \pm 0.7$ & $34.3 \pm 0.5$ & $73.3 \pm 0.2$ & $73.9 \pm 0.4$ & $58.8 \pm 0.4$ & $59.4 \pm 0.6$ \\
|
||||
DeiT-S & \cmark & $44.7 \pm 1.4$ & $37.1 \pm 1.4$ & $15.6 \pm 1.3$ & $12.1 \pm 0.9$ & $62.1 \pm 1.2$ & $57.8 \pm 1.1$ & $41.6 \pm 1.1$ & $37.9 \pm 1.2$ & $73.2 \pm 0.7$ & $73.3 \pm 0.4$ & $62.3 \pm 0.7$ & $61.4 \pm 0.9$ \\
|
||||
& & \rdtxt{$-1.6$} & \rdtxt{$-1.1$} & \grntxt{$+2.4$} & \grntxt{$+2.2$} & \rdtxt{$-0.7$} & \rdtxt{$-0.4$} & \grntxt{$+4.4$} & \grntxt{$+3.5$} & \gtxt{$-0.1$} & \rdtxt{$-0.6$} & \grntxt{$+3.5$} & \grntxt{$+2.0$} \\
|
||||
\cmidrule(r){1-2}
|
||||
DeiT-B & \xmark & $48.1 \pm 0.9$ & $40.4 \pm 2.0$ & $15.8 \pm 0.2$ & $12.9 \pm 0.6$ & $64.0 \pm 0.9$ & $59.5 \pm 1.3$ & $39.0 \pm 0.9$ & $37.2 \pm 0.8$ & $74.1 \pm 0.7$ & $74.8 \pm 0.7$ & $59.1 \pm 0.8$ & $60.0 \pm 0.6$ \\
|
||||
DeiT-B & \cmark & $50.7 \pm 0.1$ & $44.0 \pm 0.4$ & $19.3 \pm 0.2$ & $16.3 \pm 0.2$ & $66.0 \pm 0.2$ & $62.0 \pm 0.3$ & $43.4 \pm 0.3$ & $40.9 \pm 0.4$ & $75.4 \pm 0.1$ & $76.4 \pm 0.3$ & $62.8 \pm 0.2$ & $63.9 \pm 0.2$ \\
|
||||
& & \grntxt{$+2.6$} & \grntxt{$+3.6$} & \grntxt{$+3.5$} & \grntxt{$+3.5$} & \grntxt{$+2.0$} & \grntxt{$+2.5$} & \grntxt{$+4.4$} & \grntxt{$+3.8$} & \grntxt{$+1.3$} & \grntxt{$+1.6$} & \grntxt{$+3.8$} & \grntxt{$+3.9$} \\
|
||||
\cmidrule(r){1-2}
|
||||
DeiT-L & \xmark & $39.2 \pm 2.6$ & $32.6 \pm 1.5$ & $10.5 \pm 2.8$ & $9.1 \pm 2.3$ & $55.7 \pm 2.5$ & $51.0 \pm 2.7$ & $30.3 \pm 4.0$ & $29.5 \pm 3.9$ & $68.5 \pm 2.1$ & $68.1 \pm 1.7$ & $51.7 \pm 3.1$ & $52.1 \pm 2.7$ \\
|
||||
DeiT-L & \cmark & $51.9 \pm 0.7$ & $46.6 \pm 0.5$ & $21.5 \pm 1.3$ & $19.0 \pm 1.2$ & $66.6 \pm 0.6$ & $64.1 \pm 0.7$ & $45.3 \pm 1.3$ & $43.6 \pm 1.1$ & $75.6 \pm 0.4$ & $77.3 \pm 0.4$ & $63.8 \pm 0.8$ & $65.4 \pm 0.6$ \\
|
||||
& & \grntxt{$+12.8$} & \grntxt{$+14.0$} & \grntxt{$+11.0$} & \grntxt{$+9.9$} & \grntxt{$+11.0$} & \grntxt{$+13.1$} & \grntxt{$+15.0$} & \grntxt{$+14.1$} & \grntxt{$+7.1$} & \grntxt{$+9.2$} & \grntxt{$+12.1$} & \grntxt{$+13.4$} \\
|
||||
\cmidrule(r){1-2}
|
||||
Swin-Ti & \xmark & $41.2 \pm 1.8$ & $32.5 \pm 0.3$ & $17.4 \pm 2.6$ & $12.2 \pm 0.2$ & $60.0 \pm 1.6$ & $51.4 \pm 0.2$ & $39.6 \pm 2.6$ & $34.8 \pm 0.9$ & $71.7 \pm 0.8$ & $66.1 \pm 0.7$ & $58.2 \pm 1.1$ & $53.6 \pm 1.2$ \\
|
||||
Swin-Ti & \cmark & $49.8 \pm 0.6$ & $42.8 \pm 0.7$ & $24.2 \pm 0.7$ & $21.4 \pm 0.9$ & $66.4 \pm 0.6$ & $60.5 \pm 0.2$ & $47.8 \pm 0.5$ & $44.6 \pm 0.5$ & $76.0 \pm 0.3$ & $72.7 \pm 0.2$ & $65.7 \pm 0.5$ & $62.1 \pm 0.3$ \\
|
||||
& & \grntxt{$+8.5$} & \grntxt{$+10.3$} & \grntxt{$+6.8$} & \grntxt{$+9.2$} & \grntxt{$+6.4$} & \grntxt{$+9.2$} & \grntxt{$+8.2$} & \grntxt{$+9.8$} & \grntxt{$+4.3$} & \grntxt{$+6.5$} & \grntxt{$+7.5$} & \grntxt{$+8.5$} \\
|
||||
\cmidrule(r){1-2}
|
||||
Swin-S & \xmark & $41.3 \pm 0.6$ & $33.0 \pm 0.1$ & $18.4 \pm 0.7$ & $13.3 \pm 0.5$ & $59.2 \pm 0.1$ & $51.2 \pm 0.5$ & $39.1 \pm 0.2$ & $35.9 \pm 0.3$ & $71.5 \pm 0.2$ & $65.6 \pm 0.1$ & $56.8 \pm 0.5$ & $53.2 \pm 0.2$ \\
|
||||
Swin-S & \cmark & $48.6 \pm 0.7$ & $39.9 \pm 1.6$ & $22.2 \pm 0.9$ & $16.8 \pm 1.1$ & $64.4 \pm 0.9$ & $57.9 \pm 1.5$ & $43.8 \pm 1.1$ & $42.3 \pm 1.0$ & $75.7 \pm 0.2$ & $71.8 \pm 0.8$ & $63.2 \pm 0.4$ & $60.6 \pm 0.6$ \\
|
||||
& & \grntxt{$+7.3$} & \grntxt{$+7.0$} & \grntxt{$+3.8$} & \grntxt{$+3.6$} & \grntxt{$+5.1$} & \grntxt{$+6.7$} & \grntxt{$+4.7$} & \grntxt{$+6.4$} & \grntxt{$+4.2$} & \grntxt{$+6.2$} & \grntxt{$+6.4$} & \grntxt{$+7.4$} \\
|
||||
\cmidrule(r){1-2}
|
||||
ResNet50 & \xmark & $48.6 \pm 0.6$ & $35.1 \pm 0.4$ & $23.0 \pm 0.7$ & $13.0 \pm 0.3$ & $65.8 \pm 0.4$ & $58.2 \pm 0.3$ & $44.4 \pm 0.6$ & $38.1 \pm 0.5$ & $73.2 \pm 0.2$ & $69.9 \pm 0.2$ & $56.9 \pm 0.1$ & $56.9 \pm 0.1$ \\
|
||||
ResNet50 & \cmark & $52.3 \pm 0.6$ & $39.5 \pm 0.1$ & $27.4 \pm 0.6$ & $17.6 \pm 0.1$ & $68.5 \pm 0.3$ & $61.9 \pm 0.1$ & $48.5 \pm 0.4$ & $43.7 \pm 0.3$ & $75.2 \pm 0.1$ & $72.4 \pm 0.1$ & $61.7 \pm 0.3$ & $61.7 \pm 0.3$ \\
|
||||
& & \grntxt{$+3.7$} & \grntxt{$+4.4$} & \grntxt{$+4.4$} & \grntxt{$+4.6$} & \grntxt{$+2.8$} & \grntxt{$+3.8$} & \grntxt{$+4.2$} & \grntxt{$+5.5$} & \grntxt{$+2.0$} & \grntxt{$+2.5$} & \grntxt{$+4.8$} & \grntxt{$+4.8$} \\
|
||||
\cmidrule(r){1-2}
|
||||
ResNet101 & \xmark & $47.8 \pm 0.7$ & $37.2 \pm 0.5$ & $20.4 \pm 1.2$ & $14.2 \pm 0.3$ & $64.9 \pm 0.2$ & $58.6 \pm 0.5$ & $41.1 \pm 0.5$ & $38.3 \pm 0.7$ & $73.6 \pm 0.3$ & $70.5 \pm 0.3$ & $56.2 \pm 0.4$ & $57.0 \pm 0.5$ \\
|
||||
ResNet101 & \cmark & $52.3 \pm 0.1$ & $42.2 \pm 0.1$ & $24.7 \pm 0.1$ & $19.2 \pm 0.4$ & $68.8 \pm 0.6$ & $62.9 \pm 0.3$ & $46.4 \pm 1.5$ & $44.3 \pm 0.9$ & $76.0 \pm 0.4$ & $73.7 \pm 0.3$ & $61.0 \pm 1.2$ & $62.6 \pm 0.5$ \\
|
||||
& & \grntxt{$+4.4$} & \grntxt{$+5.0$} & \grntxt{$+4.3$} & \grntxt{$+5.0$} & \grntxt{$+3.9$} & \grntxt{$+4.3$} & \grntxt{$+5.3$} & \grntxt{$+6.0$} & \grntxt{$+2.4$} & \grntxt{$+3.2$} & \grntxt{$+4.7$} & \grntxt{$+5.7$} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table}
|
||||
|
||||
\Cref{tab:corner-cases} reports accuracy on the corner-cases dataset~\cite{Fatima2025} for models trained with and without \schemename.
|
||||
The dataset is constructed by pasting objects cropped by their full bounding boxes (which are available for the ImageNet validation set) onto 224$\times$224 infilled backgrounds.
|
||||
The dataset has three factors: foreground size (56, 84, 112 pixels), spatial position (center, CeX, vs.\ corner, CoX), and background type (original image background, XxO, vs.\ a random background, XxR), yielding $3 \times 2 \times 2$ controlled configurations per model.
|
||||
|
||||
Across all architectures, training with \schemename consistently improves robustness to these composition shifts.
|
||||
For ViT-S/B/L, gains range from roughly $+8$ to over $+27$ percentage points, with the largest improvements occurring in the most challenging settings with foregrounds placed in corners on random backgrounds (e.g., CoR and CeR).
|
||||
Swin and ResNet models also benefit across all configurations, with increases typically between $+3$ and $+10$ points.
|
||||
DeiT-S shows small drops on some same-background center cases (CeO/CoO), but still improves notably on random-background conditions (XxR), while DeiT-B/L gain across nearly all settings.
|
||||
|
||||
Three trends are apparent.
|
||||
First, all baselines perform substantially worse when moving from original to random backgrounds and from centered to corner placements, indicating strong background and center biases.
|
||||
Second, \schemename reduces this sensitivity: the absolute gap between center and corner, and between original and random backgrounds, shrinks for almost all models and sizes.
|
||||
Third, the relative improvements are especially pronounced for smaller objects and off-center placements, suggesting that \schemename makes models more foreground-focused and less reliant on canonical object scale and position.
|
||||
|
||||
|
||||
\section{\schemename Segmentation Samples}
|
||||
\begin{figure}[t!]
|
||||
\centering
|
||||
\begin{subfigure}{.49\textwidth}
|
||||
|
||||
\includegraphics[width=\textwidth]{img/masked_image_examples_train.pdf}
|
||||
\end{subfigure}
|
||||
\hfill
|
||||
\begin{subfigure}{.49\textwidth}
|
||||
|
||||
\includegraphics[width=\textwidth]{img/masked_image_examples.pdf}
|
||||
\end{subfigure}
|
||||
\caption{ImageNet validation samples (left) and training samples (right) of our segmentation masks with annotated bounding boxes.}
|
||||
\label{fig:mask-examples}
|
||||
\end{figure}
|
||||
We show examples of the automatically generated segmentation masks for a diverse subset of object categories (``ant,'' ``busby,'' ``bell cote,'' ``pickelhaube,'' ``snorkel,'' ``stove,'' ``tennis ``ball,'' and ``volleyball'').
|
||||
Note that ``busby,'' ``bell cote,'' ``pickelhaube,'' and ``snorkel'' are the four classes with the \textbf{worst} mean box precision and box-to-box IoU on the validation set.
|
||||
\Cref{fig:mask-examples} (right) illustrates masks from the evaluation split, while \Cref{fig:mask-examples} (left) shows examples from the training split.
|
||||
Across both sets, the masks accurately isolate foreground objects with clean boundaries, despite large variations in object scale, shape, and appearance, supporting their use for background removal and resampling in our training pipeline.
|
||||
We find that the main failure cases are:
|
||||
(\textit{i}) When the ground-truth annotation corresponds to only a part of an object, the predicted mask often expands to cover the entire object rather than the annotated region.
|
||||
See for example ``busby'' or ``bell cote''.
|
||||
(\textit{ii}) In images containing multiple instances, some objects may be missed, resulting in incomplete foreground coverage.
|
||||
This is especially visible for ``busby'' and ``pickelhaube''.
|
||||
However, note that especially for ``pickelhaube'' the training distribution is noticeably different from the validation distribution, showing many images with just the head instead of groups of people wearing it.
|
||||
(\textit{iii}) In rare cases, the predicted mask degenerates and covers nearly the entire image, effectively eliminating the background.
|
||||
This happens in $<10\%$ of all training images, and we do not use the resulting backgrounds for recombination (see \Cref{apdx:infill-ratio}).
|
||||
|
||||
\section{\schemename Sample Images}
|
||||
\begin{table*}[t!]
|
||||
\centering
|
||||
\caption{Sample Images from using \schemename on ImageNet.}
|
||||
\label{tbl:example-images}
|
||||
\resizebox{.93\textwidth}{!}{
|
||||
\begin{tabular}{ccccc}
|
||||
\toprule
|
||||
Class & \makecell{Original \\Image} & \makecell{Extracted \\Foreground} & \makecell{Infilled \\Background} & \schemename's Recombinations \\
|
||||
\midrule
|
||||
\makecell{n01531178 \\Goldfinch} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_v0_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_v0_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v11.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v13.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v14.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01531178_4963_recombined_v26.JPEG} \\
|
||||
\makecell{n01818515 \\Macaw} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_v1_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_v1_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v0.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v10.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v12.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v16.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_recombined_v28.JPEG} \\
|
||||
\makecell{n01943899 \\Conch} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v0.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v1.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v10.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v27.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_recombined_v15.JPEG} \\
|
||||
\makecell{n01986214 \\ Hermit Crab} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v12.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v21.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v9.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01986214_4117_recombined_v8.JPEG} \\
|
||||
\makecell{n02190166 \\Fly} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v1.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v23.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v7.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02190166_1208_recombined_v9.JPEG} \\
|
||||
\makecell{n02229544 \\Cricket} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v1.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v17.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v19.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v25.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_recombined_v5.JPEG} \\
|
||||
\makecell{n02443484 \\Black-Footed \\Ferret} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v16.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v24.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v27.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v3.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02443484_5430_recombined_v4.JPEG} \\
|
||||
\makecell{n03201208 \\Dining Table} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v0.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v11.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v15.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v19.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03201208_21000_recombined_v21.JPEG} \\
|
||||
\makecell{n03424325 \\Gasmask} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v10.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v11.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v12.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v13.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v15.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03424325_21435_recombined_v26.JPEG} \\
|
||||
\makecell{n03642806 \\Laptop} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v11.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v12.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v15.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v17.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v25.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_recombined_v29.JPEG} \\
|
||||
\makecell{n04141975 \\Scale} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v10.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v13.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v14.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v23.JPEG}\includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n04141975_11426_recombined_v25.JPEG} \\
|
||||
\makecell{n07714990 \\Broccoli} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v1.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v13.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v15.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v17.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v27.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07714990_7596_recombined_v29.JPEG} \\
|
||||
\makecell{n07749582 \\Lemon} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v1.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v15.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v17.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v20.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v24.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n07749582_17601_recombined_v26.JPEG} \\
|
||||
\makecell{n09332890 \\Lakeside} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_bg.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v0.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v12.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v13.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v14.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v18.JPEG} \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n09332890_27898_recombined_v20.JPEG} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table*}
|
||||
We show some example images of \schemename's recombinations for 14 random classes of ImageNet \cite{Deng2009} in \Cref{tbl:example-images}.
|
||||
% \schemename visibly varies the background, size, and position of the objects.
|
||||
The recombined samples display substantial visual diversity, with each extracted foreground appearing in multiple, clearly different background contexts.
|
||||
Foreground objects remain sharp and well‑preserved across recombinations, while backgrounds vary in texture, color, and scene type
|
||||
Images show a broad range of spatial placements and scales for the same object, resulting in noticeably different overall layouts.
|
||||
|
||||
|
||||
\FloatBarrier
|
||||
\section{Infill Model Comparison}
|
||||
\begin{table*}[h!]
|
||||
\centering
|
||||
\caption{Example infills of LaMa and Attentive Eraser.}
|
||||
\label{tab:infill-examples}
|
||||
\resizebox{.9\textwidth}{!}{
|
||||
\begin{tabular}{cc@{\hskip 0.3in}cc}
|
||||
\toprule
|
||||
LaMa & Att. Eraser & LaMa & Att. Eraser \\
|
||||
\midrule
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00000090.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00000090.JPEG} &
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00000890.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00000890.JPEG} \\
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00002106.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00002106.JPEG} &
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00005045.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00005045.JPEG} \\
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00007437.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00007437.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00008542.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00008542.JPEG} \\
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00009674.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00009674.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00002743.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00002743.JPEG} \\
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00003097.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00003097.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00011629.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00011629.JPEG} \\
|
||||
\includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00000547.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00000547.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/lama_infills/comp/ILSVRC2012_val_00025256.JPEG} & \includegraphics[width=.23\columnwidth, valign=c]{img/att_err_infills/comp/ILSVRC2012_val_00025256.JPEG} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table*}
|
||||
We visualize example infilled images for both LaMa \cite{Suvorov2022} and Attentive Eraser \cite{Sun2025} in \Cref{tab:infill-examples}.
|
||||
The side‑by‑side examples show that both methods generally produce visually consistent infills, with many pairs appearing extremely similar at a glance.
|
||||
We qualitatively find that Attentive Eraser yields slightly sharper textures or more coherent local structure, while LaMa sometimes produces smoother or more homogenized regions.
|
||||
Across the table, fine‑detail areas such as foliage, bark, and ground textures reveal the most noticeable differences between the two methods.
|
||||
% We qualitatively find that while LaMa often leaves repeated textures of blurry spots where the object was erased, Attentive Eraser produces slightly cleaner and more coherent infills of the background.
|
||||
|
||||
\FloatBarrier
|
||||
\newpage
|
||||
\section{Image Infill Ratio}
|
||||
\label{apdx:infill-ratio}
|
||||
\begin{table*}[h!]
|
||||
\centering
|
||||
\caption{Example infills with a large relative foreground area size that is infilled (infill ratio).}
|
||||
\label{tbl:high-rat}
|
||||
\resizebox{.8\textwidth}{!}{
|
||||
\begin{tabular}{ccc}
|
||||
\toprule
|
||||
Infill Ratio & LaMa & Att. Eraser \\
|
||||
\midrule
|
||||
83.7 & \raisebox{-50pt}{\includegraphics[width=.3\columnwidth]{img/lama_infills/high_rat/ILSVRC2012_val_00022522.JPEG}} & \raisebox{-50pt}{\includegraphics[width=.3\columnwidth]{img/att_err_infills/high_rat/ILSVRC2012_val_00022522.JPEG}} \\ \\
|
||||
88.2 & \raisebox{-50pt}{\includegraphics[width=.3\columnwidth]{img/lama_infills/high_rat/ILSVRC2012_val_00026530.JPEG}} & \raisebox{-50pt}{\includegraphics[width=.3\columnwidth]{img/att_err_infills/high_rat/ILSVRC2012_val_00026530.JPEG}} \\ \\
|
||||
93.7 & \raisebox{-60pt}{\includegraphics[width=.3\columnwidth]{img/lama_infills/high_rat/ILSVRC2012_val_00003735.JPEG}} & \raisebox{-60pt}{\includegraphics[width=.3\columnwidth]{img/att_err_infills/high_rat/ILSVRC2012_val_00003735.JPEG}} \\ \\
|
||||
95.7 & \raisebox{-60pt}{\includegraphics[width=.3\columnwidth]{img/lama_infills/high_rat/ILSVRC2012_val_00012151.JPEG}} & \raisebox{-60pt}{\includegraphics[width=.3\columnwidth]{img/att_err_infills/high_rat/ILSVRC2012_val_00012151.JPEG}}
|
||||
\end{tabular}}
|
||||
\end{table*}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=.9\textwidth]{img/infill_distr.pdf}
|
||||
\caption{We plot the distribution of the relative size of the detected foreground object that is infilled in our Segmentation step of ImageNet.
|
||||
While most images contain objects of smaller size, there is a peak where Grounded~SAM~\cite{Ren2024} detects almost the whole image as the foreground object. For examples of such large infills, see \Cref{tbl:high-rat}.
|
||||
}
|
||||
\label{fig:infill-distr}
|
||||
\end{figure}
|
||||
|
||||
\Cref{tbl:high-rat} shows infills for images where Grounded SAM \cite{Ren2024} marks a high percentile of the image as the foreground object (Infill Ratio), that has to be erased by the infill models.
|
||||
The examples show that when the infilled region becomes large, both methods begin to lose coherent global structure, with outputs dominated by repetitive or texture‑like patterns.
|
||||
LaMa tends to produce smoother, more uniform surfaces, like we saw in \Cref{tab:infill-examples}, while Attentive Eraser often generates denser, more regular texture patterns.
|
||||
Across the rows, increasing infill ratio corresponds to increasingly homogeneous results, with only faint hints of original scene cues remaining.
|
||||
% While LaMa tends to fill those spots with mostly black or gray and textures similar to what we saw in \Cref{tab:infill-examples}, Attentive Eraser tends to create novel patterns by copying what is left of the background all over the rest of the image.
|
||||
% We filter out such mostly infilled background using our background pruning hyperparameter $t_\text{prune} = 0.8$.
|
||||
\Cref{fig:infill-distr} plots the distribution of infill ratios in \schemename.
|
||||
While there is a smooth curve of the number of detections decreasing with the infill ratio until $\approx 90\%$, there is an additional peak at $\approx 100\%$ infill ratio.
|
||||
We hypothesize that this peak is made up of failure cases of Grounded~SAM.
|
||||
|
||||
We filter out all backgrounds that have an infill ratio larger than our pruning threshold $t_\text{prune} = 0.8$, which translates to $10\%$ of backgrounds.
|
||||
|
||||
|
||||
|
||||
21
sec/conclusion.tex
Normal file
21
sec/conclusion.tex
Normal file
@@ -0,0 +1,21 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\section{Conclusion \& Future Work}
|
||||
\label{sec:conclusion}
|
||||
|
||||
% We introduce \schemename, a novel data augmentation scheme that facilitates improved Transformer training for image classification.
|
||||
% By explicitly separating and recombining foreground objects and backgrounds, \schemename enables controlled data augmentation beyond existing image compositions, leading to significant performance gains on ImageNet and downstream fine-grained classification tasks.
|
||||
% Furthermore, \schemename provides a powerful framework for analyzing model behavior and quantifying biases, including background robustness, foreground focus, center bias, and size bias.
|
||||
% Our experiments demonstrate that training using \schemename not only boosts accuracy but also significantly reduces these biases, resulting in more robust and generalizable models.
|
||||
% In the future, we see \schemename be also applied to other datasets and tasks, like video recognition or segmentation.
|
||||
% \schemename's ability to both improve performance and provide insights into model behavior makes it a valuable tool for advancing CV research and developing more reliable AI systems.
|
||||
|
||||
|
||||
We introduced \schemename, a controlled composition augmentation scheme that factorizes images into foreground objects and backgrounds and recombines them with explicit control over background identity, object position, and object scale.
|
||||
% Empirically, \schemename consistently improves clean accuracy and robustness across architectures and scales.
|
||||
Across diverse architectures, training with \schemename on top of standard strong augmentations yields substantial gains on ImageNet (up to $+6$ p.p.) and fine-grained downstream tasks (up to $+7.3$ p.p.), and consistently improves robustness on well-recognized benchmarks (up to $+19$ p.p.).
|
||||
\schemename's compositional controls additionally provide a framework for analyzing model behavior and quantify biases, including background robustness, foreground focus, center bias, and size bias.
|
||||
This dual role of \schemename as both a training mechanism and an evaluation tool highlights the value of explicit compositional factorization in understanding and improving image classifiers.
|
||||
In future work, we aim to extend controlled composition beyond classification to multi-object and dense prediction settings, including detection, segmentation, and video recognition.
|
||||
% By coupling performance gains with interpretable, controllable evaluations, \schemename offers a practical data-centric tool for advancing robust and reliable computer vision systems.
|
||||
More generally, we believe that designing augmentations around explicitly controllable and interpretable generative setups is a promising direction for building robust and reliable vision systems.
|
||||
525
sec/experiments.tex
Normal file
525
sec/experiments.tex
Normal file
@@ -0,0 +1,525 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
|
||||
\begin{figure}[t]
|
||||
\begin{minipage}[t]{.62\textwidth}
|
||||
\captionof{table}{ImageNet results when training ViTs with different data augmentation pipelines.
|
||||
\schemename consistently improves performance in low- and mid-augmentation regimes and remains complementary to strong augmentation pipelines, with larger gains for larger models.
|
||||
}
|
||||
\label{tab:imagenet-pipelines}
|
||||
\centering
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{Augmentation} & \multirow{2.5}{*}{MixUp} & \multirow{2.5}{*}{CutMix} & \multicolumn{3}{c}{Accuracy [\%] using} \\
|
||||
\cmidrule(l){4-6}
|
||||
& & & ViT-S & ViT-B & ViT-L \\
|
||||
\midrule
|
||||
Basic & \xmark & \xmark & $71.9 \pm 0.1$ & $69.5 \pm 0.2$ & $68.3 \pm 0.4$ \\
|
||||
Basic + \schemename & \xmark & \xmark & $75.7 \pm 0.2$ & $75.5 \pm 0.6$ & $73.1 \pm 1.7$ \\
|
||||
& & & \grntxt{$+3.8$} & \grntxt{$+6.0$} & \grntxt{$+4.8$} \\
|
||||
\midrule
|
||||
RandAugment & \xmark & \xmark & $76.3 \pm 0.5$ & $75.5 \pm 0.2$ & $74.7 \pm 0.4$ \\
|
||||
RandAugment + \schemename & \xmark & \xmark & $78.0 \pm 0.1$ & $77.8 \pm 0.1$ & $78.0 \pm 0.6$ \\
|
||||
& & & \grntxt{$+1.7$} & \grntxt{$+2.3$} & \grntxt{$+3.3$} \\
|
||||
\midrule
|
||||
Basic & \cmark & \cmark & $79.8 \pm 0.3$ & $78.6 \pm 0.4$ & $78.1 \pm 1.6$ \\
|
||||
Basic + \schemename & \cmark & \cmark & $79.8 \pm 0.3$ & $81.6 \pm 0.5$ & $81.0 \pm 0.4$ \\
|
||||
& & & \gtxt{$\pm 0.0$} & \grntxt{$+3.0$} & \grntxt{$+2.9$} \\
|
||||
\midrule
|
||||
3-Augment & \xmark & \cmark & $79.1 \pm 0.1$ & $77.6 \pm 0.2$ & $75.3 \pm 0.4$ \\
|
||||
3-Augment + \schemename & \xmark & \cmark & $81.4 \pm 0.1$ & $81.1 \pm 0.4$ & $79.8 \pm 0.1$ \\
|
||||
& & & \grntxt{$+2.3$} & \grntxt{$+3.5$} & \grntxt{$+4.5$} \\
|
||||
\midrule
|
||||
RandAugment & \cmark & \cmark & $80.1 \pm 0.1$ & $81.9 \pm 0.3$ & $79.3 \pm 2.3$ \\
|
||||
RandAugment + \schemename & \cmark & \cmark & $80.0 \pm 0.3$ & $81.9 \pm 0.2$ & $82.4 \pm 0.1$ \\
|
||||
& & & \gtxt{$-0.1$} & \gtxt{$\pm 0.0$} & \grntxt{$+3.1$} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{minipage}
|
||||
\hfill
|
||||
\begin{minipage}[t]{.37\textwidth}
|
||||
\captionof{table}{ImageNet results of models trained on ImageNet with and without \schemename. \schemename improves the performance of most models, with a larger gain for larger models.}
|
||||
\label{tab:imagenet-results}
|
||||
\resizebox{\textwidth}{!}{\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{Accuracy [\%]}} & \multirow{2.5}{*}{Delta} \\
|
||||
\cmidrule(lr){2-3}
|
||||
& w/o \schemename & w/ \schemename & \\
|
||||
\midrule
|
||||
ViT-S & $79.1\pm0.1$ & $81.4\pm0.1$ & \grntxt{$+2.3$} \\
|
||||
ViT-B & $77.6\pm0.2$ & $81.1\pm0.4$ & \grntxt{$+3.5$} \\
|
||||
ViT-L & $75.3\pm0.4$ & $79.8\pm0.1$ & \grntxt{$+4.5$} \\
|
||||
\midrule
|
||||
DeiT-S & $80.1 \pm 0.1$ & $80.0\pm0.3$ & \gtxt{$-0.1$} \\
|
||||
DeiT-B & $81.9 \pm 0.3$ & $81.9\pm0.2$ & \gtxt{$\pm0.0$} \\
|
||||
DeiT-L & $79.3\pm2.3$ & $82.4\pm0.1$ & \grntxt{$+3.1$} \\
|
||||
\midrule
|
||||
Swin-Ti & $77.9\pm0.2$ & $79.7\pm0.1$ & \grntxt{$+1.8$} \\
|
||||
Swin-S & $79.4\pm0.1$ & $80.6\pm0.1$ & \grntxt{$+1.2$} \\
|
||||
\midrule
|
||||
ResNet-50 & $78.3\pm0.1$ & $78.8\pm0.1$ & \grntxt{$+0.5$} \\
|
||||
ResNet-101 & $79.4\pm0.1$ & $80.4\pm0.1$ & \grntxt{$+1.0$} \\
|
||||
\bottomrule
|
||||
\end{tabular}}
|
||||
\end{minipage}
|
||||
\end{figure}
|
||||
|
||||
% \begin{table}[t]
|
||||
% \caption{ImageNet results of models trained on ImageNet with and without \schemename. \schemename improves the performance of most models, with a larger gain for larger models.}
|
||||
% \label{tab:imagenet-results}
|
||||
% \centering
|
||||
% \begin{subfigure}{.41\textwidth}
|
||||
% \resizebox{\textwidth}{!}{\begin{tabular}{lccc}
|
||||
% \toprule
|
||||
% \multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{ImageNet Accuracy [\%]}} & \multirow{2.5}{*}{Delta} \\
|
||||
% \cmidrule(lr){2-3}
|
||||
% & w/o \schemename & w/ \schemename & \\
|
||||
% \midrule
|
||||
% ViT-S & $79.1\pm0.1$ & $81.4\pm0.1$ & \grntxt{$+2.3$} \\
|
||||
% ViT-B & $77.6\pm0.2$ & $81.1\pm0.4$ & \grntxt{$+3.5$} \\
|
||||
% ViT-L & $75.3\pm0.4$ & $79.8\pm0.1$ & \grntxt{$+4.5$} \\
|
||||
% \midrule
|
||||
% Swin-Ti & $77.9\pm0.2$ & $79.7\pm0.1$ & \grntxt{$+1.8$} \\
|
||||
% Swin-S & $79.4\pm0.1$ & $80.6\pm0.1$ & \grntxt{$+1.2$} \\
|
||||
% \bottomrule
|
||||
% \end{tabular}}
|
||||
% \end{subfigure}
|
||||
% \hspace{5pt}
|
||||
% \begin{subfigure}{.448\textwidth}
|
||||
% \resizebox{\textwidth}{!}{\begin{tabular}{lccc}
|
||||
% \toprule
|
||||
% \multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{ImageNet Accuracy [\%]}} & \multirow{2.5}{*}{Delta} \\
|
||||
% \cmidrule(lr){2-3}
|
||||
% & w/o \schemename & w/ \schemename & \\
|
||||
% \midrule
|
||||
% DeiT-S & $80.1 \pm 0.1$ & $80.0\pm0.3$ & \gtxt{$-0.1$} \\
|
||||
% DeiT-B & $81.9 \pm 0.3$ & $81.9\pm0.2$ & \gtxt{$\pm0.0$} \\
|
||||
% DeiT-L & $79.3\pm2.3$ & $82.4\pm0.1$ & \grntxt{$+3.1$} \\
|
||||
% \midrule
|
||||
% ResNet-50 & $78.3\pm0.1$ & $78.8\pm0.1$ & \grntxt{$+0.5$} \\
|
||||
% ResNet-101 & $79.4\pm0.1$ & $80.4\pm0.1$ & \grntxt{$+1.0$} \\
|
||||
% \bottomrule
|
||||
% \end{tabular}}
|
||||
% \end{subfigure}
|
||||
% \end{table}
|
||||
|
||||
\section{Experiments}
|
||||
\label{sec:experiments}
|
||||
|
||||
We conduct a comprehensive suit of experiments to validate the effectiveness of our approach,
|
||||
comparing ImageNet training with and without \schemename for 10 different models and 5 data augmentation pipelines.
|
||||
Furthermore, we assess the impact of using \schemename for pretraining on multiple fine-grained downstream datasets.
|
||||
Finally, we exploit \schemename's control over the image distribution to quantify model behaviors and biases.
|
||||
We always report the mean and standard deviation of three independent training runs.
|
||||
|
||||
\subsection{Image Classification Results}
|
||||
|
||||
\textbf{ImageNet training.}
|
||||
\Cref{tab:imagenet-pipelines} analyzes the effect of \schemename under different data augmentation pipelines:
|
||||
A \emph{basic} pipeline with RandomResizedCrop, Flip and ColorJitter, the \emph{3-Augment} pipeline from \cite{Touvron2022,Nauen2025} that also includes Grayscale, Solarization and GaussianBlur, as well as the widely used \emph{RandAugment}~\cite{Cubuk2020} based pipeline from DeiT~\cite{Touvron2021b}.
|
||||
Additionally, we include MixUp~\cite{Zhang2018a} and CutMix~\cite{Yun2019} augmentations.
|
||||
% We also include Mixup and CutMix.
|
||||
We find that the effectiveness of \schemename depends on the interplay between model capacity and baseline augmentation strength.
|
||||
When the baseline augmentation is weak or moderate, \schemename consistently improves ImageNet accuracy, with gains increasing for larger ViT models (up to $+6.0$ p.p.\ for ViT-B).
|
||||
As the augmentation pipeline becomes stronger (e.g., RandAugment with MixUp and CutMix), ImageNet improvements diminish for smaller models, indicating that the baseline augmentation already saturates their capacity.
|
||||
Importantly, even in cases where ImageNet accuracy does not improve, we consistently observe gains during downstream fine-tuning (see \Cref{tab:downstream-results}), suggesting that \schemename enhances representation quality beyond what is reflected by ImageNet accuracy.
|
||||
|
||||
\Cref{tab:imagenet-results} additionally compares performance of different model architectures.
|
||||
ViT~\cite{Dosovitskiy2021}, Swin~\cite{Liu2021} and ResNet~\cite{He2016} (representing CNNs) are trained using the ``3-augment'' strategy, while DeiT~\cite{Touvron2021b} is trained using the ``RandAugment'' strategy.
|
||||
Notably, \schemename improves performance across all tested architectures, including the ResNet models, % (up to $1$ p.p.),
|
||||
demonstrating benefits beyond Transformers.
|
||||
% We find that \schemename's improvements counteract the drop in performance for increasing model sizes.
|
||||
% Without \schemename this drop is $3.8$ p.p. (ViT-S to L), while with \schemename it is reduced to $1.6$ p.p.
|
||||
% For DeiT there is a drop of $0.8$ p.p. from small to large while when using \schemename there is a \emph{gain} of $2.4$ p.p.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Downstream accuracy in percent when finetuning on other datasets. Models are pretrained on ImageNet with and without \schemename. Pretraining using \schemename increases transformer downstream accuracy.
|
||||
% on all datasets.
|
||||
}
|
||||
\label{tab:downstream-results}
|
||||
\begin{subfigure}{.48\columnwidth}
|
||||
\resizebox{\textwidth}{!}{\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
Model & \schemename & Aircraft & Cars & Flowers & Food & Pets \\
|
||||
\midrule
|
||||
ViT-S & \xmark & $72.4\pm1.0$ & $89.8\pm0.3$ & $94.5\pm0.2$ & $89.1\pm0.1$ & $93.8\pm0.2$ \\
|
||||
ViT-S & \cmark & $78.6\pm0.5$ & $92.2\pm0.2$ & $95.5\pm0.2$ & $89.6\pm0.1$ & $94.5\pm0.2$ \\
|
||||
& & \grntxt{$+6.2$} & \grntxt{$+2.4$} & \grntxt{$+1.0$} & \grntxt{$+0.5$} & \grntxt{$+0.7$} \\
|
||||
\midrule
|
||||
ViT-B & \xmark & $71.7\pm0.5$ & $90.0\pm0.2$ & $94.8\pm0.4$ & $89.8\pm0.2$ & $94.1\pm0.4$ \\
|
||||
ViT-B & \cmark & $79.0\pm2.2$ & $93.3\pm0.1$ & $ 96.5\pm0.1$ & $90.9\pm0.1$ & $95.1\pm0.4$ \\
|
||||
& & \grntxt{$+7.3$} & \grntxt{$+3.3$} & \grntxt{$+1.7$} & \grntxt{$+1.1$} & \grntxt{$+1.0$} \\
|
||||
\midrule
|
||||
ViT-L & \xmark & $72.1\pm1.0$ & $88.8\pm0.3$ & $94.4\pm0.3$ & $90.1\pm0.2$ & $94.2\pm0.4$ \\
|
||||
ViT-L & \cmark & $77.6\pm1.2$ & $89.1\pm0.2$ & $96.6\pm0.1$ & $91.3\pm0.1$ & $95.1\pm0.1$ \\
|
||||
& & \grntxt{$+5.5$} & \grntxt{$+0.3$} & \grntxt{$+2.2$} & \grntxt{$+1.2$} & \grntxt{$+0.9$} \\
|
||||
\midrule
|
||||
Swin-Ti & \xmark & $77.0\pm0.1$ & $91.3\pm0.6$ & $95.9\pm0.1$ & $90.0\pm0.2$ & $94.2\pm0.1$ \\
|
||||
Swin-Ti & \cmark & $81.1\pm0.8$ & $92.8\pm0.4$ & $96.2\pm0.1$ & $90.4\pm0.3$ & $94.8\pm0.5$ \\
|
||||
& & \grntxt{$+4.1$} & \grntxt{$+2.5$} & \grntxt{$+0.3$} & \grntxt{$+0.4$} & \grntxt{$+0.6$} \\
|
||||
\midrule
|
||||
Swin-S & \xmark & $75.7\pm1.4$ & $91.0\pm0.3$ & $95.9\pm0.5$ & $91.1\pm0.2$ & $94.4\pm0.1$ \\
|
||||
Swin-S & \cmark & $81.4\pm0.2$ & $93.1\pm0.2$ & $96.3\pm0.3$ & $91.2\pm0.2$ & $94.9\pm0.3$ \\
|
||||
& & \grntxt{$+5.7$} & \grntxt{$+2.1$} & \grntxt{$+1.4$} & \gtxt{$+0.1$} & \grntxt{$+0.5$} \\
|
||||
\bottomrule
|
||||
\end{tabular}}
|
||||
\end{subfigure}
|
||||
\hfill
|
||||
\begin{subfigure}{.505\columnwidth}
|
||||
\resizebox{\textwidth}{!}{\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
Model & \schemename & Aircraft & Cars & Flowers & Food & Pets \\
|
||||
\midrule
|
||||
DeiT-S & \xmark & $75.3\pm0.4$ & $91.1\pm0.2$ & $94.8\pm0.4$ & $89.2\pm0.2$ & $92.4\pm0.2$ \\
|
||||
DeiT-S & \cmark & $76.8\pm0.8$ & $91.9\pm0.2$ & $95.2\pm0.3$ & $89.1\pm0.2$ & $92.3\pm0.4$ \\
|
||||
& & \grntxt{$+1.5$} & \grntxt{$+0.8$} & \grntxt{$+0.4$} & \gtxt{$-0.1$} & \gtxt{$-0.1$} \\
|
||||
\midrule
|
||||
DeiT-B & \xmark & $77.0\pm1.2$ & $92.9\pm0.2$ & $96.1\pm0.2$ & $91.2\pm0.1$ & $93.3\pm0.4$ \\
|
||||
DeiT-B & \cmark & $79.3\pm0.3$ & $93.1\pm0.1$ & $96.4\pm0.2$ & $91.3\pm0.1$ & $93.3\pm0.1$ \\
|
||||
& & \grntxt{$+2.3$} & \gtxt{$+0.2$} & \grntxt{$+0.3$} & \gtxt{$+0.1$} & \gtxt{$\pm0.0$} \\
|
||||
\midrule
|
||||
DeiT-L & \xmark & $72.8\pm5.5$ & $92.8\pm1.0$ & $95.8\pm1.5$ & $90.5\pm2.6$ & $92.4\pm2.0$ \\
|
||||
DeiT-L & \cmark & $78.8\pm0.8$ & $93.8\pm0.2$ & $97.0\pm0.2$ & $92.0\pm0.2$ & $93.5\pm0.2$ \\
|
||||
& & \grntxt{$+6.0$} & \grntxt{$+1.0$} & \grntxt{$+1.2$} & \grntxt{$+1.5$} & \grntxt{$+1.1$} \\
|
||||
\midrule
|
||||
ResNet-50 & \xmark & $78.2\pm0.5$ & $89.8\pm0.2$ & $91.7\pm0.4$ & $84.4\pm0.2$ & $93.7\pm0.3$ \\
|
||||
ResNet-50 & \cmark & $80.3\pm0.4$ & $90.4\pm0.2$ & $91.7\pm0.2$ & $84.5\pm0.2$ & $93.7\pm0.3$ \\
|
||||
& & \grntxt{$+2.1$} & \grntxt{$+0.6$} & \gtxt{$\pm0.0$} & \gtxt{$+0.1$} & \gtxt{$\pm0.0$} \\
|
||||
\midrule
|
||||
ResNet-101 & \xmark & $78.4\pm0.6$ & $90.3\pm0.1$ & $91.2\pm0.5$ & $86.0\pm0.2$ & $94.3\pm0.2$ \\
|
||||
ResNet-101 & \cmark & $81.4\pm0.5$ & $91.3\pm0.1$ & $92.9\pm0.2$ & $86.3\pm0.1$ & $94.0\pm0.3$ \\
|
||||
& & \grntxt{$+3.0$} & \grntxt{$+1.3$} & \grntxt{$+1.7$} & \grntxt{$+0.3$} & \textcolor{red}{$-0.3$} \\
|
||||
\bottomrule
|
||||
\end{tabular}}
|
||||
\end{subfigure}
|
||||
\end{table}
|
||||
|
||||
\textbf{Downstream tasks.} To assess the transferability of \schemename-trained models, we finetune models pretrained on ImageNet with and without \schemename on five fine-grained datasets:
|
||||
FGVC-Aircraft \cite{Maji2013}, Stanford Cars~\cite{Dehghan2017}, Oxford Flowers \cite{Nilsback2008}, Food-101 \cite{Kaur2017}, and Oxford-IIIT Pets \cite{Parkhi2012}.
|
||||
% While for ResNets, the performance of both training datasets is about the same,
|
||||
In \Cref{tab:downstream-results} we see transformer accuracies improve on all these datasets by up to 7.3 p.p.
|
||||
% and a reduction of error rate of up to $39.3\%$.
|
||||
% Notably, training with \name increases the downstream performance of DeiT-S and DeiT-B, even though the ImageNet results were the same.
|
||||
% This demonstrates that the improved representations from training on \name translate to superior performance beyond gains from better ImageNet performance.
|
||||
Notably, training with \schemename boosts the downstream performance of DeiT-S and DeiT-B, despite similar ImageNet accuracy.
|
||||
This shows, that the improved representations from training with \schemename translate to gains beyond better ImageNet scores.
|
||||
% not only on ImageNet, but also on fine-grained image classification tasks.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Evaluation of models trained on ImageNet with and without \schemename. \schemename generally increases models' robustness to different image distribution shifts. Note that ViT-S \emph{with} \schemename outperforms DeiT-S, the only model where \schemename does not increase robustness.}
|
||||
\label{tab:robustness-datasets}
|
||||
\begin{subfigure}{.485\textwidth}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccccccc}
|
||||
\toprule
|
||||
Model & w/ \schemename & IN-Hard & IN-A & IN-C & IN-R & IN-V2 \\
|
||||
\midrule
|
||||
ViT-S & \xmark & $18.1 \pm 0.6$ & $18.8 \pm 0.2$ & $44.7 \pm 0.8$ & $41.6 \pm 0.6$ & $67.3 \pm 0.4$ \\
|
||||
ViT-S & \cmark & $21.0 \pm 0.4$ & $26.5 \pm 0.4$ & $52.6 \pm 0.6$ & $49.8 \pm 0.3$ & $70.6 \pm 0.1$ \\
|
||||
& & \grntxt{$+2.9$} & \grntxt{$+7.7$} & \grntxt{$+7.9$} & \grntxt{$+8.1$} & \grntxt{$+3.3$} \\
|
||||
\midrule
|
||||
ViT-B & \xmark & $17.0 \pm 0.4$ & $15.8 \pm 0.7$ & $40.4 \pm 0.8$ & $38.4 \pm 0.7$ & $65.1 \pm 0.6$ \\
|
||||
ViT-B & \cmark & $22.0 \pm 0.9$ & $31.9 \pm 1.5$ & $51.6 \pm 1.8$ & $48.7 \pm 1.7$ & $70.3 \pm 0.9$ \\
|
||||
& & \grntxt{$+5.0$} & \grntxt{$+16.0$} & \grntxt{$+11.2$} & \grntxt{$+10.3$} & \grntxt{$+5.2$} \\
|
||||
\midrule
|
||||
ViT-L & \xmark & $15.6 \pm 0.4$ & $11.3 \pm 0.9$ & $38.4 \pm 1.0$ & $36.8 \pm 0.8$ & $61.6 \pm 0.8$ \\
|
||||
ViT-L & \cmark & $20.6 \pm 0.1$ & $30.4 \pm 0.5$ & $48.2 \pm 0.7$ & $46.0 \pm 0.4$ & $68.7 \pm 0.3$ \\
|
||||
& & \grntxt{$+5.0$} & \grntxt{$+19.0$} & \grntxt{$+9.8$} & \grntxt{$+9.3$} & \grntxt{$+7.1$} \\
|
||||
\midrule
|
||||
Swin-Ti & \xmark & $16.2 \pm 0.4$ & $15.0 \pm 0.3$ & $36.0 \pm 0.8$ & $36.6 \pm 0.2$ & $65.5 \pm 0.4$ \\
|
||||
Swin-Ti & \cmark & $18.3 \pm 0.3$ & $20.3 \pm 0.4$ & $41.4 \pm 0.8$ & $41.4 \pm 0.2$ & $68.2 \pm 0.4$ \\
|
||||
& & \grntxt{$+2.2$} & \grntxt{$+5.4$} & \grntxt{$+5.4$} & \grntxt{$+4.8$} & \grntxt{$+2.7$} \\
|
||||
\midrule
|
||||
Swin-S & \xmark & $18.2 \pm 0.3$ & $19.4 \pm 0.3$ & $39.0 \pm 0.7$ & $39.1 \pm 0.2$ & $67.5 \pm 0.1$ \\
|
||||
Swin-S & \cmark & $20.5 \pm 0.1$ & $27.7 \pm 0.4$ & $45.6 \pm 0.8$ & $44.1 \pm 0.3$ & $69.6 \pm 0.1$ \\
|
||||
& & \grntxt{$+2.2$} & \grntxt{$+8.4$} & \grntxt{$+6.6$} & \grntxt{$+5.0$} & \grntxt{$+2.2$} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{subfigure}
|
||||
\hfill
|
||||
\begin{subfigure}{.505\textwidth}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccccccc}
|
||||
\toprule
|
||||
Model & w/ \schemename & IN-Hard & IN-A & IN-C & IN-R & IN-V2 \\
|
||||
\midrule
|
||||
DeiT-S & \xmark & $19.5 \pm 0.2$ & $18.4 \pm 0.3$ & $58.8 \pm 0.7$ & $43.0 \pm 0.1$ & $68.8 \pm 0.2$ \\
|
||||
DeiT-S & \cmark & $18.5 \pm 0.5$ & $17.3 \pm 1.0$ & $57.0 \pm 0.9$ & $43.8 \pm 0.2$ & $68.7 \pm 0.6$ \\
|
||||
& & \rdtxt{$-1.0$} & \rdtxt{$-1.1$} & \rdtxt{$-1.8$} & \grntxt{$+0.8$} & \gtxt{$-0.1$} \\
|
||||
\midrule
|
||||
DeiT-B & \xmark & $22.6 \pm 0.2$ & $26.0 \pm 0.2$ & $62.1 \pm 1.0$ & $45.6 \pm 1.9$ & $70.6 \pm 0.9$ \\
|
||||
DeiT-B & \cmark & $22.6 \pm 0.2$ & $25.0 \pm 0.3$ & $62.8 \pm 0.6$ & $47.7 \pm 0.8$ & $70.8 \pm 0.5$ \\
|
||||
& & \gtxt{$\pm 0.0$} & \rdtxt{$-1.0$} & \grntxt{$+0.8$} & \grntxt{$+2.0$} & \gtxt{$+0.2$} \\
|
||||
\midrule
|
||||
DeiT-L & \xmark & $21.2 \pm 2.0$ & $20.2 \pm 3.4$ & $59.3 \pm 4.3$ & $41.3 \pm 2.7$ & $66.9 \pm 2.8$ \\
|
||||
DeiT-L & \cmark & $23.4 \pm 0.3$ & $28.8 \pm 2.0$ & $63.4 \pm 0.7$ & $47.8 \pm 0.6$ & $71.6 \pm 0.5$ \\
|
||||
& & \grntxt{$+2.2$} & \grntxt{$+8.7$} & \grntxt{$+4.1$} & \grntxt{$+6.5$} & \grntxt{$+4.7$} \\
|
||||
\midrule
|
||||
ResNet50 & \xmark & $16.1 \pm 0.2$ & $9.7 \pm 0.1$ & $38.0 \pm 1.0$ & $40.5 \pm 0.6$ & $66.8 \pm 0.4$ \\
|
||||
ResNet50 & \cmark & $17.2 \pm 0.1$ & $10.8 \pm 0.4$ & $41.0 \pm 0.7$ & $43.7 \pm 0.3$ & $67.5 \pm 0.1$ \\
|
||||
& & \grntxt{$+1.1$} & \grntxt{$+1.1$} & \grntxt{$+3.0$} & \grntxt{$+3.2$} & \grntxt{$+0.7$} \\
|
||||
\midrule
|
||||
ResNet101 & \xmark & $18.2 \pm 0.4$ & $14.3 \pm 0.1$ & $41.7 \pm 0.7$ & $42.3 \pm 0.1$ & $67.7 \pm 0.5$ \\
|
||||
ResNet101 & \cmark & $19.9 \pm 0.2$ & $17.6 \pm 0.5$ & $46.3 \pm 0.6$ & $46.3 \pm 0.3$ & $69.5 \pm 0.3$ \\
|
||||
& & \grntxt{$+1.7$} & \grntxt{$+3.2$} & \grntxt{$+4.6$} & \grntxt{$+4.0$} & \grntxt{$+1.8$} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{subfigure}
|
||||
\end{table}
|
||||
|
||||
\subsection{Bias and Robustness Evaluation}
|
||||
Beyond its use for training, \schemename's unique properties and controlled data generation capabilities make it a powerful tool for analyzing behavior and biases of black-box models.
|
||||
We exploit this in two complementary ways.
|
||||
First, we ask whether \schemename-trained models are more robust on \emph{external} ImageNet robustness benchmarks that are not generated by our pipeline.
|
||||
Second, we use \schemename's fine-grained control for targeted evaluation of specific dimensions of model bias, such as background reliance and center/size bias.
|
||||
% Together, these experiments allow us to both \emph{probe} and \emph{improve} robustness along clearly defined axes.
|
||||
% This combination of standard benchmarks and controlled probes allows us to both quantify robustness improvements and attribute them to changes in particular model behaviors.
|
||||
|
||||
\textbf{Robustness on External Distribution Shifts.}
|
||||
\Cref{tab:robustness-datasets} summarizes accuracy on five widely used ImageNet robustness benchmarks: ImageNet-Hard~\cite{Taesiri2023}, ImageNet-A~\cite{Hendrycks2021}, ImageNet-C~\cite{Hendrycks2019}, ImageNet-R~\cite{Hendrycks2021a}, and ImageNetV2~\cite{Recht2019}.
|
||||
Across ViTs, Swin Transformers, and ResNets, incorporating \schemename during training generally improves robustness to all considered distribution shifts.
|
||||
For ViTs, the gains are substantial: for example, ViT-B improves from $15.8\%$ to $31.9\%$ accuracy on ImageNet-A ($+16.0$ p.p.) and from $40.4\%$ to $51.6\%$ on ImageNet-C ($+11.2$ p.p.), with similar improvements for ViT-S and ViT-L.
|
||||
Swin also benefits consistently, with increases of roughly $2$--$8$ p.p. on most benchmarks, and ResNet sees smaller but steady gains (e.g., up to $+4.6$ points on ImageNet-C).
|
||||
|
||||
For DeiT, the picture is more nuanced: DeiT-B and DeiT-L still enjoy robustness improvements, whereas DeiT-S exhibits small decreases on several benchmarks.
|
||||
Interestingly, however, ViT-S trained with \schemename outperforms the DeiT-S baseline.
|
||||
This suggests that controlled composition can partially close the robustness gap between lightly and heavily regularized models.
|
||||
Overall, the consistent improvements on corruption-based, natural and hard examples indicate that the compositional invariances induced by \schemename extend beyond the specific foreground/background manipulations used in its construction.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/bg_robustness.pdf}
|
||||
\caption{Evaluation of background robustness on ImageNet + \schemename, ImageNet9~\cite{Xiao2020} and CounterAnimal~\cite{Wang2024f}.
|
||||
We plot the in-distribution (top of arrow) and the out-of-distribution (bottom of arrow) accuracy when training with and without \schemename.
|
||||
We annotate each arrow with its length $\Delta$.
|
||||
Training with \schemename improves the background robustness of all transformers by mostly boosting the out-of-distribution accuracy.
|
||||
}
|
||||
\label{fig:background-robustness}
|
||||
\end{figure*}
|
||||
|
||||
\textbf{Background Robustness.}
|
||||
% By adjusting the background distribution from using a background from an image of the same class as the foreground to using any background, we can evaluate the robustness of models to shifts in the background distribution.
|
||||
% We assess background robustness by changing the background distribution, comparing accuracy with backgrounds of the same class as the foreground to using any background.
|
||||
We assess the robustness of models to shifts in the background distribution from a class-related background to any background.
|
||||
% We define the background robustness coefficient to be the accuracy of a model on \name when using the same class background divided by the accuracy when using any background:
|
||||
% Background robustness is defined to be the ratio of accuracy on \name with same-class backgrounds to accuracy with any background:
|
||||
% \begin{align}
|
||||
% \text{Background Robustness} = \frac{\text{Acc}(\name_\text{all})}{\text{Acc}(\name_\text{same})}
|
||||
% \end{align}
|
||||
% It represents the relative drop in performance under a background distribution shift.
|
||||
\Cref{fig:background-robustness} presents the background robustness results for three datasets: ImageNet with \schemename (all backgrounds vs. backgrounds of same class), ImageNet9~\cite{Xiao2020} (random backgrounds vs. original backgrounds), and CounterAnimal~\cite{Wang2024f} (counter vs. common background).
|
||||
The top triangle of each arrow represents the in-distribution backgrounds and the bottom triangle represents the out-of-distribution ones.
|
||||
We follow ImageNet9 and CounterAnimal and assess the background robustness in terms of the accuracy gap when evaluating a model on images of normal background distribution compared to out-of-distribution backgrounds (length of each arrow; $\Delta$).
|
||||
% When trained on ImageNet, smaller models generally exhibit greater robustness to changes in the background distribution than larger models and ResNet is more robust than the tested Transformer models.
|
||||
Crucially, \schemename improves the background robustness of all models and across datasets, reducing the background-gap by boosting the performance on the out-of-background-distribution samples more than the in-distribution ones.
|
||||
We find a similar trend for the Corner-Cases~\cite{Fatima2025} dataset (see supplementary), highlighting the generalization benefits of \schemename to unusual image compositions.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/fg_focus.pdf}
|
||||
\caption{Evaluation of the foreground focus (\Cref{eq:fg-focus}) using GradCam, GradCam++ and IntegratedGradients (IG) of models trained on ImageNet. Training with \schemename improves the foreground focus of almost all models.}
|
||||
\label{fig:foreground-focus}
|
||||
\end{figure*}
|
||||
|
||||
\textbf{Foreground Focus.}
|
||||
Leveraging our inherent knowledge of the foreground masks when using \schemename, as well as common XAI techniques~\cite{Selvaraju2016,Chattopadhay2018,Sundararajan2017}, we can evaluate a model's focus on the foreground object.
|
||||
% I.e. we measure how much the model's decision depends on the foreground.
|
||||
We can directly evaluate ImageNet-trained models, but this technique can also be extended to other datasets without relying on manually annotated foreground masks.
|
||||
To evaluate the foreground focus, we employ Grad-CAM \cite{Selvaraju2016}, Grad-CAM++ \cite{Chattopadhay2018} and IntegratedGradients (IG) \cite{Sundararajan2017} to compute the per-pixel importance of an image for the model's prediction.
|
||||
The foreground focus is defined to be the ratio of the foreground's relative importance to its relative size in the image:
|
||||
\begin{align} \label{eq:fg-focus}
|
||||
\text{FG Focus}(\text{img}) = \frac{\text{Area}(\text{img}) \hspace{3pt} \text{Importance}(\text{fg})}{\text{Area}(\text{fg}) \hspace{3pt} \text{Importance}(\text{img})}
|
||||
\end{align}
|
||||
If all pixels uniformly receive the same importance value, the foreground focus is one.
|
||||
The foreground focus of a model is its average focus over all test images.
|
||||
\Cref{fig:foreground-focus} presents our findings.
|
||||
Using \schemename significantly increases the foreground focus of ViT, DeiT and ResNet across all XAI metrics.
|
||||
% I.e. \schemename-trained models base their decision more on the foreground object compared to the background than models trained without \schemename.
|
||||
% For Swin, the foreground focus stagnates when measured using GradCam and GradCam++, but almost doubles when using IG.
|
||||
% We hypothesize that Swin's below-uniform foreground focus reported with GradCam is due to its specific implementation for Swin.
|
||||
We hypothesize Swin's below-uniform foreground focus with GradCam is due to its specific implementation.
|
||||
% These differences might be due to the way GradCam is calculated for Swin \todo{cite package website where this is from} and the \todo{common critique of GradCam}.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{
|
||||
% Evaluation of the center bias.
|
||||
Accuracy relative to the center accuracy of multiple instantiations of the models when the foreground objects is in different cells of a $3 \times 3$ grid.
|
||||
We calculate center bias according to \Cref{eq:center-bias}.
|
||||
Using \schemename significantly reduces models' center bias.}
|
||||
\label{tab:center-bias}
|
||||
\begin{subfigure}{.48\columnwidth}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{Center Bias [\%] when trained}} & \multirow{2.5}{*}{Delta} \\
|
||||
\cmidrule(lr){2-3}
|
||||
& w/o \schemename & w/ \schemename \\
|
||||
\midrule
|
||||
ViT-S & \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_ImageNet_v3.pdf} & \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-S_RecombNet_all_v3.pdf} \\
|
||||
& $25.5\pm0.8$ & $22.0\pm0.3$ & \grntxt{$-3.5$} \\
|
||||
ViT-B & {\includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_ImageNet_v3.pdf}} & \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-B_RecombNet_all_v3.pdf} \\
|
||||
& $25.4\pm0.4$ & $19.0\pm0.2$ & \grntxt{$-6.4$} \\
|
||||
ViT-L & \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_ImageNet_v3.pdf} & \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ViT-L_RecombNet_all_v3.pdf} \\
|
||||
& $24.3\pm1.1$ & $11.7\pm0.7$ & \grntxt{$-12.6$} \\
|
||||
\midrule
|
||||
Swin-Ti & {\includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_ImageNet_v3.pdf}} & {\includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-Ti_RecombNet_all_v3.pdf}} \\
|
||||
& $25.0\pm0.7$ & $16.5\pm0.2$ & \grntxt{$-8.5$} \\
|
||||
Swin-S & {\includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_ImageNet_v3.pdf}} & {\includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/Swin-S_RecombNet_all_v3.pdf}} \\
|
||||
& $23.2\pm0.1$ & $15.6\pm0.2$ & \grntxt{$-7.6$} \\
|
||||
\bottomrule
|
||||
\end{tabular} }
|
||||
\end{subfigure}
|
||||
\hfill
|
||||
\begin{subfigure}{.497\columnwidth}
|
||||
\resizebox{\textwidth}{!}{
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\multirow{2.5}{*}{Model} & \multicolumn{2}{c}{\makecell{Center Bias [\%] when trained}} & \multirow{2.5}{*}{Delta} \\
|
||||
\cmidrule(lr){2-3}
|
||||
& w/o \schemename & w/ \schemename \\
|
||||
\midrule
|
||||
DeiT-S & {\includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_ImageNet_vNone.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_ImageNet_v3.pdf} } & {\includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_fornet_all_linear_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_fornet_all_linear_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-S_fornet_all_linear_v3.pdf}} \\
|
||||
& $20.4 \pm 0.2$ & $21.2 \pm 0.1$ & \gtxt{$+0.8$} \\
|
||||
DeiT-B & {\includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_ImageNet_vNone.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_ImageNet_v3.pdf} } & {\includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_fornet_all_cos_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_fornet_all_cos_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-B_fornet_all_cos_v3.pdf}} \\
|
||||
& $19.0 \pm 0.7$ & $19.0 \pm 0.2$ & \gtxt{$\pm0.0$} \\
|
||||
DeiT-L & { \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_ImageNet_v3.pdf} } & { \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_fornet_all_cos_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_fornet_all_cos_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/DeiT-L_fornet_all_cos_v3.pdf} } \\
|
||||
& $21.2 \pm 0.2$ & $18.0 \pm 0.2$ & \grntxt{$-3.2$} \\
|
||||
\midrule
|
||||
ResNet50 & {\includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_ImageNet_v3.pdf}} & {\includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet50_RecombNet_all_v3.pdf}} \\
|
||||
& $26.3\pm0.3$ & $19.7\pm0.3$ & \grntxt{$-6.6$} \\
|
||||
ResNet101 & {\includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_ImageNet_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_ImageNet_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_ImageNet_v3.pdf}} & {\includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_RecombNet_all_v1.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_RecombNet_all_v2.pdf} \includegraphics[width=.08\columnwidth, valign=c]{img/ResNet101_RecombNet_all_v3.pdf}} \\
|
||||
& $23.0\pm0.3$ & $19.9\pm0.2$ & \grntxt{$-3.1$} \\
|
||||
\bottomrule
|
||||
\end{tabular} }
|
||||
\end{subfigure}
|
||||
\centering
|
||||
\includegraphics[width=.5\columnwidth]{img/colorbar_horizontal.pdf}
|
||||
\end{table}
|
||||
|
||||
\textbf{Center Bias.}
|
||||
With \schemename we have unique control over the position of the foreground object in the image.
|
||||
This lets us quantify the center bias of models trained with and without \schemename.
|
||||
We divide the image into a $3 \times 3$ grid and evaluate model accuracy when the (scaled-down) foreground object is in each of the $9$ grid cells.
|
||||
Each cell's accuracy is divided by the accuracy in the center cell for normalization, which gives us the relative performance drop when the foreground is in each part of the image.
|
||||
The center bias is calculated as one minus the average of the minimum performance of a corner cell and the minimum performance of a side cell:
|
||||
% \begin{align}
|
||||
% \begin{split}
|
||||
% & \text{Center Bias} = \\
|
||||
% & \hspace{7pt} 1 - \frac{\min\limits_{a, b \in \{0, 2\}} \text{Acc}(\text{cell}_{(a, b)}) + \min\limits_{\substack{a=1 \text{ or } b=1 \\ a \neq b}} \text{Acc}(\text{cell}_{(a, b)})}{2 \text{Acc}(\text{cell}_{(1, 1)})}
|
||||
% \end{split}
|
||||
% \end{align}
|
||||
\begin{align} \label{eq:center-bias}
|
||||
\text{Center Bias} = 1 - \frac{\min\limits_{c \in \text{sides}} \text{Acc}(c) + \min\limits_{c \in \text{corners}} \text{Acc}(c)}{2 \text{Acc}(c_\text{center})}
|
||||
\end{align}
|
||||
\Cref{tab:center-bias} visualizes the center bias of three instantiations of each model.
|
||||
Performance is generally highest in the center and lowest in the four corners.
|
||||
Interestingly, ImageNet-trained models perform slightly better when the foreground object is on the right side of the image, compared to the left side, despite our use of random flipping with a probability of $0.5$ during training.
|
||||
% Training on \name reduces the center bias of all models by at least half.
|
||||
Using \schemename significantly reduces center bias across models, with a more uniform performance especially across the middle row.
|
||||
% On corner-cases (see supplementary) we find that
|
||||
% Their accuracy is higher in the center left and right cells than in the center top and bottom ones, which is not the case for ImageNet-trained models.
|
||||
% This demonstrates that \schemename promotes a more uniform spatial attention distribution, counteracting the center-bias of ImageNet.
|
||||
Thus, \schemename makes the model recognize objects across a wider spatial distribution, counteracting the center-bias of ImageNet.
|
||||
|
||||
\begin{figure}[t!]
|
||||
\centering
|
||||
\includegraphics[width=\columnwidth]{img/size_bias_wide.pdf}
|
||||
\caption{Evaluation of the size bias of models trained on ImageNet. We plot the accuracy relative to the accuracy when using the default size ($f_\text{size} = 1.0$).}
|
||||
\label{fig:size-bias}
|
||||
\end{figure}
|
||||
|
||||
\textbf{Size Bias.}
|
||||
Finally, we evaluate the impact of different sized foreground objects on the accuracy.
|
||||
For this evaluation, we use the \emph{mean} foreground size strategy.
|
||||
We introduce a size factor $f_\text{size}$ by which we additionally scale the foreground object before pasting it onto the background.
|
||||
Results are normalized by the accuracy when using $f_\text{size} = 1.0$.
|
||||
\Cref{fig:size-bias} shows the size bias curves of models trained with and without \schemename.
|
||||
% When training on \name, the resulting model keeps it's good performance on smaller foreground objects, while models trained on ImageNet fall of faster and lower.
|
||||
Models trained using \schemename perform better, especially with smaller foreground objects.
|
||||
%, when ImageNet-trained models exhibit a more rapid performance decline.
|
||||
Therefore, \schemename-training improves robustness to variations in object scale, especially for larger models.
|
||||
|
||||
|
||||
\subsection{Design Choices of \schemename}
|
||||
We next analyze key components of \schemename, focusing on three questions: how it compares to simple copy-paste, how background choice affects performance, and how reliably labels are preserved after recomposition.
|
||||
Additional ablations over variants and hyperparameters are provided in the supplementary material.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Comparison of \schemename and simple Copy-Paste methods. We train ViT-S on ImageNet using the same 3-augment data augmentation on top of the copy-paste augmentation.}
|
||||
\label{tab:copy-paste-comparison}
|
||||
\centering
|
||||
\resizebox{.66\columnwidth}{!}{
|
||||
\begin{tabular}{lcc S[table-format=+2.1,retain-explicit-plus,detect-inline-weight=math,detect-weight=true]}
|
||||
\toprule
|
||||
Augmentation & labels & \makecell{ Accuracy [\%]} & {\makecell{Delta \\to Prev.}} \\
|
||||
\midrule
|
||||
% Baseline & & $79.1 \pm 0.1$ \\
|
||||
3-Augment + \textbf{Simple Copy-Paste} & bg & $31.3 \pm 0.6$ & \\
|
||||
+ mixed labels & fg + bg & $32.0 \pm 0.8$ & +0.7 \\
|
||||
+ fg labels & fg & $31.6 \pm 0.9$ & -0.4 \\
|
||||
+ \emph{range} foreground size variation & \gtxt{fg} & $43.0 \pm 1.2$ & \bfseries +11.4 \\
|
||||
+ infilled backgrounds & \gtxt{fg} & $68.7 \pm 0.2$ & \bfseries +25.7 \\
|
||||
+ \emph{cos} mixing strategy & \gtxt{fg} & $81.2 \pm 0.1$ & \bfseries +12.5 \\
|
||||
+ edge smoothing & \gtxt{fg} & $81.3 \pm 0.1$ & +0.1 \\
|
||||
+ background pruning$=$ \textbf{\schemename} & \gtxt{fg} & $81.4 \pm 0.1$ & +0.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}}
|
||||
\end{table}
|
||||
\textbf{Comparison to Simple Copy-Paste.}
|
||||
We compare \schemename to a simple adaption of the Copy-Paste augmentation inspired by \cite{Ge2023,Ghiasi2021,Shermaine2025} in \Cref{tab:copy-paste-comparison}.
|
||||
Contrary to semantic segmentation we do not have foreground masks available.
|
||||
Thus, we paste the extracted objects from \textbf{\schemename's segmentation stage} onto normal ImageNet images.
|
||||
% Since such images do not have straight forward classification labels, we test multiple possibilities.
|
||||
We observe 3 large jumps in accuracy: (\textbf{1}) From our \emph{range} foreground size variation (+11.4\%), (\textbf{2}) from using our infilled backgrounds instead of images from the dataset (+25.7\%), and (\textbf{3}) from our \emph{cos} mixing strategy with non-augmented images (+12.5\%).
|
||||
\schemename's changes to the naive copy-paste augmentation are thus imperative for good classification performance.
|
||||
|
||||
\begin{figure}[t]
|
||||
\begin{minipage}[c]{.49\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/strategy.pdf}
|
||||
\captionof{figure}{We compare Original, Same-class, and All-classes background selection using ViT-Ti and ViT-S backbones on TinyImageNet.
|
||||
Increasing background diversity consistently improves classification accuracy.
|
||||
}
|
||||
\label{fig:background-strategy}
|
||||
\end{minipage}
|
||||
\hfill
|
||||
\begin{minipage}[c]{.49\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/mask_expansion.pdf}
|
||||
\captionof{figure}{
|
||||
We vary the foreground mask area for TinyImageNet by shrinking or expanding masks relative to the original outline and report accuracy when training on $100\%$ augmented samples.
|
||||
Performance is stable for expanded masks and degrades rapidly after shrinking masks.
|
||||
}
|
||||
\label{fig:mask-expansion}
|
||||
\end{minipage}
|
||||
\end{figure}
|
||||
|
||||
\textbf{Background Choice Strategy.}
|
||||
\Cref{fig:background-strategy} shows the effect of background selection on TinyImageNet accuracy, where we trade off diversity against context plausibility.
|
||||
% Using the original inpainted background yields the lowest accuracy, indicating limited regularization from contextual cues.
|
||||
% Sampling backgrounds from the same class provides a modest but consistent improvement, suggesting that mild context variation encourages robustness while preserving semantic plausibility.
|
||||
The best performance is achieved by sampling backgrounds from all classes, which introduces substantial context shifts, but leads to the strongest accuracy gains for both ViT-Ti and ViT-S.
|
||||
Thus, aggressive background diversification is more important than context plausibility and acts as an effective form of context-based regularization rather than introducing harmful noise.
|
||||
|
||||
\textbf{Label Integrity.}
|
||||
% We assess the label integrity of \schemename, i.e., whether object labels remain correct after recombination, by verifying that the intended object is accurately extracted.
|
||||
% To this end, we leverage the object bounding box annotations provided in the ImageNet validation set.
|
||||
% Specifically, we compute the \emph{box precision}, defined as the fraction of the predicted mask area that lies within the ground-truth bounding box, obtaining a mean value of $91\%$.
|
||||
% In addition, we measure the \emph{box-to-box IoU}, computed as the IoU between the tight bounding box enclosing the predicted mask and the tight bounding box of the ground-truth annotation, which yields a high $76.1\%$.
|
||||
% Qualitative examples of the predicted masks and bounding boxes are provided in the supplementary material.
|
||||
% We additionally test label integrity under systematic mask perturbations by expanding or shrinking the foreground masks before composition.
|
||||
% Concretely, starting from the original outline, we erode or dilate the mask such that the foreground area changes by some percentage.
|
||||
% \Cref{fig:mask-expansion} shows that accuracy is relatively stable for expanded masks, but drops off significantly for eroded masks, consistent with cropping away semantically important object parts.
|
||||
% This experiment suggests, that \schemename is relatively robust to artifacts from including an object's original background in the foreground mask.
|
||||
% Overall, these results indicate that the segmentation stage of \schemename reliably isolates the target class object, thereby preserving label correctness after recombination.
|
||||
To quantify whether recombined images still depict the intended class, we evaluate the segmentation stage of \schemename on ImageNet validation boxes.
|
||||
Our predicted masks achieve a mean box precision of $91.0\%$ (fraction of mask area inside the ground-truth bounding boxes of the ImageNet validation set) and a high box-to-box IoU of $76.1\%$, indicating that they tightly capture the target object.
|
||||
Qualitative examples of the predicted masks and bounding boxes are provided in the supplementary material.
|
||||
We further probe robustness to mask imprecision by eroding or dilating masks such that the foreground area changes by a fixed percentage before composition.
|
||||
As shown in \Cref{fig:mask-expansion}, accuracy remains stable for expansions but drops sharply under erosion, consistent with removing semantically important object parts.
|
||||
Together, these results suggest that (\textit{i}) \schemename reliably isolates the target objects and preserves label integrity and that (\textit{ii}) \schemename is robust to artifacts from an object's original background and degrades mainly when the foreground no longer contains the full object.
|
||||
|
||||
11
sec/future_work.tex
Normal file
11
sec/future_work.tex
Normal file
@@ -0,0 +1,11 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\section{Future Work}
|
||||
\label{sec:future-work}
|
||||
|
||||
While \name demonstrates significant improvements in image classification performance and provides valuable tools for model analysis, several avenues for future research remain.
|
||||
Investigating more sophisticated segmentation methods, could improve the quality of foreground/background separation and further enhance \name's effectiveness.
|
||||
Developing strategies for adaptively selecting backgrounds and recombination parameters (size, position, etc.) based on the specific foreground object or model performance could lead to more efficient training.
|
||||
This could involve reinforcement learning or other online learning techniques.
|
||||
We also want to explore the potential of \name for other datasets and for unsupervised or self-supervised learning tasks.
|
||||
Another potential fruitful direction is exploring the use of \name, and the insights gained from the bias analyses, to develop specific training techniques or architectural modifications that explicitly mitigate identified biases
|
||||
71
sec/intro.tex
Normal file
71
sec/intro.tex
Normal file
@@ -0,0 +1,71 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
% \begin{figure}
|
||||
% \centering
|
||||
% \includegraphics[width=.5\columnwidth]{img/fig-1.pdf}
|
||||
% \caption{\schemename factorizes each training image into a foreground object and a background, then recombines them on the fly while controlling background identity, object position, and object scale. Standard, strong augmentations are applied afterwards.}
|
||||
% \label{fig:fig-1}
|
||||
% \end{figure}
|
||||
\begin{table}[t]
|
||||
\caption{Examples of \schemename generated images (center cropped) from ImageNet.
|
||||
We successfully segment even multiple objects (\textit{Macaw}) and complex shapes (\textit{Cricket}).}
|
||||
\label{tab:foraug-examples}
|
||||
\centering
|
||||
\resizebox{.9\textwidth}{!}{
|
||||
\begin{tabular}{ccccc}
|
||||
\toprule
|
||||
Class & \makecell{Original \\Image} & \makecell{Extracted \\Foreground} & \makecell{Infilled \\Background} & Recombined Examples \\
|
||||
\midrule
|
||||
Macaw & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_v1_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01818515_31507_v1_bg.JPEG} & \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v12.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v15.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v18.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v3.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v4.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01818515_31507_recombined_v6.JPEG} \\
|
||||
% Conch & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n01943899_20070_bg.JPEG} & \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v9.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v10.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v11.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v12.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v17.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n01943899_20070_recombined_v8.JPEG} \\
|
||||
Cricket & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n02229544_6170_bg.JPEG} & \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v0.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v10.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v15.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v16.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v2.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n02229544_6170_recombined_v6.JPEG} \\
|
||||
Laptop & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615.JPEG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_fg.PNG} & \includegraphics[max width=.1\columnwidth, max height=2cm, valign=c]{img/appendix_examples/n03642806_3615_bg.JPEG} & \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v0.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v1.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v11.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v14.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v15.JPEG} \includegraphics[width=.1\columnwidth, valign=c]{img/foraug_examples/n03642806_3615_recombined_v2.JPEG} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table}
|
||||
|
||||
|
||||
Large-scale image classification is a central driver of modern computer vision: it benchmarks progress in computer vision~\cite{Khan2022,Rangel2024}, powers model pretraining~\cite{Dosovitskiy2021,Liu2021,Touvron2021b}, and yields representations that transfer broadly and underpin applications like medical diagnosis~\cite{Sanderson2022,Vezakis2024}, autonomous driving~\cite{Wang2023a}, and object recognition~\cite{Carion2020,He2017,Girshick2014}.
|
||||
However, classification supervision is weak in an important sense: the label does not specify \emph{how} the class-object should appear.
|
||||
In ImageNet~\cite{Deng2009} for example, objects often occur at characteristic positions and scales and co-occur with correlated scene context~\cite{Fatima2025,Barbu2019}.
|
||||
% In datasets such as ImageNet, objects often occur at characteristic positions and scales and co-occur with correlated scene context~\cite{Fatima2025,Barbu2019}.
|
||||
As a result, models rely on shortcuts like background cues, center bias, or size bias, that boost in-distribution accuracy but hurt robustness and transfer~\cite{Geirhos2020,Fatima2025,Barbu2019}.
|
||||
|
||||
Here, data augmentation is the default defense.
|
||||
Standard transformations (crop/flip/color jitter) and stronger policies such as MixUp~\cite{Zhang2018a}/CutMix~\cite{Yun2019} and automated augmentation search~\cite{Cubuk2019,Cubuk2020} expand appearance diversity~\cite{Shorten2019,Xu2023d}. % , yet they largely preserve the original \emph{composition} of each image~\cite{Shorten2019,Xu2023d}.
|
||||
However, their ability to teach spatial and compositional invariances is limited.
|
||||
This constraint matters especially for Vision Transformers (ViTs)~\cite{Dosovitskiy2021}: with weaker built-in spatial inductive biases than Convolutional Neural Networks (CNNs), ViTs must learn key equivariances (e.g., translation and scale robustness) primarily from data.
|
||||
Copy-paste style augmentations~\cite{Ghiasi2021,Kang2022} alter composition more aggressively by overlaying segmented objects onto other images.
|
||||
These are typically designed for detection or instance segmentation and rely on dense human annotations available for these tasks or use unconstrained dataset images as backgrounds.
|
||||
As a result, they do not offer fine-grained control of object position and scale, and they do not explicitly enforce that the pasted background is semantically neutral, creating ambiguous labels for classification.
|
||||
|
||||
To encode compositional invariances directly in the training data, we propose \emph{Foreground-Background Augmentation} (\schemename), a controlled composition augmentation that \emph{explicitly factorizes each image into foreground and background, then recombines them for label-preserving, interpretable distribution shifts}.
|
||||
Concretely, \schemename uses off-the-shelf segmentation and inpainting models to (i) extract a foreground object and synthesize a class-consistent, semantically neutral background, and (ii) paste the foreground onto diverse neutral backgrounds while controlling its position and scale (see \Cref{tab:foraug-examples}).
|
||||
Unlike prior copy-paste methods that simply overlay objects onto arbitrary scenes~\cite{Ghiasi2021,Ghiasi2021,Kang2022}, \schemename first removes and neutralizes the original background, then samples from well-defined distributions of backgrounds, object positions, and object sizes.
|
||||
This explicit factorization preserves a clean label for the recombined image while providing direct control over compositions, enabling us to break spurious correlations while still fitting seamlessly into modern strong augmentation pipelines. % (see \Cref{fig:fig-1}).
|
||||
% Throughout, we apply \schemename on top of strong augmentation pipelines (RandAugment, Mixup, CutMix), so any gains are complementary to these widely used techniques.
|
||||
% As it is important that any gains are complementary to strong augmentation pipelines (RandAugment, MixUp, CutMix), we apply \schemename on top of these widely used techniques.
|
||||
To ensure that all gains are complementary to strong augmentation pipelines (RandAugment, MixUp, CutMix), we apply \schemename on top of these widely used techniques.
|
||||
|
||||
Empirically, \schemename yields consistent accuracy gains across architectures, improving ImageNet top-1 accuracy by up to 6 p.p. and fine-grained downstream accuracy by up to 7.3 p.p., and even improving transfer when ImageNet accuracy is matched.
|
||||
Beyond accuracy, training with \schemename substantially improves robustness on standard distribution-shift benchmarks, where we observe gains of roughly $2-19$ p.p. across ViT, Swin, and ResNet architectures.
|
||||
|
||||
Finally, the same control knobs enable \schemename to become a targeted diagnostic tool of shortcut reliance and model robustness.
|
||||
We quantify background reliance via controlled background swaps, and probe center and size biases through systematic position and scale sweeps, showing that training with \schemename reduces model biases.
|
||||
|
||||
\medskip
|
||||
\noindent
|
||||
\textbf{Contributions}
|
||||
\begin{itemize}[topsep=0pt]
|
||||
\item \textbf{Controlled composition augmentation for classification.}
|
||||
We introduce \schemename, a foreground-background factorization and recombination scheme for image classification that creates label-preserving training samples with explicit control over background identity, object position, and object scale.
|
||||
\item \textbf{Accuracy and transfer gains.}
|
||||
Training with \schemename, in addition to standard strong augmentation pipelines, improves ImageNet top-1 accuracy by up to 6 p.p., boosts fine-grained downstream accuracy by up to 7.3 p.p. and increases accuracy on shifted distributions by up to $19$ p.p.
|
||||
\item \textbf{Controlled bias diagnostics and mitigation.}
|
||||
Using the same controls during evaluation, we measure background reliance, foreground focus, and position/scale biases through targeted distribution shifts.
|
||||
\schemename systematically reduces shortcut behaviors and model biases.
|
||||
\end{itemize}
|
||||
73
sec/intro_old.tex
Normal file
73
sec/intro_old.tex
Normal file
@@ -0,0 +1,73 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
% \begin{itemize}
|
||||
% \item General Into Image classification
|
||||
% \item ImageNet
|
||||
% \item CNNs $\to$ Transformers
|
||||
% \item Traditional Data Augmentation: CNNs
|
||||
% \item Problems with that: Other model properties of Transformers
|
||||
% \item Our approach: Recombining ImageNet forgrounds and backgrounds
|
||||
% \end{itemize}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=.5\columnwidth]{img/fig-1.pdf}
|
||||
\caption{Comparison of traditional image classification training and training when using \schemename. \schemename recombines foreground objects with different backgrounds each epoch, thus creating a more diverse training set. We still apply strong traditional data augmentation afterwards.}
|
||||
\label{fig:fig-1}
|
||||
\end{figure}
|
||||
|
||||
Image classification, a fundamental task in computer vision (CV), involves assigning labels to images from a set of categories.
|
||||
It underpins a wide range of applications, like medical diagnosis~\cite{Sanderson2022,Vezakis2024}, autonomous driving~\cite{Wang2023a}, and object recognition~\cite{Carion2020,He2017,Girshick2014} and facilitates large-scale pretraining~\cite{Dosovitskiy2021,Liu2021,Touvron2021b}, and progress evaluation in CV~\cite{Khan2022, Rangel2024}.
|
||||
% Furthermore, image classification is used for large-scale pretraining of vision models~\cite{Dosovitskiy2021,Liu2021,Touvron2021b} and to judge the progress of the field of CV \cite{Khan2022, Rangel2024}.
|
||||
The advent of large-scale datasets, particularly ImageNet~\cite{Deng2009}, served as a catalyst for the rise of large-scale CV models~\cite{Krizhevsky2012, He2016} and remains the most important CV benchmark for more than a decade \cite{Krizhevsky2012,Touvron2022, Wortsman2022, He2016}.
|
||||
% containing millions of labeled images across thousands of categories, has been instrumental in driving significant progress in this field.
|
||||
% ImageNet served as a catalyst for the rise of large-scale CV models~\cite{Krizhevsky2012, He2016} and remains the most important CV benchmark for more than a decade \cite{Krizhevsky2012,Touvron2022, Wortsman2022, He2016}.
|
||||
% It is used to train and evaluate the best models in the field.
|
||||
While traditionally, convolutional neural networks (CNNs) have been the go-to architecture in CV, Transformers \cite{Vaswani2017}, particularly the Vision Transformer (ViT) \cite{Dosovitskiy2021}, have emerged as a powerful alternative and go-to architecture, demonstrating
|
||||
% These attention-based models have demonstrated
|
||||
superior performance in various vision tasks, including image classification \cite{Wortsman2022,Yu2022,Carion2020,Zong2023,Wang2023b}.
|
||||
|
||||
|
||||
|
||||
Data augmentation is a key technique for training image classification models.
|
||||
% A key technique for training image classification models, especially with limited data, is data augmentation.
|
||||
Traditional augmentation methods, such as cropping, flipping, or color shifts, are commonly employed to increase data diversity~\cite{Xu2023d, Shorten2019}, but remain bound to existing image compositions.
|
||||
While these preserve the images' semantic meaning, their ability to teach spatial invariances is limited.
|
||||
% the diversity of the training data and improve the model's performance~\cite{Xu2023d, Shorten2019}.
|
||||
% These basic transformations, originally designed for CNNs, change the input images in a way that preserves their semantic meaning~\cite{Alomar2023}, but are limited to existing image compositions.
|
||||
While combinations of these data augmentations are still used today, they originally were proposed to benefit CNNs.
|
||||
However, the architectural differences of CNNs and Transformers suggest that the latter might benefit from different data augmentation strategies.
|
||||
In particular, the self-attention mechanism, unlike a CNN, is not translation equivariant~\cite{RojasGomez2023,Ding2023a}, meaning that the model is not designed to understand the spatial relationships between pixels.
|
||||
% This creates the need for novel data augmentation strategies tailored to the Transformer architecture.
|
||||
% This fact opens a new design space for data augmentation strategies to help Transformers understand the basic invariances of image classification.
|
||||
% Note that these traditional data augmentations are also limited by existing image compositions.
|
||||
|
||||
Recognizing that Transformers need to learn spatial relationships directly from data,
|
||||
% and in general are usually trained on larger datasets~\cite{Kolesnikov2020},
|
||||
we propose \schemename, a data augmentation method that makes these relationships explicit by recombining foreground objects with diverse backgrounds.
|
||||
Thus, \schemename goes beyond existing image compositions and encodes desired invariances directly into the training data (see \Cref{fig:fig-1}).
|
||||
% Inspired by this inductive bias of CNNs, that is not inherent to ViTs, we propose \schemename, a novel data augmentation scheme for image classification which makes the translation equivariance of CNNs explicit in the training data by recombining foreground objects at varying positions with different backgrounds.
|
||||
% In this paper, we address the challenge of effectively training Transformers for image classification by proposing \schemename, a novel data augmentation scheme for image classification, which combines foreground objects with different backgrounds.
|
||||
% Applying \schemename to ImageNet gives rise to \name, a novel dataset that enables this data augmentation with with fine-grained control over the image composition.
|
||||
Applying \schemename to a dataset like ImageNet is a two-step process:
|
||||
(1)~We separate the foreground objects in ImageNet from their backgrounds, using an open-world object detector~\cite{Ren2024} and fill in the background in a neutral way using an object removal model~\cite{Sun2025,Suvorov2022}.
|
||||
(2)~This allows us to then recombine any foreground object with any background on the fly, creating a highly diverse training set.
|
||||
% During recombination, we can control important parameters, like the size and position of the foreground object, to help the model learn the spatial invariances necessary for image classification.
|
||||
By exploiting the control over foreground size and position during recombination, \schemename explicitly teaches spatial invariances that image classification models typically must learn implicitly.
|
||||
We show that using \schemename additionally to strong traditional data augmentation increases the model accuracy of Transformers by up to 4.5 p.p. on ImageNet and reduces the error rate by up to $7.3$ p.p. in downstream tasks.
|
||||
|
||||
Beyond training, \schemename becomes a diagnostic tool for analyzing model behavior and biases, when used during evaluation.
|
||||
We utilize our control over the image distribution to measure a model's background robustness (by varying the choice of background), foreground focus (by leveraging our knowledge about the placement of the foreground object), center bias (by controlling position), and size bias (by controlling size).
|
||||
These analyses provide valuable insights into model behavior and biases, which is crucial for model deployment and future robustness optimizations.
|
||||
We show that training using \schemename significantly reduces all of these biases.
|
||||
We make our code for \schemename and the output of \schemename's segmentation phase on ImageNet publicly available\footnote{Link will go here.} to facilitate further research.
|
||||
|
||||
\subsection*{Contributions}
|
||||
\begin{itemize}
|
||||
\item We propose \schemename, a novel data augmentation scheme, that recombines objects and backgrounds. \schemename allows us to move beyond the (possibly biased) image compositions in the dataset while preserving label integrity.
|
||||
\item We show that training a standard ViT using \schemename leads to up to 4.5 p.p. improved accuracy on ImageNet-1k and 7.3 p.p. on downstream tasks.
|
||||
\item We propose novel \schemename-based metrics to analyze and quantify fine-grained biases of trained models: Background Robustness, Foreground Focus, Center Bias, and Size Bias. We show that \schemename significantly reduces these biases by encoding invariance that benefits ViT into the training data.
|
||||
\end{itemize}
|
||||
137
sec/method.tex
Normal file
137
sec/method.tex
Normal file
@@ -0,0 +1,137 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
%\begin{figure*}[ht!]
|
||||
% \centering
|
||||
% \includegraphics[width=.9\textwidth]{img/fig-2.pdf}
|
||||
% \caption{Overview of \name. The data creation consists of two stages: (1, offline) Segmentation, where we segment the foreground objects from the background and fill in the background. (3, online) Recombination, where we combine the foreground objects with different backgrounds to create new samples. After recombination, we apply strong, commonly used augmentation policies.}
|
||||
% \label{fig:method}
|
||||
%\end{figure*}
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/fig-2.pdf}
|
||||
\caption{Overview of \schemename.
|
||||
We segment the foreground object and inpaint the removed region to obtain a neutral background (Offline, \Cref{sec:segmentation}).
|
||||
We then paste the foreground onto a sampled background while controlling position and scale, then apply standard strong traditional augmentations (Online, \Cref{sec:recombination}).}
|
||||
\label{fig:method}
|
||||
\end{figure*}
|
||||
|
||||
\section{\schemename}
|
||||
\label{sec:method}
|
||||
|
||||
% \begin{itemize}
|
||||
% \item[1.] Segment ImageNet
|
||||
% \item Detect and Cutout Foreground
|
||||
% \item Multiple foreground possibilities
|
||||
% \item Foreground mask merging
|
||||
% \item Background infills
|
||||
% \item Foreground/Background Filtering
|
||||
% \item [2.] Recombination
|
||||
% \item Which foreground \& Background
|
||||
% \item Background pruning
|
||||
% \item size
|
||||
% \item positioning
|
||||
% \item Border smoothing
|
||||
% \item Dealing with other data augmentations/transformations
|
||||
% \end{itemize}
|
||||
|
||||
% We propose a novel dataset, called \name, that improves image classification performance by explicitly separating and recombining foreground objects and plain backgrounds.
|
||||
% \name consists of two stages: Segmentation and recombination. Both are visualized in \Cref{fig:method}.
|
||||
% We introduce \schemename, a data augmentation scheme designed to enhance Transformer training by explicitly separating and recombining foreground objects and backgrounds.
|
||||
% \schemename enhances transformer training by explicitly encoding spatial invariances that these need to learn explicitly in the data.
|
||||
% \schemename involves two stages: Segmentation and Recombination, both visualized in \Cref{fig:method}.
|
||||
We introduce \schemename, a data augmentation designed to enhance training by embedding spatial invariances, which Transformers would otherwise need to learn implicitly, directly into the training data.
|
||||
% It operates by explicitly segmenting and recombining foreground objects and backgrounds.
|
||||
\schemename comprises two distinct stages: Segmentation and Recombination. Both are illustrated in \Cref{fig:method}.
|
||||
|
||||
|
||||
\subsection{Segmentation}
|
||||
\label{sec:segmentation}
|
||||
The offline segmentation stage produces reusable assets for recombination.
|
||||
% The segmentation stage isolates the foreground objects and their corresponding backgrounds.
|
||||
For each labeled training image, we create a pair $(\mathrm{fg},\mathrm{bg})$ consisting of (\textit{i}) a foreground cut-out $\mathrm{fg}$ with an alpha mask and (\textit{ii}) an inpainted background image $\mathrm{bg}$ where the foreground region has been removed.
|
||||
This stage is computed once offline and the results are stored for the recombination stage.
|
||||
|
||||
\textbf{Generate candidate foreground masks.}
|
||||
We obtain foreground candidates with Grounded SAM~\cite{Ren2024} (Grounding DINO~\cite{Liu2024a} + SAM~\cite{Kirillov2023}).
|
||||
We leverage the dataset label by prompting the model with ``\code{a <class name>, a type of <object category>}''.
|
||||
Here \code{<object category>} is the immediate WordNet hypernym of the class (e.g., ``sorrel'' $\rightarrow$ ``horse''), which improves robustness when the class name is rare or overly specific.
|
||||
This can be the case with prompts like ``sorrel'' or ``guenon'', where the more general name ``horse'' or ``monkey'' is more ubiquitous.
|
||||
To increase recall, we generate up to $N=3$ masks per image by iteratively moving one level up the hypernym chain (e.g., ``sorrel'' $\rightarrow$ ``horse'' $\rightarrow$ ``equine'' $\dots$).
|
||||
We merge near-duplicate masks with pairwise IoU $\ge 0.9$, yielding a small set of $n_i<N$ candidate masks per image $i$.
|
||||
We select the best mask per image (according to \Cref{eq:filtering-score}) in a later filtering step, described below.
|
||||
|
||||
\textbf{Create neutral backgrounds via object removal.}
|
||||
Given a candidate mask, we remove the masked region and inpaint it using an object-removal model (LaMa~\cite{Suvorov2022} or Attentive Eraser~\cite{Sun2025}).
|
||||
This produces a visually plausible, ``neutral'' candidate background that can be paired with many foregrounds.
|
||||
For an image $i$ we now have $n_i$ foreground objects, extracted from $i$ by cutting out the masked region, each paired with a background where the same mask has been infilled.
|
||||
|
||||
\textbf{Select a high-quality pair.}
|
||||
Different masks can trade off including the full object versus leaking class cues into the background.
|
||||
We therefore score each candidate pair using an ensemble $E$ of six pretrained classifiers (ViT/ResNet/Swin) trained on the original dataset.
|
||||
Intuitively, we prefer (\textit{i}) foregrounds that strongly support the ground-truth class and (\textit{ii}) backgrounds that do \emph{not} support the ground-truth class, while (\textit{iii}) discouraging overly large foreground regions.
|
||||
For each model $m \in E$, we compute the class scores of the ground truth class $c$, $\P[m(\mathrm{fg})=c]$ on the foreground (with solid-gray background) and $\P[m(\mathrm{bg})=c]$ on the background and combine them with a prior $\operatorname{size}(\cdot)$ (pixel count):
|
||||
|
||||
\begin{align} \begin{split} \label{eq:filtering-score}
|
||||
\text{score}(\mathrm{fg}, \mathrm{bg}, c) &= \log \left( \sum_{m \in E} \frac{\P[m(\mathrm{fg}) = c]}{\abs{E}} \right)
|
||||
+ \log \left( 1 - \sum_{m \in E} \frac{\P[m(\mathrm{bg}) = c]}{\abs E} \right) \\
|
||||
& + \lambda \log \left( 1 - \abs{\frac{\operatorname{size}(\mathrm{fg})}{\operatorname{size}(\mathrm{bg})} - \eps} \right).
|
||||
\end{split} \end{align}
|
||||
% We set $\lambda = 2$ and $\eps = 0.1$ via a small hyperparameter search on a manually annotated subset.
|
||||
We run a hyperparameter search using a manually annotated subset of foreground/background variants to find the factors in \Cref{eq:filtering-score}: $\lambda = 2$ and $\eps = 0.1$.
|
||||
For each image, we keep the candidate mask with the highest score.
|
||||
|
||||
\textbf{Filter low-quality backgrounds.}
|
||||
Finally, we discard backgrounds that are heavily ($\geq 80\%$) inpainted, as they tend to look synthetic and provide little useful diversity (see supplementary).
|
||||
This step filters out $10\%$ of backgrounds.
|
||||
|
||||
Although segmentation is the main computational overhead, it is performed once offline and reused across all training runs.
|
||||
On NVIDIA H100 GPUs, the segmentation stage computes at a rate of $5 338.6 \frac{\text{img}}{\text{GPU} \times \text{h}}$ when inpainting with LaMa.
|
||||
For ImageNet this comes down to just under $30$ hours on a single node.
|
||||
At roughly twice the cost of a single ViT-B training run ($\approx 14$ hours), this is a modest investment that is amortized over every subsequent experiment the dataset is used in.
|
||||
For details see the supplementary material.
|
||||
% Compare this to $\approx 14$ hours for training ViT-B on ImageNet once.
|
||||
The output of the segmentation stage is a collection of foreground cut-outs (with transparency) and a pool of diverse, neutral backgrounds, which we use in the online recombination stage.
|
||||
For ImageNet, we provide pre-computed segmentation output\footnote{\code{URL will go here}}.
|
||||
|
||||
\subsection{Recombination}
|
||||
\label{sec:recombination}
|
||||
In each epoch, the recombination stage generates a recombined training sample for each foreground by (\textit{i}) choosing a background, (\textit{ii}) choosing a target foreground size, (\textit{iii}) sampling a placement, and (\textit{iv}) pasting the foreground using its alpha mask.
|
||||
This exposes the model to controlled changes in context and spatial layout that are largely absent from standard augmentation.
|
||||
|
||||
\textbf{Background sampling.}
|
||||
For each foreground object, we draw a background using one of three increasingly challenging strategies:
|
||||
(\textit{i}) \textit{Original}: use the object's own inpainted background (no context shift);
|
||||
(\textit{ii}) \textit{Same-class}: sample a background from the pool of backgrounds belonging to the same class (slight, but plausible context shift);
|
||||
(\textit{iii}) \textit{All-classes}: sample from the pool of all inpainted backgrounds (large context shift).
|
||||
These strategies trade off context diversity against semantic plausibility.
|
||||
We ensure that each foreground is used exactly once per epoch; backgrounds may repeat.
|
||||
|
||||
\textbf{Foreground scaling.}
|
||||
Let $r_{\text{fg}}$ denote the relative foreground area in the source image of the foreground, and $r_{\text{bg}}$ the relative foreground area in the source image of the background. % of the \emph{original} foreground (before inpainting) in the chosen background image.
|
||||
We compute the lower/upper size limits $(s_l, s_u)$ from these two ratios using one of two variants:
|
||||
(\textit{i}) \emph{mean} sets $(s_l, s_u)$ using the mean of $r_{\text{fg}}$ and $r_{\text{bg}}$, while
|
||||
(\textit{ii}) \emph{range} uses the min/max to preserve a wider scale range.
|
||||
Then, we sample the final scale from a $\pm 30\%$ interval around them and resize the foreground to this scale, while keeping the aspect ratio.
|
||||
|
||||
|
||||
\textbf{Placement and boundary smoothing.}
|
||||
We paste the resized foreground at a uniformly random location within the background.
|
||||
To reduce cut-and-paste artifacts, we slightly soften the alpha mask boundary by applying a Gaussian blur with $\sigma \in [\frac{\sigma_{\text{max}}}{10}, \sigma_{\text{max}}]$, following the range used in modern augmentation~\cite{Touvron2022}.
|
||||
% For example recombined images see \Cref{tab:foraug-examples}.
|
||||
|
||||
% \textbf{Interaction with standard augmentation.}
|
||||
% We support two augmentation orders:
|
||||
% (\textit{i}) apply the full augmentation pipeline after recombination; or
|
||||
% (\textit{ii}) apply crop+resize to the background first (to keep the full foreground visible), then recombine, then apply the remaining augmentations.
|
||||
% The former matches standard training exactly; the latter isolates composition changes from random cropping.
|
||||
|
||||
\textbf{Mixing with original images.}
|
||||
We optionally mix recombined samples with unmodified dataset images.
|
||||
A mixing ratio $p$ acts as the probability of drawing the original image; otherwise we use its foreground and apply \schemename.
|
||||
We consider constant $p$ as well as linear/cosine schedules that increase $p$ over training.
|
||||
Finally, we apply standard data augmentation techniques on the resulting images.
|
||||
|
||||
The online recombination is CPU-parallel and does not measurably increase training time.
|
||||
We find a $\approx 1\%$ increase in average step-time (see supplementary).
|
||||
|
||||
120
sec/method_old.tex
Normal file
120
sec/method_old.tex
Normal file
@@ -0,0 +1,120 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
%\begin{figure*}[ht!]
|
||||
% \centering
|
||||
% \includegraphics[width=.9\textwidth]{img/fig-2.pdf}
|
||||
% \caption{Overview of \name. The data creation consists of two stages: (1, offline) Segmentation, where we segment the foreground objects from the background and fill in the background. (2, online) Recombination, where we combine the foreground objects with different backgrounds to create new samples. After recombination, we apply strong, commonly used augmentation policies.}
|
||||
% \label{fig:method}
|
||||
%\end{figure*}
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{img/fig-2.pdf}
|
||||
\caption{Overview of \schemename. The data creation consists of two stages: Segmentation (offline, \Cref{sec:segmentation}), where we segment the foreground objects from the background and fill in the background. Recombination (online, \Cref{sec:recombination}), where we combine the foreground objects with different backgrounds to create new samples. After recombination, we apply strong, commonly used augmentation policies.}
|
||||
\label{fig:method}
|
||||
\end{figure*}
|
||||
|
||||
\section{\schemename (Method)}
|
||||
\label{sec:method}
|
||||
|
||||
% \begin{itemize}
|
||||
% \item[1.] Segment ImageNet
|
||||
% \item Detect and Cutout Foreground
|
||||
% \item Multiple foreground possibilities
|
||||
% \item Foreground mask merging
|
||||
% \item Background infills
|
||||
% \item Foreground/Background Filtering
|
||||
% \item [2.] Recombination
|
||||
% \item Which foreground \& Background
|
||||
% \item Background pruning
|
||||
% \item size
|
||||
% \item positioning
|
||||
% \item Border smoothing
|
||||
% \item Dealing with other data augmentations/transformations
|
||||
% \end{itemize}
|
||||
|
||||
% We propose a novel dataset, called \name, that improves image classification performance by explicitly separating and recombining foreground objects and plain backgrounds.
|
||||
% \name consists of two stages: Segmentation and recombination. Both are visualized in \Cref{fig:method}.
|
||||
% We introduce \schemename, a data augmentation scheme designed to enhance Transformer training by explicitly separating and recombining foreground objects and backgrounds.
|
||||
% \schemename enhances transformer training by explicitly encoding spatial invariances that these need to learn explicitly in the data.
|
||||
% \schemename involves two stages: Segmentation and Recombination, both visualized in \Cref{fig:method}.
|
||||
We introduce \schemename, a data augmentation designed to enhance Transformer training by embedding spatial invariances--which Transformers would otherwise need to learn implicitly--directly into the training data.
|
||||
% It operates by explicitly segmenting and recombining foreground objects and backgrounds.
|
||||
\schemename comprises two distinct stages: Segmentation and Recombination. Both are illustrated in \Cref{fig:method}.
|
||||
|
||||
|
||||
\subsection{Segmentation}
|
||||
\label{sec:segmentation}
|
||||
The segmentation stage isolates the foreground objects and their corresponding backgrounds.
|
||||
% We then fill in the background in a visually plausible way~\cite{Sun2025} using a pretrained object-removal model.
|
||||
We then fill the background using a pretrained object-removal model, producing visually plausible~\cite{Sun2025}, neutral scenes ready for recombination.
|
||||
This stage is computed once offline and the results are stored for the recombination stage.
|
||||
|
||||
First, foreground objects are detected and segmented from their backgrounds using a prompt-based segmentation model to exploit the classification datasets labels.
|
||||
We use the state-of-the-art Grounded SAM~\cite{Ren2024}, which is based on Grounding DINO~\cite{Liu2024a} and SAM~\cite{Kirillov2023}.
|
||||
The prompt we use is ``\code{a <class name>, a type of <object category>}'', where \code{<class name>} is the specific name of the objects class as defined by the dataset and \code{<object category>} is a the broader category of the object.
|
||||
The \code{<object category>} guides the segmentation model towards the correct object in case the \code{<class name>} alone is too specific.
|
||||
This can be the case with prompts like ``sorrel'' or ``guenon'', where the more general name ``horse'' or ``monkey'' is more helpful.
|
||||
We derive the \code{<object category>} from the WordNet hierarchy, using the immediate hypernym.
|
||||
|
||||
% We iteratively extract up to $n$ foreground masks for each dataset-image, using different more and more general prompts based on the more general synsets of WordNet (e.g. ``a sorrel, a type of horse'', ``a horse, a type of equine'', ...).
|
||||
We iteratively extract $n$ foreground masks for each dataset-image, creating prompts by going one hypernym up the WordNet-tree each step (e.g. ``a sorrel, a type of horse'', ``a horse, a type of equine'', ...).
|
||||
Masks that are very similar, with a pairwise IoU of at least $0.9$, are merged.
|
||||
The output is a set of masks delineating the foreground objects and the backgrounds.
|
||||
We select the best mask per image (according to \Cref{eq:filtering-score}) in a later filtering step, described below.
|
||||
|
||||
First, an inpainting model that is specifically optimized to remove objects from images, such as LaMa~\cite{Suvorov2022} or Attentive Eraser~\cite{Sun2025}, is used to inpaint the foreground regions in the backgrounds.
|
||||
Then, to ensure the quality of the foregrounds and the neutral background images, we select a foreground/background pair (for each dataset-image) from the $\leq n$ variants we have extracted and infilled in the previous steps.
|
||||
Using an ensemble $E$ of six ViT, ResNet, and Swin Transformer models pretrained on the original dataset, we select the foreground/background pair that maximizes foreground performance while minimizing the performance on the background and size of the foreground.
|
||||
For each model $m \in E$, we predict the score of the ground truth class $c$ on the foreground $\mathrm{fg}$ and background $\mathrm{bg}$ and weigh these with the size $\operatorname{size}(\cdot)$ in number of pixels according to:
|
||||
% $c$ is the correct foreground class, $\mathrm{fg}$, and $\mathrm{bg}$ are the foreground and background and $\operatorname{size}(\cdot)$ is the size in number of pixels.
|
||||
\begin{align} \begin{split} \label{eq:filtering-score}
|
||||
\text{score}(\mathrm{fg}, \mathrm{bg}, c) &= \log \left( \sum_{m \in E} \frac{\P[m(\mathrm{fg}) = c]}{\abs{E}} \right)
|
||||
+ \log \left( 1 - \sum_{m \in E} \frac{\P[m(\mathrm{bg}) = c]}{\abs E} \right) \\
|
||||
& + \lambda \log \left( 1 - \abs{\frac{\operatorname{size}(\mathrm{fg})}{\operatorname{size}(\mathrm{bg})} - \eps} \right).
|
||||
\end{split} \end{align}
|
||||
% We use $E$ is the ensemble of models and $m$ is a pretrained model, $c$ is the correct foreground class, $\mathrm{fg}$, and $\mathrm{bg}$ are the foreground and background and $\operatorname{size}(\cdot)$ is the size in number of pixels.
|
||||
We run a hyperparameter search using a manually annotated subset of foreground/background variants to find the factors in \Cref{eq:filtering-score}: $\lambda = 2$ and $\eps = 0.1$.
|
||||
% The \textit{optimal foreground size} of $10\%$ of the full image balances the smallest possible foreground size that encompasses all the respective class information in the image with still conveying the foreground information after pasting it onto another background.
|
||||
% This filtering step ensures we segment all the relevant foreground objects.
|
||||
|
||||
Finally, we filter out backgrounds that are largely infilled, as these tend to be overly synthetic and do not carry much information (see the supplementary material).
|
||||
% We ablate this choice in \Cref{sec:ablation}.
|
||||
% While the computational cost for the segmentation stage is significant, this is a one-time calculation whose results can be reused in subsequent experiments (see the supplementary material for details).
|
||||
Although the segmentation stage is computational overhead, it is a one-time cost with results that can be reused across experiments (see the supplementary material for details).
|
||||
In summary, we factorize the dataset into a set of foreground objects with a transparent background and a set of diverse backgrounds per class.
|
||||
The next step is to recombine these, before applying other common data augmentation operations during training.
|
||||
|
||||
\subsection{Recombination}
|
||||
\label{sec:recombination}
|
||||
The recombination stage, performed online during training, combines the foreground objects with different backgrounds to create new training samples.
|
||||
For each object, we follow the pipeline of: Pick an appropriate background, resize it to a fitting size, and place it in the background image.
|
||||
Through this step, we expose the model to variations beyond the image compositions of the dataset.
|
||||
|
||||
For each foreground object, we sample a background using one of the following strategies:
|
||||
(1) the original image background, (2) the set of backgrounds from the same class, or (3) the set of all possible backgrounds.
|
||||
These sets are trading off the amount of information the model can learn from the background against the diversity of new images created.
|
||||
In each epoch, each foreground object is seen exactly once, but a background may appear multiple times.
|
||||
|
||||
The selected foreground is resized based on its relative size within its original image and the relative size of the original foreground in the selected background image.
|
||||
The final size is randomly selected from a 30\% range around upper and lower limits ($s_u$ and $s_l$), based on the original sizes.
|
||||
% \begin{align}
|
||||
% s \sim \mathcal U \left[ (1 - 0.3) s_l, (1 + 0.3) s_u \right].
|
||||
% \end{align}
|
||||
To balance the size of the foreground and that of the backgrounds original foreground, the upper and lower limit $s_u$ and $s_l$ are set to the mean or range of both sizes, depending on the foreground size strategy: \emph{mean} or \emph{range}.
|
||||
|
||||
The resized foreground is then placed at a random position within the background image.
|
||||
To more seamlessly integrate the foreground, we apply a Gaussian blur with ${\sigma \in [\frac{\sigma_{\text{max}}}{10}, \sigma_{\text{max}}]}$, inspired by the standard range for the Gaussian blur operation in \cite{Touvron2022}, to the foreground's alpha-mask.
|
||||
|
||||
We can apply standard data augmentation techniques in two modes:
|
||||
Either we apply all augmentations to the recombined image, or we apply the cropping and resizing to the background only and then apply the other augmentations after recombination.
|
||||
% While for the second mode, the foreground object will always be fully visible, the first mode uses the data augmentations in the same way they would be used for the baseline dataset.
|
||||
% The second mode ensures the foreground object remains fully visible, while the first mode mirrors standard data augmentation practices.
|
||||
The first mode mirrors standard augmentation practice, whereas the second one ensures the foreground object remains fully visible.
|
||||
|
||||
We experiment with a constant mixing ratio, or a linear or cosine annealing schedule that increases the amount of images from the original dataset over time.
|
||||
The mixing ratio acts as a probability of selecting an image from the original dataset;
|
||||
otherwise, an image with the same foreground is recombined using \schemename, ensuring each object is seen once per epoch.
|
||||
% Thus, we still ensure each foreground is seen once per epoch.
|
||||
The recombination stage is designed to be parallelized on the CPU during training and thus does not impact training time (see supplementary material for details).
|
||||
|
||||
46
sec/related_work.tex
Normal file
46
sec/related_work.tex
Normal file
@@ -0,0 +1,46 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\section{Related Work}
|
||||
\label{sec:related_work}
|
||||
|
||||
\textbf{Data Augmentation for Image Classification.}
|
||||
Data augmentation is a crucial technique for improving the model performance and generalization.
|
||||
Traditional augmentation strategies rely on simple geometric or color-space transformations like cropping, flipping, rotation, blurring, color jittering, or random erasing~\cite{Zhong2020} to increase training data diversity without changing the semantic meaning.
|
||||
With the advent of ViTs~\cite{Dosovitskiy2021}, new data augmentation operations like PatchDropout~\cite{Liu2022d} have been proposed.
|
||||
Other transformations like MixUp~\cite{Zhang2018a}, CutMix~\cite{Yun2019}, or random cropping and patching~\cite{Takahashi2018} combine multiple input images.
|
||||
These simple transformations are usually bundled to form more complex augmentation policies like AutoAugment~\cite{Cubuk2019} and RandAugment~\cite{Cubuk2020}, or 3-Augment~\cite{Touvron2022}. %, which is optimized to train a ViT.
|
||||
For a general overview of data augmentation for image classification, we refer to Shorten et al.~\cite{Shorten2019} and Xu et al.~\cite{Xu2023d}.
|
||||
|
||||
We advance these general augmentations by introducing \schemename to explicitly separate objects and backgrounds for image classification, allowing us to move beyond image compositions from the dataset.
|
||||
Thus, \schemename unlocks performance improvements and bias reduction not possible with traditional data augmentation.
|
||||
% \schemename is used additionally to traditional augmentation techniques to improve performance and reduce biases.
|
||||
|
||||
\textbf{Copy-Paste Augmentation.}
|
||||
The copy-paste augmentation~\cite{Ghiasi2021}, which is used only for object detection~\cite{Shermaine2025,Ghiasi2021} and instance segmentation~\cite{Werman2022,Ling2022}, involves copying segmented objects from one image and pasting them onto another.
|
||||
While typically human annotated segmentation masks are used to extract the foreground objects, other foreground sources have been explored, like 3D models~\cite{Hinterstoisser2019} and pretrained object-detection models for use on objects on white background~\cite{Dwibedi2017} or synthetic images~\cite{Ge2023}.
|
||||
Kang et al.~\cite{Kang2022} apply copy-paste as an alternative to CutMix in image classification, but they do not shift the size or position of the foregrounds and use dataset images (with object) as backgrounds.
|
||||
|
||||
Unlike prior copy-paste methods that overlay objects, \schemename extracts foregrounds and replaces their backgrounds with semantically neutral fills, thereby preserving label integrity while enabling controlled and diverse recombination.
|
||||
|
||||
\textbf{Generative data augmentation.}
|
||||
Recent work uses generative models to synthesize additional training images, e.g., via GANs or diffusion models driven by text prompts or attribute labels~\cite{Lu2022,Trabucco2024,Islam2024}.
|
||||
Concurrently to our work, AGA~\cite{Rahat2025} combines LLMs, diffusion models, and segmentation to generate fully synthetic backgrounds from text prompts, onto which real foregrounds are pasted.
|
||||
These synthetic images are appended to the original training set.
|
||||
|
||||
While AGA focuses on increasing diversity via prompt-driven background synthesis, \schemename uses generative models differently:
|
||||
We apply inpainting only to locally neutralize the original object region, yielding semi-synthetic backgrounds that preserve the global layout, style, and characteristics of real dataset images.
|
||||
% AGA's focus on synthetic background is likely to produce a shifted, or even collapsed background image distribution~\cite{Zverev2025,Shumailov2024,Adamkiewicz2026}.
|
||||
Fully synthetic, prompt-generated backgrounds are likely to change, the effective background distribution, especially when prompts or generators are biased~\cite{Zverev2025,Shumailov2024,Adamkiewicz2026}.
|
||||
We then do online recombination of real foregrounds with these neutralized, dataset-consistent backgrounds under explicit control of object position and scale.
|
||||
Thus, \schemename acts as a dynamic large-scale augmentation method while AGA is statically expanding small-scale training sets with synthetic data.
|
||||
|
||||
\textbf{Model robustness evaluation.}
|
||||
Evaluating model robustness to various image variations is critical for understanding and improving model generalization.
|
||||
Datasets like ImageNet-A~\cite{Hendrycks2021}, ImageNet-C~\cite{Hendrycks2019} and ImageNet-P~\cite{Hendrycks2019} introduce common corruptions and perturbations.
|
||||
ImageNet-E~\cite{Li2023e} evaluates model robustness against a collection of distribution shifts.
|
||||
Other datasets, such as ImageNet-D~\cite{Zhang2024f} and ImageNet-R~\cite{Hendrycks2021a}, focus on varying background, texture, and material, but rely on synthetic data.
|
||||
Stylized ImageNet~\cite{Geirhos2019} investigates the impact of texture changes.
|
||||
ImageNet-9~\cite{Xiao2020} explores background variations using segmented images for a 9-class subset of ImageNet with artificial backgrounds.
|
||||
|
||||
In contrast to these existing datasets, which are used only for evaluation, \schemename provides fine-grained control over foreground object placement, size, and background selection, enabling a precise and comprehensive analysis of specific model biases within the context of a large-scale, real-world image distribution.
|
||||
As \schemename also provides controllable training data generation, it goes beyond simply measuring robustness to actively improving it through training.
|
||||
228
sec/reproducability.tex
Normal file
228
sec/reproducability.tex
Normal file
@@ -0,0 +1,228 @@
|
||||
% !TeX root = ../main.tex
|
||||
|
||||
\makeatletter
|
||||
% \@ifundefined{isChecklistMainFile}{
|
||||
% % We are compiling a standalone document
|
||||
% \newif\ifreproStandalone
|
||||
% \reproStandalonetrue
|
||||
% }{
|
||||
% We are being \input into the main paper
|
||||
\newif\ifreproStandalone
|
||||
\reproStandalonefalse
|
||||
% }
|
||||
\makeatother
|
||||
|
||||
\ifreproStandalone
|
||||
\documentclass[letterpaper]{article}
|
||||
\usepackage[submission]{aaai2026}
|
||||
\setlength{\pdfpagewidth}{8.5in}
|
||||
\setlength{\pdfpageheight}{11in}
|
||||
\usepackage{times}
|
||||
\usepackage{helvet}
|
||||
\usepackage{courier}
|
||||
\usepackage{xcolor}
|
||||
\frenchspacing
|
||||
|
||||
\begin{document}
|
||||
\fi
|
||||
\setlength{\leftmargini}{20pt}
|
||||
\makeatletter\def\@listi{\leftmargin\leftmargini \topsep .5em \parsep .5em \itemsep .5em}
|
||||
\def\@listii{\leftmargin\leftmarginii \labelwidth\leftmarginii \advance\labelwidth-\labelsep \topsep .4em \parsep .4em \itemsep .4em}
|
||||
\def\@listiii{\leftmargin\leftmarginiii \labelwidth\leftmarginiii \advance\labelwidth-\labelsep \topsep .4em \parsep .4em \itemsep .4em}\makeatother
|
||||
|
||||
\setcounter{secnumdepth}{0}
|
||||
\renewcommand\thesubsection{\arabic{subsection}}
|
||||
\renewcommand\labelenumi{\thesubsection.\arabic{enumi}}
|
||||
|
||||
\newcounter{checksubsection}
|
||||
\newcounter{checkitem}[checksubsection]
|
||||
|
||||
\newcommand{\checksubsection}[1]{%
|
||||
\refstepcounter{checksubsection}%
|
||||
\paragraph{\arabic{checksubsection}. #1}%
|
||||
\setcounter{checkitem}{0}%
|
||||
}
|
||||
|
||||
\newcommand{\checkitem}{%
|
||||
\refstepcounter{checkitem}%
|
||||
\item[\arabic{checksubsection}.\arabic{checkitem}.]%
|
||||
}
|
||||
\newcommand{\question}[2]{\normalcolor\checkitem #1 #2 \color{blue}}
|
||||
\newcommand{\ifyespoints}[1]{\makebox[0pt][l]{\hspace{-15pt}\normalcolor #1}}
|
||||
|
||||
\section*{Reproducibility Checklist}
|
||||
|
||||
\vspace{1em}
|
||||
\hrule
|
||||
\vspace{1em}
|
||||
|
||||
\textbf{Instructions for Authors:}
|
||||
|
||||
This document outlines key aspects for assessing reproducibility. Please provide your input by editing this \texttt{.tex} file directly.
|
||||
|
||||
For each question (that applies), replace the ``Type your response here'' text with your answer.
|
||||
|
||||
\vspace{1em}
|
||||
\noindent
|
||||
\textbf{Example:} If a question appears as
|
||||
%
|
||||
\begin{center}
|
||||
\noindent
|
||||
\begin{minipage}{.9\linewidth}
|
||||
\ttfamily\raggedright
|
||||
\string\question \{Proofs of all novel claims are included\} \{(yes/partial/no)\} \\
|
||||
Type your response here
|
||||
\end{minipage}
|
||||
\end{center}
|
||||
you would change it to:
|
||||
\begin{center}
|
||||
\noindent
|
||||
\begin{minipage}{.9\linewidth}
|
||||
\ttfamily\raggedright
|
||||
\string\question \{Proofs of all novel claims are included\} \{(yes/partial/no)\} \\
|
||||
yes
|
||||
\end{minipage}
|
||||
\end{center}
|
||||
%
|
||||
Please make sure to:
|
||||
\begin{itemize}\setlength{\itemsep}{.1em}
|
||||
\item Replace ONLY the ``Type your response here'' text and nothing else.
|
||||
\item Use one of the options listed for that question (e.g., \textbf{yes}, \textbf{no}, \textbf{partial}, or \textbf{NA}).
|
||||
\item \textbf{Not} modify any other part of the \texttt{\string\question} command or any other lines in this document.\\
|
||||
\end{itemize}
|
||||
|
||||
You can \texttt{\string\input} this .tex file right before \texttt{\string\end\{document\}} of your main file or compile it as a stand-alone document. Check the instructions on your conference's website to see if you will be asked to provide this checklist with your paper or separately.
|
||||
|
||||
\vspace{1em}
|
||||
\hrule
|
||||
\vspace{1em}
|
||||
|
||||
% The questions start here
|
||||
|
||||
\checksubsection{General Paper Structure}
|
||||
\begin{itemize}
|
||||
|
||||
\question{Includes a conceptual outline and/or pseudocode description of AI methods introduced}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results}{(yes/no)}
|
||||
yes
|
||||
|
||||
\question{Provides well-marked pedagogical references for less-familiar readers to gain background necessary to replicate the paper}{(yes/no)}
|
||||
yes
|
||||
|
||||
\end{itemize}
|
||||
\checksubsection{Theoretical Contributions}
|
||||
\begin{itemize}
|
||||
|
||||
\question{Does this paper make theoretical contributions?}{(yes/no)}
|
||||
no
|
||||
|
||||
\ifyespoints{\vspace{1.2em}If yes, please address the following points:}
|
||||
\begin{itemize}
|
||||
|
||||
\question{All assumptions and restrictions are stated clearly and formally}{(yes/partial/no)}
|
||||
Type your response here
|
||||
|
||||
\question{All novel claims are stated formally (e.g., in theorem statements)}{(yes/partial/no)}
|
||||
Type your response here
|
||||
|
||||
\question{Proofs of all novel claims are included}{(yes/partial/no)}
|
||||
Type your response here
|
||||
|
||||
\question{Proof sketches or intuitions are given for complex and/or novel results}{(yes/partial/no)}
|
||||
Type your response here
|
||||
|
||||
\question{Appropriate citations to theoretical tools used are given}{(yes/partial/no)}
|
||||
Type your response here
|
||||
|
||||
\question{All theoretical claims are demonstrated empirically to hold}{(yes/partial/no/NA)}
|
||||
Type your response here
|
||||
|
||||
\question{All experimental code used to eliminate or disprove claims is included}{(yes/no/NA)}
|
||||
Type your response here
|
||||
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
|
||||
\checksubsection{Dataset Usage}
|
||||
\begin{itemize}
|
||||
|
||||
\question{Does this paper rely on one or more datasets?}{(yes/no)}
|
||||
yes
|
||||
|
||||
\ifyespoints{If yes, please address the following points:}
|
||||
\begin{itemize}
|
||||
|
||||
\question{A motivation is given for why the experiments are conducted on the selected datasets}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{All novel datasets introduced in this paper are included in a data appendix}{(yes/partial/no/NA)}
|
||||
no
|
||||
|
||||
\question{All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{All datasets drawn from the existing literature (potentially including authors' own previously published work) are accompanied by appropriate citations}{(yes/no/NA)}
|
||||
yes
|
||||
|
||||
\question{All datasets drawn from the existing literature (potentially including authors' own previously published work) are publicly available}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives are not scientifically satisficing}{(yes/partial/no/NA)}
|
||||
NA
|
||||
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
|
||||
\checksubsection{Computational Experiments}
|
||||
\begin{itemize}
|
||||
|
||||
\question{Does this paper include computational experiments?}{(yes/no)}
|
||||
yes
|
||||
|
||||
\ifyespoints{If yes, please address the following points:}
|
||||
\begin{itemize}
|
||||
|
||||
\question{This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{Any code required for pre-processing data is included in the appendix}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{All source code required for conducting and analyzing the experiments is included in a code appendix}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
\question{This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics}{(yes/partial/no)}
|
||||
yes
|
||||
|
||||
\question{This paper states the number of algorithm runs used to compute each reported result}{(yes/no)}
|
||||
yes
|
||||
|
||||
\question{Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information}{(yes/no)}
|
||||
yes
|
||||
|
||||
\question{The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank)}{(yes/partial/no)}
|
||||
no
|
||||
|
||||
\question{This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments}{(yes/partial/no/NA)}
|
||||
yes
|
||||
|
||||
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
\ifreproStandalone
|
||||
\end{document}
|
||||
\fi
|
||||
Reference in New Issue
Block a user