\author{
Lu Qi$^{1}$\thanks{Equal contribution.},~
Jason Kuen$^{2*}$,~
Weidong Guo$^{3*}$,~
Tiancheng Shen$^4$,~
Jiuxiang Gu$^{2}$,~
Wenbo Li$^{4}$,~\\
Jiaya Jia$^{4}$~
Zhe Lin$^2$,~
Ming-Hsuan Yang$^{1}$
\\[0.2cm]
$^1$The University of California, Merced~~\\
$^2$Adobe Research~~\\
$^3$QQ Browser Lab, Tencent~~\\
$^4$The Chinese University of Hong Kong~~
\vspace{-0.4in}
}
\maketitle

\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

Motivated by the above-discussed observations, we propose a new large-scale, high-quality dataset for entity segmentation. Our dataset has three important characteristics: 
1) we collect about 33,227 images from public datasets and the Internet. These images cover various domains such as landscapes, outdoor/indoor scenes, cartoons, graphical illustrations, and computer games; 
2) the annotations are open-world without any predefined class restrictions and densely cover
the entire images; 
3) we elevate the segmentation quality objective of entity segmentation~\cite{qi2021open} to the next level by annotating high-quality masks with fine-grained details on high-resolution images. There are over 70\% images that 
that have high image resolutions, spanning from 
2K to 15K, with significantly more accurate boundary annotations compared to existing panoptic datasets. 
As shown in Figure~\ref{fig:comp_coco}, our diversified dataset provides fine-grained segmentation masks on high-res images, along with open-world annotations on low-res images from public datasets.

\section{Related Work}
\paragraph{Image Segmentation Dataset}
Numerous datasets have been proposed for semantic, instance, and panoptic segmentation,~\emph{e.g}., Microsoft COCO~\cite{lin2014microsoft}, ADE20K~\cite{zhou2017scene}, KITTI~\cite{geiger2012we}, LVIS~\cite{gupta2019lvis}, PASCAL-VOC~\cite{everingham2010pascal}, Open-Images~\cite{kuznetsova2020open},  Cityscapes~\cite{cordts2016cityscapes},~\emph{etc}. 
In addition, there are some datasets designed for specific scenarios, such as amodal segmentation (COCO-Amodal~\cite{zhu2017semantic}, KINS~\cite{qi2019amodal}), human segmentation (CelebAMask-HQ~\cite{CelebAMask-HQ}, LIP~\cite{gong2017look}, MHP~\cite{zhao2018understanding}) and domain adaptation (Synscapes~\cite{wrenninge2018synscapes}, GTA5~\cite{richter2016playing}). 
Despite significant contributions from these datasets, it is of great interest to fulfill the needs of real-world applications with high-quality images with large diversity. 
For example, a segmentation model should be robust to high-resolution images from different domains. 
Meanwhile, models should be able to segment unseen objects in the open-world setting. 
Most relevant to our work is the  ADE20K dataset~\cite{zhou2017scene}, which has large-scale open-vocabulary categories. 
However, the collected images in ADE20K are of low resolution and from narrowed domains. 
In our dataset, we collect the images from multiple image domains, including indoor, outdoor, street scenes, and even cartoon and remote image domains. 
In addition, over 80\% of the images resolution
fall within the high-resolution range of 2000px and 8000 px. Compared to the ADE20K~\cite{zhou2017places} and LVIS~\cite{gupta2019lvis} dataset with a predefined category list, our annotation process is different.
We first conduct class-agnostic mask annotation on each entity. After that, the category information is then labeled in an open-vocabulary manner.

The EntitySeg dataset contains 33,227 images with high-quality mask annotations. 
Compared with existing dataets, there are three distinct properties in EntitySeg. 
First, 71.25\% and 86.23\% of the images are of high resolution with at least 2000px$\times$2000px and 1000px$\times$1000px which is more consistent with current digital imaging trends.
Second, our dataset is open-world and is not limited to predefined classes. We regard 
every semantically-coherent region in the images as an entity, even 
if it is a blurred region or it cannot be semantically recognized easily. Third, the mask annotation along the boundaries are more accurate than existing datasets, as shown in Figure~\ref{fig:comp_coco}. 
In the following, we describe our EntitySeg dataset and provide  comprehensive analysis.

\subsection{Image Collection}
Inspired by the collection criterion of COCO~\cite{lin2014microsoft}, we collect most non-iconic images with more contextual information and non-canonical viewpoints. 
In addition, the image domains should be as diversified as possible to guarantee substantial domain diversity. Therefore, the image sources of our dataset include several public datasets and the Internet where the images permitted for academic research use. For public dataset sources, we select part of the images from
COCO~\cite{lin2014microsoft}, ADE20K~\cite{zhou2017scene}, 
Pascal VOC~\cite{everingham2010pascal}, Cityscapes~\cite{cordts2016cityscapes}, 
Mapillary Vistas~\cite{neuhold2017mapillary}, ImageNet~\cite{krizhevsky2017imagenet}, 
Open Images~\cite{kuznetsova2020open}, DIV2K~\cite{agustsson2017ntire}, Flick2K~\cite{wang2019flickr1024}, Clipart~\cite{inoue2018cross},
Comic~\cite{inoue2018cross}, DOTA~\cite{xia2018dota} and some computer game screenshots from Synscapes~\cite{wrenninge2018synscapes} and GTA5~\cite{richter2016playing}. From the Internet, we crawled mainly high-resolution images from Pixabay~\cite{pixabay}, Unsplash~\cite{Unsplash}, and LAION-400M~\cite{schuhmann2021laion}. In Figure~\ref{fig:statistics_01}(c), we show the distribution of image sources in our dataset and see that most of the images are from high-resolution sources like Pixabay, Unsplash, and LAION-400M. In Figure~\ref{fig:statistics_01}(a), our dataset has more high-resolution images whose resolutions are normally distributed in the resolution range between 0px to 15,000px, whereas all the images of ADE20K~\cite{zhou2017scene} and COCO~\cite{lin2014microsoft} are under 1000px.

\subsection{Image Annotation}
The annotation process for our dataset mainly consists of three steps. For an image, we firstly annotate all entities with class-agnostic pixel-level masks where each mask has no overlaps with others. After that, each mask is annotated with a class label based on a large vocabulary class set, which is not fixed but updated continuously over time. There are two special considerations here.
On one hand, we annotate the entity as `unknown' or `blurred' if it cannot be name easily or severely blurred. On the other hand, the entity is annotated as a  `\textit{supercategory}\_other' if we merely know it is a \textit{supercategory} but unable to identify its fine-grained class. Our annotation process is the direct opposite of those of the popular segmentation datasets like COCO~\cite{lin2014microsoft} and ADE20K~\cite{zhou2017places}, where class sets are predefined first before annotating masks based on the predefined classes. In Table~\ref{Tab:comp_entity_other}, it shows that our dataset has 
a greater proportion of covered image areas and number of masks on average than COCO- and ADE20K-Panoptic. This is by virtue of our novel annotation process that allows us to take into account all the semantically-coherent regions, and then assign class labels to them. 
In addition, our annotation process is more similar to the the human vision system. This is consistent with human vision system that is inherently class-agnostic and is able to recognize entities without understanding its use and purpose, as explained by Marr~\cite{Marr1982Vision}.

Annotation consistency is a crucial aspect of any human-labeled dataset since it tells whether the annotation task is well-defined and reasonable. We randomly selected 500 images from our dataset ($1.5\%$ of the entire dataset) and asked another two annotators to 
separately annotate them again after four months. Table~\ref{tab:consistency}(a) shows the class-agnostic mask mAP in the first step of the annotation process compared to those of the other two annotators. We use one as ground truth, and another one as prediction results. 
In addition, we evaluate the category consistency by accuracy (ACC) under the same mask annotations in Table~\ref{tab:consistency}(b). 
These two tables indicate our dataset has a high degree of annotation consistency both in the mask and category labeling stages.

\begin{tabular}{c|ccc}
             & \cellcolor{lightgray!30} ann$_{1}$ & \cellcolor{lightgray!30} ann$_{2}$ & \cellcolor{lightgray!30} ann$_{3}$\\ \hline
            \cellcolor{lightgray!30} ann$_{1}$ & - & - & -\\ 
            \cellcolor{lightgray!30} ann$_{2}$ & 90.6 & - & -\\ 
            \cellcolor{lightgray!30} ann$_{3}$ & 91.2 & 92.1 & -\\ \hline
            \end{tabular}

You are provided with a table and supplementary text from a machine learning paper. Here are JSON templates for four types of numeric cells: "Other", "Data Stat.", "Hyper-parameter/Architecture", and "Result":
{"value": "xx", "type": "Other"}
{"value": "xx", "type": "Data Stat.", "dataset": "xx", "attribute name": "xx", "sub-set/group name": "xx", "dataset features": {"xx": "yy"}}
{"value": "xx", "type": "Hyper-parameter/Architecture", "model": "xx", "parameter/architecture name": "xx", "dataset": "xx"}
{"value": "xx", "type": "Result", "task": "xx", "metric": "xx", "training data/set": "xx", "test data/set": "xx", "model": "xx", "model settings": {"xx": "yy"}, "experimental settings": {"xx": "yy"}}

Please describe all numeric cells in the above latex table following the JSON templates (proceeding by row in a left-right, top-down direction). For each cell, output one JSON description per line. For any unanswerable attributes in the templates, set their value to the placeholder "xx" if it is of string type and {"xx": "yy"} if it is of dictionary type. Numeric cells, which describe performance/error analysis, should be labeled as "Other".

Cell Description:
{"value": "90.6", "type": "Other"}
{"value": "91.2", "type": "Other"}
{"value": "92.1", "type": "Other"}