\section*{Acknowledgments}
We thank Gianluca Calcagni and Florian Nortier for valuable comments. AK is supported by FCT Portugal investigator project IF/01607/2015.

\section*{Appendix: Confinement in local Yang-Mills theory}
\label{AppendixC}

Confinement in SU(N) Yang-Mills theory was investigated and proven in Ref. \cite{Chaichian:2018cyv}. Utilising the Kugo-Ojima formalism \cite{Kugo:1977zq,Kugo:1979gm}, we have
\begin{widetext}
\begin{eqnarray}
\int d^4xe^{ipx}\langle D_\mu\bar{c}^{a} (x),D_\nu c^{b} (0) \rangle&=&-\delta^{ab}\frac{p_\mu p_\nu}{k^2}\\
&+&\frac{(N^2-1)^2}{2N}g^2\delta^{ab}
\left(\delta_{\mu\nu} - \frac{p_{\mu} p_{\nu}}{p^{2}}\right)
\int\frac{d^4p'}{(2\pi)^4}K_2(p-p')G_2(p').\nonumber
\end{eqnarray}
\end{widetext}
The form of the propagators look like:
\begin{equation}
K_2(p)=-\frac{1}{p^2+i\epsilon}
\end{equation}
and that of the ghost field and
\begin{equation}
G_2(p)=\frac{\pi^3}{4K^3(i)}
	\sum_{n=0}^\infty\frac{e^{-(n+\frac{1}{2})\pi}}{1+e^{-(2n+1)\pi}}(2n+1)^2\frac{1}{p^2-m_n^2+i\epsilon}
\end{equation}
and that of the gauge field,
provided the mass spectrum
\begin{equation}
m_n=(2n+1)\frac{\pi}{2K(i)}\left(\frac{Ng^2}{2}\right)^\frac{1}{4}\mu,
\end{equation}
where $K(i)$ represents the complete elliptical integral of the first kind and $\mu$ is one of the integration constants of the theory.
This is a good and natural approximation to the full propagator as long as we omit the mass shift induced due to quantum corrections. Then, the Kugo-Ojima confinement condition looks:
\begin{widetext}
\begin{equation}
\label{eq:NL}
u(0)=-\frac{(N^2-1)^2}{2N}g^2
\int\frac{d^4p}{(2\pi)^4}\frac{1}{p^2+i\epsilon}\frac{\pi^3}{4K^3(-1)}
	\sum_{n=0}^\infty\frac{e^{-(n+\frac{1}{2})\pi}}{1+e^{-(2n+1)\pi}}(2n+1)^2\frac{1}{p^2-m_n^2+i\epsilon}=-1.
\end{equation}
\end{widetext}
Evaluating the integral one obtains the $\beta$-function in a closed form
\begin{equation}
\beta_{YM}=-\beta_0\frac{\alpha_s^2}{1-\frac{1}{2}\beta_0\alpha_s},
\end{equation}
with $\beta_0=(N^2-1)^2/8\pi N$. This beta function grants confinement of the theory with the coupling running to infinity at lower momenta with no Landau pole. In the ultraviolet we recover the asymptotic freedom as expected.


\section{Downloading the Dataset and the Software Package}
\input{where_to_download}

\clearpage

\section{Datasheet}

\definecolor{darkblue}{RGB}{46,25, 110}

\newcommand{\dssectionheader}[1]{%
\phantomsection
   \noindent\framebox[\columnwidth]{%
      {\fontfamily{phv}\selectfont \textbf{\textcolor{darkblue}{#1}}}
   }
  \addcontentsline{toc}{subsection}{#1}
}

\newcommand{\dsquestion}[1]{%
\medskip
\phantomsection
    {\noindent \fontfamily{phv}\selectfont \textcolor{darkblue}{\textbf{#1}}}
  \addcontentsline{toc}{subsubsection}{#1}
\smallskip
    }

\newcommand{\dsquestionex}[2]{%
\medskip
\phantomsection
    {\noindent \fontfamily{phv}\selectfont \textcolor{darkblue}{\textbf{#1} #2}}
  \addcontentsline{toc}{subsubsection}{#1}
    \smallskip
}
\input{datasheet_content}

\clearpage
\listoffigures
\addcontentsline{toc}{section}{List of Figures}
\listoftables
\addcontentsline{toc}{section}{List of Tables}

\end{document}
\section*{Checklist}

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \answerYes{To the best of our assessment}
  \item Did you describe the limitations of your work?
    \answerYes{Those we could think about, see Discussion}
  \item Did you discuss any potential negative societal impacts of your work?
    \answerYes{See Discussion}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerYes{Yes, and we discussed abundantly with our advisors sepcialized in human rights and geographical information systems.}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerNA{No theoretical results}
	\item Did you include complete proofs of all theoretical results?
    \answerNA{No theoretical results}
\end{enumerate}

\item If you ran experiments (e.g. for benchmarks)...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerYes{Coming in the supplementary material on June 16th.}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerYes{Coming in in the supplementary material on June 16th.}
	\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerYes{We went further and compared the distribution of the means and the distributions over the whole validation set.}
	\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerYes{We indicated the main speed of training, and will include details in the supplementary material. We are a small operation, running on one single Amazon instance, so will not be a bottleneck to reproducibility}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerYes{See references for all datasets used.}
  \item Did you mention the license of the assets?
    \answerYes{In the abstract and under section 1.2.}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerYes{A whole new dataset, to be hosted on Zenodo with a DOI, and temporarily in a private Google Drive whose URL will be supplied to the reviewers.}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerYes{Mentioned our providers in the text.}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerYes{See final section.}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerNA{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerNA{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerNA{}
\end{enumerate}

\end{enumerate}

\section{Introduction}

\subsection{The Problem}

Computer vision and satellite imagery seem to be a match made in heaven. The idea to automatically process the growing amount of imagery collected has been lingering for decades in the remote sensing and the earth observation communities. The appeal of seeing the whole planet, and analyzing it at scale, is akin to few others. Many attempts have been made throughout the last thirty years. The ever-higher resolution of imagery available to civilians, and the last decade of improvement in Machine Learning and Computer Vision, have brought tools that could be brilliantly assistive in that regard. Some very visible scientific successes have been  published, such as\citep{jean_combining_2016}. Even prominent Deep Learning innovators cut their teeth early on as Master students on applications to satellite imagery, e.g.  \citet{mnih_learning_2010}. And prominent tools like Google Earth, which gives everyone access to high-resolution aerial imagery (although not all of it obtained by satellite), is a trigger for imagination. The dream runs deep.

However, the full potential of these two fields has been hindered by a combination of factors, in particular data access and the associated costs.

Broadening access to cutting edge technology is a hacker's delight: some satellite imagery can actually be received by any accessible for any amateur with an antenna up their roof -- back as an undergrad the first author built a cheap reception, storage, and processing station for low-resolution Meteosat Second Generation imagery \citep{beaudoin_meteosat_2005}, undercutting ten-fold the cost of commercial stations at the time. Fun and games. And while such hijinks on reception stations are not feasible for high-accuracy digital imagery, since 2015, the European Sentinel 2 satellites have been providing medium-resolution imagery (10 m/pixel) for free every five days accross the world for anyone who knows how to access them.

Yet, high-resolution imagery (1 m/pixel) or very-high-resolution (sub 1 m/pixel) are still out of easy reach. In a more recent work with Amnesty International to detect destroyed villages in conflict zones, \citep{cornebise_witnessing_2018}, we discovered that the cost to purchase a single very-high-resolution mosaic of the whole of Darfur, akin to Google Earth at maximum zoom, would cost 4 Million USD, even including a generous discount for charities. This makes it a tremendous challenge to even experiment with computer vision for high resolution.

Even setting cost aside, and assuming, as some hope, that the thunderous technological advances in launch technologies unlock a deluge a high-resolution imagery at a smaller price point, the key material for Machine Learning is still simply not there: carefully curated datasets to train on!  Accessing satellite imagery, even Sentinel 2, requires a certain amount of domain knowledge, more so than for natural images that can be sourced pretty much anywhere. The barrier to entry is real.

Indeed open high-resolution satellite imagery datasets are still quite rare, and the few who do exist tend to be either small, cover few unique locations, ad-hoc locations, or are designed for very specific uses. The SpaceNet challenge datasets \citep{van_etten_spacenet_2018} are possibly the most widely used satellite imagery datasets. Their combined unique location area is close to WorldStrat with a bit more than 10 000 km², but it is mainly focused on urban structures and made for specific tasks like building or road detection, with varying data providers and resolutions, and with no paired multi-temporal low-resolution imagery.

We mention paired lower-resolution, because another hope to work around the lack of access lies in the field of Super-Resolution: being able to derive from (possibly multiple) free low-resolution satellite revisits the same insights that would be available from a single high-resolution satellite visit of the corresponding area.  While we want to build datasets that can be used for a whole breadth of applications, making them suitable for super-resolution brings a swath of extra benefits.

On that topic, the  ESA Kelvins PROBA-V dataset \citep{martens_super-resolution_2019} and the associated competition have been a boon for such multi-frame super-resolution, but is single-channel and most importantly is very low resolution at 300 m/pixel and 100 m/pixel. It is also not georeferenced or time-referenced. However, it allowed us to develop the high-performing HighRes-Net multi-frame super-resolution algorithm \citep{deudon_highres-net_2020}. There is a an abundant literature on multiframe super-resolution for natural images and videos, but quite not as much in satellite imagery -- which we will cover in Section~\ref{sec:superres}.

The very recent dataset by \citet{michel_sen2vens_2022} promises to enable single-image super-resolution. covers only 29 unique locations, on 806 unique \sqkm, at 5 meter/pixel. It offers nine revisits for each location, remarkably pairing one single low-resolution and one high-resolution for each visit. This is somewhat promising, but still lacks the breadth we hope for.

To the best of our knowledge, no dataset tries to be generic and cover systematically the whole type of land-use across the world. Even fewer are explicitly designed with the aim of transferring learning to high-availability lower-resolution data: low-resolutions from Sentinel 2 can be manually added, but at the price of  extra work and expertise.

\subsection{Our Contributions}

\begin{figure}[htbp]
  \centering
  \includegraphics[height=1\textwidth]{summary_main_fig.pdf}
  \caption{Summarizing the construction and classes of the WorldStrat dataset.\label{fig:summary_construction}}
\end{figure}

We aim to empower the development of machine learning for satellite imagery that can be used for a wide range of applications: from ecology and climate change monitoring, to urbanization, to sociology, to disaster preparation, to agriculture, etc.  Our focus while building this dataset and its accompanying benchmarks and software package has therefore been to:
\begin{itemize}
  \tightlist
  \item Maximise the representation of all possible features of interest, for the widest possible use-cases.
  \item Have a decent worldwide representativity -- especially relevant in light of the need for Fairness Accountability and Transparency in ML, which includes the problem datasets often biased towards the Global North.
  \item And provide a pipeline that allows easy reproducibility by others, and extension if extra budget becomes available.
\end{itemize}

Our resulting dataset, the \textbf{World Stratified Dataset} (or WorldStrat for short) \citep{cornebise_worldstrat_2022} covers \textbf{~10,000 \ssqkm~, and 4,000 distinct locations, specially curated for the highest diversity of possible uses}.   In particular, as visualised in Figure~\ref{fig:summary_construction}, we separated our image acquisition budget into three parts:
\begin{itemize}
\tightlist
  \item One part focusing on human settlements further stratified by population density: filtering the world for settlements according to the ESA CCI LandCover product \citep{esa_land_v20} (which is licensed without restrictions under the CCI Data Policy v1.1 ), then sub-stratifying according to the Urban density class of the  Global Human Settlement Layer SMOD product \citep{florczyk_ghsl_2019} (which is licensed under CC-BY).
  \item A second part focused on non-settlement areas, using stratified sampling and class-rebalancing across the Land Cover Classification System labels of the ESA CCI LandCover product, and more precisely its aggregation into classes provided by the International Panel on Climate Change.
  \item Finally, a third part is focused on use cases typically under-served by usual datasets and not covered by the above. We sourced points of interests from the United Nations High Commissioner for Refugees (UNHCR) for populations of concerns (which is licensed under CC-BY-IGO), from Amnesty International for human rights sites of interest (with permission), and ESA ASMSpotter for illegal mining (with permission).
\end{itemize}

At each resulting location, along with the label, we provide the following imagery:
\begin{itemize}
  \tightlist
  \item One High-Resolution multispectral image from Airbus SPOT 6/7 (licensed a paid-for extension to ESA TPM license granting redistribution under CC-BY-NC), in RGB (6 m/pixel), Near Infrared (6 m/pixel), and Pan-chromatic channels (1.5m / pixel), at 1054x1054 pixels at the highest resolution.
  \item 16 Low-Resolution revisits from Copernicus Sentinel 2 (licensed without restrictions under Copernicus Sentinel Data Legal Notice and Service Information), temporally matched to the High-Resolution image -- within 5 days for the temporally closest. All 12 spectral bands are covered, at up to 10 m/pixel.
\end{itemize}

The rest of the article is structured as follows. In Section~\ref{sec:stratif}, we present how we have curated the parts of the world we cover, aka Areas Of Interest (AOIs), to offer maximum representativity of the world and of use cases.   In Section~\ref{sec:imagery}, we describe the characteristics of the paired imagery available at every AOI, both in low- and high-resolution, and how it can be easily extended.   In Section~\ref{sec:benchmarks}, to illustrate one possible use of this dataset, we establish baselines on multi-frame super-resolution tasks using several architectures, with an emphasis on compute efficiency. We also present the toolbox integrated with popular package EO-learn to use, reproduce, and extend, all our work.
We conclude in Section~\ref{sec:discussion} by discussing ideas tried and discarded along the way, as well as possible extensions.

\section{Curating Highly Representative Locations}
\label{sec:stratif}
The JIF dataset covers almost 10,000 \sqkm. Each base AOI is 2.5 \sqkm, i.e. 1,581 meters per side, the minimum contiguous order size allowed by Airbus, provider of the high-resolutiom imagery. This maximizes the number of AOIs within the allotted budget.

\subsection{Stratifying The World}
We use the first half of the dataset to attempt a systematic, stratified coverage of the world.
 The question becomes: how do we chose these locations to ensure a ''best'' application-agnostic dataset for super-resolution?

Sixty precent will be taken from the ``Settlement'' class from the ESA CCI LandCover Product,
  which we then stratify according to the Global Human Settlement Layer
  SMOD for different types of urban density, and with marginal distribution
  proportional to the cubic root of the actual distribution -\/- to
  keep the order of classes but diminish the overall imbalance.

 Fourty  percent o will be taken from all the other IPCC classes, i.e.
  non-settlement,
stratified according to (non-settlement) IPCC class, marginal
  distribution proportional to the cubic root of the actual
  distribution,  .and within each (non-settlement) ~IPCC class, again stratifying,
  according to the LCCS class (thinner vegetation typology), again with
  cubic root proportions.

\subsection{Why stratifying by land-use}
In an optimal sampling entirely focused on super-resolution, we would first find a latent-space feature-based clustering of satellite imagery, not unlike \citep{jean2019tile2vec}. Then, inspired by the classical statistical tools variance-minimizing design of experiments \citep{atkinson2007optimum}, we would sample each semantic cluster proportionally to how difficult it is to super-resolve.

Quantifying this would, at best, be done via some form of active learning, with the problem of being dependent on the actual super-resolution algorithm used in the process, or at worst, require having first solved the super-resolution problem, neither of which is quite satisfying.
 Besides, this would, again, be optimized solely for super-resolution: we aim for broader.

A proxy frequently used in statistical design of experiment is instead be to sample each semantic class proportionally to its variance. While this works well for numerical features, variance is not a clearly defined quantity for visual features. There would be an interesting methodological question to be investigated as to what a form of “semantic variance” would look like, but is quite a deep methodological topic and would take us into a rabbit hole out of scope of this project.

This ideal schema being quite ill-defined, we move on to a more pragmatic choice.

A pragmatic way to decide on an optimal dataset suitable for a broad range of applications is instead to ensure a representation of every type of land. This prepares equally for studies of urban space, of ice coverage, of limnology, etc.

We could sample POIs uniformly on POIs uniformly on the WSG84 ellipsoid, conditionally on being on land (by, say, rejection sampling), then filter the resulting POIs by SPOT availability. However this would not provide any guarantee as to how much each type of land use would be represented, nor in which proportion. The law of large numbers would guarantee a representation asymptotically proportional to the real world, but this does not provide any
fine-grained control, nor any representation of rare classes: random fluctuations would not be controlled for, which is problematic when the number of samples is ``far from asymptotic'' -- as in our case.

Instead, we can stratify by ``land use'', for some definition thereof, exploiting ample prior art on land use and land cover classifications.

The ESA Climate Change Initiative Land Cover Products (ESA CCI LCP) \citep{esa_land_v20}. This worldwide land cover map is based on Food and Agriculture Organization of the United Nations (FAO) Land Cover Classification System (LCCS) hierarchical system \citep{di2005land}. Moreover, it nests it hierarchy within a coarser classification suggested by the International Panel for Climate Change \cite{esa_land_v20}[page 30.], reflecting the preoccupations of climate change studies.

\subsubsection{Sampling the Non-Urban World}

We sampled 2,000 \sqkm at 800 POIs and stratified them using the IPCC classification found in the ESA Climate Change Initiative (CCI) Land Cover 2019 dataset. We then rebalanced the classes using cubic-root importance sampling, e.g. to under-represent on-purpose the surprisingly large planet-wide proportion of moss/lichen-covered land. The points were enriched to include ice-caps, noticeably absent from the original classification but of core importance both for climate analysis, geopolitical implications, and new logistics routes.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=.5\textwidth]{images/lccs_ipcc_classes.png}\hfill
  \includegraphics[width=.5\textwidth]{images/lccs_ipcc_cbrt_classes.png}
  \caption{Distribution of the IPCC classes over the whole world (left), and the class-rebalanced sample using cubic root (right). \label{fig:lccs_ipcc_classes}}
\end{figure}

The rationale for using importance sampling with the cubic root of
the population distribution as the proposal is the following:

\vspace{-1em}
\begin{itemize}
\tightlist
\item We want to make sure we have a higher representation of rare classes,
  i.e. not being dominated by the classes that are already highly
  present naturally, because we wouldn't want our super-resolution to
  miss a rare object that would be out of place.
\item But we do still want to acknowledge that classes that are highly
  present naturally are possibly be super-resolved quite often, so we do
  not want to completely flatten the histogram by sampling uniformly.
\end{itemize}
Hence the cubic root as a compromise, which boosts rare classes but does keep the monotonous order relationship between the classes. The result is visible in Figure~\ref{fig:lccs_ipcc_classes}.

\subsubsection{Sampling the Urban World: Nested Stratification of Urban Density with GHSL SMOD}

We sampled 3,000 \sqkm at 1,200 POIs and filtered them by the "Urban" class found in the ESA CCI dataset. The points were then stratified by the density in Global Human Settlement Layer S-MOD, which combines census and build-up data. The S-MOD classes were also rebalanced by cubic-root importance sampling. We wanted to oversample the Settlement class, as it is quite crucial for a very wide range of applications, from geography to economics to demographics to aid planning to disaster recovery.

The Global Human Settlement Layer, an in particular its SMOD product \citep{florczyk_ghsl_2019}, focuses on density of building and population. A titanic work, it is not without  shortcomings, documented by~\citet{van_den_hoek_satellite-based_2021}, especially regarding informal settlements. Those would make it ill-suited for our needs, were it not complemented by the UNHCR-provided POIs.

Within the 1200 POIs classified as Settlement under the IPCC classification in ESA CCI LC, we stratify with respect to GHSL SMOD.  ESA CCI LC is of higher spatial resolution than SMOD.  But IPCC is of coarser semantic resolution than SMOD. SMOD has indeed multiple settlement classes, visible in Figure~\ref{fig:summary_construction}.

The relation in spatial resolution is inverse of the relation in land
use resolution. Therefore it makes extra sense to stratify IPCC settlement tiles (which
are of a smaller size than SMOD tiles) according to the type of (larger)
SMOD tile they fall into. This guarantees that we will have sampled
``IPCC settlements tiles'' from a varying type of surrounding urban
environment (as provided by SMOD).

\subsection{AOIs from Under-Represented Key Users}

The second half of the dataset is obtained by sourcing 3,895 sq km around 1,062 Points Of Interest (POIs) from specialists of use-cases ignored by most existing datasets. For the rarer type of POIs, we sample 9 actual images in a non-overlapping grid centered on the POI.

NGOs and charities in Earth Observation interested about using recent Machine Learning articles are often met with the sobering answer ``Sorry, our models have never been trained on this type of landscape''. This is for example the case for informal settlements, refugee camps.  Most existing satellite imagery datasets, such as SpaceNet \citep{van_etten_spacenet_2018}, are often centred on urban features in large cities -- when they even cover beyond North America.

We also know that some of the highest benefits of this technique will go to social change actors who otherwise absolutely cannot afford high-resolution imagery.

For that reason,  we contacted high-potential users to inquire about the type of landscape they operate in. By obtaining examples of their past and ongoing points of interest (POIs) we ensure that our dataset will contain examples of the type of terrains in regions typically relevant to those users.

\textbf{Amnesty International}, focusing on human rights violations and conflict areas: 495 \sqkm at 22 POIs, with 9 AOIs around each POIs.  Amnesty International’s geospatial specialist Micah Farfour, specialised in satellite monitoring of human rights abuse and environmental abuse, provided us with 22 POIs across the globe, ranging from barracks to prisons to mass grave sites. To make up for their small number, we ordered each of them on a 3x3 grid of minimum-size tiles, i.e. a total of 22.5 \sqkm per POI, with a total of 495 \sqkm.

\textbf{ASMSpotter}, focusing on illegal mining in remote areas. 900 \sqkm at 40 POIs, with 9 AOIs around each POIs, mostly in South America.  ASMSpotter's data specialist  Moritz Besser provided 40 POIs which we ordered similarly to Amnesty's, covering 900 \sqkm.

\textbf{United Nations High Commissioner for Refugees (UNHCR)}, focusing on informal human settlements in disaster and conflict areas. 2,500 \sqkm at 1,000 POIs, one AOI per POI. Oregon State University professor Jamon Van de Hoek, specialised in human settlements and in particular informal settlements -- see e.g. his latest publication \cite{van_den_hoek_satellite-based_2021} studying the Global Human Settlement Layer --
kindly provided us with 3,000 POIs from the Persons of Concerns UNHCR Dataset~\citep{unhcr_2021}), which we downsampled to 1,000.

Since certain parts of the world have a particularly high density of UNHCR-register location (e.g. in the Middle East), we avoided overlapping AOIs by ensuring all our AOIs are at least 10 km apart, using rejection sampling.

\section{Imagery}
\label{sec:imagery}

\vspace{-1em}
\subsection{High-Resolution: Single Visit SPOT 6/7}

\vspace{-1em}
Each AOI has a single visit of SPOT 6/7 high-resolution imagery, over five spectral bands.  The panchromatic band has a resolution of 1.5 m/pixel, hence a 1,054x1,054 pixel image per 2.5 \sqkm AOI. The Red, Green, Blue, and Near Infrared bands are each at 6 m/pixel.
The date of the visit has been picked at random between 2017 and 2019 amongst the visits whose whole-scene cloud-cover is lower than 5\%. Because our AOIs are much smaller than a full SPOT scene, it is not absolutely guaranteed that the actual image has precisely 5\% cloud -- it is likely to be entirely empty of clouds. This provides a good target image to reconstruct in the case of super-resolution.
A glimpse of the imagery is show in Figure~\ref{fig:tile}.

\subsection{Low-Resolution: Multiple Revisits Sentinel 2}

\vspace{-1em}
For each high-resolution visit, we have 16 revisits. We picked revisits’ dates centered on SPOT visit. If more than 16 revisits per SPOT visit are needed, they are available via SentinelHub. The average time between revisits is 5 days. All 12 bands are available for each revisit. The resolution ranges from from 10 m/pixel (for RGB) to 60 m/pixel.

 We chose to not filter the low resolution Sentinel 2 revisits by their cloud coverage. This is to try and ensure the training distribution on the low resolution is similar to the real world use cases, where the user will want to rebuild at a given place at a given time. Algorithms should learn to ignore clouds and be able to assemble a view from the cloudless parts of the cloudy revisits.

\subsection{Temporal Selection: Matching AOI, Low-Res, and High-Res Imagery}

Unlike Sentinel2, SPOT is only available where and when it has been tasked. This raises two questions: How available and at which dates is SPOT high-resolution imagery of the POIs we have sampled? For each SPOT visit, how available is the Sentinel2 imagery, within which time window around the date of the SPOT visit?

Figure~\ref{fig:revisits} illustrates the number of Sentinel2 revisits (Y-axis) over each start and end date (up to +/- 6 months, coloured lines) around each SPOT visit (X axis at 0) of  each of the 22 Amnesty POIs (cell), sorted by latitude.
\begin{figure}
    \centering
    \includegraphics[width=1\textwidth]{images/n_s2_revisists.png}
    \caption{Number of Sentinel 2 visits for each SPOT visits on Amnesty International's AOIs}
    \label{fig:revisits}
\end{figure}

The takeaway from this admittedly elaborate figure is that in theory we could try to pick POIs and SPOT visit times that maximise the number of Sentinel2 imagery available within a fixed length time window, so as to have the richest training set.  We do indeed observe some variation in that regard: some of these lines go much higher than others.

However, this would induce an implicit bias of a nature hard to interpret. We also observe that within a POI, the discrepancy between the number of S2 revisits, while clearly present, is reasonable, with multiple SPOT revisits offering similar S2 availability.

Finally, biasing per Sentinel2 imagery would be akin to biasing per Sentinel2 cloud coverage: this would not be a fair representation of real-world use cases, and we would therefore be training our models for the wrong problem.

We therefore took the decision yet again to solve a harder problem than an optimally-curated dataset would make for, so as to be the closest to reality. To that effect, within a POI, we pick uniformly at random the SPOT visit to use as a reference.

Of course, one bias remains: we will not have imagery of POIs that have never been tasked by SPOT customers. While unfortunate, there is no way around it, short of using another high resolution product. We do have hope in two mitigating factors:
\begin{itemize}
\tightlist
  \item SPOT swath covers more than just the single POI, so we cover areas that are possibly more diverse than just the one precise point of interest to the SPOT customer.
  \item SPOT tasking means the POI exhibits features of activity interesting to at least the SPOT customer. SPOT customers might not have entirely the same interests as the users of our open-source package, but it is not unreasonable to assume that the features will be transferable. Therefore, this implicit sampling is actually a positive way to ensure interesting features.
\end{itemize}

\section{Putting it to Use: Baselines, Benchmarks, and Source Code}

\subsection{Super-Resolution Benchmark}
\label{sec:superres}\label{sec:benchmarks}
We  illustrate the use of this dataset on the task of superresolution. While there has been considerable recent progress in multi-frame super-resolution (see \citet{freddie0, freddie1, freddie2, freddie3}), since this is not meant as an exhaustive benchmark article, we only focus on three architectures: the single-image super-resolution architecture SRCNN \citep{dong2015image}, our multi-frame extension of SRCNN by collating revisits as channels, and a multi-spectral modification of the original HighResNet \citep{deudon_highres-net_2020} to handle multiple bands similarly to \citep{razzak_multi-spectral_2021}. We accelerated the latter by replacing the learned ShiftNet by a simple cached alignment search. The two core architectures for (multi-frame) SRCNN and HighResNet are visualised in Figure~\ref{fig:architectures}.
\begin{figure}
  \centering
  \includegraphics[width=.5\textwidth,valign=t]{MultiFrame_SRCNN.pdf}\hfill%
  \includegraphics[width=.5\textwidth,valign=t]{MultiSpectralHRNet.pdf}
  \caption{Multi-frame super-resolution architectures. Left: Multi-Frame SRCNN. Right: HighResNet. Single-frame SRCCN is achieved by using Multi-Frame SRCNN with a single revisit.\label{fig:architectures}}
\end{figure}

All implementations are our own, and the code as well as the trained models are released as part of our EOLearn plugin, described in the following section. Figure~\ref{fig:ci_metrics} shows the results. We refer to~\cite{martens_super-resolution_2019} for the metrics.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=.5\textwidth]{mean_with_ci.pdf}\hfill%
  \includegraphics[width=.5\textwidth]{over_images.pdf}\hfill
  \caption{Comparing three architectures and two metrics over the validation set for three independent training runs. PSNR: higher is better. SSIM: lower is better. Left: comparing the 95\% confidence interval of the mean of the metric. Right: comparing the 95\% quantiles of the metric over the whole distribution of the validation set. Note how, while the means seem significantly different, the variability across the distribution absolutely dwarfs any impact of the algorithms, pointing to the need for moving away from means-based benchmark, and to the need for better algorithms.\label{fig:ci_metrics}}
\end{figure}

\subsection{Toolbox and EO-Learn Plugin}
In addition to the dataset, we open-source a Python package designed for ease of use. It integrates with the widely EO-Learn Python package as one in EO-Learn pipeline, and provides
abundant tutorial notebooks.  We made this dataset to be reproducible and extensible, and thus include
notebooks covering data collection, training, and inference of all baselines, with standardized interfaces in Pytorch Lightning. We provide the ability to  sample new training data as needed.
 Particular care was paid to High-Efficiency Training, to make this world accessible with modest computing budget: our variant of HighResNet \textbf{trains in  30 minutes on a single V100 GPU}, thanks to our fast caching mechanism for 95\% average GPU usage.

\section{Discussion}
\label{sec:discussion}

\vspace{-1em}
\textbf{Potential social impact:}
A whole coverage of the potential social impact of computer vision and remote sensing could cover a whole book. However, in our specific case, the only sensitive data are the UNHCR locations, but these have already been released by UNHCR themselves.

\textbf{Known limitations and future work:}
Although we want a broad range of applications to be possible with our trained models, we know that the frequency of revisits constrains us to structures that change at a slower pace than Sentinel2 revisits -- one revisit every five days. This puts more emphasis on permanent structures, such as buildings or a form of land occupation, and rules out higher temporal variability use cases, such as immediate state of crops.

We did not enrich the dataset with rivers/harbours and liminal coastal space. This could be done by finding maps of water courses, and selecting locations at these intersections.

Stratification could be endlessly refined. For example we considered but discarded stratifying by use of the  Local Climate Zones (LCZ) database of the World Urban Database
\citep{stewart_local_2012}. LCZ tries to identify the type of buildings on a
local scale, and is available for Europe, The Americas, and contributed zones
\citep{demuzere_lcz_2021}. However, the coverage of the contributed zones is not as wide as the GHSL.

The UNHCR Persons of Concerns dataset is not guaranteed to be extremely precise: its POIs are more likely ``in the neighbourhood'' of the actual settlements. We mitigate this by covering a 2.5 \sqkm area, and by accounting for the fact that temporary settlements are often similar to the surrounding settlements, which are therefore representative.

\acksection{Project empowered by the ESA Phi Lab (\url{https://philab.phi.esa.int}) as part of the ESA-funded QueryPlanet project 4000124792/18/I-BG CCN3.
\ifdefined\decrementstc
\decrementstc
\fi

We are very grateful to Pierre-Philippe Matthieu (ESA Phi-Lab) and Nicolas Longepe (ESA Phi-Lab) for their support and belief in this project. To Grega Milcinski (Sinergise) and SentinelHub for taking us on board QueryPlanet. To Jamon Van Den Hoek (Oregon State University) for his expertise on GHSL and providing the UNHCR POIs dataset. To  Micah Farfour (Amnesty International) for POIs of humanitarian interest. To Moritz Besser (dida Datenschmiede GmbH) for their ASMSpotter data. And Peggy Fischer, Bryan Keary, and Montserrat Del Riego (ESA TPM) for their support in making the imagery available publicly.}