\section{Introduction}
The ability to detect and track the unfolding of disasters is often enabled by the Internet of Things (IoT), where devices sit out \emph{on the edge}, and are used to monitor a series of sensors. Whether it be detecting wildfires, earthquakes, extreme weather, or excessive pollution, these systems must work reliably within challenging tolerances. One of the most challenging aspects is that, because they are often battery powered, the IoT devices must work efficiently yet draw minimal amounts of power. Desktop and server processor designs, such as the Intel i7 and Xeon, have significant power draw requirements and as such are completely inappropriate for such workloads.

Micro-core architectures look to deliver high performance whilst addressing power consumption issues by implementing large numbers of simple cores running in parallel on a single chip. There have been some successes in deploying such architectures in the data centre for HPC workloads, such as the Sunway SW26010 of the Sunway TaihuLight  (\#3 June 2019 Top500 \cite{June2019TOP5002019}) and the 2048 core PEZY-SC2 of the Shoubu system B (\#1 June 2019 Green500 \cite{June2019GREEN5002019}). However, more generally micro-cores deliver significant benefits out in the field \cite{labowski2016implementing} running embedded style workloads. Whilst power is a critically important consideration, this is combined with the challenge that these workloads often operate within specific time constraints and as such an architecture that can deliver performance and power efficiency is critical.

It has already been demonstrated that reconfigurable architecture, such as field programmable gate arrays (FPGAs), can deliver IoT based solutions with significantly lower power consumption compared to central processing units (CPUs) or graphics processing units (GPUs), however a major restriction to leveraging FPGAs are the challenges in programming. An alternative is the use of soft-cores, where an FPGA is configured to appear and be programmed like traditional CPU core(s). This has resulted in the ability to specialise CPU architectures for the environment in question, in our focus the specific disaster which is of interest, without the impediment of expensive manufacturing costs associated with taping out physical chips. Furthermore, FPGA-based soft-cores enable researchers to experiment with CPU core designs, and this is important to understand the optimal design of an embedded micro-core architecture that will be deployed in the field.

When choosing an IoT architecture, whether it be a physical chip or soft-core, it is important that the choice made is a good one, however with over 40 implementations of the RISC-V architecture alone, the ability to assess competing designs simply and quickly is crucial. Furthermore, it is also important to consider not only different micro-core instruction set architectures (ISAs) but also variants within a particular processor ISA. This is especially true for RISC-V based designs due to the rich micro-architecture ecosystem. For instance, when selecting a RISC-V CPU there is a choice between many important aspects which all exhibit performance and power trade-offs. Without hard numbers from benchmarking to quantify the impact of such choices, it is difficult to make informed decisions.

Whilst it would appear that the best approach would be to run a set of the large number of currently available benchmarks on the micro-cores, the process isn't as simple as it would at first seem. There are a number of architectural features common to micro-cores that makes them significantly different from traditional CPUs and difficult to benchmark, including tiny amounts of on-chip RAM, and low-level knowledge specific to each device such as memory maps and linker files. Therefore, running existing benchmarks as-is on micro-cores is at best difficult and more than not often impossible. In order to be able to meaningfully compare and contrast different micro-core architectures, a benchmark framework is required to abstract over each architecture's idiosyncrasies and complexities.

It is such a framework and exploration of common micro-core architecture that this paper describes. This text is organised as follows, in Section \ref{background} we describe the background and related work already done in the community around benchmarking soft-cores, before we describe the selected CPUs in Section \ref{hardware} and our Eithne framework and benchmarks in Section \ref{benchmarks}. We provide a comparison of benchmark results, including soft-core performance, power consumption, and code density in Section \ref{results}. We highlight some of the challenges posed by the immaturity of the RISC-V ecosystem in Section \ref{challenges} and then conclude in Section \ref{conclusions}, before discussing further work.

\section{Background and related work}\label{background}
Whilst micro-core architectures have been around for for some time, the ability to perform a detailed exploration of the characteristics of the different options and how they compare and contrast against each other is somewhat limited. The experiments performed are often centred around very specific benchmarks or codes, and difficult to recreate on other technologies. This is typically even more pronounced for soft-cores, as there are often numerous configuration options to tune the architecture. For instance, in \cite{makni_comparison_2016} the authors detail a performance and resource comparison between Xilinx's MicroBlaze\cite{seely_microblaze_2017} and LEON3\cite{noauthor_leon3_nodate} to determine their suitability for multi-core embedded processors. These are both soft-cores and whilst they conclude that the performance between these technologies is similar, the survey omits to explore any of the configuration options provided by the MicroBlaze, which can have a significant impact on overall performance.

By providing configurable logic blocks sitting within a sea of interconnect, FPGAs enable a chip to be programmed so that it operates electronically based upon a software design. By including other facets, such as fast memory and digital signal processor (DSP) slices within these chips, complex and high performance circuitry can be temporarily configured. Soft-cores are a software description of a CPU, which is then used by the tooling to configure the FPGA to represent this electronically. Whilst it is generally accepted that FPGAs are more power efficient than CPUs and GPUs\cite{kestur_blas_2010}, work done in \cite{castells-rufas_energy_nodate} concluded that this power efficiency extends to soft-core processors running on FPGAs. This is important in IoT use for disaster detection, as it potentially addresses the programmability challenges of FPGAs, whilst maintaining many of the power efficiency benefits of FPGAs.

A key feature of the existing body of work benchmarking micro-core architectures is that comparisons have been performed on a very piecemeal basis, with little thought in making the process reusable across other technologies, benchmarks or metrics. Therefore, a key question for the urgent computing community when considering whether this is an appropriate technology to deploy in the field, is firstly whether micro-cores do provide performance and power benefits, secondly whether soft-cores can provide improved power efficiency due to their use of FPGAs, and lastly if one was looking to choose such a technology then what characteristics would they be most interested in.

\section{Hardware and metrics surveyed in this paper}\label{hardware}
The key objective of this research has been to determine the impact of micro-core architecture features and configurations, such as pipeline depth, and hardware floating point support, on metrics of interest to use of these architectures for disaster detection. Based on this application, we are interested in the following metrics:

\begin{itemize}
  \item Performance, detailing the performance that the micro-core can deliver
  \item Power consumption, exploring the amount of power that the micro-core draws
  \item Energy cost, which combines performance and power consumption to consider the overall energy used for a workload
  \item Resource usage, which is for soft-cores only and explores the amount of resources that are required. This is important as increased resources result in larger, more expensive chips.
  \item Code density, which explores the memory size required for the specific codes being studied. This depends heavily on the ISA, and increased memory requirements result in increased cost and power consumption
  \item Ecosystem maturity, exploring the availability of and support for, the general software ecosystem on the technology in question.
\end{itemize}

These criteria directly influence the suitability of micro-cores for execution on the edge for disaster detection.

\subsection{CPUs}
In this paper various micro-core architectures, both physical chips and soft-cores, along with numerous configurations have been compared against to understand the benefits and negatives of deploying the technologies in this specific area.
The following micro-core CPUs were chosen for comparison, based on availability and, for soft-cores, ease of integration into an FPGA multi-core design:
\begin{itemize}
	\item PicoRV32 (soft-core)
	\item Xilinx MicroBlaze (soft-core)
	\item ARM Cortex-A9 (hard processor)
	\item Adapteva Epiphany-III (hard processor)
\end{itemize}

Whilst this list is a fraction of the available soft-cores, within the context of this paper only a limited set can be realistically benchmarked and we selected such a short-list due to their differences and interest. However, the Eithne benchmark framework described in Section \ref{eithne} is extensible and can can built upon to perform analysis of other technologies of interest.

The PicoRV32\cite{wolf_picorv32:_2018}  is a von Neumann, non-pipelined open source RISC-V soft-core that supports the RV32IMC instruction set\cite{waterman_design_2016}. This is the simplest soft-core considered in this paper, and the von Neumann architecture means that both code and data resides in the same memory space. This is at odds with many other micro-core technologies which are based on a Harvard architecture which allocate code and data in different memories. The Xilinx MicroBlaze is an example of this Harvard architecture approach, but furthermore it is pipelined. This means that, unlike the PicoRV32 which must wait until each instruction has entirely finished executing before the next can start, the processing of an instruction is split up across multiple stages. Each pipelined stage can execute different instructions, completing a specific subset before passing it onto the next. Whilst this makes the design more complex, increasing power consumption and resource usage, it means that multiple instructions are being processed concurrently which can result in performance benefits.

The PicoRV32 only provides hardware support for integer arithmetic, and floating point arithmetic must be implemented in software, typically provided by the underlying runtime. However, the Xilinx MicroBlaze contains an optional hardware floating point unit (FPU), and enables direct execution of hardware floating point arithmetic by the CPU. This increases performance, but at the cost of increasing complexity and potentially power usage, so a key question is whether such a trade-off is worth it given the typical workloads found in edge-based disaster detection and tracking.

The ARM Cortex-A9 and Adapteva Epiphany-III are physical micro-core CPUs. An important question is whether physical chips, which run at higher clock frequencies, exhibit significant benefits over their soft-core brethren for IoT workloads. The Cortex-A9 is a Harvard, superscalar, out of order, pipelined, architecture with FPU. Superscalar means that the architecture is capable of completing more than one instruction per clock cycle, and it typically achieves this by leveraging out or order execution, where the order of physical instruction execution is determined only by dependencies rather than the order in which they are issued by the programmer. This is a very common approach, and the CPU will commit completed instructions in program order to ensure a consistent view of execution on behalf of the programmers. The Epiphany-III is rather similar, with 16 von Neumann, superscalar cores, and 32KB of scratchpad static RAM (SRAM) per core.

\subsection{Hardware boards}
The Xilinx PYNQ-Z2\cite{noauthor_pynq-z2_nodate} single board computer (SBC) was selected as the platform for all soft-core devices explored in this paper. The on-board Xilinx Zynq 7020 FPGA contains a dual-core 650MHz ARM Cortex-A9 running Linux, accessing 512MB of on-board but off-chip dynamic RAM (DRAM). The programmable logic of the Zynq 7020 contains 53,200 configurable LookUp Tables (LUTs), 4.9Mb of block RAM (BRAM) and 220 DSP slices which are commonly used for floating point arithmetic. Whilst one can configure a small number of cores with a large amount of memory per core, realistically eight cores, each with 32KB memory, is the best balanced that can be achieved, and that is the configuration we have selected. However, the reader is able to reuse our benchmarks and framework to explore the performance of other configurations. The Adapteva Parallella \cite{noauthor_parallella_2013} was used to host the Epiphany-III, providing a 650MHz dual-core ARM Cortex-A9 running Linux, with 1GB of DRAM and the 600 MB/s link to the Epiphany co-processor.

\section{Software benchmarks}\label{benchmarks}
Three benchmarks have been selected to compare the overall performance of the selected technologies. We have selected these benchmarks to test different facets of the technology, firstly the LINPACK single-precision \cite{dongarra_linpack_2003} benchmark has been chosen due to its role in stressing raw floating point performance, which will be interesting to explore in the context of the micro-cores. LINPACK determines the performance of a system in millions of floating point operations per second (MFLOPS) by performing LU factorization as follows \cite{linpack}

\begin{enumerate}
    \item Set up a random matrix $A$ of size $N$
    \item Set up the vector $X$ which has all values set to $1$
    \item Set up a vector $B$ which is the product of $A$ and the vector $X$
    \item Compute an LU factorization of $A$
    \item Use the LU factorization of $A$ to solve $A * X = B$
\end{enumerate}

The number of floating point operations required for the two LU factorizations is \[ops = 2 * N*N*N / 3 + 2 * N * N\] and the MFLOPS value is calculated by \[MFLOPS = ops / ( t * 1000000 )\]

In addition to LINPACK we have also implemented the Discrete Fourier Transform (DFT) and Fast Fourier Transform (FFT) benchmarks \cite{fftbenchmark}. These were chosen due to their relevance in the embedded community, and also represent a very common workload across numerous disaster scenarios, for instance interpreting streaming sensor data to identify any anomalies that might represent some sort of emergency, such as an earthquake. These Fourier benchmarks compute the forward and backwards transform of data.

Whilst we have chosen these particular benchmarks to drive the exploration of micro-core characteristics in this paper, it is important to note that the Eithne benchmark framework as described in Section \ref{eithne} is easily extensible with other benchmarks that suit specific disaster use-cases.

\subsection{Eithne Benchmark framework}\label{eithne}
To minimise the impact of different micro-core architectures on the benchmark results, the Eithne\footnote{Eithne (/\textipa{En{\textsuperscript{j}}9}/ "enya"):  Gaelic for "kernel" or "grain".} framework has been developed which enables a suite of benchmarks to be run across many different devices with limited or no modification required. The framework uses a single execution model across all devices, where the kernels are transferred to the device to be benchmarked, and a listener is launched awaiting data transfers and launch requests from the host benchmark application. This ensures that the communications link architecture, such as shared memory or universal asynchronous receiver-transmitter (UART), does not significantly impact the measurement of kernel execution. Data transfers are separated from kernel launch requests to enable the measurement of the communications link latency and bandwidth. Most importantly, this framework has been developed with extensibility in mind, where new devices, benchmarks, or metrics can be trivially added.

 \begin{figure}[h!]
	\centering
	\includegraphics[width=0.45\textwidth]{images/Eithne_arch_v1d6.png}
	\caption{Eithne framework architecture}
	\label{fig:eithne}
\end{figure}

Figure \ref{fig:eithne} illustrates the Eithne framework's layered architecture, provided as a stack of functionality. Adding a new benchmark,
device or communication mechanism only requires that specific layer of the framework is modified, with all other layers remaining unchanged. This not only simplifies supporting new technologies, benchmarks and metrics, but also isolates the remaining code, reducing the regression testing effort required.

The high-level flow for the LINPACK benchmark is outlined in Figure \ref{fig:linpackflow} and in this paper we use LINPACK to outline the modifications required to run benchmarks on micro-core architectures using the Eithne framework.
\begin{figure}[h!]
	\centering
	\includegraphics[width=0.45\textwidth]{images/Framework_flow_v1d0.png}
	\caption{LINPACK benchmark host / device  flow}
	\label{fig:linpackflow}
\end{figure}

\subsection{LINPACK}\label{linpack}
A single-precision C version of the LINPACK\_BENCH benchmark \cite{linpack} was modified to run on the Eithne framework by separating out the \texttt{sgesl} and \texttt{sgefa} kernels, and their support functions, such as \texttt{saxpy} and \texttt{ddot}, from the rest of the initialisation, verification and timing code. These kernel codes were placed in a \texttt{kernels.c} file, along with the Eithne framework initialisation function which is outlined in Listing \ref{lst:kernelinit}. This code first registers the kernel input and output data variables;  \emph{a}, \emph{b}, \emph{ipvt}, \emph{job}, and \emph{info} in Listing \ref{lst:kernelinit}. These are then compiled and downloaded to the device, with kernel execution and data transfers being performed by the framework.

\begin{lstlisting}[style=cstyle, caption={Kernel framework initialisation function}, label={lst:kernelinit}]
void kernel_init(EithneTargetId id, EithneSharedMem buffer) {
  EithneKernel kernels[] = { sgefa, sgesl };

  EITHNE_INIT_DEVICE(vars,id,buffer+EITHNE_DATA_OFFSET,buffer,kernels);

  EITHNE_REGISTER_ARRAY(vars,A,EITHNE_FLOAT_ARRAY,a,N*LDA);
  EITHNE_REGISTER_ARRAY(vars,B,EITHNE_FLOAT_ARRAY,b,N);
  EITHNE_REGISTER_ARRAY(vars,IPVT,EITHNE_INTEGER_ARRAY,ipvt,N);
  EITHNE_REGISTER_SCALAR(vars,JOB,EITHNE_INTEGER,job);
  EITHNE_REGISTER_SCALAR(vars,INFO,EITHNE_INTEGER,info);

  EITHNE_START_LISTENER;
}
\end{lstlisting}

The remaining LINPACK code was modified to use the Eithne framework API calls to allocate memory, register variables, transfer data and launch kernels. The host initialisation code for kernels running on the Adapteva Epiphany co-processor, Xilinx MicroBlaze and PicoRV32 soft-cores is outlined in Listing \ref{lst:hostinit}.

\begin{lstlisting}[style=cstyle, caption={Host framework initialisation code}, label={lst:hostinit}]
buffer = EITHNE_ALLOC_MEM(sizeof(float)*N*LDA);

EITHNE_INIT_HOST(vars,HOST_ID,buffer+EITHNE_DATA_OFFSET,buffer);
EITHNE_INIT_CORES(16);
EITHNE_START_CORES(16);

EITHNE_REGISTER_ARRAY(vars, A, EITHNE_FLOAT_ARRAY, a, N*LDA);
EITHNE_REGISTER_ARRAY(vars, B, EITHNE_FLOAT_ARRAY, b, N);
EITHNE_REGISTER_ARRAY(vars, IPVT, EITHNE_INTEGER_ARRAY, ipvt, N);
EITHNE_REGISTER_SCALAR(vars, JOB, EITHNE_INTEGER, job);
EITHNE_REGISTER_SCALAR(vars, INFO, EITHNE_INTEGER, info);
\end{lstlisting}

Listing \ref{lst:sgefa} outlines the addition of the required Eithne framework API calls to launch and time the \texttt{sgefa} kernel to the existing LINPACK code. The kernel function parameters are replaced by the \texttt{EITHNE\_SEND} and \texttt{EITHNE\_RECV} API calls. However, due to the previous registration API calls, the underlying kernel input and output variables such as \emph{a} and \emph{ipvt} are used unchanged, thereby minimising the impact to the existing codes.

\begin{lstlisting}[style=cstyle, caption={Executing and timing the LINPACK sgefa kernel}, label={lst:sgefa}]
/* Input to SGEFA */
EITHNE_SEND(vars, TARGET_ID, A);

t1 = cpu_time ( );
EITHNE_EXECUTE(TARGET_ID, SGEFA);
t2 = cpu_time ( );

/* Output variables from SGEFA */
EITHNE_RECV(vars, TARGET_ID, A);
EITHNE_RECV(vars, TARGET_ID, IPVT);
EITHNE_RECV(vars, TARGET_ID, INFO);
\end{lstlisting}

In all experiments \emph{N} was set to 20, this was found to be an appropriate parameter setting which takes into account the fact that the micro-cores have very limited memory. As the LINPACK matrix order size \emph{N} impacts the overall MFLOPS result, we felt it was important to keep this consistent across all devices to enable a like-for-like comparison of performance.

\subsection{DFT and FFT}\label{fft}
Both DFT and FFT benchmarks \cite{dft_fft_benchmark} were implemented using the Eithne framework. The main \texttt{dtf} and \texttt{fft} kernels and support functions were extracted from the surrounding initialisation and timing code and placed in a \texttt{kernels.c} file, along with the kernel framework code initialisation code similar to that outlined in Listing \ref{lst:kernelinit} but updated to reflect the FFT and DFT variables and kernels.

Eithne kernels have a void parameter list as the input and output variables are transferred by the framework. Since the \texttt{fft} kernel is recursive, a wrapper function was created to isolate the changes to the kernel code, as shown in Listing \ref{lst:fftkernel}.

\begin{lstlisting}[style=cstyle, caption={FFT kernel and wrapper}, label={lst:fftkernel}]
void fft_wrapper(void) {
  fft(xfer_sig, xfer_f, xfer_s, 1<<N, xfer_inv);
}

void fft(const Comp *sig,Comp *f,int s,int n,int inv) {
    int i, hn = n >> 1;
    Comp ep = comp_euler((inv ? PI : -PI)/(float)hn),ei;
    Comp *pi = &ei, *pp = &ep;
    if (!hn) *f = *sig;
    else
    {
        fft(sig, f, s << 1, hn, inv);
        fft(sig + s, f + hn, s << 1, hn, inv);
        pi->a = 1;
        pi->b = 0;
        for (i = 0; i < hn; i++)
        {
            Comp even = f[i], *pe = f + i, *po = pe + hn;
            comp_mul_self(po, pi);
            pe->a += po->a;
            pe->b += po->b;
            po->a = even.a - po->a;
            po->b = even.b - po->b;
            comp_mul_self(pi, pp);
        }
    }
}
\end{lstlisting}

\section{Benchmark results}\label{results}
Based upon the software framework described in Section \ref{eithne}, the benchmarks of Section \ref{linpack} and Section \ref{fft} have been executed on the hardware described in Section \ref{hardware}. In this section we use these results as a tool to explore, and compare and contrast, the different characteristics of these technologies and consider their suitability for use on the edge in the context of urgent computing.

\subsection{Performance}
\label{sec:performance}
\subsubsection{LINPACK} Figure \ref{fig:perfpower} illustrates the performance of different micro-core technologies running the LINPACK benchmark via the Eithne framework. It can be seen that the physical micro-core processors have significantly higher performance than soft-cores, which can be mainly explained by the higher clock rates of between five and six times. However, for the Epiphany-III, there are other performance design features that expand this gap that will be covered in Section \ref{sec:dft_and_fft} below.

It is interesting to note that the power consumption for all soft-cores running on the Zynq 7020 is almost the same, even when the soft-cores have very different performance profiles, as is the case for the integer-only PicoRV32 and MicroBlaze with FPU. This initial analysis would seem to support the view that FPGA-based designs are more power efficient than physical CPUs, per \cite{kestur_blas_2010}. However, as we will discover in Section \ref{sec:energy_cost}, the overall power required to execute the code to completion is perhaps a more important figure influencing the choice of micro-cores deployed in the field.

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.45\textwidth]{images/microcore_performance_power_v4.png}
	\caption{Micro-core LINPACK benchmark performance / power consumption}
	\label{fig:perfpower}
\end{figure}

\subsubsection{DFT and FFT}
\label{sec:dft_and_fft}
The performance results for the DFT and FFT benchmarks detailed in Table \ref{tbl:fftperf} are somewhat similar to those seen for LINPACK, where the Epiphany-III physical processor is 653 times faster than the PicoRV32. The integer-only MicroBlaze is 2.6 times faster than the PicoRV32 due to its pipelined architecture, and the hardware floating point MicroBlaze is 13.7 times faster than the PicoRV32. Assuming that the performance scaled linearly with clock frequency, the Epiphany would be 109 times faster than the PicoRV32, and 7.5 times faster than the MicroBlaze, when running at a clock speed of 100MHz.

\begin{table}[h!]
\caption{DFT / FFT Micro-core Performance}
\centering%
\footnotesize
\begin{tabular}{| c c c |}
\hline
\textbf{Device} & \textbf{DFT (seconds)} & \textbf{FFT (seconds)} \\
\hline
\textbf{PicoRV32} & 0.11096 & 0.11130 \\
\textbf{MicroBlaze} & 0.04259 & 0.04266 \\
\textbf{MicroBlaze \& FPU} & 0.00808	& 0.00825 \\
\textbf{Epiphany-III} & 0.00017 & 0.00017 \\
\hline
\end{tabular}
\label{tbl:fftperf}
\end{table}

When comparing the PicoRV32 against the MicroBlaze, one can observe the performance advantages of a pipelined architecture and hardware floating point support. However, results from the Epiphany-III highlight the further benefits of a 64 register, superscalar CPU that can execute two floating point and a 64-bit memory load operation every clock cycle \cite{noauthor_epiphany_2014}.

\subsection{Power consumption}
In Section \ref{sec:performance} it was seen that more complex architectures deliver significantly better performance than simpler ones, which is not a major surprise. Furthermore, the fact that physical processors can operate at a much higher clock frequency also delivers numerous performance benefits. However, a key question is whether there are any power consumption disadvantages of such designs, and this was calculated by measuring the voltage and amperage of each board running the benchmarks using a wiring harness and two UNI-T UT60E multimeters.

\subsubsection{LINPACK}
Power consumption for the LINPACK benchmark, along with the measured floating point performance (in MFLOPS), is illustrated in Figure \ref{fig:perfpower}. From the results, it can be seen that the (integer only) MicroBlaze soft-core is five times more energy efficient than the Epiphany-III and 4 times more so than the Cortex-A9. The overall number of cores is likely to play a factor here, namely the fact that the Epiphany-III has 16 cores, the Cortex-A9 has 2, and the MicroBlaze was configured with 8 cores.

When the clock rate (MicroBlaze 100Mhz, Epiphany 600MHz and Cortex-A9 650MHz) and the number of cores is taken into account, then we find for that each core, the Epiphany is approximately two times more power efficient than the MicroBlaze, but the MicroBlaze is six and a half times more power efficient than the Cortex-A9. By contrast, the PicoRV32 is about two times less power efficient than the MicroBlaze, and six times less power efficient than the Epiphany-III, but is still around two times more power efficient than the Cortex-A9.

The fact that the much simpler PicoRV32 drew more power than the more complex MicroBlaze surprised us, but it can most likely be explained by the fact that more complex (AXI4) bus support logic is required for a multi-core PicoRV32 design, whereas the multi-core MicroBlaze uses a much simpler Local Memory Bus (LMB) design.

As expected, power consumption increases with clock rate. However, as outlined, the Watts, MHz and core results for the Epiphany-III are twice as efficient than for the MicroBlaze, making it the most power efficient core design of those tested. This may be due in part to general power inefficiencies in the soft-core designs or the impact of the greater instruction decoding logic of the soft-cores versus the Epiphany-III. For instance, the MicroBlaze has 129 instructions and the Epiphany-III only has 41, and this is a consideration when one is running a micro-core in the field for this workload, namely based on the specialised nature is it possible to significantly limit the number of instructions?

\subsubsection{DFT / FFT}
The larger DFT / FFT benchmark kernel binaries, due to the inclusion of \texttt{sin()} and \texttt{cos()} functions, required more BRAM than is available on the 8 soft-core devices. Therefore, 4 core designs for the MicroBlaze and PicoRV32 were synthesised, each with 128KB of scratch-pad memory. The MicroBlaze design also included hardware floating point support, MicroBlaze+FPU, implemented using FPGA DSP slices, unlike the integer-only MicroBlaze used in the 8-core bitstream for the LINPACK benchmark.

It is interesting to compare the impact of enabling hardware floating point support in the MicroBlaze, and this impacted power consumption significantly, resulting in the simpler PicoRV32 drawing 14\% less power than the MicroBlaze+FPU running the DFT and FFT benchmarks.

\begin{table}[h!]
\caption{DFT and FFT Benchmark Power Consumption}
\centering%
\footnotesize
\begin{tabular}{| c c c |}
\hline
\textbf{Device} & \textbf{Idle (Watts)} & \textbf{Load (Watts)} \\
\hline
\textbf{PicoRV32} & 2.05 & 2.19 \\
\textbf{MicroBlaze} & 2.36 & 2.54 \\
\textbf{Epiphany-III} & 3.46 & 4.36 \\
\hline
\end{tabular}
\label{tbl:fftpower}
\end{table}

As detailed in Table \ref{tbl:fftpower}, the Epiphany-III uses approximately 1.8 times the power of both the PicoRV32 or MicroBlaze+FPU whilst running the benchmark. However, as for LINPACK, when we consider overall efficiency per core, we find that the Epiphany delivers a lower figure of 0.27 Watts/core at 600MHz against 0.55 Watts/core at 100MHz for the PicoRV32 and 0.63 Watts/core at 100MHz for the MicroBlaze+FPU. Bearing in mind that CPU power consumption increases with frequency \cite{datta_cpu_2014}, the Epiphany-III draws significantly less power than the soft-cores when scaled to the same clock rate of 100MHz, estimated at 0.045 Watts/core at 100MHz, a fourteen times greater power efficiency than the PicoRV32.

\subsection{Energy cost}
\label{sec:energy_cost}
Whilst the absolute power consumption of a processor is important, the \textit{power consumption to solution} is also of great interest. For instance, the power consumption required for processing streams of input data to generate a result which determines whether a disaster is unfolding or not. Effectively, such a metric describes the overall energy cost of the system, and power consumption to solution or energy (E) is defined as: \[E = Pt\]

Figure \ref{fig:power2solution} outlines the overall energy consumption for the FFT benchmark running on the selected micro-cores. The poor performance of the PicoRV32 results in a 328 times greater overall energy consumption than the much more powerful Epiphany-III processor for the same FFT kernel. The MicroBlaze+FPU, which is around fourteen times faster that the PicoRV32, uses around twelve times less energy overall to run the FFT benchmark than the PicoRV32. Therefore it can be seen here that the significant performance advantages of the MicroBlaze+FPU and Epiphany-III, mean that whilst the absolute power consumption is larger than the PicoRV32, there is still an overall energy benefit.

\begin{figure}
	\centering
	\includegraphics[width=0.45\textwidth]{images/Power2solution_v2.png}
	\caption{Micro-core FFT benchmark energy consumption}
	\label{fig:power2solution}
\end{figure}

\subsection{FPGA area / resources}
\label{sec:usage}
Resource usage is important, as it dictates the overall size of the FPGA chip required to host specific soft-cores. Put simply, small FPGAs such as the Spartan are cheap \cite{alfke2009xilinx}, whereas as the number of resources scale up, the overall cost increases significantly. Table \ref{tbl:fpgaarea} details the FPGA resources utilised by each of the 8-core soft-core designs, where all of the soft-cores were configured to have a total of 64KB of on-chip RAM. This use of block RAM (BRAM) on-chip memory is the primary limiting factor in scaling the number of micro-cores further on the Zynq 7020.

The integer-only MicroBlaze and PicoRV32 cores have a comparable LUT (look-up table) utilisation of 38\% and 35\% respectively. The PicoRV32 has been configured to support the RISC-V M (ISA MUL, DIV and REM instructions) and uses 15\% DSP slices for its implementation, whereas the MicroBlaze integer-only core does not use any DSP slices. The slightly increased LUTRAM and FF (flip-flop) requirements of the MicroBlaze over the PicoRV32 are likely to be attributable to the pipeline support and additional decoding logic required for the larger MicroBlaze instruction set versus the simpler RISC-V IMC instruction set.

The hardware floating point version of the MicroBlaze, MicroBlaze+FPU, uses 47\% more LUTs, 30\% more FF and 22 times more DSP slices than the integer-only MicroBlaze design. This represents a very significant increase in resources, and whilst the increase in DSP slice utilisation is to be expected, as this is the primary way in which floating point is executed by the FPGA, the increase in LUT usage was unexpected. Given an unlimited amount of on-chip memory, these figures would mean that the hardware floating point MicroBlaze could scale to 36 cores and the PicoRV32 to 53 cores on the Zynq 7020.

\begin{table}[h!]
\caption{MicroBlaze and PicoRV32 Soft-core Z7020 Resource Percentage Utilisation}
\centering%
\footnotesize
\begin{tabular}{| c c c c c c |}
\hline
\textbf{Soft-core} & \textbf{LUT} & \textbf{LUTRAM} & \textbf{FF} & \textbf{BRAM} & \textbf{DSP} \\
\hline
\textbf{PicoRV32} & 35\% & 3\% & 13\% & 91\% & 15\% \\
\textbf{MicroBlaze} & 38\% & 6\% & 23\% & 91\% & 0\% \\
\textbf{MicroBlaze+FPU} & 56\% & 7\% & 30\% & 91\% & 22\% \\
\hline
\end{tabular}
\label{tbl:fpgaarea}
\end{table}

\subsection{Code density}
On-chip memory is often a major limiting factor when it comes to micro-cores. This is especially true with soft-cores, as from a resource usage perspective it was demonstrated in Section \ref{sec:usage} that BRAM is the factor that determines the overall number of soft-cores that can be implemented on a particular FPGA. Therefore, the size of the resultant kernel binaries is an important consideration with respect to the choice of processor instruction set architecture.

All of the micro-cores selected for the benchmarks use the GNU C Compiler (GCC). Whilst it would be ideal to use the same version of GCC across all devices, some micro-cores, such as the Epiphany-III and RISC-V, only support specific versions of GCC. Therefore, for the benchmarks, we used the recommended version of GCC for each micro-core architecture. Apart from micro-core specific linker files and compiler flags (little-endian for the MicroBlaze), the GCC compiler options were identical across all architectures. Due to the aforementioned RAM limitations, the kernel codes were optimised for size (GCC option \texttt{-Os}), rather than for speed (GCC option \texttt{-O3}).

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.45\textwidth]{images/fft_kernel_size.png}
	\caption{Micro-core FFT benchmark kernel size}
	\label{fig:kernelsize}
\end{figure}

Figure \ref{fig:kernelsize} illustrates the FFT kernel binary size produced by GCC for the micro-cores. The three soft-core options are roughly equivalent at 109KB for the PicoRV, 102KB for the integer-only MicroBlaze and 100KB for the hardware floating point MicroBlaze. The kernel binary for the MicroBlaze with hardware floating point is slightly smaller than the binary for the integer only core, as that must include software floating point emulation routines. However, at 23KB, the FFT kernel binary size for the Epiphany is significantly smaller than for the soft-cores.

This difference is stark, and a surprising aspect of this comparison is that a larger instruction set does not seem to result in a smaller binary file size. For example, the RV32IMC ISA has 76 instructions, the MicroBlaze ISA 129 and the Epiphany-III ISA 41 instructions. Therefore, one would assume that the Epiphany-III binary would be the largest, as fewer instructions must be explicitly composed together. Yet the Epiphany has, by far, the smallest binary file size.

In fact, this supports the assertion in \cite{mutigweInstructionSetUsage2013} where the authors state that one does not need more than 100 instructions, and further research in  \cite{mutigweInstructionSetUsage2013} also demonstrates that GCC only uses between 5-20\% of all the instructions across a range of processors (x86, x86-64, PowerPC and MIPS). When one considers the silicon area and power requirements of a CPU's instruction set decoding logic, it would seem prudent to keep a micro-core's ISA as small as possible. This could be a key factor in the Epiphany-III's impressive performance and power consumption figures that we obtained for the LINPACK, DFT and FFT benchmarks.

\subsection{Maturity of the software ecosystem}\label{challenges}
The Epiphany-III and MicroBlaze are commercial products and, the MicroBlaze especially which is supported by Xilinx, offer a fairly large software ecosystem. This is especially the case for C compilation, which is very popular in embedded computing. Furthermore, both these technologies can be obtained \emph{off the shelf}, and simple to operate. However, whilst there are over forty RISC-V based implementations available, for this paper, we have found that actually being able to configure multi-core FPGA designs with these is a different matter. There are a number of challenges to successfully using RISC-V soft-cores, ranging from the ability to synthesise the source files, often designed for simulation rather than for use as an IP component within an FPGA bitstream, to the immaturity of the development tools.

During the work conducted in this paper, we have observed a number of idiosyncrasies of the RISC-V ecosystem that we found especially challenging.

\subsubsection{Lack of RISC-V soft-core verification}
Many of the available open source RISC-V soft-cores have not been verified against the published RISC-V ISA standards. For example, the VectorBlox ORCA \cite{noauthor_risc-v_2019} is stated \cite{risc-v-list_2019} to support the RV32IM ISA and provides options for hardware multiply and divide instructions, but does not implement the REM instruction specified in the M ISA option. The  consequence is that that codes compiled by the official RISC-V GCC toolchain with this option enabled will freeze on the ORCA, and resulted in it being excluded from our comparisons.  Tracking down this type of issue is time consuming and beyond the expertise of many programmers due to the lack of support for debugging tools on a number of the available soft-cores, such as the PicoRV32 and ORCA.

\subsubsection{Low-level GCC linker file knowledge}
The RISC-V uses register \texttt{x2} as the stack pointer, which needs to be initialised before C functions are called. This is usually performed by an assembly language routine called when the binary is loaded onto the core. For simple codes, this initialisation routine can be compiled or linked without issue. However, for more complex codes requiring the \texttt{-ffunction-sections -Wl,-gc-sections} GCC optimisation flags to reduce the size of the binary, the GCC optimiser will remove the initialisation code because it is managed in the linker file and the code will fail to run on the core. Ensuring that GCC does not remove this code but in a manner where the compiler can still perform size optimisation requires in-depth understanding of segments and modification of the RISC-V linker file. Therefore, compiling and running codes such as the LINPACK and FFT benchmarks on RISC-V soft-cores is far more involved than simply taking existing codes and recompiling them for the RISC-V.

\subsubsection{Inconsistent RISC-V compressed instruction set support}
Although RISC-V supports a compressed instruction set (RV32C) that can reduce the size of a compiled binary by up to 30\% \cite{patterson_risc-v_2017}, the majority of the 32 bit RISC-V soft-cores do not support compressed instructions. When we consider that the FFT kernel binaries compiled using the RV32IMC ISA were too large to fit in 64KB, and as such we had to reduce our design down to four 128KB cores, the provision of compressed instruction support could provide significant benefits for code density.

\section{Conclusions and further work}\label{conclusions}
Micro-core architectures have significant potential in the monitoring of the environment, operating on streams of sensor data to detect disasters. Running on the edge, important characteristics of the technology differ greatly from those typically found in the data-centre. However, given the choice of architectures available, an important question is which one is most applicable, the answer to which will often depend upon the situation in question.

In this paper, to address this, we have introduced a framework that greatly simplifies the benchmarking of micro-cores. Layering our design makes it far easier to plugin new architectures, benchmarks, and metrics, which until now has been very difficult to achieve. We then ran a series of benchmarks across some of the more popular micro-core architectures, exploring resulting measurements based upon characteristics most important in the IoT domain.

We demonstrated that, irrespective of clock frequency, the Epiphany-III physical chip delivers significant performance and is more energy efficient than the other architectures. Given FPGA processor implementations have between eighteen and twenty six times greater circuit delay than their custom CMOS \cite{wong_comparing_2011} equivalents, this is not surprising, however it does illustrate that performance is important when optimising for energy efficiency. However, in terms of absolute power draw the PicoRV32 consumed around half the power of the Epiphany-III. We have also shown that the most important limiting factor for soft-cores tends to be the amount of on-chip memory, BRAM, present. Therefore, the size of kernel binaries produced for a particular processor's ISA is a critical limitation to the number of viable cores that can be implemented.

Whilst \cite{castells-rufas_energy_nodate} found that their many soft-core processor was highly energy efficient, our results paint a different picture. They estimated the energy efficiency of their customised soft-core was 1623 MOPS/Watt, compared to 26 MOPS/Watt for an i7 running eight threads. However, we measured the MicroBlaze soft-core at 6 MFLOPS/Watt and the ARM Cortex-A9 at 55 MFLOPS/Watt. There is a difference here, for instance \cite{castells-rufas_energy_nodate} measured operations per second, and us explore floating point operations per second, but it is demonstrated by this work that the power efficiency of physical processors is at least nine times higher than the soft-cores.

Further work includes extending our benchmarking framework to include other micro-core architectures, and to explore other relevant codes to disaster detection. Specifically, we think that connecting to real-world sensors and supporting the streaming in of data would be a sensible next benchmark. This would be another possible metric, how fast data can be streamed into a chip, and one where the soft-cores might have an edge due to the large number of IO connections that some FPGAs possess. Eithne currently separates data communications / transfers from the execution of kernels, therefore it has the support to enable the measurement of data bandwidth. Furthermore, there are embedded GPUs, such as NVIDIA's Jetson that would be interesting to also compare against. In terms of the micro-core architectures selected, there are higher performance RISC-V implementations, and exploring some of the customisable CPUs developed by SiFive would also be of interest.

Therefore, we conclude that micro-core architectures have potential for use in disaster detection, however this is best done with physical chips rather than soft-cores. Our hypothesis that soft-cores could provide the best of all worlds; high performance, energy efficiency and programmability is simply not the case. For production edge computing then one should utilise physical chips, such as the Epiphany-III, with soft-cores useful for prototyping and the exploration of architectural configurations.

\bibliographystyle{./bibliography/IEEEtran}


\section{Introduction}

To start off, we introduce a class $\mathcal{C}$ of open 3-manifolds which we view as a candidate for `the smallest class of open 3-manifolds for which the classification problem is interesting.' It is large enough so that exotic phenomena due to the topology at infinity can occur; yet it is small enough so that algorithmic problems---in particular the homeomorphism problem---make sense, and seem to have a decent chance of being decidable.

Throughout the paper we work in the PL category. Let $(X,Y,F_+,F_-,f,g)$ be a 6-tuple with the following properties:
\begin{itemize}
\item Both $X$ and $Y$ are connected, orientable, compact 3-manifolds.
\item $\partial X$ is connected.
\item $\partial Y$ has exactly two components, which are $F_+$ and $F_-$.
\item $f$ is an orientation-reversing homeomorphism from $\partial X$ to $F_-$.
\item $g$ is an orientation-preserving homeomorphism from $F_+$ to $F_-$.
\end{itemize}

To such a 6-tuple, we associate an open 3-manifold obtained by gluing together $X$ and an infinite sequence of copies of $Y$ using the homeomorphisms $f,g$. More precisely, we set
$$M(X,Y,F_+,F_-,f,g) = X \cup_{f_0} Y \times \{0\} \cup_{g_0} T \times \{1\} _{g_1} Y \times \{2\} \cup_{g_2} \cdots$$
where the gluing homeomorphisms are given by $f_0(x)=(f(x),0)$ for all $x\in X$, and $g_n(x,n)=(g(x),n+1)$ for all $x\in Y$ and $n\in\mathbf{N}$. We denote by $\mathcal{C}$ the class of 3-manifolds obtained in this way.

\begin{rems}
\item Every 3-manifold in $\mathcal{C}$ is orientable, connected, and one-ended. Moreover, it has an exhaustion by compact submanifolds with connected boundary of fixed genus. We can call \bydef{genus} of a manifold $M\in\mathcal{C}$ the minimal genus of $\partial X$ in a presentation of $M$ as $M(X,Y,F_+,F_-,g,f)$.
\item We can fix triangulations of $X$ and $Y$ with respect to which $f$ and $g$ are simplicial, so that algorithmic problems are well-defined. For the same reason, there are countably many manifolds in $\mathcal{C}$ up to homeomorphism.
\item The class $\mathcal{C}$ contains the original Whitehead manifold~\cite{whitehead:unity} as well as many other contractible 3-manifolds not homeomorphic to $\mathbf{R}^3$. It does not contain those with infinite genus. Nor does it contain all Whitehead manifolds of genus~1, since there are  uncountably many of those, as shown by McMillan~\cite{mcmillan}.
\item One can define in a similar way a class $\mathcal{C}_n$ for each dimension $n$, so that $\mathcal{C}=\mathcal{C}_3$. This construction is a special case of that of~\cite{maillot:homeosurf} of manifolds associated to a topological automaton. The automaton has two states corresponding to $X,Y$. Thus the classification for $\mathcal{C}_2$ follows from the main theorem of~\cite{maillot:homeosurf}. The one-ended case, however, is much simpler than that; it is a straightforward consequence of the Kerekjarto classification theorem. Indeed, an orientable, one-ended surface is classified by its genus $h\in [0,+\infty]$, which is easily seen to be $+\infty$ if $Y$ has positive genus, and equal to the genus of $X$ otherwise.
\end{rems}

In this paper, we are interested in the question of which, among the exotic phenomena concerning open 3-manifolds, occur in the class $\mathcal{C}$. We already mentioned that $\mathcal{C}$ contains contractible manifolds which are not homeomorphic to $\mathbf{R}^3$. For instance, the original Whitehead manifold has genus 1, the manifold $X$ being a solid torus, and $Y$ being the exterior of the Whitehead link in $S^3$.

Likewise, the manifold $M_1$ constructed by the author in~\cite{maillot:examples} is easily seen to belong to $\mathcal{C}$. It has genus $1$, the manifold $X$ being again a solid torus, and $Y$ being the product of a circle by a compact orientable surface of genus $1$ with two boundary components. This manifold has the property that it is impossible to split it as a connected sum of prime manifolds, even allowing infinitely many factors, and allowing the factors to be noncompact. Note that the first such example was constructed by P.~Scott~\cite{scott:noncompact}. Scott's example has infinitely many ends (in fact, its space of ends is a Cantor set due to the treelike nature of the construction.) Thus it does not belong to $\mathcal{C}$.

The manifold $M_3$ from~\cite{maillot:examples} does not have a locally finite splitting along 2-tori into submanifolds that are Seifert-fibered or atoroidal, thus showing that the theory of Jaco-Shalen~\cite{js:seifert} and Johannson~\cite{joh:hom} is difficult to extend to open 3-manifolds. This construction was inspired by Scott's work, so $M_3$ also has a Cantor set's worth of ends, and does not belong to the class $\mathcal{C}$. The goal of this paper is to give an example of a manifold in $\mathcal{C}$ with the same property.

In order to state the result, we recall some terminology. Let $M$ be an orientable 3-manifold. It is \bydef{irreducible} if every embedded 2-sphere in $M$ bounds a 3-ball. An embedded torus in $M$ is \bydef{incompressible} if it is $\pi_1$-injective. Following W.~Neumann and G.~Swarup~\cite{ns:canonical}, we call an embedded torus $T$ in $M$  \bydef{canonical} if it is incompressible and for every embedded, incompressible torus $T'\subset M$, there is an embedded torus $T''$ isotopic to $T'$ such that $T\cap T''=\emptyset$.

\begin{defi}
Let $\mathcal{T}=\{T_i\}_{i\in I}$ be a family of pairwise disjoint canonical tori in $M$. We say that $\mathcal{T}$ is \bydef{complete} if every canonical torus in $M$ is isotopic to $T_i$ for some $i\in I$.
\end{defi}

\begin{theo}\label{thm:manifoldone}
There is an open 3-manifold $M_0$ in the class $\mathcal{C}$ with the following properties.
\begin{enumerate}
\item $M_0$ is irreducible.
\item Every complete family of canonical tori in $M_0$ fails to be locally finite.
\end{enumerate}
\end{theo}

The key to constructing such a manifold is to ensure that there is a sequence $\{T_n\}$ of pairwise nonisotopic canonical tori in $M_0$ which fails to be locally finite no matter how the representatives of the various isotopy classes are chosen, because they are \bydef{trapped} by some compact subset $K\subset M_0$ which has to intersect all of them. In the paper~\cite{maillot:examples}, the $T_n$'s are pairwise nonhomologous, and separate different ends of the manifold $M_3$. By contrast, $M_0$ has only one end, and its $T_n$'s are all null-homologous, making it harder to prove that they are not isotopic.

In order to construct $M_0$, we first construct a \emph{$3$-orbifold} $\mathcal{O}_0$ with similar properties, interpreted in the orbifold sense. It is irreducible, and contains a sequence $\{P_n\}$ of \bydef{pillows}, i.e.~spheres with four conical points of order 2, which is trapped by some compact set. The orbifold $\mathcal{O}_0$ has underlying space $\mathbf{R}^3$ and is designed so that it is easy to see that the $P_n$'s are pairwise nonisotopic. Then the manifold $M_0$ is constructed as a 2-fold cover of $\mathcal{O}^3$. The $T_n$'s are taken to be the preimages of the $P_n$'s under the covering map.

The structure of the paper is as follows: in Section~\ref{sec:orbi} we construct the $3$-orbifold $\mathcal{O}_0$ and prove its various properties. In Section~\ref{sec:man} we will prove Theorem~\ref{thm:manifoldone} by constructing the manifold $M_0$ and showing that it has the required properties.

\paragraph{Acknowledgements}
This work is partially supported by Agence Nationale de la Recherche through Grant~ANR-12-BS01-0004.

I would like to thank Gr\'egoire Montcouquiol, Michel Boileau and Luisa Paoluzzi for useful conversations.

\section{The orbifold case}\label{sec:orbi}

\subsection{Definition of the orbifold $\mathcal{O}_0$}
Throughout the paper, we work in the PL or smooth category, and all manifolds and orbifolds are assumed to be connected and orientable. For terminology about 3-orbifolds, we refer to~\cite{bmp}.

Let $\mathcal{O}$ be a $3$-orbifold. Two 2-sub\-or\-bi\-fold s $F,F'$ are \bydef{isotopically disjoint} if there is a sub\-or\-bi\-fold\ $F''$ isotopic to $F'$ and disjoint from $F$. An incompressible toric sub\-or\-bi\-fold\  is \bydef{canonical} if it is isotopically disjoint from every incompressible toric 2-sub\-or\-bi\-fold.

Let $K$ be a compact subset of $\mathcal{O}$. A sequence $\{F_n\}$ of $2$-sub\-or\-bi\-fold s is said to be \bydef{trapped by}  $K$ if no $2$-sub\-or\-bi\-fold\ isotopic to any $F_n$ is disjoint from $K$. Thus, it is impossible to make $\{F_n\}$ locally finite by choosing different representatives of the various isotopy classes.

Let $B$ be a $3$-manifold homeomorphic to the $3$-ball and let $\alpha$ (resp.~$c$) be a properly embedded arc (resp.~circle) in $B$. Assume that $\alpha$ and $c$ are disjoint. Then we say that $\alpha \cup c$ is \emph{trivial} (resp.~a \bydef{Hopf tangle}) if $\alpha$ is trivial and $c$ bounds an embedded disk disjoint from $\alpha$ (resp.~meeting $\alpha$ transversally in a single point.)
To justify the terminology, note that if another $3$-ball $B'$ is glued to $B$ and $\alpha$ is extended to an unknot $c'$ in $B\cup B'$ in the obvious way, then $c\cup c'$ is an unlink (resp.~a Hopf link.)

We now come to our main construction: throughout the article we let $\mathcal{O}_0$ be a $3$-orbifold with the following properties: its underlying space is Euclidean $3$-space. Its singular locus $\Sigma_{\oo_0}$
consists in three unknotted, properly embedded lines $L_0$, $L_{12}$, and $L_{34}$, and four sequences of embedded circles $(c_i^{n})$ with $n\in\mathbf{N}$ and $i\in \{1,2,3,4\}$. The general relative positions of the various components of $\Sigma_{\oo_0}$ are shown on Figure~\ref{fig:general}. For each $n\in\mathbf{N}$, $c_1^n \cup c_2^n \cup c_3^n \cup c_4^n$ is a four component link contained in a `box' $B_n$ whose intersection with $L_{12} \cup L_{34}$ consists of a trivial $4$-tangle $\alpha_1^n \cup \alpha_2^n \cup \alpha_3^n \cup \alpha_4^n$, with $\alpha_1^n \cup \alpha_2^n \subset L_{12}$ and
 $\alpha_3^n \cup \alpha_4^n \subset L_{34}$.

\begin{figure}[ht]
\begin{center}
\includegraphics{general}
\end{center}
\caption {\label{fig:general} General configuration of $\Sigma_{\oo_0}$}
\end{figure}

Furthermore, we will assume that in each box, the $c_i^n$'s and the $\alpha_j^n$ satisfy the following properties:
\begin{enumerate}
\item each $c_i^n$ is unknotted;
\item $c_1^n\cup c_2^n$, $c_2^n\cup c_3^n$, $c_3^n\cup c_4^n$, and $c_4^n\cup c_1^n$ are Hopf links;
\item  $c_1^n\cup c_3^n$ and $c_2^n\cup c_4^n$ are unlinks;
\item for each $(i,j)\in \{1,2,3,4\}^2$, $c_i \cup \alpha_j$ is trivial if $i=j$, and a Hopf tangle otherwise.
\end{enumerate}

It is straightforward to see that such configurations exist, though they are far from being unique. Throughout the text, we assume that a choice has been made once and for all. Moreover, the choice is the same for all values of $n$, since we want the double manifold cover of $\mathcal{O}_0$ to belong to the class $\mathcal{C}$.

What matters is not the precise nature of the content of the boxes, but rather the values of the linking numbers of the various components of $\Sigma_{\oo_0}$; those are summarized by the graph depicted on~Figure~\ref{fig:linking}, with the convention that the linking number between two components is zero if there is no edge drawn between them, and one otherwise.

\bigskip

\begin{figure}[ht]
\begin{center}
\includegraphics{linking}
\end{center}
\caption {\label{fig:linking} Linking numbers of components of $\Sigma_{\oo_0}$}
\end{figure}

Finally, we assume that $\mathcal{O}_0$ is a so-called $\pi$-orbifold, i.e.~each nontrivial local group is cyclic of order 2.

\begin{lem}\label{lem:ooirr}
$\mathcal{O}$ is irreducible.
\end{lem}

\begin{proof}
Let $S\subset \mathcal{O}$ be a $2$-sub\-or\-bi\-fold\ of positive Euler characteristic. Then $|S|$ is a $2$-sphere intersecting $\Sigma_{\oo_0}$ transversally. For homological reasons, the number of singular points of $S$ is even. Thus $S$ is either a nonsingular $2$-sphere or a football. Furthermore, by Alexander's Theorem, $|S|$ bounds a $3$-ball $B\subset\mathbf{R}^3$.

Suppose that $S$ is nonsingular. If $B$ contains a singular point, then it contains some compact component $c_i^n$ of $\Sigma_{\oo_0}$. Since $c_i^n$ is unknotted, it bounds a disk $D\subset\operatorname{Int} B$. Either $L_{12}$ or $L_{34}$ has linking number one with $c_i^n$. Thus for $L$ equal to $L_{12}$ or $L_{34}$ we have $D\cap L\neq\emptyset$, hence $B\cap L\neq\emptyset$. Since $L$ is noncompact, this contradicts the assumption that $S$ is nonsingular. Therefore, $B$ is nonsingular.

Suppose now that $S$ has two singular points. Those two points must belong to either $L_{12}$ or $L_{34}$. By symmetry we may assume it is $L_{12}$. Then $B\cap L_{12}$ is an unknotted arc. We need to show that $B\cap \Sigma_{\oo_0}$ is in fact equal to this arc. Arguing by contradiction, assume that some $c_{i_0}^n$ is contained in $B$. Arguing as above using linking numbers, we can show that for every $i$, $c_i^n$ is contained in $\operatorname{Int} B$. Thus for the same reason $L_{34}$ intersects $B$, leading to a contradiction.
\end{proof}

\subsection{The pillows $P_n$ and their first properties}

Let us define the sequence of pillows $(P_n)$ and the compact subset $K'$ which traps them. As shown on Figure~\ref{fig:pillows} each $P_n$ meets $L_{12}$ and $L_{34}$ both twice; the $3$-ball bounded by $|P_n|$ contains a given box $B_m$ if and only if $m\le n$; finally, $K'$ is a $3$-ball meeting each of $L_0$, $L_{12}$, and $L_{34}$ in an unknotted arc, and for every $n$ the intersection of $K'$ with $P_n$ is a nonsingular disk.

For future reference, we let $\Pi$ denote the properly embedded plane shown on Figure~\ref{fig:pillows} and $H_0,H_1$ the closed half-spaces bounded by $\Pi$, so that $L_0\subset H_0$ and $H_1$ contains all other components of $\Sigma_{\oo_0}$.

\begin{figure}[ht]
\begin{center}
\includegraphics{pillows}
\end{center}
\caption {\label{fig:pillows} The canonical pillows $P_n$, the compact set $K'$, and the plane $\Pi$}
\end{figure}

For future reference, for each $n$ we let $U_n$ denote the $3$-ball bounded by $|P_n|$ and set $X_n:=|\mathcal{O}_0| \setminus \operatorname{Int} B_n$. Thus $X_n$ is noncompact and $\partial X_n=P_n$.

We collect in the next proposition some facts about the $P_n$'s which are fairly easy to prove.
\begin{lem}\label{lem:first}
\begin{enumerate}
\item Each $P_n$ is incompressible.
\item The $P_n$'s are pairwise nonisotopic.
\item The $P_n$'s are trapped by $K'$.
\end{enumerate}
\end{lem}

\begin{proof}
Proof of~assertion~(i): If some $P_n$ were compressible, then $P_n$ would bound a solid pillow or be contained in a discal $3$-sub\-or\-bi\-fold. Now $U_n$ contains at least four closed components of $\Sigma_{\oo_0}$, so this is impossible.

Assertion~(ii) follows immediately from the fact that for $n\neq m$, the number of closed components of $\Sigma_{\oo_0}$ contained in $U_n$ and $U_m$ are different.

Proof of~assertion~(iii): let $n$ be a natural number. Seeking a contradiction, we assume there is a pillow $P'_n$ isotopic to $P_n$ such that $P'_n\cap K'=\emptyset$. By Alexander's Theorem, the $2$-sphere $|P'_n|$ bounds a $3$-ball $U'_n$ in $\mathbf{R}^3$. There are two cases:

\paragraph{Case 1} $K'$ is contained in $U'_n$. Then $|P'_n|$ is homologous to $\partial K'$ in $\mathbf{R}^3\setminus \operatorname{Int} K'$. In this case, $P'_n$ must hit $L_0$, which contradicts our assumption that $P'_n$ is isotopic to $P_n$.

\paragraph{Case 2} $U'_n$ is disjoint from $K'$. Let $L_1,L_2$ (resp.~$L_3,L_4$) be the connected components of $\operatorname{Int} K'\cap L_{12}$ (resp.~$\operatorname{Int} K'\cap L_{34}$.) Again by homological reasons, and using the fact that $P'_n$ is isotopic to $P_n$, among the four singular points of $P'_n$, two must lie on some $L_i$ with $1\le i\le 2$, and the other two on some $L_j$ with $3\le j\le 4$. Now $\operatorname{Int} U'_n$ contains all $c_i^n$ for $1\le i\le 4$. As we have already seen several times, this leads to a contradiction.
\end{proof}

\begin{rem}\label{rem:irr}
The argument used to prove Lemma~\ref{lem:ooirr} shows that every $X_n$ is irreducible.
\end{rem}

\subsection{The $P_n$'s are canonical}

If $V$ is a solid torus and $V'$ a solid torus contained in $V$, then the \bydef{geometric winding number} of the pair $(V,V')$, denoted by $\operatorname{gw} (V,V')$, is the smallest natural number $n$ such that there is a meridian disk $D$ of $V$ with the property that $D\cap V'$ has $n$ connected components. A fundamental fact, due to Schubert, is the multiplicativity of $\operatorname{gw}$:

\begin{lem}[Schubert~\cite{schubert:vollringe}]\label{lem:schu}
Let $V,V',V''$ be solid tori such that $V\subset V'\subset V''$. Then $$\operatorname{gw}(V'',V) = \operatorname{gw}(V'',V') \cdot \operatorname{gw}(V',V).$$
\end{lem}

If $V$ is a solid torus and $c$ is an embedded circle in $V$, then the geometric winding number of the pair $(V,V')$ where $V'$ is a tubular neighbourhood of $c$ does not depend on the choice of $V'$. We shall denote this number by $\operatorname{gw}(V,c)$

Let $c,c'$ be two disjoint unknots in $\mathbf{R}^3$. Choose two solid tori $V,V'$ such that $V\cap V'=\emptyset$ and $V$ (resp.~$V'$) is a tubular neighborhood of $c$ (resp.~$c'$.) Viewing $\mathbf{R}^3$ as $S^3$ minus a point $\infty$, the complement of $\operatorname{Int} V$ in $S^3$ is also a solid torus, which we denote by $V_1$. By hypothesis, we have $V'\subset V_1$. We shall denote by $\operatorname{gw} (c^*,c')$ the geometric winding number of the pair $(V_1,V)$. Again, this does not depend on the various choices. If $c$ is an unknot in $\mathbf{R}^3$ and $V$ is a solid torus contained in $\mathbf{R}^3 \setminus c$, then $\operatorname{gw}(c^*,V)$ is defined similarly.

Note that if $c \cup c'$ is an unlink (resp.~a Hopf link), then $\operatorname{gw}(c^*,c')$ equals zero (resp.~one). Hence all geometric winding numbers involving compact components of $\Sigma_{\oo_0}$ are determined by our construction.

By extension, we can define $\operatorname{gw}(L^*,c)$ when $L$ is an unknotted properly embeded line in $\mathbf{R}^3$ and $c$ is an embedded circle in $\mathbf{R}^3$ that misses $L$~: in the one-point compactification $S^3=\mathbf{R}^3\cup \{\infty\}$, $L$ compactifies to a circle $L\cup\{\infty\}$. Let $V$ be the complement of an open tubular neighborhood of this circle such that $c\subset V$. Then we set $\operatorname{gw}(L^*,c):=\operatorname{gw}(V,c)$.
Again, all such numbers involving components of $\Sigma_{\oo_0}$ are determined by the construction, and are equal to either zero or one.
Likewise, if $V'$ is a solid torus which misses $L$, then $\operatorname{gw}(L^*,V')$ is defined as $\operatorname{gw}(V,V')$ for a suitable choice of $V$.

\begin{lem}\label{lem:solid}
Let $V$ be a solid torus in $|\mathcal{O}_0|$ such that no point of $\partial V$ is singular. Then $\operatorname{Int} V$ contains at most one component of $\Sigma_{\oo_0}$. If it does, then this component is compact and homotopic to the soul of $V$.
\end{lem}

\begin{proof}
Assume that $\operatorname{Int} V$ contains some component of $\Sigma_{\oo_0}$. Then this component is compact. Up to changing the notation, we may assume it is $c_1^n$ for some $n$. For brevity, we drop the superscript $n$ in the sequel.

Observe that $c_1\subset V\subset S^3\setminus L_{12}$. Hence by Lemma~\ref{lem:schu}, we have $\operatorname{gw}(L_{12}^*,c_1) = \operatorname{gw}(L_{12}^*,V)\cdot
\operatorname{gw}(V,c_1)$. Since $\operatorname{gw}(L_{12}^*,c_1)=1$, we deduce $\operatorname{gw}(L_{12}^*,V)=\operatorname{gw}(V,c_1)=1$. We also have  $c_1\subset V\subset S^3\setminus L_{34}$. Since $\operatorname{gw}(L_{34}^*,c_1)=0$ and $\operatorname{gw}(V,c_1)=1$, Lemma~\ref{lem:schu} implies that $\operatorname{gw}(L_{34}^*,V)=0$.

Seeking a contradiction, assume that $c_3\subset V$. Then using the chain of inclusions $c_3\subset V\subset S^3\setminus L_{34}$ and the facts that $\operatorname{gw}(L_{34}^*,c_3)=1$ and $\operatorname{gw}(L_{34}^*,V)=0$ we get a contradiction. Hence $V\subset S^3\setminus c_3$. Using the chain $c_1\subset V\subset S^3\setminus c_3$ we obtain $\operatorname{gw}(c_3^*,V)=0$. The same argument applies, \emph{mutatis mutandis}, to $c_4$.

Again we prove by contradiction that $c_2\not\subset V$: otherwise using the chain $c_2\subset V\subset S^3\setminus c_3$ and the fact that $\operatorname{gw}(c_3^*,c_2)=1$, Lemma~\ref{lem:schu} would lead to a contradiction.

By similar arguments, we exclude $c_3^m$ and $c_4^m$ for $m\neq n$ (using $L_{34}$), then $c_2^m$ (using $c_3^m$), and finally $c_1^m$ (using $c_4^m$.)
\end{proof}

\begin{lem}\label{lem:noannular}
For each $n\in\mathbf{N}$, $X_n$ does not contain any essential, properly embedded annular $2$-sub\-or\-bi\-fold.
\end{lem}

\begin{proof}
Arguing by contradiction, we let $A$ be such an annular $2$-sub\-or\-bi\-fold. Thus $A$ is either a nonsingular annulus or a disk with two singular points.

\paragraph{Case 1} Suppose that $A$ is an annulus. An innermost disk argument shows that after an isotopy we may assume that $A\cap\Pi=\emptyset$. Then $\partial A$ is a union of two circles, which viewed on $P_n$ are parallel and have two singular points on each side. Let $Y$ be the manifold obtained from $|H_1\cap X_n|$ by removing small open tubular neighborhoods of $L_{12}$ and $L_{34}$. Then $Y$ has a manifold compactification diffeomorphic to a product $F\times [0,1]$, where $F$ is a sphere with four holes, and $F\times \{0\}$ is the image of a subset of $P_n$. Such a manifold does not contain any essential annulus with both boundary components in $F\times \{0\}$.

Since $A$ is incompressible in $Y$, we deduce that it is boundary-parallel in $Y$. Let $V$ be a parallelism region in $Y$ between $A$ and some annulus $A'$ in $P_n$. Let us show that no component of $\Sigma_{\oo_0}$ lies in $V$. By Lemma~\ref{lem:solid}, $V$ contains at most one such component, say $c_1^m$ for some $m>n$. For simplicity of notation, set $c_i:=c_i^m$. Let $\gamma$ be a core of $A'$. Then we apply Lemma~\ref{lem:schu} to the chain $\gamma \subset V\subset S^3\setminus c_2$: from $\operatorname{gw}(V,\gamma)=1$ and $\operatorname{gw}(c_2^*,\gamma)=0$ we deduce $\operatorname{gw}(c_2^*,V)=0$. Then Lemma~\ref{lem:schu} applied to $c_1\subset V\subset S^3\setminus c_2$ yields a contradiction. This finishes the proof of Lemma~\ref{lem:noannular} in Case 1.

\paragraph{Case 2} Suppose that $A$ is a disk with two singular points. Again, we may assume that $A$ misses $\Pi$, and therefore lies in $H_1$. The intersection of $X_n$ with $L_{12} \cup L_{34}$ consists of four properly embedded half-lines, which we denote by $L_1,L_2,L_3,L_4$ in such a way that for each $m>n$ and each $1\le i\le 4$, $\alpha_i^m$ is contained in $L_i$.

By hypothesis, there exist two different indices $i_0,j_0$ in $\{1,2,3,4\}$ such that $A$ meets $L_i$ if and only if $i\in\{i_0,j_0\}$. Let $k_0,l_0$ be the two remaining indices. Let $Y$ be the $\pi$-orbifold whose underlying space is obtained from $|H_1 \cap X_n|$ by removing small open tubular neighborhoods of $L_{k_0}$ and $L_{l_0}$, and whose singular locus consists of $L_{i_0} \cup L_{j_0}$. Then $Y$ compactifies as $F\times [0,1]$, with $F$ an annulus (two-holed sphere) with two singular points, $F\times \{0\}$ corresponding to a subset of $P_n$.

As before, we deduce that $A$ is parallel in $Y$ to some disk $D\subset P_n$ with two singular points, which necessarily belong, one  to $L_{i_0}$ and the other to $L_{j_0}$. Let $B$ be the product region between $A$ and $D$, this time viewed as a sub\-or\-bi\-fold\ of $\mathcal{O}_0$.
Observe that $B$ is a $3$-ball whose singular locus contains a trivial $2$-tangle consisting of the union of a subarc of $L_{i_0}$ and a subarc of $L_{j_0}$. Since $A$ is essential in $X_n$, $\operatorname{Int} B$ must in addition contain  $c_i^m$ for some $1\le i\le 4$ and $m>n$.

Arguing as in the proof of the irreducibility of $\mathcal{O}$, we see that
$\operatorname{Int} B$ contains $c_i^m$ for \emph{i}; in particular it contains $c_{k_0}^m$. We deduce that $c_{k_0}^m$, being an unknot in the ball $|B|$, bounds a disk $D'\subset |B|$. Such a disk must meet either $L_{k_0}$ or $L_{l_0}$. This contradiction completes the proof of
Case~2.
\end{proof}

At last we can prove the main result of this subsection:
\begin{prop}\label{prop:oocan}
Every $P_n$ is canonical.
\end{prop}

\begin{proof}
Assume that $P_n$ is not canonical. Let $P'_n$ be an incompressible toric $2$-sub\-or\-bi\-fold\ of $\mathcal{O}_0$ which cannot be isotoped off $P_n$. Since $\mathcal{O}$ does not contain any turnover, $P'_n$ is a nonsingular torus or a pillow. Assume that $P'_n$ has been isotoped so as to intersect $P_n$ transversally and minimally.

Since both $P_n$ and $P'_n$ are incompressible, a component of $P_n\cap P'_n$ is essential on $P_n$ if and only if it is essential on $P'_n$. Using the irreducibility of $\mathcal{O}_0$ and a standard argument, one can remove any inessential intersection component between $P_n$ and $P'_n$. Hence, by our assumption of minimality, each component of $P_n\cap P'_n$ is essential.
As a consequence, some connected component $A$ of $X_n\cap P'_n$ is an annular $2$-sub\-or\-bi\-fold, which is essential as a sub\-or\-bi\-fold\ of $X_n$. This contradicts Lemma~\ref{lem:noannular}.
\end{proof}

\section{The manifold $M_0$}\label{sec:man}
In this section, we let $p:M_0\to\mathcal{O}_0$ be the only two-fold manifold cover of $\mathcal{O}_0$. In other words, $M_0$ is the two-fold branched cover over $\mathbf{R}^3$ with branching locus $\Sigma_{\oo_0}$ and $p$ is the canonical projection. As in the compact case, this cover can be constructed by splitting $\mathcal{O}_0$ along a Seifert surface for $\Sigma_{\oo_0}$, taking two copies of the resulting manifold, and gluing them appropriately. Alternatively, the orbifold fundamental group $\pi_1\mathcal{O}_0$ is (infinitely) generated by \emph{meridians}, i.e.~small circles around each component of $\Sigma_{\oo_0}$. All meridians have order~2. There is an index~2 normal subgroup $\Gamma$ of $\pi_1\mathcal{O}_0$ which is the kernel of a homomorphism from $\pi_1\mathcal{O}_0$ to $\mathbf{Z}/2\mathbf{Z}$ sending all meridians to the nontrivial element. This group is torsion~free, and $p:M_0\to\mathcal{O}_0$ is the corresponding manifold cover. We denote by $\tau:M_0\to M_0$ the involution such that $\mathcal{O}_0=M_0/\tau$.

By construction, $\mathcal{O}_0$ is obtained by gluing $K'$ and countably many copies of some compact 3-orbifold $Y'$ with two boundary components, glued to each other `in the same way'. Moreover, the preimage of $K'$ by $p$ is connected and has connected boundary, and the preimage of each copy of $Y'$ is connected and has exactly two boundary components. It follows that $M_0$ belongs to the class $\mathcal C$ defined in the introduction.

For each $n\ge 0$, set $T_n:=p^{-1}(P_n)$ and $Y_n:=p^{-1}(X_n)$. Thus each $T_n$ is an embedded $2$-torus in $M$. Set $K:=p^{-1}(K')$. Then $K$ is a compact subset of $M$. We need to prove that $M$ has all the properties stated in Theorem~\ref{thm:manifoldone}.

Since $\mathcal{O}_0$ is irreducible, $M_0$ is irreducible~(\cite[Theorem~3.23]{bmp}.) For the same reason, Remark~\ref{rem:irr} implies that $Y_n$ is irreducible for every $n$. By the equivariant Dehn Lemma, incompressibility of every $T_n$ follows from Lemma~\ref{lem:first}(i).

\begin{lem}
The $T_n$'s are pairwise nonisotopic.
\end{lem}

\begin{proof}
Assume that there exist $n,m$ with $n\neq m$ such that $T_m$ is isotopic to $T_n$. By~\cite[Prop~4.5]{bz:link}, there is a $\tau$-equivariant isotopy. Hence $P_m$ is isotopic to $P_n$, contradicting Lemma~\ref{lem:first}(ii).
\end{proof}

Our next goal is to show that every $T_n$ is canonical. For this we need a lemma.

\begin{lem}\label{lem:noannman}
There is no properly embedded essential annulus in $Y_n$.
\end{lem}

\begin{proof}
Put a $\tau$-invariant riemannian metric on $Y_n$ with mean-convex boundary. Arguing by contradiction, we assume that $Y_n$ contains a properly embedded essential annulus. Let $A$ have least area among such annuli. Set $A':=\tau(A)$. If $A\cap A'=\emptyset$ or $A=A'$, then $p(A)$ is an annular $2$-sub\-or\-bi\-fold\ in $X_n$. Thus by Lemma~\ref{lem:noannular}, we deduce that $p(A)$ is inessential. This implies that $A$ is inessential, contradicting our hypothesis.

Hence by the Meeks-Yau trick we may assume that $A$ and $A'$ intersect transversally in a nonempty disjoint union of curves and arcs. Since both $A$ and $A'$ are essential, a curve or arc is essential on $A$ if and only if it is esssential on $A'$.

If there exists an inessential curve in $A\cap A'$, we let $D$ be a disk of minimal area on $A$ or $A'$ bounded by such a curve. Note that $D$ is automatically innermost. Thus a classical exchange/roundoff argument leads to a contradiction (the disk exchange can be realized by an isotopy because $Y_n$ is irreducible.)

If there is an inessential arc in $A\cap A'$, we can argue similarly using a disk of minimal area cobounded by such an arc and an arc in $\partial Y_n$. In order to ensure that the disk exchange can be realized by an isotopy, we use the irreducibility of $Y_n$ and the incompressibility of $\partial Y_n$.

Suppose there is an essential curve in $A\cap A'$. Let $\gamma$ be such a curve. Pick a basepoint on $\gamma$ and consider the covering space $Y_\gamma$ of $M$ whose fundamental group is the infinite cyclic group generated by $\gamma$. Let $\tilde A$ and $\tilde A'$ be lifts to $Y_\gamma$ of $A$ and $A'$ respectively. Observe that the inclusions of $\tilde A$ and $\tilde A'$ in $Y_\gamma$ are homotopy equivalences. In particular,  $\tilde A$ and $\tilde A'$ are incompressible.

\begin{slem}\label{slem:boundary}
\begin{enumerate}
\item The two boundary components of $\tilde A$ lie on distinct components of $\partial Y_\gamma$.
\item The same property holds for $\tilde A'$.
\end{enumerate}
\end{slem}

\begin{proof}
It suffices to prove the first point. We argue by contradiction, letting $F$ be a component of $\partial Y_\gamma$ containing all of $\partial \tilde A$. Observe that $F$ is an open annulus whose inclusion in $Y_\gamma$ is a homotopy equivalence. Hence there is an annulus $A''$ contained in $F$ and such that $\partial A''=\partial \tilde A$. Thus $T:=\tilde A \cup A''$ is an embedded torus. Since $\pi_1(Y_\gamma)$ is cyclic, $T$ is compressible. As $T$ contains an incompressible annulus, it does not lie in a $3$-ball. Therefore, $T$ bounds a solid torus $V$, which is a parallelism region between $\tilde A$ and $A''$. In particular, $\tilde A$ is isotopic to $A''$ relative to the boundary. Projecting such an isotopy to $Y_n$ leads to a contradiction with the fact that $A$ is essential. This completes the proof of Sublemma~\ref{slem:boundary}.
\end{proof}

Note that in the previous discussion, $\gamma$ was an arbitrary essential curve of intersection between $A$ and $A'$. Now we make a more specific choice of $\gamma$: we assume that some annulus cobounded by $\gamma$ and some boundary component of $A$ or $A'$ has least possible area. Without loss of generality, we assume that such an annulus, called $A_1$, is contained in $A$. We let $A_2$ be the subannulus of $A$ whose boundary is the union of $\gamma$ and the other component of $\partial A$, and $A_3,A_4$ be the annuli such that $A_3\cap A_4=\gamma$ and $A_3\cup A_4=A'$.

We set $A_5:=A_1\cup A_3$ and $A_6:=A_1\cup A_4$. Then $A_5$ and $A_6$ are properly embedded annuli in $X_n$. As before we consider lifts $\tilde A_5$ and $\tilde A_6$ of those annuli to $Y_\gamma$. By the second assertion of Sublemma~\ref{slem:boundary}, at least one of $\tilde A_5$ and $\tilde A_6$ has its two boundary components on different components of $\partial Y_\gamma$. Assume $\tilde A_5$ has this property. Then $A_5$ is essential: otherwise one could lift a boundary-parallelism region for $A_5$ to $M_\gamma$ and obtain one for $\tilde A_5$. Now by choice of $A_1$, the area of $A_5$ is less than or equal to that of $A$, and by rounding the corner, we obtain an essential annulus which contradicts the minimizing property of $A$.

Thus we are left with the case where $A\cap A'$ contains an essential arc.

\begin{slem}
There are at least two distinct essential arcs in $A\cap A'$.
\end{slem}

\begin{proof}
Let $\gamma_1,\gamma_2$ denote the boundary components of $A$ and $\gamma_3,\gamma_4$ denote those of $A'$. Thus $\gamma_1$ (resp.~$\gamma_3$) is isotopic to $\gamma_2$ (resp.~$\gamma_4$) on $T_n$. If $A\cap A'$ consists of only one essential arc $\alpha$, then there are only two points in $\partial A\cap \partial A'$. If $\gamma$ is not isotopic to $\gamma_3$, then $\gamma_1\cap \gamma_3$ is nonempty, and so are $\gamma_1\cap\gamma_4$, $\gamma_2\cap\gamma_3$, and $\gamma_2\cap\gamma_4$, resulting in a contradiction.

Hence the $\gamma_i$'s are all pairwise isotopic, and the two intersection points belong to the same two arcs, which means that $\alpha$ is inessential, again a contradiction.
\end{proof}

Choose as basepoint an endpoint of some arc in $A\cap A'$ and let $Y_T$ be the covering space of $Y_n$ such that $\pi_1(Y_T)=\pi_1(T_n)$. Let $U_n$ be a torus component of $\partial Y_T$ whose projection is $T_n$, the said projection being a homeomorphism. Let $\tilde A$ and $\tilde A'$ be lifts to $Y_T$ of $A$ and $A'$ respectively, which both meet $U_n$.

\begin{slem}
\begin{enumerate}
\item The two components of $\partial\tilde A$ lie on distinct components of $\partial Y_T$.
\item The same holds for $\tilde A'$.
\end{enumerate}
\end{slem}

\begin{proof}
Again we only have to prove the first statement. Arguing by contradiction, assume both components of $\partial\tilde A$ belong to the same component of $\partial Y_T$. By construction, this component is $U_n$. Then there is an annulus $A''\subset U_n$ such that $\partial A''=\partial \tilde A$. Set $T=A''\cup \tilde A$. Then $T$ is an embedded torus in $Y_T$. By pushing $T$ slightly we get $T'\subset \operatorname{Int} Y_T$.

Suppose that $T$ (or equivalently $T'$) is compressible. Since $A''$ is not contained in any $3$-ball, neither is $T$, so $T$ bounds a solid torus $V$. Since the core of $A''$ is a primitive element of $U_n$, $V$ is a parallelism region between $A''$ and $\tilde A$. This shows that $\tilde A$ is inessential, and so is $A$, a contradiction.

Hence $T$ is incompressible. Note that $H_2(Y_T)$ is isomorphic to $\mathbf{Z}$ and generated by the class of $U_n$. Thus $T$ is either null-homologous or homologous to $U_n$.

If $T$ is null-homologous, then $T$ bounds some compact submanifold $V$. Since $T$ is incompressible in $Y_T$, it is also incompressible in $V$, and by van Kampen's theorem, the induced homomorphisms $\pi_1 T\to \pi_1V \to \pi_1 Y_T$ are both injective. It follows that $\pi_1V$ is isomorphic to $\mathbf{Z}^2$, which is impossible with $\partial V$ connected.

Thus $T$ is homologous to $U_n$, and so is $T'$. Let $X$ be a compact submanifold whose boundary is $U_n\cup T'$. Arguing as before, we see that $\pi_1X$ is isomorphic to $\mathbf{Z}^2$. Since $X$ is irreducible, $X$ is homeomorphic to $T^2\times I$. Again this implies that $\tilde A$ is boundary-parallel.
\end{proof}

We can now finish the proof of Lemma~\ref{lem:noannman}. Since $Y_T$ covers a noncompact manifold, it is itself noncompact. In particular, $Y_T$ is not homeomorphic to $T^2\times I$. Thus, homology considerations as above show that $U_n$ is the only torus in $\partial Y_T$. It follows that the boundary components $\gamma,\gamma'$ of $\tilde A$ and $\tilde A'$ that do not lie on $U_n$ are contained in some annulus component $A''$ of $\partial Y_T$, and are freely homotopic (and noncontractible) there. Hence there is a bigon between $\gamma$ and $\gamma'$, and we can use an exchange/roundoff argument to get a contradiction. This completes the proof of Lemma~\ref{lem:noannman}.
\end{proof}

The fact that the $T_n$'s are canonical now follow as Proposition~\ref{prop:oocan} followed from Lemma~\ref{lem:noannular}. Hence to prove Theorem~\ref{thm:manifoldone}, we only need to show the following statement:

\begin{lem}\label{lem:trapped}
The compact subset $K=p^{-1}(K')\subset M$ traps each $T_n$.
\end{lem}

\begin{proof}
Arguing by contradiction, we assume that for some $n$ there exists an embedded torus $T'\subset M\setminus K$ which is isotopic to $T_n$. Our aim is to show that after isotopy, $T'$ can be chosen to be invariant under $\tau$. Then by~\cite[Proposition~4.5]{bz:link}  $T'$ and $T_n$ are equivariantly isotopic. This implies that the image of $T'$ is an embedded pillow in $\mathcal{O}\setminus K'$ isotopic to $P_n$, which contradicts the fact that $P_n$ is trapped by $K'$.

Let $F$ be a parallel copy of $\partial K$ lying outside $K$, and $X$ be the (compact) product submanifold whose boundary is the disjoint union of $\partial K$ and $F$. Thus $X\cap K=\partial K$. Without loss of generality we may assume that $T'\cap X=\emptyset$. We are going to use the Jaco-Rubinstein theory of PL minimal surfaces, as modified in~\cite{maillot:center}. Choose a triangulation $\mathcal{T}$ of $M$ which is in general position with respect to $T'$ and invariant under $\tau$. Let $w_0$ be the weight of $T'$ with respect to $\mathcal{T}$. By repeatedly subdividing $\mathcal{T}$ on $X$, with may assume that any normal surface which meets both components of $\partial X$ has weight strictly greater than $w_0$, keeping the invariance under $\tau$. Then choose a $\tau$-invariant regular Jaco-Rubinstein metric in the sense of~\cite{maillot:center} so that the PL area is well-defined.

Let $T''$ be a normal torus of least PL area in the isotopy class of $T'$. We claim that $T''$ misses $K$. Indeed, $K$, being the double branched cover of a 3-ball with branching locus a trivial tangle, is a handlebody. The same is true for $K\cup X$. Since $T''$ is an incompressible torus, it cannot be contained in $K\subset X$. Hence if it met $K$, it would meet both boundary components of $X$, contradicting the fact that its weight is at most $w_0$.

Since the Jaco-Rubinstein metric we use is $\tau$-invariant, $\tau T''$ also has least PL area in its isotopy class. Now $T''$ is isotopic to $T_n$, which is $\tau$-invariant, so $\tau T''$ is isotopic to $T''$.
By Jaco-Rubinstein's version~\cite{jr:min} of a Theorem of Freedman-Hass-Scott, it follows that $T''$ and $\tau T''$ are disjoint or equal, i.e. $T''$ is invariant or equivariant. We still have to rule out the latter possibility.

Suppose that $T''\cap \tau T''=\emptyset$. Since $T''$ and $\tau T''$ are isotopic, they are parallel. Let $Y$ be the compact submanifold bounded by $T''\cup \tau T''$ and diffeomorphic to $T^2\times I$. Note that $Y$ is unique because $M$ is noncompact. Hence $Y$ is $\tau$-invariant. By the classification of involutions of $T^2\times I$, noting that $\tau$ exchanges the two boundary components, this implies that $\operatorname{Int} X$ contains an invariant embedded torus whose projection in $\mathcal{O}$ is a one-sided, nonorientable $2$-suborbifold. Since $\mathcal{O}$ does not contain any such suborbifold, we have reached the desired contradiction. Hence the proof of Lemma~\ref{lem:trapped} is complete.
\end{proof}

\bibliographystyle{alpha}