text:
\section{Introduction}

Radio-frequency interference broadly refers to all signals that negatively impact radio astronomical observations, the vast majority of which emanate from artificial sources. In radio transient and pulsar searches, idealised data consist of the superposition of a weak astrophysical signal with uncorrelated Gaussian noise in all frequency channels. In this case, the theoretically optimal detection method is known both for individual pulses \citep{CordesMcLaughlin2003} and periodic sources \citep{Morello2020}. Search codes thus generally work under the assumption of a pure Gaussian noise background, which is, however, never realised in practice. The calculation of the detection statistic is dependent on estimating the mean and standard deviation of the background noise from the data; this process is perturbed by the presence of Radio Frequency Interference (RFI), which can act to reduce the estimated signal-to-noise ratio of a genuine source below the detection threshold. It is necessary to subtract away or mask undesirable signals from the data before they are searched, in order to remain acceptably close to the ideal data assumptions and approach the theoretical sensitivity of the instrument.

RFI mitigation can be performed at all stages of the signal chain, with complementary benefits \citep[see e.g.][for an overview]{Baan2010}. One can make a broad distinction between pre-detection techniques, which act on the voltage data stream while it is still available at its highest time resolution, and post-detection methods that operate on the channelised, two-dimensional time-frequency data just before they are searched for radio transients and pulsars. The former are better suited to remove short bursts of RFI with minimal data loss, however they are required to handle large volumes of data in real time and thus tend to be limited to simple thresholding and replacement schemes. A notable exception is that interferometers may implement an additional technique called spatial filtering \citep[e.g.][]{Leshem2000}, where a null in the beam pattern can be placed in the estimated direction of a strong interference source.

On the other hand, the detection stage involves integrating the baseband data in time, making weaker sources of RFI accessible for removal and reducing the data rate, which in turn enables more sophisticated processing. Post-detection methods usually attempt to blank sections of data across the time and/or frequency dimension, based on some specific RFI signatures. Two simple but highly popular methods are the application of a fixed channel mask which permanently discards the most RFI-occupied portions of the band, and the so-called zero-DM filter of \citet{Eatough2009} that aims to subtract away broadband signals that show no cold-plasma dispersion expected from propagation through the interstellar medium. An entire class of algorithms involve flagging sections of data based on whether some statistical property exits the range expected for clean data: for example, a rolling sum over multiple time scales \citep[the \textsc{sumthreshold} algorithm of][]{Offringa2010}, or a higher-order statistical moment \citep{Nita2010}. Convolutional neural networks have also been trained to perform intelligent clipping of time-frequency data \citep[e.g.][]{Akeret2017}. Fourier transforming the data and finding Fourier bins with excessive power is effective against weaker periodic RFI and well-suited to periodicity searches \citep{Fridman2001, Maan2020}.

While there is a trade-off between RFI removal effectiveness and execution speed, the pressure to find methods that perform well on both fronts is increasing, and not all the aforementioned methods are fast enough to be used in a streaming environment. Indeed, on the latest and upcoming generations of radio telescopes, an RFI mitigation algorithm should aim to be:
\begin{itemize}
    \item Fast enough to be used on massively multi-beam systems, where searching large data streams in real-time has now become a fundamental requirement \citep[e.g.][]{CHIMEOverview, SPIEMeerTRAP}
    \item Able to increase the detection probability of genuine sources \textit{and} to reduce the spurious candidate rate at the same time
    \item Transferable between telescopes with limited adjustments to the algorithm's parameters, which would make it more useful to the community as a whole and avoid duplication of work.
\end{itemize}
The challenge lies in developing methods that fulfil all three criteria.

Static channel masks may be a blunt tool, but are extremely easy to implement and optimal against portions of the band permanently ridden with RFI. The shape and portion of the band occupied by RFI varies as a function of pointing position and of time however; a common situation is that few channels are permanently unusable, while most of them can be occasionally affected with narrow-band RFI; this is arguably the case on MeerKAT for example \citep[][see also \S \ref{subsec:bandpass_tests_meertrap}]{Sihlangu2020}. The ability to automatically and accurately adapt a channel mask to the current data on a time scale of a few seconds is quite desirable; here we propose an algorithm called IQRM that performs this task much faster than real time; IQRM stands for Inter-Quartile range RFI Mitigation, for reasons that will become clear below.

The outline of the paper is as follows. In \S\ref{sec:method}, we explain the rationale behind the design of IQRM, before describing the algorithm in detail. In \S\ref{sec:bandpass_tests}, we test the ability of IQRM to derive a time-variable channel mask from spectral statistics of data obtained with the MeerKAT and Lovell radio telescopes; its execution speed is also measured. We then evaluate the impact of IQRM in a blind search for single pulses in \S\ref{sec:sps_tests}; using over two hours of known pulsar observations taken with the Lovell telescope, we compare the number of genuine astrophysical pulses and false positives reported, with and without IQRM applied. We conclude in \S\ref{sec:discussion} after discussing the performance, limitations and potential extensions to the method.

\section{Method}
\label{sec:method}

\subsection{General principle}

Search-mode data is a sequence of dynamic spectra recorded with integration times of typically a few tens to hundreds of microseconds. Given a short-duration block of such time-frequency data, the goal of IQRM is to identify all frequency channels affected by narrow-band RFI. The idea is to reduce the entire block to a single summary statistic per frequency channel (i.e. a \textit{spectral moment}), and then to determine for which channels the value is unlikely to be associated with clean data. Nita et al. have previously proposed a method where the data are flagged based on their short-term spectral kurtosis \citep[][]{Nita2007, Nita2010}, for which the acceptable range of values can be determined from first principles, independently of the specific data analysed. IQRM could be described as a generalisation attempt of the aforementioned work where:
\begin{itemize}
    \item Any spectral statistic can be used as the basis to decide which channels are contaminated by RFI, with the important requirement that higher values of the statistic must indicate a higher probability of contamination.
    \item The rejection threshold value for this statistic is inferred from the data themselves.
\end{itemize}

Although IQRM is a generic method, it was originally developed for MeerTRAP \citep[see][and \S \ref{subsec:bandpass_tests_meertrap}]{Rajwade2020}, the commensal real-time radio transient search backend of the MeerKAT radio telescope \citep{Jonas2016}. On MeerTRAP search-mode data, the short-term spectral standard deviation has been found to be an excellent indicator of RFI, for technical reasons related to how the beamformer scales and digitises the data (see \S \ref{sec:bandpass_tests} for full details). This can be visualised on an example pulse from a known pulsar recorded with MeerTRAP at L-Band (Fig. \ref{fig:J1226_before_after}, left panel). Using standard deviation instead of kurtosis as proxy for interference contamination comes with an additional challenge: there is no analytical method to determine what constitutes a threshold value above which a channel should be deemed contaminated. Although the beamformer~\citep[see][and the references therein]{Barr2018} aims to produce data with a fixed standard deviation in all channels, in practice the scale of the data fluctuates measurably over time, and a frequency-dependent background trend or ``baseline" may be present in the sequence of spectral standard deviation values. The baseline is difficult to model, and its shape changes as a function of time and pointing position. Anything that alters the effective system temperature unequally across the band has the potential to generate a baseline: imperfect bandpass filters, variations in receiver response, the diffuse Galactic radio emission \citep[e.g.][]{Haslam1981}, the presence of a bright radio source with a steep spectral index in the telescope field of view, or broadband RFI.

IQRM is an algorithm designed to identify channels whose short-term standard deviation, or any other spectral moment chosen as a contamination measure, is abnormally high compared to some arbitrary, slow-varying trend along the observing frequency axis, i.e. the baseline; IQRM is non-parametric and requires no external information to make that decision, i.e. what constitutes an abnormally high value is inferred from the data itself. The fundamental idea of IQRM is to subtract away the baseline problem by taking the so-called lagged differences of the spectral moment values, and then finding high outliers in these difference values, which is a simpler task. This raises the matter of choosing the spectral moment that correlates best with the presence of narrow-band RFI, which will be covered in \S \ref{sec:bandpass_tests} when testing the method on real-world data.

\begin{figure*}
    \centering
    \includegraphics[width=1.00\textwidth]{Figures/J1226_before_after.png}
    \caption{A single pulse from RRAT J1226$-$3223 recorded with MeerKAT at L-Band during the commissioning phase of MeerTRAP in July 2019. Left: original time-frequency data; here, the short-term spectral standard deviation of the data is an excellent indicator for the presence of RFI. Right: same data after masking the channels flagged by IQRM. The colour scale and axis ranges are identical. 17\% of the channels were masked in this example.}
    \label{fig:J1226_before_after}
\end{figure*}

\subsection{The IQRM algorithm}
\label{subsec:algorithm_statement}

Formally, the IQRM algorithm takes as its input an ordered sequence of real-valued numbers $(x_i)_{i=0}^{n-1}$ and identifies values significantly larger than their close neighbours in the sequence. $i$ represents a channel index and $x_i$ the value of the chosen spectral moment (the measure of RFI contamination) in that channel. IQRM has two adjustable parameters:
\begin{itemize}
    \item The radius $r$: the distance (in number of sequence elements) to the furthest neighbour being considered when evaluating the outlier status of a given data point in the sequence.
    \item The threshold $t$: a significance level in number of Gaussian sigmas, which controls by how much a data point must exceed one of its neighbours to be categorised as an outlier.
\end{itemize}
The first step of the algorithm is to calculate a set of lagged differences of the input sequence $(x_i)$. Here we define the lagged difference of $(x_i)$ for a given lag $k \in \mathbb{Z}$ the sequence $(\Delta_i^k)_{i=0}^{n-1}$ such that

\begin{equation}
\label{eq:lagged_difference}
    \Delta_i^k = x_{i} - x_{i-k},
\end{equation}
where the boundary conditions are handled by setting $x_j = x_0$ for $j < 0$ and $x_j = x_{n-1}$ for $j \geq n$ in the expression above. The usefulness of the lagged difference operation when a trend is present is illustrated in Fig. \ref{fig:lagged_diff_trend_elimination}. It is necessary to calculate lagged differences using multiple trial values for $k$, so that a set of consecutive outliers with very similar values does not escape detection. Trial lag values are selected within the range $[-r, +r]$, excluding $0$. To save computation time, we do not try every possible integer value in that range, but instead arrange the trial values in a geometric progression using the following recurrence relation:

\begin{equation}
\begin{split}
    k_0 &= 1 \\
    k_{m+1} &= \mathrm{max}(\lfloor 1.5 \times k_m \rfloor, k_m + 1),
\end{split}
\end{equation}
where $\lfloor \rfloor$ denotes the floor function, and noting that all $-k_m$ are also added to the sequence of trial values. For example, the sequence corresponding to $r=10$ is $(-9, -6, -4, -3, -2, -1, 1, 2, 3, 4, 6, 9)$.

Then, for each trial lag value $k$ separately, \textit{high} outliers in the sequence $(\Delta_i^k)_{i=0}^{n-1}$ are then identified using a criterion similar to Tukey's rule for outliers, also known as Tukey's fences \citep{TukeyFences}. Here we assume that the $\Delta_i^k$ values (for fixed $k$) contain:
\begin{itemize}
    \item A majority of inliers that follow some underlying distribution with mean $\mu_k$ and standard deviation $\sigma_k$.
    \item A minority of outliers that span a range of values much greater than $\sigma_k$
\end{itemize}
$\mu_k$ and $\sigma_k$ need to be measured using robust estimators, i.e. that are not easily biased by outliers. We use the following:

\begin{equation}
\label{eq:robust_mean}
    \mu_k = Q_k(0.5)
\end{equation}

\begin{equation}
\label{eq:robust_stddev}
    \sigma_k = \frac{Q_k(0.75) - Q_k(0.25)}{\Phi^{-1}(0.75) - \Phi^{-1}(0.25)} \approx \frac{\mathrm{IQR}}{1.349}
\end{equation}
where $Q_k(z)$ denotes the $z$-th empirical quantile of the $\Delta_i^k$, and $\Phi^{-1}$ is the inverse cumulative distribution function of the normal distribution. In other words, the mean of the inliers is taken to be the median of the $\Delta_i^k$; the standard deviation of the inliers is taken to be proportional to the empirical inter-quartile range $\mathrm{IQR} = Q_k(0.75) - Q_k(0.25)$, where we used the fact that the inter-quartile range of the normal distribution is $\mathrm{IQR}_\mathrm{norm} = \Phi^{-1}(0.75) - \Phi^{-1}(0.25) \approx 1.349$. For every pair ($i$, $k$), we then test the condition

\begin{equation}
\label{eq:outlier_definition}
    (\Delta_i^k - \mu_k) > t \times \sigma_k,
\end{equation}
which if true, denotes that $x_i$ is abnormally larger than $x_{i-k}$. In this case, we say thereafter that $x_{i-k}$ casts a vote against $x_i$, which we note $(i-k) \xrightarrow[]{} i$. The full set of votes can be represented as the edges of a directed graph, where the nodes are channel indices. In the initial version of the IQRM algorithm, an input data point $x_i$ was deemed to be an outlier if Eq. \ref{eq:outlier_definition} was true for any trial lag value $k$, or in other words, if it received just a single vote. This however created a moral equivalence problem where the algorithm was unable to distinguish between a legitimate outlier lying far above the local trend, and an acceptable point that happens to be the neighbour of another lying far below the local trend. In fact, the presence of such a \textit{low} outlier created a pathological case where the algorithm would mask most of its neighbours against any common sense. Keeping a full record of votes provides a means to solve this issue: a vote $i \xrightarrow[]{} j$ is considered valid if and only if $i$ has cast \textit{strictly} less votes in total than $j$ has received. Any point that receives at least one \textit{valid} vote is finally marked as an outlier, and the algorithm returns a binary mask with the same size as the input data sequence. The vote pruning process is illustrated on an example in Fig. \ref{fig:iqrm_voting}.

\begin{figure}
    \centering
    \includegraphics[width=1.00\columnwidth]{Figures/LaggedDiffTrendElimination.png}
    \caption{The fundamental idea of IQRM: the lagged difference operation eliminates a slowly varying trend in sequential data and facilitates the procedural identification of outliers. Top panel: artificially generated input data. These are the sum of two sine functions and gaussian noise; an additional 40 points selected at random were then incremented by an exponentially distributed random variate, to simulate high outliers. Bottom panel: lagged difference of the input data (Eq. \ref{eq:lagged_difference}) with a lag of $k=1$ bin; the red dashed lines represent $\mu_k \pm 3 \times \sigma_k$, i.e. the acceptable range of values empirically inferred using the robust mean and scale estimators of Eqs. \ref{eq:robust_mean} and \ref{eq:robust_stddev}. Points highlighted in orange are those deemed abnormally larger than their previous neighbour according to Eq. \ref{eq:outlier_definition}. IQRM repeats this process for multiple values of the lag $k$, in order to overcome cases where consecutive points are outliers with similar values.}
    \label{fig:lagged_diff_trend_elimination}
\end{figure}

\subsection{Practical implementation and usage}

A minimal open-source implementation of IQRM in the \textsc{Python} language has been made available\footnote{\url{https://github.com/v-morello/iqrm}}. It is minimal in the sense that it provides only the outlier flagging capability specified in \S \ref{subsec:algorithm_statement}. Furthermore, identifying sections of data affected by RFI is only half the battle; this information must then be passed on to a search code. The difficulty that arises here comes from the fact that IQRM is meant to generate a channel mask that varies in time: read consecutive data blocks of a few seconds in length, calculate their spectral standard deviation (or other contamination statistic of choice), and obtain a mask adapted to each block by applying IQRM. However, the dedispersion stages of most widely-used search pipelines are not designed to deal with time-variable masks, and instead only accept a fixed list of channels to ignore; such is the case of \textsc{sigproc} \citep{SIGPROC}, \textsc{presto} \citep{Ransom2002}, \textsc{heimdall}\footnote{\url{https://sourceforge.net/projects/heimdall-astro/}}, \textsc{peasoup}\footnote{\url{https://github.com/ewanbarr/peasoup}}, and \textsc{astroaccelerate} \citep{AASPS}. In many use cases it is thus necessary to replace the bad sections of input data with adequate substitution values, before sending the edited data for processing. For that purpose, we provide a \textsc{C++} implementation, \textsc{iqrm apollo}\footnote{\url{https://gitlab.com/kmrajwade/iqrm_apollo}}, that performs the full sequence of operations: reading, calculation of the spectral moment, flagging and replacing. It currently processes \textsc{sigproc} filterbank files and saves cleaned copies, noting that interfaces to other formats can be added. There are three possible replacement policies for flagged data:

\begin{enumerate}
    \item Replace by a constant value of the user's choice.
    \item Replace by the mean of the medians of the non-flagged channels within the block.
    \item Replace by artificial Gaussian noise, with mean and variance equal to the global median and global variance of the non-flagged channels within the block.
\end{enumerate}
Mean-of-medians is the default and recommended option, noting that the code is modular and other data replacement strategies can be implemented.

\begin{figure*}
    \centering
    \includegraphics[width=1.00\textwidth]{Figures/IQRMVotingGraph.png}
    \caption{Illustration of the IQRM ``voting" system on a simple test case. Left panel: input data, deliberately generated so that all non-zero lagged differences are considered significant by the algorithm (through Eq. \ref{eq:outlier_definition}). IQRM was then run with a threshold of 3.0 and a radius of 2 channels, which corresponds to trial lag values of \{-2, -1, +1, +2\}. Middle panel: summary of votes cast by channel indices on their neighbours, represented as a directed graph. A vote $i \xrightarrow[]{} j$ is considered valid if and only if $i$ has cast \textit{strictly} less votes in total than $j$ has received; this ensures that neighbours of \textit{low} outliers are not abusively flagged. Here every vote cast by channel 6 is invalid. Right panel: final vote graph after pruning invalid votes; only channel 4 gets marked as a high outlier, reasonably so.}
    \label{fig:iqrm_voting}
\end{figure*}

\section{Tests on spectral statistic samples}
\label{sec:bandpass_tests}

In this section we test the efficacy of the IQRM masking process described in the previous section on spectral statistic samples obtained from real-world observations.

\subsection{MeerTRAP}
\label{subsec:bandpass_tests_meertrap}

MeerKAT is an array of 64 antennas in the Karoo region of South Africa \citep{Jonas2016}, each 13.5 metres in diameter and with the capacity to host four different receivers; both an L-Band (856$-$1712 MHz) and an UHF (544$-$1088 MHz) receiver are currently operational. The raw voltage signals are digitized at the antennas then streamed to the central correlator and beamformer (CBF), which handles channelisation followed by correlation and/or beamforming. The CBF is designed to cover the needs of most telescope users while remaining cost-effective and does not feature a massively multi-beam mode for radio transient searches, which greatly benefit from a wide field of view. However, MeerKAT can host additional backends called USEs (for user-supplied equipment), that may receive a copy of the antenna signals via the CBF and implement additional functionality. MeerTRAP is the commensal, real-time radio transient and pulsar search processor for the MeerKAT radio telescope. It is the association of two sets of user-supplied equipment:
\begin{itemize}
    \item FBFUSE: a GPU-based beamforming cluster capable of tiling the primary field of view of the telescope with over a thousand tied-array beams~\citep{Barr2018}.
    \item TUSE: a 66-node search cluster that ingests the beamformed data streams from FBFUSE and searches them for radio transients in real time. The raw data are immediately discarded after processing due to their large volume, and only a limited number of data products are kept.
\end{itemize}
A more detailed system description can be found in \citet{SPIEMeerTRAP}. The configuration of both components is flexible, but in most observations they are set so that TUSE searches 768 beams tiled in an hexagonal pattern centered on boresight position, and where the data are digitised to 8-bit precision, contain 1024 frequency channels and have a sampling interval of either 306 $\upmu$s (at L-Band) or 480 $\upmu$s (at UHF). In parallel to the search pipeline, each TUSE node runs a so-called bandpass monitor that measures and records spectral statistics of the data for every beam every 6 seconds, which has provided the test dataset for this section. A very important detail to mention here is how the mean and scale of the beamformed data are set on FBFUSE before they are 8-bit digitised. The process is as follows:

\begin{enumerate}
    \item A command to initiate a data rescaling is received, which happens when moving to a new source or at the user's request.
    \item FBFUSE waits until its first input ring buffer block has been filled with data from the CBF. The block has three dimensions: antenna index, observing frequency and time. It contains approximately 0.3 seconds worth of data.
    \item The complex gain corrections are applied to the block.
    \item The mean and standard deviation of the block are calculated for each channel across \textit{all} antennas taken together.
    \item Using these estimates, a scale and an offset term are chosen such that the beamformed data has an expected mean of 127 and standard deviation of 7.0. The scale and offset terms are kept constant and applied to all subsequent blocks until the next rescaling command is received.
\end{enumerate}
The calculation of the spectral scale and offsets assumes that the input data is idealised Gaussian noise: that is, the addition of N signals with unit scale from N independent antennas is expected to result in a beamformed output with a scale of $\sqrt{N}$. However, this assumption breaks down in any channel containing highly directional RFI: in these, the signals from distinct antennas retain a certain level of coherence, and the output scale is thus significantly larger than $\sqrt{N}$. Because of this specific data scaling procedure, the spectral standard deviation of the beamformed data received by TUSE has been found to be an excellent indicator of RFI contamination, and can be passed directly to IQRM. The efficacy of IQRM on MeerKAT is shown in Fig. \ref{fig:meertrap_before_after}, on a 5-hour sequence of spectral standard deviation samples acquired at L-Band.
\begin{figure*}
    \centering
    \includegraphics[width=1.00\textwidth]{Figures/MeertrapIQRMBeforeAfterRed.png}
    \caption{Test of IQRM on 5 hours of MeerTRAP L-Band, 1024-channel data with a 306.24 $\upmu$s sampling interval. Top panel: spectral standard deviation of the data stream from a single tied-array beam as a function of time, measured on consecutive 6-second data blocks. The beamformer aims to produce 8-bit sampled data scaled to a mean of 127 and a standard deviation of 7.0, which in practice happens only in channels dominated by background noise; higher values are a reliable indicator of RFI (see \S \ref{subsec:bandpass_tests_meertrap} for further details). The colour scale has been clipped for readability; some channels can reach a standard deviation $> 100$. Vertical lines correspond to short processing interruptions caused by target changes, and adjustments of the beam tiling pattern on sky that occur every 10 minutes.  Bottom panel: same plot after running IQRM on every spectral standard deviation sample (i.e. every column in the top panel plot), using a radius $r=100$ and a threshold $t=3.0$. Channels flagged by IQRM are shown in orange.}
    \label{fig:meertrap_before_after}
\end{figure*}

\subsection{Lovell 76-m radio telescope}
\label{subsec:Lovell}
The 76m Lovell telescope (LT) is located at the Jodrell Bank Observatory in the UK. It lies approximately 25 km away from Manchester's city centre and 15 km away from its international airport. The RFI environment is thus particularly adverse compared to MeerKAT or any telescope located in a sparsely populated, radio quiet area. The observing system has been previously described in \citet{Rajwade2020}. The effective observing band spans 336 MHz between 1396 and 1732 MHz divided into 672 frequency channels, and the data are digitised to 8-bit precision. In contrast to MeerTRAP however, the spectral standard deviation of the data does \textit{not} correlate well with RFI contamination here; the variance of the data was often found to be smaller in interference-ridden channels. The most plausible explanation is that the low-noise amplifiers of the receiver operate outside of their linear range in the presence of strong RFI, a regime beyond which their output signal power may become a decreasing function of input signal power, contrary to design expectations (M. Mickaliger, priv. comm.). It was thus necessary to use another spectral moment as an input to IQRM. We found a good alternative to be the absolute value of the spectral autocorrelation with a 1-sample delay, |ACF1| hereafter, defined as:
\begin{equation}
    \mathrm{|ACF1|} = \frac{\bigl\lvert  \sum_{i=0}^{n-2} (X_{i+1} - \mu)(X_i - \mu) \bigr\rvert}{n \sigma^2},
\end{equation}
where the $X_i$ denote a sequence of $n$ samples within a given frequency channel, and $\mu$ and $\sigma^2$ are the empirical mean and variance of the $X_i$ respectively. |ACF1| takes values in the range [0, 1] and has all the required properties to be used as a proxy for RFI contamination with IQRM:
\begin{itemize}
    \item Higher values indicate a larger departure from ideal uncorrelated Gaussian noise, for which the expected ACF is 0 regardless of time lag.
    \item It has a computational cost only marginally larger than the spectral standard deviation, which makes it practical for real-time processing.
    \item It is independent from the scale of the data, and thus insensitive to changes in spectral standard deviation caused by the occasional level compression issues faced on LT data.
\end{itemize}

A test on a 32-minute sequence of spectral |ACF1| from an observation of PSR B0611$+$22 can be seen in Fig. \ref{fig:jodrell_before_after}, and shows that IQRM can be used with different spectral statistics as an input. There are however short time periods in LT data during which they become essentially worthless across the whole band, as can be seen for example around 1100 seconds after the start of the test observation. This is a case that IQRM cannot properly handle by design, since a fundamental assumption is that the majority of the frequency channels are reasonably clean. In this case, IQRM tends to recommend leaving most or all of the band unmasked, whereas the sensible action would be to discard the data block entirely.

\begin{figure*}
    \centering
    \includegraphics[width=1.00\textwidth]{Figures/JodrellB0611_BeforeAfter_ACF1.png}
    \caption{Application of IQRM to a 32-minute Lovell telescope observation of PSR B0611+22, with a 256 $\upmu$s sampling interval and 672 frequency channels. On Lovell data the spectral standard deviation is a much less powerful RFI indicator than on MeerKAT (see \S \ref{subsec:Lovell} for an explanation), and we have thus selected another spectral statistic: the absolute value of the autocorrelation, with a delay of 1 sample. Top panel: spectral autocorrelation measured on consecutive 0.8-second blocks. Bottom panel: same data where IQRM has been applied to every column in the top panel; we used a threshold $t=3.0$, and radius $r=67$, i.e. 10\% of the number of channels. Flagged channels are shown in orange. A pathological case of the algorithm is visible on a few short time intervals here: it fails when a majority of channels are dominated by RFI.}
    \label{fig:jodrell_before_after}
\end{figure*}

\subsection{Execution speed}

We measured the run times of the \textsc{iqrm apollo} implementation on search-mode data files with different numbers of frequency channels, all containing artificially generated Gaussian noise. These test files were 60~seconds long with a sampling interval of $\tau = 256~\upmu s$, and with a number of channels equal to all powers of two between 256 and 16384 inclusive. We set the IQRM parameters to a radius equal to 10\% of the number of channels and a threshold of $t = 3.0$. The data were processed in blocks of 4096 samples (1.05-second long), and for each block a channel mask was calculated. We ran the code twice on each file, once using spectral standard deviation as the spectral statistic of choice, and once using the spectral autocorrelation with a lag of 1 sample (|ACF1| as described above). In both cases we recorded separately the total time taken by the calculation of the spectral statistic and the total time consumed by the IQRM algorithm \textit{per se}, i.e. the process described in \S \ref{subsec:algorithm_statement}. A single Intel\textregistered~Xeon E5-2630 CPU core was used.

The benchmarks results can be seen in Fig. \ref{fig:runtime_vs_nchan}; rather than displaying the raw execution times, the ratio between the total data duration and the total run time of each task is shown, which shows how much faster than real time they can be completed in a streaming environment such as MeerTRAP. We note that these results must be linearly extrapolated for a data sampling interval different from $\tau = 256~\upmu s$; indeed the run time scales linearly with the total number of time samples, but not necessarily with the data duration. On both the MeerTRAP and Lovell telescope observing systems described above, IQRM can be run approximately 100 times faster than real-time on a single core of a recent CPU. It should be noted that the calculation of the spectral statistic dominates the total cost, and in principle, this task could be significantly accelerated using many-core architectures compared to the single CPU core used here. A further speedup of one order of magnitude may be achievable assuming that computing resources are plentiful enough.

\begin{figure}
    \centering
    \includegraphics[width=1.00\columnwidth]{Figures/IQRM_cpp_benchmarks.png}
    \caption{Benchmarks of the \textsc{iqrm apollo} implementation. Using a single CPU core, we ran the code on several test data files containing artificial Gaussian noise, with a sampling time $\tau = 256~\upmu s$ and a number of frequency channels ranging from 128 to 16,384. The data were processed in consecutive blocks of 4096 time samples, approx. 1.05-second long, and a channel mask calculated for each block. We separately measured the average time consumed by the calculation of the spectral statistic (we tested standard deviation and |ACF1| separately), and by the IQRM masking algorithm \textit{per se}. The graph shows the ratio between the effective data length and the execution time of each task, i.e. how much faster than real-time each task would run in a streaming environment.}
    \label{fig:runtime_vs_nchan}
\end{figure}

\section{Tests on single pulse searches with the Lovell telescope}
\label{sec:sps_tests}

Although the experiments done in the previous section are an important visual check of the IQRM algorithm behaviour on its input, the spectral moment passed to the algorithm constitutes only a proxy for interference contamination. Masking all channels for which the spectral moment of choice is deemed high constitutes no guarantee that RFI has been eliminated. The ground truth, that is which exact sections of data are affected by RFI, is never available in practice. The only rigorous way to judge the efficacy of an RFI mitigation method is by measuring how it improves the tangible outcome of the scientific experiment being conducted. In a search for radio transients, one wants to maximize the number of detected pulses from genuine sources while minimizing the number of false positives. In this section, we set out to measure how IQRM improves a standard single pulse search compared to a fixed channel mask, using observations of known pulsars taken with the Lovell telescope.

\subsection{Experimental setup}

In order to test the RFI mitigation efficiency of IQRM, we gathered 30-minute long search-mode observations of four known pulsars recorded with the Lovell 76-m radio telescope at 1.4~GHz, with the goal of submitting them to a single pulse search. The observing setup was the same as described above (\S \ref{subsec:Lovell}), with a time sampling interval set to $\tau = 256~\upmu$s. The test sources were PSR~B0611+22, PSR~B0919+06, PSR~J1819-1458 and PSR~B0531+21 (the Crab pulsar), whose parameters are shown in Table \ref{tab:sps_test_pulsars}. These were selected such that we would be able to detect a large sample of individual pulses within the combined two hours of data available, and where a significant fraction of these pulses would be registered with a signal-to-noise ratio near the typical detection threshold of a transient search. The processing pipeline consisted of three stages, following the usual processing model for large-scale radio transient searches:
\begin{itemize}
    \item Search. The data were passed to the GPU-based \textsc{heimdall} search code; the signal-to-noise ratio threshold was set to 6, and the search dispersion measure (DM) range to $[0, 350] \mathrm{pc~cm}^{-3}$. The default internal RFI mitigation options of \textsc{heimdall} were enabled, which consist of narrow-band RFI clipping and a zero-DM filter \citep{Eatough2009}.
    \item Automated candidate classification. The candidates returned by \textsc{heimdall} were passed to \textsc{FETCH} \citep{Agarwal2020}, a radio transient classifier based on a deep convolutional neural network.
    \item Manual candidate vetting. The candidates reported as positive by FETCH were visually inspected; the ones confirmed to be originating from the pulsar were given a final positive label, the others were overruled as negative.
\end{itemize}
On each test data file, we ran two identical instances of the pipeline, which processed copies of the data masked with different methods:
\begin{enumerate}
    \item One instance processed the original data files with only a static channel mask applied. This mask is the ``standard" for the Lovell telescope at 1.4~GHz and was determined independently of the present work, mainly via visual inspection of candidates produced by FRB searches conducted with the same backend \citep[][Mickaliger et al. in prep.]{Rajwade2020}.
    \item The other instance processed copies of the data files that had first been cleaned with the \textsc{iqrm apollo} implementation, where we used |ACF1| as the contamination metric. No static mask was applied.
\end{enumerate}
A comparison of the static channel mask with the average IQRM mask inferred for each test pulsar observation can be found in Fig. \ref{fig:iqrm_vs_static_mask}, where changes of RFI environment between observations are evident.

\begin{table}
    \centering
    \caption{Test pulsars chosen for the single pulse search experiment with the Lovell 76-m telescope at L-Band. DM denotes the dispersion measure and $S_{\mathrm{1400}}$ the average flux density at 1400 MHz. These sources were chosen so that: a) a large sample of detectable single pulses would be recorded in a short amount of observing time (30 minutes for each source), and b) the majority of detectable pulses would be near the typical detection threshold of a search, in a regime where the benefits of improved RFI mitigation are most evident. The data in this table are from the ATNF pulsar catalogue \citep{PSRCAT}.}
    \label{tab:sps_test_pulsars}

    \begin{tabular}{lS[table-format=4.1]S[table-format=3.2]S[table-format=2.1]}
        Pulsar           & {Period (ms)} & {DM ($\mathrm{pc~cm}^{-3}$)} & {$S_{\mathrm{1400}}$ (mJy)} \\
        \hline
        PSR B0531$+$21   & 33.4   & 56.77  & 14  \\
        PSR B0611$+$22   & 334.9  & 96.91  & 3.3 \\
        PSR B0919$+$06   & 430.6  & 27.30  & 10  \\
        PSR J1819$-$1458 & 4263.2 & 196.0  & {N/A} \\
        \hline
    \end{tabular}
\end{table}

\begin{figure}
    \centering
    \includegraphics[width=1.00\columnwidth]{Figures/Masking_Frequencies_Jodrell.png}
    \caption{Changes in RFI environment between test pulsar observations taken with the Lovell telescope, underlining the need for adaptive time-dependent RFI masking. The percentage of time intervals masked by IQRM as a function of channel frequency is shown in blue. The static channel mask shown in grey is the one commonly used for Lovell L-band data and was determined prior to, and independently of this work (see text). Note that many RFI-contaminated channels were saturating near the end of the PSR B0919$+$06 observation and were thus left unmasked by IQRM, which explains the apparent clipping of the graph.}
    \label{fig:iqrm_vs_static_mask}
\end{figure}

\subsection{Results}

The outcome of the search is plotted in detail in Fig. \ref{fig:sps_summary}, where for each pulsar and channel masking method, the candidates reported as positive by FETCH and confirmed to be genuine pulses by visual inspection are shown in a dispersion measure (DM) versus time plane as blue circles. Grey crosses denote the candidates labelled as negatives by FETCH, most of which emerge from RFI that was not entirely masked. Table \ref{tab:sps_candidate_counts} shows the total number of confirmed pulses and of negative candidates produced using either RFI mitigation method.

It is worth discussing briefly two experimental outcomes that significantly deviate from the average in Table \ref{tab:sps_candidate_counts}. Firstly, the exceptionally large decrease of 96\% in negative candidates in the PSR B0531$+$21 observation; closer inspection of the data showed that in this case, most candidates produced in the static mask run are caused by the RFI source occupying the 1450$-$1470 MHz frequency range, which manifests itself as a square wave with a period usually equal to two time samples. This interference source was active in other observations, for example in the PSR B0611$+$22 one, which can be seen in Fig. \ref{fig:jodrell_before_after}, where it did not cause a significant portion of the spurious candidates; however, it was unsually bright during the PSR B0531$+$21 observation, and the fact that the incriminated frequency band is \textit{not} part of the static channel mask had a significant impact. In contrast, IQRM nearly always masked this portion of the spectrum.
Secondly, the larger than average increase in genuine pulse detections for PSR B0611$+$22 is worth mentioning. We attribute it to the fact that, during this observation, the vast majority of pulses from the source had signal-to-noise ratios (SNRs) concentrated near the threshold of the search code, which was set to 6. In this regime, the consistent increase in SNR provided by the use of IQRM pushed dozens of pulses just above said threshold.

From the above results it is apparent that replacing the commonly-used Lovell static channel mask by IQRM is clearly beneficial and that there is no trade-off in doing so: for every test source, less negative candidates are reported by the search code and more importantly, a significantly larger number of genuine pulses are found. There is overall a threefold decrease in the negative candidate rate and a nearly threefold increase in the number of astrophysical events detected.

\begin{figure*}
    \centering
    \includegraphics[width=1.00\textwidth]{Figures/IQRM_SPS_Summary.png}
    \caption{Summary plot of the test single pulse searches. The four known pulsars in Table \ref{tab:sps_test_pulsars} were observed for 30 minutes with the Lovell 76-m telescope at L-Band, and the resulting data were searched with two distinct pipelines: one using a static channel mask, and the other using IQRM instead. Both pipelines were otherwise identical. Confirmed pulse detections are shown as blue circles, with the dispersion measure of the pulsar shown as horizontal dashed lines. Grey crosses correspond to candidates labelled as negatives by the \textsc{fetch} classifier. Time is measured in seconds since the start of each 30-minute observation. All marker sizes are proportional to the signal-to-noise ratio of the event as reported by the \textsc{heimdall} search code. The total number of pulses and of negative candidates for each case is reported in Table \ref{tab:sps_candidate_counts}.}
    \label{fig:sps_summary}
\end{figure*}

\begin{table*}
    \centering
    \caption{Summary of candidate numbers produced by the test single pulse searches. Using IQRM instead of the static channel mask resulted both in a reduction in the number of spurious candidates reported, and in a significantly increased number of astrophysical events identified.}
    \label{tab:sps_candidate_counts}
    \begin{tabular}{lllllll}
                     & \multicolumn{3}{c}{\textbf{Confirmed pulse detections}} & \multicolumn{3}{c}{\textbf{Negative candidates}}  \\
    Source           & Static Mask   & IQRM           & Change                 & Static Mask    & IQRM          & Change           \\ \hline
    PSR B0531$+$21   & 51            & 108            & $+112\%$               & 7079           & 312           & $-$96\%          \\
    PSR B0611$+$22   & 19            & 95             & $+400\%$               & 2883           & 2423          & $-$16\%          \\
    PSR B0919$+$06   & 5             & 14             & $+180\%$               & 1442           & 941           & $-$35\%          \\
    PSR J1819$-$1458 & 3             & 9              & $+200\%$               & 1134           & 629           & $-$45\%          \\ \hline
    \textbf{Total}   & \textbf{78}   & \textbf{226}   & $\mathbf{+190\%}$      & \textbf{12538} & \textbf{4305} & $\mathbf{-66\%}$ \\
    \end{tabular}
\end{table*}

\section{Discussion and Conclusion}
\label{sec:discussion}

Implementing effective RFI mitigation is now a necessity on any radio telescope; the task is made even more challenging on recent massively multi-beam systems where real-time processing is required. The widely-used static channel masks cannot effectively cope with the RFI environment variations as a function of pointing position or time of the day for example. We have introduced IQRM, an algorithm that infers a time-dependent, adaptive channel mask directly from consecutive blocks of search-mode data. Moreover, it can perform this task much faster than real-time (approximately 100 times faster on Lovell telescope data, with a single CPU core); it should be possible to integrate it into any existing data processing pipeline without measurably affecting the speed of the overall search process. Our tests have shown that in a radio transient search, IQRM is able to largely reduce the number of false positives but also enhance the S/N of genuine astrophysical events; on Lovell telescope data, the number of individual pulses detected from a set of four known pulsars increased by an average factor of 3 when replacing the standard, commonly-used static channel mask by IQRM.

While IQRM is non-parametric and should transfer well between different telescopes, we do note that one important task is still left to the user: finding a spectral moment or statistic that correlates well with the presence of RFI on the observing system at hand. Our analysis above indicates that the correct choice depends chiefly on the dynamic range of the input data. We found that on MeerKAT, the specific method used to determined the data digitisation levels is such that the standard deviation of the data in a frequency channel is an excellent proxy for RFI contamination. On the Lovell telescope however, the dynamic range of the data was found to be unexpectedly \textit{reduced} in the presence of strong RFI bursts, due to a technical limitation of the low-noise amplifiers; this motivated using the spectral autocorrelation of the data as an input to IQRM instead. With that in mind, the IQRM \textsc{C++} implementation provides multiple spectral statistics to choose from. The code is modular and users may add new ones if need be.

Much like all existing techniques, IQRM is not universally effective against all forms of RFI and is primarily used as a bright narrow-band interference masking algorithm. Its main limitation is that the estimator it uses to infer the acceptable range for its input spectral statistic (Eq. \ref{eq:outlier_definition}) assumes that at least 50\% of the channels are clean at any point in time; the method becomes less effective otherwise and masks less data than expected, a situation that was found to occasionally occur on Lovell telescope data. A possible solution would be to add a short-term memory to the algorithm, i.e. take into account the spectral statistics of past data blocks when determining the acceptable range for said statistic in the most recent block; this could help in making the correct decision when all channels become temporarily contaminated (see Fig. \ref{fig:jodrell_before_after}). Another potential application of IQRM would be to use the outlier detection algorithm on the data integrated in time, i.e. the zero-DM time series. This could make it useful against broadband, non-dispersed RFI pulses, much like the popular zero-DM filter devised by \citet{Eatough2009}, but without systematically reducing the S/N of genuine, low DM astrophysical events. The two aforementioned extensions will be considered for a later update of the \textsc{iqrm apollo} implementation.

\section*{Acknowledgements}

VM, KMR and BWS acknowledge funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 694745). The authors thank Mitchell Mickaliger for useful technical discussions about the Lovell telescope backend. The authors would  like  to  thank  the  South African Radio Astronomy Observatory (SARAO)  for  the  approval to use MeerKAT data for the analysis presented in this paper. The MeerKAT telescope is operated by SARAO, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Pulsar research at Jodrell Bank and data acquisition with the Lovell Telescope is supported by a consolidated grant from the UK Science and Technology Facilities Council (STFC).

\section*{Data Availability}
The data used in the analysis presented here are publicly available in a repository on Zenodo\footnote{\url{https://doi.org/10.5281/zenodo.5256885}}.

\bibliographystyle{mnras}

redpajama_set_name: RedPajamaArXiv


text:
Lily Ex Machina
Lily Ex Machina Fanart
Seeking Elysium
A 3% Chance for Glory
Seeking Elysium Volume 1 Chapter 23ExtantLily2017-08-07T02:22:46+08:00
The Tax Collector Cometh
Rosalind and I were drinking coffee in the tavern, waiting for the first guests to arrive. It was a little past two in the afternoon, but we had been open for several hours. In those days, with decreasing patrons, we had been closing up earlier, and thus going to bed earlier – as a result, we ended up waking up earlier, and therefore opening earlier. Not that there was much point to doing so. Nobody came to a tavern in the middle of the day. As a result, we spent our time idly, an air of gloom settling over us. Both of us knew why our patrons had decreased – the new tax increases had cut down on the amount of disposable income our patrons held. There was simply less money to spare for frivolities like drink. Due to the initial popularity of our shop, we were still sitting rather comfortably above the cost line, but those customers that came were often rowdy and noisy, spending all their money on drinking to forget some grievance or other. Our female customer base had declined considerably, such that Rosalind and I adjusted our shifts such that she spent more time entertaining, while I spent more time in the kitchen.
However, while we had not yet entered into the red, there was no guarantee that such a situation would last. The continued decline of our consumer base was one issue, but on top of that, we had yet to be re-evaluated by the tax agents. Due to constitutional rules, the new Lord Rishard was unable to make changes to existing tax percentages for a period of at least three years. Instead, what his purported "economic reforms" had done was to introduce new taxable items, and tax agents were thus being sent to every household in the land to do a reassessment of the tax amount owed by each household. We passed every day awaiting the arrival of this agent, whose evaluation would then decide our fate.
As Rosalind and I were making small talk at the counter, the door opened. We turned towards it, taking in the polished shoes, the black slacks, the white tunic and the smart black necktie of the tax agent that stood there. Rosalind greeted him first.
"Mister Edgar. It's been a while."
The tax agent, Edgar, took off his fedora and placed it across his chest, revealing his slicked-back brown hair and his sharp green eyes. He bowed his head slightly.
"Miss Rosalind. Miss Elysium. I'd like to say 'always a pleasure', but I'm afraid I've come on rather unsavoury business."
Edgar placed his fedora on a hook by the door and took a seat at one of the tables. As the tax collector assigned to this town, I had met him on a number of occasions, each time when he came by to collect the monthly taxes. He struck me as a professional, well-mannered man who took pride in his work, but not necessarily in his job. I briefly wondered what manner of fate led him to become a tax collector, when his disposition and abilities seemed to lend themselves to bureaucracy. Putting that aside, I went over to the table and sat down next to him – I was better at business talk than Rosalind.
"I suppose you're here to perform the tax evaluation?"
Edgar smiled at me apologetically.
"I'm afraid so. However, if it's possible, could I perhaps have my meal before making the evaluation? I'm rather afraid you might poison me otherwise, and I'm terribly hungry."
I raised an eyebrow.
"That bad?"
"Oh, yes. The grocer tried to lock me in his storeroom – would have done it, too, if his wife hadn't stepped in. And the butcher took up his knives with a dangerous look – suffice to say, I left as soon as I could. So you can see my concern."
I looked with some amusement at the troubled expression that was showing on Edgar's face. But at the same time, I felt a tinge of worry. What kind of tax additions could spark off such a response? I decided that I would at least help the man relax.
"Well then, good sir, may I take your order?"
While Edgar recited his order, Rosalind looked at me in surprise – clearly, she had expected me to refuse, so as to get the matter with as quickly as possible. In truth, that was my first instinct. But seeing Edgar, sitting in the chair and sighing, well… my heart went out to the poor guy. He had a hard job. I looked at Rosalind and met her eyes, trying to convey my intention to at least help him calm down. Maybe she understood, but Rosalind gave me a smile and nodded.
Once Edgar's lunch and drink had been served and consumed, he coughed and sat up straight. Rosalind and I both watched him attentively. He pulled out a piece of parchment and read off it.
"In the name of the lord of the region, Lord Rishard, I shall hereby reveal the results of the tax bureau's evaluation of this premises. First off, may I have your income declaration for this month?"
Rosalind, rummaging through the drawer hidden behind the counter, pulled out a sheet of parchment with this month's expenses and earnings handwritten on it and placed it in Edgar's waiting hands.
Edgar confirmed the contents, then tied it off with a string and placed it in his bag. He then pulled out another piece of parchment, one that had a table on it, and began to read off items from it while glancing around the shop.
"Business tax, 15%"
That was normal. The business tax was the tax we paid for being allowed to do business on the grounds of this town. That was what we had been paying all along, so there was no change there.
"Land tax, 10%"
This was the tax that we paid as citizens living on the land owned by the Lord. Once again, this was an older item, so we had been paying this tax the whole time.
"That is all for the old tax items. Next is for the new taxable additions."
As expected, there was no change from the existing amounts for the older taxables. So far, we were sitting at the old amount of 25% of profits being due as taxes. Of course, the moment of truth would come after this – just what were Rishard's new taxes, and how would we be affected? Edgar continued speaking.
"Employee tax, 10%"
"Employee tax? What's that?"
Rosalind raised a voice in question. Of course, in my home country, we had employee taxes, but that was largely for foreign workers, meant to be a disincentive for companies to import labour. It seemed counterproductive to national employment to impose a tax on hiring local workers. Edgar put on an uncomfortable expression while explaining.
"Well, according to Lord Rishard, 'being able to hire an employee means you have money to spare. If you have money to spare, you should be giving it to me.' Of course, there are… many big problems with his reasoning, but he refuses to listen to counsel, and it's unfortunate, but the right to set taxes in this region ultimately falls to him, so that's the way it is. We did, at least, convince him to exclude family members from being considered as hired help, but it's still rather unreasonable."
I frowned. This was clearly an abuse of power. Since it only affected a small amount of the population, most of us would just endure it – and 10%, while being non-negligible, was not devastating, either. But I could foresee a future where if Rishard continued abusing power like this to fatten his coffers, he might face an uprising sooner or later. I looked at Rosalind, who was looking similarly troubled. We met gazes and shrugged in unison. 10% of profits would hurt, but we could still deal with it. I turned back to Edgar.
Edgar fidgeted around, clearly unwilling to carry on. However, as we silently waited for him to speak, he sighed and stowed away the parchment.
"Alcohol tax… 40%"
Rosalind's jaw dropped; I spontaneously stood up from my seat at the table. We were a tavern – our main product was alcohol. We had been selling alcohol for years – to suddenly impose a tax, and more to the point, what was with that ridiculous number, 40%?! That was too much, way too much. Edgar shrank back. I glared at him, even though I knew that it was not his fault.
"…explain."
"W-Well, Lord Rishard claims that the number of alcoholics in our land is too high, and so he's imposing a tax to try and drive the price up so that shops will stop selling it; he says he intends to make the country sober."
"…Is the tax imposed on the product cost or on our sales?"
"…it's 40% of your sales."
A tax on alcohol had its merits as a demerit tax, and if the intention was truly to sober up the country, a demerit tax on alcohol was common – however, such a tax was usually levied on the purchase of alcohol, that is, a flat tax on alcohol bought from suppliers – this would cause retailers to offset the increased cost by selling at a higher price. But taxing the retailer based on a percentage of earnings did nothing, aside from possibly causing the retailer to simply cut alcohol from its shelves. However, would the Lord Rishard who supposedly killed a barkeep for not carrying his favorite brand of alcohol truly want all the taverns in the land to stop selling liquor? It didn't add up. Even if his only intention was to fatten his coffers, this method was too irrational. However, before I thought on it further, Edgar spoke up again.
"And… the above, new taxations… are taxations on your earnings, not your profits."
Rosalind raised a shout from inside the kitchen and stormed out, looking like she intended to punch Edgar. Edgar took in a deep breath and closed his eyes, as if preparing to get hit. I held my arm across, blocking Rosalind's way. Rosalind glared at me, but I returned her stare with calmness, keeping my anger bottled inside.
"Ros. Calm down. He's just doing his job. Hitting him won't do anything."
Edgar breathed a sigh of relief and crossed over to the door, collecting his fedora.
"Because of the sudden changes, you have a month to raise the money. I'll be back a month from now to collect the taxes for this month, but if at that time you are unable to pay according to what you've listed here, well… I'm afraid we'll have to confiscate your property."
Rosalind surged forward, but I held her back by forcefully clamping a hand on her shoulder. I bowed my head slightly.
"Thank you for the consideration. Please take care on the road."
Edgar donned his fedora while opening the door.
"Yes, thank you for understanding. And… well… I'm sorry."
With that, the door closed, leaving Rosalind and myself in the empty tavern. Rosalind walked over to a chair and slumped down, holding her head in her hands.
"50% of earnings… that's way too much."
Leaning on the wall, I crossed my arms and said nothing. I was too busy thinking.
PREVIOUS CHAPTER | INDEX | NEXT CHAPTER
Subscribe and be the first to know whenever I publish any new content ^^
Support the Author!
Become a Patron and receive early access to new content!
Powered by GLORIOUS RUSSIAN LOLIS

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: In the command center in x-com enemy unknown the view is stuck. How to resolve that? When I open the command center in enemy unknown, the view is stuck at always the same place. I cannot move the globe and generally don't see anything else than north america. This seems to be the case after I lost an interceptor.
Any ideas how to resolve that?

redpajama_set_name: RedPajamaStackExchange


text:
Review of the Eligibility of Persons Converted From State Disability Rolls to the Supplemental Security Income Program
HRD-78-97: Published: Apr 18, 1978. Publicly Released: Apr 18, 1978.
Social Security Administration's (SSA) activities relating to an assessment of the continued medical eligibility of over 2 million disabled Supplemental Security Income (SSI) recipients were reviewed. SSA was asked to evaluate two samples of SSI disabled recipients including an evaluation of: (1) medical evidence supporting the disability determination of 402 recipients who were converted to the SSI program from state disability programs; and (2) more recent medical evidence on 175 recipients. Of the 402 converted recipients, only 152, 38 percent, had sufficient medical evidence in their files to support a disability decision. Thirty-six of the 152 cases, 24 percent, were not disabled as defined by appropriate state disability criteria. About 10 percent of the 175 recipients for whom current medical evidence was obtained were no longer disabled. Once they are approved for the program, the vast majority of SSI disabled recipients are not subject to medical reexaminations. SSA assumes that these recipients have impairments which will not improve. The Commissioner, SSA, should immediately establish appropriate mechanisms for systematically reviewing the disabled recipients' caseload so that persons no longer disabled can be removed from the rolls. SSA should establish and implement systems for periodically reassessing the adequacy of guidelines for establishing diaries for the total disability caseload and reviewing, on a priority basis, the disability determinations for converted recipients.
Find Recent Work on Worker and Family Assistance »
Fair Labor Standards Act:
Tracking Additional Complaint Data Could Improve DOL's Enforcement
GAO-21-13: Published: Dec 9, 2020. Publicly Released: Jan 8, 2021.
Amateur Athletes:
The U.S. Center for SafeSport's Response and Resolution Process for Reporting Abuse
GAO-21-128R: Published: Dec 18, 2020. Publicly Released: Dec 18, 2020.
Military Child Care:
Off-Base Financial Assistance and Wait Lists for On-Base Care
GAO-21-127R: Published: Dec 1, 2020. Publicly Released: Dec 1, 2020.
Drug Misuse:
Agencies Have Not Fully Identified How Grants That Can Support Drug Prevention Education Programs Contribute to National Goals
Federal Social Safety Net Programs:
Millions of Full-Time Workers Rely on Federal Health Care and Food Assistance Programs
GAO-21-45: Published: Oct 19, 2020. Publicly Released: Nov 18, 2020.
Child Care Facilities:
Federal Agencies Need to Enhance Monitoring and Collaboration to Help Assure Drinking Water is Safe from Lead
GAO-20-597: Published: Sep 28, 2020. Publicly Released: Oct 1, 2020.
Federal Employees' Compensation Act:
Comparisons of Benefits in Retirement and Actions Needed to Help Injured Workers Choose Best Option
GAO-20-523: Published: Jul 23, 2020. Publicly Released: Aug 21, 2020.
Social Security Disability:
Information on Wait Times, Bankruptcies, and Deaths among Applicants Who Appealed Benefit Denials
GAO-20-641R: Published: Aug 13, 2020. Publicly Released: Aug 13, 2020.
Child Welfare and Aging Programs:
HHS Could Enhance Support for Grandparents and Other Relative Caregivers
Economic Adjustment Assistance:
Actions Needed to Better Address Workers' Needs and Assess Program Effectiveness
Explore our Key Issues on Worker and Family Assistance
Employment in a Changing Economy
Improving and Modernizing Federal Disability Programs - High Risk Issue


text:
Events2HVAC Results: University of Southern California and Penn State University
University of Southern California and Pennsylvania State University are in the process of implementing Events2HVAC on their campuses. They recently shared the following energy-savings results.
Craig Shultenburg, Building Controls Systems Manager at USC, said they are achieving twice as much energy savings as they projected in their original business case for Events2HVAC.
USC has 242 classrooms controlled by Events2HVAC, out of their 501-room license. There are 29 air handlers (AHUs) serving those 242 rooms, and Events2HVAC returns them to unoccupied settings between scheduled events. Shultenburg used the horsepower ratings of the AHUs converted to KW and accumulated runtime hours to determine energy use. Results were then compared to baseline numbers to determine dollar savings.
Shultenburg said the combined 29 AHUs are averaging about $1,700/week in energy savings, which would be more than $80,000/year if the savings proves steady through the seasons. USC had originally projected $20,000 in energy savings in fiscal year 2019 and $40,000 in energy savings each year thereafter.
But this is only part of the total energy savings because there will be savings related to boilers and chillers too. And they still only have about half of their licensed rooms implemented.
Kevin Lynch, Direct Digital Controls Specialist at Penn State University, provided two examples of Events2HVAC energy-savings results.
The first analysis is based on a single air handler (AHU) serving 40 VAV boxes. When nothing is scheduled in the rooms the AHU serves, Events2HVAC automatically returns it to unoccupied settings. Since Events2HVAC implementation, the fan power for the AHU has been reduced an average of 22.5 kilowatt-hours (kWh) per day. Lynch expects annual savings for the fan to be roughly 8,200 kWh or more, with most of the savings occurring in the summer.
The second analysis is based on comparing chilled water use in a classroom building during August 2018 to chilled water use in the same building during August 2017. Lynch explained that the average temperature in August 2018 was higher than in August 2017, and other buildings on campus used more chilled water this year than last year for the month of August. But the classroom building where Events2HVAC was implemented to return HVAC equipment to unoccupied settings between events used 10,000 ton-hours less chilled water, a savings of about $2,300 for the month of August. Fall classes began on the 20th of August. The building was not heavily used until then.
In September 2018, Lynch said the same classroom building used 1,000 ton-hours less chilled water than the previous year, and the average temperature was higher than in September 2017.
Categories: News, Energy Efficiency, Articles, Case Studies, Higher Education
Location: Blogs Events2HVAC Product Blog


text:
Parliament - functions
Understanding Parliamentary Government
Parliament - structure
Parliament - committees
Parliament - legislative function
Parliament - Scrutiny
Parliament - legitimacy
Parliament - Representation
Parliament - Reforms
Parliament - an evaluation
REVISION TASKS
PARLIAMENT REVISION
Attempt the (a) & (b) question from this section A question & then 2 of the section B questions attached
Source question
Despite the arrival of potentially weaker coalition government in 2010, the UK parliament remains relatively powerless in the face of government power. The power of the party whips is based on prime ministerial patronage, party loyalty, collective responsibility & the constant threat that rebellious MP's will ultimately be dropped by their local party & so lose their seat. In the past, large government have enabled the government to drive through its legislation, despite determined opposition. Departmental select committees & a more active House of Lords are rare examples of parliamentary power. The select committees cannot enforce their recommendations & the House of Lords powers are limited by law. Even under coalition government, most legislative proposals are safe from parliamentary interference. New reforms now being proposed, however, suggest that the tide may be turning & that parliamentary power may increase in the future.
(a) With reference to the sources, outline two reasons why Parliament is dominated by the government
(b) With reference to the source & your own knowledge, how can Parliament check the power of the government?
(c) How has it been proposed to strengthen Parliaments control of the executive?


text:
Q: TinyMCE + Jquery + PHP + AJAX Special chars issue i have a TinyMCE implementation on my website that works great. But if the user enters a special char like "<" the query send to my MySQL database gets cropped just when the special char appears. Let me show you whats happining:
This is how i call TinyMCE:
tinyMCE.init({
        mode : "exact", elements : "e_text", theme : "simple", oninit : "setPlainText", plugins : "paste" });
        function setPlainText() {
            var ed = tinyMCE.get('e_text');
            ed.pasteAsPlainText = true;
            if (tinymce.isOpera || /Firefox\/2/.test(navigator.userAgent)) {
                ed.onKeyDown.add(function (ed, e) {
                if (((tinymce.isMac ? e.metaKey : e.ctrlKey) && e.keyCode == 86) || (e.shiftKey && e.keyCode == 45))
                    ed.pasteAsPlainText = true;
                });
            } else {
                ed.onPaste.addToTop(function (ed, e) {
                ed.pasteAsPlainText = true;
            });
            }
        }

Dont worry about searching for an error there, that code renders the TinyMCE with no errors. Then i have a jquery function that holds the ajax process:
    $("#save-btn").click(function() {
            tinyMCE.triggerSave();
            alert($('#e_text').val());
            $("#status").html(' Saving...');
            $.ajax({type: "POST",   url: "handler.php", data: "e_text="+ $('#e_text').val() + "&e_title="+ $('#extra_title').val(),
                        success: function(server_response){
                                $("#status").ajaxComplete(function(event, request){
                                alert(server_response);
                                        if(server_response.substring(0,4) == 'Last') {
                                            $("#status").html('  '+ server_response +'   ');
                                        }
                                        else {
                                            $("#status").html(' Not saved, please try again later. '+server_response+'');
                                        }

                                });
                        }

                });
    });

This call handler.php wich handle the ajax call and then return a result. For this particular problem i modify the php script just to print the POST var and see what is going on, or in other hands, what is arriving to the php script:
} elseif(isset($_POST['e_text'])) {
    $fgmembersite->DBLogin();
    echo($_POST['e_text']);
    mysql_query("INSERT INTO ".table_extra." (id,id_user,title,content) VALUES (NULL,'".$_SESSION['id_of_user']."','".mysql_real_escape_string($_POST['e_title'])."','".mysql_real_escape_string($_POST['e_text'])."') ON DUPLICATE KEY UPDATE title='".mysql_real_escape_string($_POST['e_title'])."', content='".mysql_real_escape_string($_POST['e_text'])."';");
    mysql_query("UPDATE ".$fgmembersite->tablename." SET last_update_cv = NOW();");
    $result = mysql_fetch_array(mysql_query("SELECT NOW();"));

    //echo 'Last saved ' . $result['NOW()'];
}

Ignore everything but the echo instruction. Returning to my jquery code you remember the two alert functions? Well one of them show me the data returned by the TinyMCE before sending it to the php script and the second one show the data returned data by the php script.
Lets suppose i insert the phrase "Hello world >>> Hello again!" into the TinyMCE, this is what i see:

Then i hit Save, and i got the alert number one:

I hit ok and i get the alert number two (the server response, what php got):

SO, WHY?!!! why is the php script (just and echo of what it gets via POST) is returning a cropped phrase? Just when the special char appear?
Thanks for any help you can offer me!

A: Those ampersands are messing with the URL format for the data property.
Why not change it to something like this instead?
data: {
    "e_text": $('#e_text').val(),
    "e_title": $('#extra_title').val()
}


text:
Equity, Outreach, & Wellness
Mathematics and Statistics Pictures
Current academic postings in Mathematics and Statistics
Position in Geometry
Tenure Track Position in Geometry
Tenure Track Positions
Wellness advice
www.mcgill.ca/mathsta/pwww.mcgill.ca/mathstat/postgraduate/current-students/graduate-orientationostgraduate/current-students/graduate-orientation
Eva Miranda (Polytechnic University of Catalonia, Spain)
14Jan202211:00
TITLE : Looking at hydrodynamics through a contact mirror: From Euler to Turing and beyond
PLACE : ZOOM
https://umontreal.zoom.us/j/93983313215?pwd=clB6cUNsSjAvRmFMME1PblhkTUtsQT09
Code secret : 096952
RESUME / ABSTRACT :
What physical systems can be non-computational? (Roger Penrose, 1989). Is hydrodynamics capable of calculations? (Cris Moore, 1991). Can a mechanical system (including the trajectory of a fluid) simulate a universal Turing machine? (Terence Tao, 2017).
The movement of an incompressible fluid without viscosity is governed by Euler equations. Its viscid analogue is given by the Navier-Stokes equations whose regularity is one of the open problems in the list of problems for the Millenium by
the Clay Foundation. The trajectories of a fluid are complex. Can we measure its levels of complexity (computational, logical and dynamical)?
In this talk, we will address these questions. In particular, we will show how to construct a 3-dimensional Euler flow which is Turing complete. Undecidability of fluid paths is then a consequence of the classical undecidability of the halting
problem proved by Alan Turing back in 1936. This is another manifestation of complexity in hydrodynamics which is very different from the theory of chaos.
Our solution of Euler equations corresponds to a stationary solution or Beltrami field. To address this problem, we will use a mirror [5] reflecting Beltrami fields as Reeb vector fields of a contact
structure. Thus, our solutions import techniques from geometry to solve a problem in fluid dynamics. But how general are Euler flows? Can we represent any dynamics as an Euler flow? We will address this universality problem using the Beltrami/Reeb mirror again and Gromov's h-principle. We will also consider the non-stationary case. These universality features illustrate the complexity of Euler flows. However, this construction is not "physical" in the sense that the associated metric is not the euclidean metric. We will announce an euclidean construction and its implications to complexity and undecidability.
These constructions [1,2,3,4] are motivated by Tao's approach to the problem of Navier-Stokes [7,8,9] which we will also explain.
[1] R. Cardona, E. Miranda, D. Peralta-Salas, F. Presas. Universality of Euler flows and flexibility of Reeb
embeddings. https://arxiv.org/abs/1911.01963.
[2] R. Cardona, E. Miranda, D. Peralta-Salas, F. Presas. Constructing Turing complete Euler flows in
dimension 3. Proc. Natl. Acad. Sci. 118 (2021) e2026818118.
[3] R. Cardona, E. Miranda, D. Peralta-Salas. Turing universality of the incompressible Euler equations
and a conjecture of Moore. Int. Math. Res. Notices, , 2021;, rnab233,
https://doi.org/10.1093/imrn/rnab233
[4] R. Cardona, E. Miranda, D. Peralta-Salas. Computability and Beltrami fields in Euclidean space.
https://arxiv.org/abs/2111.03559
[5] J. Etnyre, R. Ghrist. Contact topology and hydrodynamics I. Beltrami fields and the Seifert conjecture.
Nonlinearity 13 (2000) 441–458.
[6] C. Moore. Generalized shifts: unpredictability and undecidability in dynamical systems. Nonlinearity
4 (1991) 199–230.
[7] T. Tao. On the universality of potential well dynamics. Dyn. PDE 14 (2017) 219–238.
[8] T. Tao. On the universality of the incompressible Euler equation on compact manifolds. Discrete
Cont. Dyn. Sys. A 38 (2018) 1553–1565.
[9] T. Tao. Searching for singularities in the Navier-Stokes equations. Nature Rev. Phys. 1 (2019) 418–419.
Dept. of Mathematics and Statistics
/mathstat
Burnside Hall
McGill Home Page
Minerva Login
Restricted Pages
Department Meetings


The objective of this cooperative research project is to develop cool-season grass germplasm for use in dryland semiarid growing environments in the USA and PR China.
Collaborative experiments will be jointly designed and implemented in the field in Logan, Utah, and in Ningxia Hui Autonomous Region (Province), China, where research concerning the Germplasm evaluation of U.S. grass accessions (cultivars and experimental lines) and selection of superior germplasm for plant stand establishment and persistence under abiotic stressed environments will be conducted under dryland semiarid growing conditions. This research will develop improved plant materials (grasses) with improved: (1) stand establishment; (2) persistence; (3) forage yield; and (4) seed yield for dryland semiarid environments in the United States and China.


text:
Digital health and wellness continues to grow at CES 2020
Digital health and wellness features heavily at CES 2020, the gigantic consumer technology show that takes over Las Vegas this week and will be attended by around 170,000 people.
Away from the latest in high-end entertainment hardware and gadgets, peripherals and tech innovation from every facet of the economy, health forms a major part of the broader themes of smart homes, pervasive sensors, ever-smarter wearables and internet of things.
Long-term CES watchers note that the show is moving from focusing on the tech giants to how more traditional companies like Delta airlines and Daimler are innovating.
The CES Digital Health Summit, now in its tenth year, has a programme focused on how trying to make sense of the massive waves of disruption that the convergence of healthcare and technology are causing across the healthcare system, and how the citizen and consumer needs tools to navigate the fragmented landscape.
Much of the health tech featured here at CES 2020 is emerging from the blurring of the boundaries of consumer health and wellness and based on rapid evolution of AI, robotics, sensors and IOT.
What is stunning is the sheer variety of innovation and start-ups at the show, with 4,500 exhibitors and 1,200 start-ups, creating a blizzard of novel technology-based solutions for health, pointing to a fast-evolving mosaic of imaginative solutions to current health challenges.
Working out what will be the trends to watch and which innovations will win out is perhaps the biggest challenge of this fast-changing landscape.
Bisu Body Coach, for instance, is the world's first smart urine analyser that provides personalised health advice. As the company says "Every day we flush huge amounts of info on our bodies down the toilet".
Another sensor tech launched at the show from Brilliant Company, uses sensors, cloud and analytics "to track the quality of the air you breathe", it's Pico Go device is a digital therapeutic for paediatric asthma patients, offering data tracking in real-time and interventions to help management of asthma.
One UK start-up at the show is DnaNudge, which is offers consumers suggestions on what groceries to buy based on their DNA, taken by a swab, and linked to a smart band.
There are many examples to refine better technology solutions to medical conditions. Glutrac, falls in this category, as a smart wearable featuring Non-Invasive Continuous Glucose Monitoring.
Groovex's Lovot, which is described the world's first truly friendly robot. was on display at CES 2020.
Or there are any number of Carebots, robotic companions for nursing homes and home care to both monitor their 'owners' and tackle loneliness. Groovex's Lovot, for instance, is described the world's first truly friendly robot.
Other themes pushing the boundaries in Sleeptech, Sextech (which caused a storm last year when a CES innovation award was withdrawn from Lori DiCarlo) and Cannabistech, seem uniquely well-suited to Sin City.
A wide variety of sleeptech on show at CES forms part of a much wider range of wellbeing technologies, all aiming to help us better deal with the constant demands and stresses and of our 24/7 sleepless digital lives, through more tech.
Innovators in this area include Pulsewear, maker of Dreamon, "a wearable that that helps users fall asleep faster without the use of drugs".
The decision to invite the President's daughter Ivanka Trump to give a 2020 keynote address, overlooking many high profile female tech leaders has caused a lot of bad feeling.
Digital Health News will be reporting from CES across the week, so stay tuned.
Emerging tech and trends to watch from CES 2020
Taiwan targets digital health and smart tech for exports
Mobile health and apps news in brief
CES 2020 | digital health
Jon Hoeksma
More in Digital Patient, News
Number of NHS App users double in three months as ambassadors get to work
The number of NHS App users has risen from 91,000 in September 2019 to over 220,000, following the launch of a campaign to promote the...


(S//SI) Hand held finishing tool used for geolocating targeted handsets in the field.
(S//SI) Tactical Operators use WATERWITCH to locate handsets (last mile) where handset is connected to Typhon or similar equipment interrogator. WATERWITCH emits tone and gives signal strength of target handset. Directional antenna on unit allos operator to locate specific handset.
There is definitely a logic to these codenames. WATERWITCH is used by "Tactical Operators" (commandos) for wetwork, the assassination or rendition of persons using targeted phones. It's good to know that NSA aren't entirely beholden to drone technology; as in the title of a Petri movie, "We Still Kill The Old Way".
It's also clear that 'active interrogator' is a description of a basestation that impersonates a cell tower. By controlling the timing and power of the captive phone's RF emissions, it interrogates actively. This is analogous to passive and active attacks on cryptosystems.
now what is active interrogator?
It's also clear that 'active interrogator' is a description of a basestation that impersonates a cell tower.
so the question is, what this is. Is it a reaper or a predator drone? Or is it just the cell tower implant of the drone?
so, reaper?, predator?, or just a cell tower for a drone?
It could even be a hand held unit.
The point is GSM will "handover" to whichever is the strongest BTS it receives.
from the capabilities, certainly. But from the viewpoint that you need a proof of nsa associated with killing people, it would be interesting, if we could know on what airoplaine active interrogator is mounted.
Given that the display is an oddball size of e-ink, and that there are only one or two companies making e-ink displays, it seems like it would be easy to figure out (at least for people in the vending company) who is procuring the parts, and maybe unmask the company that is making the stuff for the NSA, or at least unmask a front company used for procurement.
Dowsing is a magical technique for finding water. The name WaterWitch makes good sense if you know of this bit of folklore.
Well i think it is safe to say, as long as it is like this, all these TAO implants must be published in the open.
Hamid Karzai was in the midst of negotiating a security agreement with the United States when he met a 4-year-old girl who had lost half her face in an American airstrike.Five months later, the Afghan president's eyes welled with tears as he described visiting the disfigured little girl at a hospital. He took long pauses between words. Sitting behind his desk Saturday night, the man who has projected a defiant image toward the West suddenly looked frail.
That day, I wished she were dead, so she could be buried with her parents and brothers and sisters" — 14 of whom had been killed in the attack — he said.
In an unusually emotional interview, the departing Afghan president sought to explain why he has been such a harsh critic of the 12-year-old U.S. war effort here. He said he's deeply troubled by all the casualties he has seen, including those in U.S. military operations. He feels betrayed by what he calls an insufficient U.S. focus on targeting Taliban sanctuaries in Pakistan. And he insists that public criticism was the only way to guarantee an American response to his concerns.
What we see here is a spectacular compromise of extremely important classified information. Enemies of the US now know a lot more about American methods and capablities than they may have known before.
One can be sure that all of these documents have been gone over with a fine-toothed comb by Moscow and Beijing.
Yes, probably four or five years before Ed Snowden started collecting the info. Simple logic dictates that if Ed could gather that amount of information and the NSA did not have a clue, then their internal controls are grossly deficient to the point of uslessness.
And a study of human nature will tell you that others will have exploited such failings for profit or ideology.
As for "capabilities" nearly all of them are stolen / copied from other people in the private sector, usually without accreditation or payment.
For this TAO device go and read the 1996 book "Takedown" by Tsutomu Shimomura and journalist John Markoff.
In there you will find a description of Shimomura modifing a mobile phone into a tracking device. He is also known to have worked for and with the NSA, as well as testifing before Congress on the security --or lack there of-- cellular phones.
All of which raises the "Chicken and egg" question over this device and Shimomura... Did Shimomura think up the attack himself or was it something he saw in the NSA and copied, and then wrote up as a self promotion and in the process perform what you call "a spectacular compromise of extremely important classified information".
The simple fact is all the TAO catalogue has shown is just how far behind the curve the NSA is, and just what a bunch of idea stealing low lifes they are, and thus not due any respect what so ever.
Signal strength is not the point.
Yes, Snowden robbed the cookie jar and he got away. A series of safeguards must have failed: counter-intelligence, physical security, etc.
But one does not usually think of "NSA" and "behind the curve" in the same sentence.


text:
Archaeology within Memory
2 Comments on Archaeology within MemoryPosted in BotP By Ed PevelerPosted on 22nd November 2019 22nd November 2019
Whilst most of Wendy's and my time is spent considering prehistory, i.e. archaeology from before the Roman invasion, the LiDAR data is showing us monuments and sites from far more recent periods, and even earthworks created within living memory. I think it's a really interesting question as to when "archaeology" stops and when the "modern" begins. This is a question that we have been asked by a number of people, wanting to know whether to record certain things in the LiDAR portal.
The Cruise Missile shelter complex at Greenham Common, Berks., possibly the youngest Historic England "Scheduled Monument," having still been in use as a military base into the early '90s. It was scheduled in 2004.
Photo © Copyright Trevor Hickman and licensed for reuse under a Creative Commons Licence.
As you can see from the example above, perhaps the nation's youngest scheduled monument, age isn't necessarily a primary consideration in whether a site might be considered of historic or archaeological significance. Greenham Common's Ground Launched Cruise Missile silos were only emptied of their nuclear weapons between 1989 and 1991, just 13 years before the site was Scheduled by Historic England.
And we have plenty of other 20th Century archaeology that we certainly want to record. During the First World War a significant number of troops passed through training camps in the Chilterns, including more than 12,000 who trained at Berkhamsted over the course of the war. Fitness, discipline, marksmanship, manoeuvres, and trench-digging might have been core elements. And some of these have left their mark written into the landscape.
Larger and more permanent camps elsewhere in the country have left extensive remains including the imprint of building foundations: Cannock Chase in Staffordshire was home to one of the largest First World War training camps in the country, built to house around 40,000 troops during their training. These sites consisted not only of barrack buildings, but also infrastructure such as roads and drainage, and auxiliary buildings such as canteens, hospitals, theatres, post offices, and clubs. Cannock Chase has been surveyed by LiDAR, and you can view this and read more about the amazing finds discovered there on Historic England's website:
https://historicengland.org.uk/research/current/discover-and-understand/landscapes/cannock-chase/
Besides structures, the construction of shooting ranges, and the digging of practice trenches will also have left their mark on the landscape, and the Chilterns LiDAR is providing an excellent means for finding and surveying these. Notable examples include training trenches on Berkhamsted Common, and others in Pullingshill Wood near Marlow. Perhaps the Chilterns provided a useful training ground for this activity, given that the geology of the British Sector of the Western Front is in many ways essentially identical to that of Chilterns.
WWI training trenches in Pullingshill Wood, near Marlow, Bucks. Imagery copyright CCB.
World War II also had a similar impact on our landscape that survives in places through to today, with, for example, the ghostly traces of barracks buildings and parade grounds showing up under young woodland at Ashridge, Hertfordshire, and on Kingwood Common, Oxfordshire.
WWII barracks buildings and parade ground, now under woodland at Ashridge, Herts.
Other interesting landscape features we might want to record include the traces of a now abandoned ploughing technique: the marks left by (apparently so-called) "square ploughing" are visible in many locations in the Chiltern LiDAR: an envelope-like shape in fields, created by diagonal furrows having been left running from the corners to the centre of the land parcel.
"Envelope" shapes: traces of square-ploughing in the Local Relief Model visualisation of the Chilterns LiDAR. Imagery copyright CCB.
These traces stem from the system of ploughing in a square-sided spiral running from the edge of the field in towards the centre, leaving a narrow furrow where the plough is turned at each right-angle. I have struggled to find anything written about the technique, apparently in use well into the 20th century, as evidenced by historic aerial photographs on Google Earth showing the system in action.
Square-ploughing in action in aerial photographs from the 1940s. Image from Google Earth.
So how do we decide what is worth recording, and what isn't? Rather than relying entirely on a date as a cut-off, we must consider any thing or activity that has past… things that perhaps haven't been recorded, something that is perhaps poorly understood, or indeed things which are at risk of being destroyed or forgotten from our collective memory.
The Grand Plan
2 thoughts on "Archaeology within Memory"
Susan Holmes says:
Or you could just make a mistake about the age – these remains are labelled EAW014221 ENGLAND (1948). Roman site to the north of Berkhamsted Castle, Berkhamsted, 1948 – but I'm pretty sure they are WWI trenches as that was where King George V inspected them during the war
https://britainfromabove.org.uk/en/image/EAW014221
Daverrm says:
I've often wondered, when you are buried, how long does it take before you become archaeology…


text:
Q: NSURLCache does not work when response header value for transfer-encoding is chunked I found an issue with (possibly) NSURLCache today while inspecting request and response headers in Charles Proxy. The issue is a little perplexing, but I'm able to repro it consistently:
In a nutshell, the issue has to do with caching of NSURLRequests using iOS's native NSURLCache with the default policy. It turns out that the request is not cached whenever the response has the header transfer-encoding: chunked. But if the response header is content-length: xxx instead, caching works fine. Specifically, it seems that when the response is chunked, NSURLCache doesn't save the eTag and also neglects appending the if-none-match header to subsequent requests to the same url, and consequently, caching fails (as it should), i.e. a 200 is returned instead of a 304.
I'm testing on the iOS8.2 simulator. Even if you don't have a solution, I'd love to hear if you've experienced the same issue. I've found at least one similar report), and here's a related thread posted by my back-end engineer.

A: It should work if you manually add the response data to the cache. I've got an image loading class where I want to make sure everything is cached, so I do something like this:
- (void)getImageWithURL:(NSURL *)url onCompletion:(void (^)(UIImage *image, NSError *error))completion {
    NSURLRequest *request = [NSURLRequest requestWithURL:url];

    UIImage *cachedImage = [self cachedImageForURLRequest:request];
    if (cachedImage) {
        NSLog(@"Got image from cache.");
        completion(cachedImage, nil);
        return;
    }

    [[[NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]] dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
        // Manually cache the response.
        NSCachedURLResponse *cachedResponse = [[NSCachedURLResponse alloc] initWithResponse:response data:data userInfo:nil storagePolicy:NSURLCacheStorageAllowed];
        [[NSURLCache sharedURLCache] storeCachedResponse:cachedResponse forRequest:request];
        NSLog(@"Got a fresh image.");
        completion([UIImage imageWithData:data], error);
    }] resume];
}

- (UIImage *)cachedImageForURLRequest:(NSURLRequest *)urlRequest {
    NSCachedURLResponse *cachedResponse = [[NSURLCache sharedURLCache] cachedResponseForRequest:urlRequest];
    return [UIImage imageWithData:cachedResponse.data];
}


text:
Video: Former Ethiopian Paramilitary commander tells right abuses in Ogaden
Translated by Mohamed Faarah
(Ogadentoday Press)- Illays Program, a London based Somali Ogaden web journalists have presented a video about human right abuses in Ogaden.
Commander Hassan Abdi Mohamud has revealed reports of arrest, rape and civilian killing in remote Somali ethnic populated region in eastern Ethiopia Ogaden Region.
Mr. Hassan 27 old tells that he joined with the paramilitary forces in 2009 after the Ethiopian defense forces announced the regional federal troops are needed.
Many people are killed, looted their property and raped, I was a witness while dozes of women are raped somewhere minors, I have raised the issue to the regional administration and they did not take an action, even they encouraged the campaign.
Let me give you an example, Sahra was raped, she is well-known, and after she wanted a justice about her rape, she has been accused of as ONLF supporter and even she would be killed if she talks about defaming the troops, said former commander defected from the Paramilitary forces.
The major issue is when a member of Liyu Police wants to get married a woman, if she refuses she is blamed to be married a member of ONLF, and she is an anti-peace element, she is arrested and raped, 16 cases happened in Guna-Gado District
Hassas claim that he defected after he could not stop the right abuses committed his troops and the regional administration ignored the accusation.
There are injustices in Ogaden Region, and I have been waiting a long time how to escape, said Mr. Hassan. A human cannot accept just to be a witness about the crimes, added Hassan.
I have been stationed in Dhagagbour Zone, Bulale vicinity; there have been a looting and committing crimes in the area.
People are killed, raped, and arrested because of their family members are ONLF fighters, Ogaden National Liberation Front, a group fighting for self-determination of Ogaden Region.
Speaking about the especial target women, Mr. Hassan said women who are married ONLF members are commonly raped.
Mr. Hassan said there is no independent media presence in the region and I believe that the reason the government banned is to fear to reveal the human right abuses taking place in the region.
After years of human rights abuses in Ogaden region without international independent investigation, Ogaden region seems to finally have a good news story. Sweden will be the first European country that is investigating what Swedish Prosecutors say a serious human rights abuses committed at Ogaden region.
n 2007, Ethiopia crackdown against insurgent of ONLF, Ogaden National Liberation Front, a group fighting for self-determination of Ogaden region has left thousands of Ogaden civilians who are mainly women and children.
Ethiopia government refuses the wide accusations against their mistreatment of civilians in Ogaden region.
Media and journalist are banned in the Ogaden region and that is why the conflict in Ogaden is not getting international attention.
The past few days the region has seen an increase in heavy fighting.
Since 2005, Ethiopia government isolated Ogaden region from the World, Ethiopia imposed a ban all international aid and media organizations in Ogaden despite some are operating under the permit of intelligence surveillances.
Ogaden National Liberation Front (ONLF) fighting for the self-determination of Ogaden Region in Ethiopia since 1994.
Source: Ogadentoday Press