By the end of the 1959 Southern Cross Crusade, it's estimated that half the people in Australia had heard Billy Graham's Gospel message either at a live event or over the airwaves and telephone lines.
The Melbourne Crusade began at an indoor stadium, but it couldn't hold the crowds. The events were moved to larger and larger venues, with the final service taking place at the Melbourne Cricket Ground.
"In 1959 Billy Graham conducted the largest Crusades in our history in Sydney, Australia," William Petterson recently shared with the Billy Graham Evangelistic Association. "I surrendered my life to Jesus and responded with hundreds of others, as Billy told us of God's great love for us in the cross of Jesus Christ." Read more stories about the impact of the 1959 Crusades.
Many of Billy Graham's messages were broadcast across Australia, allowing the Gospel to penetrate the whole country. Watch a portion of one of Mr. Graham's messages from Melbourne.
Cliff Barrows, Billy Graham's longtime music director, passionately led massive Crusade choirs at each event.


2 tier childrens birthday cakes short article.
Instead, the AIs have to offer simple dialogue choices from where you select your responses. I look at the advantages and disadvantages. Here are only a few ideas.


text:
\section{Introduction}
\label{sec:introduction}

Machine learning approaches, in particular Deep Learning, dominate
many areas of Natural Language Processing (NLP). To reach peak
performance, they require large data sets to train models. It is thus
common to continuously collect user data after a model has been
deployed in order to augment existing training data. This practice raises a
clear need for protecting the privacy of the users whose data are
collected. For instance, commercial providers of voice assistants have
been criticized for recording and transcribing conversations of their
users\footnote{\url{https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio}}. But other domains are affected as well, for instance, patients'
health records in medical applications.

For text collections, one way to respect the users' privacy is to
sanitize each document through a de-identification process before
adding it to a data collection. De-identification requires to either
delete all sensitive information in a text, or to replace it with
benign surrogates. \todo{Insert citations.} Arguably, strict deletion
is the less desirable option because without an indication of where
the edit was made, texts can become impossible to understand or,
perhaps worse, change their meaning. To illustrate, consider the
following example from the medical domain, where the names of specific
medications are considered sensitive information:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\item Besides \textit{warfarin}, the patient is not taking any medication.
\vspace{-1mm}
\item Besides, the patient is not taking any medication .
\end{enumerate}
A more commonly used alternative to deletion is \textit{redaction},
where relevant text portions are blackened rather than deleted, but
this can still impact readability. For instance, when dates, times, and
locations are considered sensitive with respect to a person's
whereabouts, a sentence such as \ref{item:ex:redacted_before} would be
rather useless in a text corpus in its redacted form
\ref{item:ex:redacted_after}:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\setcounter{enumi}{2}
\item \label{item:ex:redacted_before} How about \textit{Rick's
    Caf\'{e}} around \textit{noon} on \textit{the 15th} ?
\item \label{item:ex:redacted_after} How about \censor{Rick's
    Caf\'{e}} around \censor{noon} on \censor{the 15th} ?
\end{enumerate}
This example illustrates that de-identification can imply a trade-off
between \emph{privacy} and \emph{utility}.
In order to gauge the latter, Tang et al.\@\xspace measure the impact of three
alternative methods for masking sensitive words on a subsequent
machine learning task \cite{Tang2004PreservingPI}. The methods consist
of two different ways of replacing words with other, randomly selected
words from the same category, and of a method for replacing words with
a specific category marker. Applying these strategies to the previous
example could lead e.g.\@\xspace to the sentences in \ref{item:ex:vd} and
\ref{item:ex:vcm} respectively:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\setcounter{enumi}{4}
\item \label{item:ex:vd} How about \textit{London} around \textit{4
    o'clock} on \textit{the 3rd of May} ?
\item \label{item:ex:vcm} How about \textit{$<$\footnotesize{LOCATION}$>$} around
  \textit{$<$\footnotesize{TIME}$>$} on \textit{$<$\footnotesize{DATE}$>$}?
\end{enumerate}
\vspace{-1mm}
However, Tang et al.\@\xspace~\cite{Tang2004PreservingPI} do not give any formal privacy guarantees for
their methods, making it difficult to judge the privacy-utility
trade-off.

In this paper, we fill this gap by deriving formal privacy guarantees
for general text transformation-based de-identification methods on the
basis of \emph{Differential Privacy}, a well-established framework for
quantifying privacy leakage \cite{DworkC+14}. In addition, we show the
impact of five different text transformation strategies on three
common NLP tasks, when the transformed texts are used as training data
for machine learning approaches. Unlike Tang et al.\@\xspace, we perform our
experiments on six different corpora to gain more balanced evidence.

\section{Related Work}
\label{sec:related-work}

Text de-identification, also known as sanitization, is
well established in highly sensitive domains, such as e.g.\@\xspace, for
patient health records \cite{UzunerO+07}. A large number of
de-identification methods have been suggested in that area in the
past e.g.\@\xspace
\cite{MeystreSM+10:de-identification-review,DernoncourtF+17,KhinK+18}. While such methods are important for a number of domains, we focus here on the privacy issues associated with cloud-based dialog systems, such as e.g.\@\xspace \cite{ChenH+17}, which are generally acknowledged (e.g. [8]) but have not yet received wide-spread attention.

In contrast, the necessity for protecting private information has long
been realized in the data mining community \cite{AggarwalCC+08}. Our
task is, however, quite different from data mining. For instance,
data mining transformations oftentimes pay special attention to
the preservation of certain statistical properties of the underlying
data, which is not a primary concern in our work, thus allowing us to
explore simpler approaches.

Probably the closest work to ours is \cite{Tang2004PreservingPI} where
the impact of data sanitization has been investigated on Automatic Speech Recognition (ASR) and call
classification tasks. It has been shown there that the spoken dialog
system trained on sanitized data achieves a comparable accuracy. In
contrast, we apply text replacement to Natural Language Understanding (NLU) tasks and show that some
replacement strategies can have a large destructive effect on the
performance of the model. We verify this by applying a
state-of-the-art deep learning model to train the NLU tasks by
fine-tuning BERT~\cite{Bert_jacob} embeddings on three tasks that include
named entity recognition, intent detection and dialog act
classification.

Recently, Carrell et al.\@\xspace~\cite{parrot_attack} proposed
an attack to leak sensitive information in a transformed text,
however, this attack only works on a small dataset. The attack does
not scale to large datasets because it requires the attacker to
perform annotation of private tokens, which is costly and tedious.

\section{Privacy}
\label{sec:privacy}

A general framework for protecting privacy is \textit{Differential
  Privacy} (DP) introduced in \cite{Dwork}. DP quantifies to what
extent privacy in statistical queries is preserved while extracting
useful information from a dataset and has received increasing
attention recently as a rigorous privacy methodology. In this section,
we clarify the connection between DP and text replacement methods but
first provide some technical background on general DP.

Let $\mathcal{D}$ be the set of all possible datasets for a given
domain of data points. A key concept in DP is neighboring datasets. We
call two datasets $D_1, D_2 \in \mathcal{D}$ neighboring if they are
the same except for one data point. For example, $D_1$ and $D_2$ could
be two text corpora which differ only in one single word. The intuition
behind differential privacy, as defined below, is a guarantee that a
randomized algorithm behaves similarly on similar input datasets to a
point where the output of the algorithm does not allow to infer which
dataset was used with any relevant degree of certainty. Therefore, an
attacker cannot tell whether the aforementioned data point is
contained in the algorithm's dataset or not.

\begin{definition}{(Differential Privacy).}
  A randomized algorithm $\mathcal{M}$ is $(\varepsilon, \delta)$
  private with domain $\mathcal{D}$ if \, for all measurable sets
  $S \in \text{Range}(\mathcal{M})$ and for all neighboring datasets
  $D_1$ and $D_2$ differing in at most one data point, we have
  \vspace{-1mm}
\begin{equation}
    \Pr[\mathcal{M}(D_1) \in S] \leq \exp{(\varepsilon)} \Pr[\mathcal{M}(D_2) \in S] + \delta
    \vspace{-1mm}
\end{equation}

\end{definition}

\noindent
Intuitively, a $(\varepsilon, \delta)$ differential private mechanism
guarantees that the absolute value of privacy leakage will be bounded
by $\varepsilon$ with probability at least $1-\delta$ for adjacent
datasets. The higher the value of $\varepsilon$, the higher the chance
of data re-identification

\smallskip

\vspace{-3mm}
\renewcommand{\thealgocf}{}
\begin{algorithm}
\SetAlgoLined
\KwIn{dataset $\mathcal{D}$, token replacement policy $\pi$,
  probability parameter $p$.}

\For{$t'$ in sensitive data}
{
$r \sim  U(0, 1)$
\If{$r\leq p$}
{replace $t'$  with $t \sim \pi(t|t')$}
}
 \caption{Probabilistic Text De-identification}
\end{algorithm}
\vspace{-2mm}

To define a general algorithm for de-identification,
let $\mathcal{T}$ denote the vocabulary of private tokens and consider
a token replacement policy
$\pi: \mathcal{T} \longrightarrow \mathcal{T}$, where $\pi(t|t')$ is
the probability of replacing $t'$ in the original text with $t$.
We introduce a parameter $p$ to model the probability that a token
gets replaced:

\begin{lemma}
\vspace{-1mm}
  If token replacement policy $\pi$ in the algorithm is independent
  of the token to replace, i.e.\@\xspace $\pi(t|t')=\pi(t)$, the algorithm is
  $(\varepsilon, 0)$ differentially private with:
  \vspace{-1mm}
\begin{equation}
    \varepsilon = \max_t \log\frac{1-p+p \,\pi(t)}{p \, \pi(t)}.
    \label{eqn:lemmaEqn}
    \vspace{-1mm}
\end{equation}
\end{lemma}

To prove it, we consider two neighboring datasets $D_1$ and $D_2$, which are the same except in one token. In other words, $D_2$ can be
obtained from $D_1$ by replacing a token $t_1$ in $D_1$ with $t_2$.
Using this notation, we may compute the privacy loss as follows. Let
\begin{equation}
    \varepsilon = \log \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
\end{equation}
where $\mathcal{M}(D)$ is the dataset obtained by applying the
de-identification algorithm to the original dataset $D$, and $t$ is the
observed token in the resulting text. If $t_1$ and $t_2$ are not equal
to $t$ (i.e., $t_1$ and $t_2$ are replaced by $t$), we have :
\begin{equation}
  \label{eq:case1}
     \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
    =\frac{p\, \pi(t|t_1)}{p\, \pi(t|t_2)} = 1
\end{equation}
where we have used this fact that replacement policies are independent
of the original tokens. On the other hand, if $t$ is equal to $t_1$,
we arrive at the following expression for the privacy loss:

\begin{equation}
  \label{eq:case2}
 \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
    =\frac{1-p+p\, \pi(t)}{p\,\pi(t)}.
\end{equation}

We get the inverse of this expression in the opposite case $t=t_2$.
The overall privacy bound is given by the maximum of \eqref{eq:case1}
and \eqref{eq:case2} over the private tokens as stated in
\eqref{eqn:lemmaEqn}.
\qed

\medskip

The algorithm is a variant of \textit{randomized response} \cite{RR}
whose connection to differential privacy has been studied before (e.g.\@\xspace
\cite{DworkC+14,RR,WangY+16}), although not in the context of text
de-identification. The probability parameter $p$ gives data curators
fine-grained control over the privacy-utility trade-off: an
\textit{ideal} text replacement, corresponding to $p=1$, has zero
privacy loss ($\epsilon=0$) but in cases where the replacement noise
harms the performance of models trained on the resulting data too
much, the curator might choose to use a lower probability $p$ if reduced
privacy is deemed acceptable. Our lemma allows to quantify this effect
and compare different de-identification options.

In practice, $p=1$ cannot be achieved very easily if the sensitive
tokens are identified automatically as in e.g.\@\xspace
\cite{Tang2004PreservingPI}'s and our own experiments below. Instead,
the \textit{recall} value of the employed identification method
defines an upper bound for $p$, e.g.\@\xspace, a recall value of $0.8$
implies that an expected $20\%$ of the sensitive tokens will not be
replaced. As $p$ approaches $0$, the $\epsilon$ value approaches
infinity, meaning that no privacy is provided.

In order to interpret our result, we consider the case where an
attacker gets hold of the fully transformed data set. The level of
privacy expressed by the lemma
refer to the possibility of
reversing the replacement in order to reconstruct the source from a
transformed sentence, which is difficult when the privacy loss is small.
Context information might be helpful, but in general,
original tokens can only be guessed according to their prior
probabilities which we assume to be uniform in this paper. However, the algorithm allows for certain sentences to
appear in the output untransformed, either because of the value of the
randomized response value $r$, or when the randomly chosen replacement
token happens to be identical to the source token. The privacy
guarantee given by our lemma arises from the fact that transformed and
untransformed sentences are not obviously distinguishable. In fact,
the DP parameter, $\epsilon$ can be seen as a measure of the certainty with
which an attacker can judge whether a sentence from the output was
actually part of the source text.

Referring to a specific instance of the above algorithm, i.e.\@\xspace a fixed
choice for $p$ and $\pi$ (called a text replacement \emph{strategy}), with
tokens being either single words or multi-word expressions, we examine some straight-forward replacement strategies and the level of privacy they present in the light of our results.
\begin{description}[style=unboxed,leftmargin=0cm]
\item[\textbf{Redact}] Here, the private tokens are replaced with a
  non-word placeholder that is typically not part of the vocabulary of
  the source text e.g.\@\xspace
  {\setlength{\fboxsep}{2pt}\colorbox{lightgray}{\texttt{IIIII}}}.
  Hence, we only fall into case \eqref{eq:case1} above, implying $\epsilon=0$
   under the interpretation outlined above: an attacker can
  decide with certainty which of the tokens were part of
  the original text but cannot infer the replaced tokens.
\item[\textbf{Typed placeholder} (aka \textbf{value-class
    membership}~\cite{Tang2004PreservingPI})] This is akin to using private category markers like LOCATION as the replacement token. This is a strategy
  similar to redaction, providing the same level of privacy.
  However, it provides additional information about a replaced token's
  category and might thus be more useful than redaction for certain
  NLP tasks.
\item[\textbf{Named placeholder}] A fixed category exemplar is used to
  replace all private tokens of that
  category~\cite{pestian-etal-2007-shared_named_Placeholder}, e.g, all
  locations are replaced by ``London''. This strategy makes it
  slightly more difficult to judge which sentence was transformed and
  which was not, i.e.\@\xspace $\epsilon > 0$. But for all instances that differ
  from the exemplar, it is clear that they must have been part of the
  source.
\item[\textbf{Word-by-word replacement}] We can distinguish between
  \textit{value distortion}~\cite{Tang2004PreservingPI} if the
  replacement tokens are from an external source, and \textit{value
    dissociation}~\cite{Tang2004PreservingPI} when the surrogate
  tokens are from the same corpus. The latter keeps the distribution
  of tokens in the resulting document unchanged, which might be
  relevant for some tasks. Both variants make it hard to identify
  untransformed sentences, which is reflected in lower $\epsilon$
  values.
\item[\textbf{Full entity replacement}] Text coherence could be
  improved if source tokens were consistently replaced by the same
  surrogates. However, this case is not supported by our lemma where
  we require $\pi(t|t')=\pi(t)$. Another downside of the word-by-word
  strategy is that multi-word expressions could lead to nonsensical
  replacements, e.g.\@\xspace ``Frankfurt Airport'' could be transformed to
  ``New Francisco''. A variant is thus to replace full entities
  instead of single words. In terms of what can be captured by our lemma, this does not lead to more privacy,
  but the expected
  gain in coherence might benefit downstream tasks.
\end{description}

\begin{table}[t]
  \centering
  \caption{Examples of the replacement strategies, using color codes
    for \annoPER{PER}, \annoLOC{LOC}, \annoORG{ORG}, and \annoTIME{TIME}.
    \vspace{-2mm}}
  \label{tab:transformation_example}
  \scalebox{0.7}{

  \def1.3{1.3}
  \begin{tabularx}{0.65\textwidth}{|p{.15\textwidth}X|}
    \hline
    \textbf{Replacement strategy} & \textbf{Transformed text} \\
    \hline
    No Replacement & Hi Mister \annoPER{Miller}, the                      \annoORG{Lufthansa} flight from \annoLOC{Frankfurt Airport} to \annoLOC{Rome} is leaving by \annoTIME{six pm} \\
    \hline
    Redact & Hi Mister \annoPER{IIIII}, the                      \annoORG{IIIII} flight from \annoLOC{IIIII} to \annoLOC{IIIII}  is leaving by \annoTIME{IIIII} \\
    \hline
    Typed-Placeholder & Hi Mister \annoPER{PER }, the                      \annoORG{ORG} flight from \annoLOC{LOC} to \annoLOC{LOC} is leaving by \annoTIME{TIME} \\
    \hline
    Named-Placeholder & Hi Mister \annoPER{Smith}, the                      \annoORG{SAP} flight from \annoLOC{London} to \annoLOC{London} is leaving by \annoTIME{afternoon} \\
    \hline
    Word by word & Hi Mister \annoPER{John}, the                      \annoORG{BOSCH} flight from \annoLOC{New Boston} to \annoLOC{Berlin} is leaving by \annoTIME{eleven morning} \\
    \hline
    Full entity & Hi Mister \annoPER{John},  the                      \annoORG{BOSCH} flight from \annoLOC{New York} to \annoLOC{Berlin} is leaving by \annoTIME{twelve pm} \\
    \hline
  \end{tabularx}
}
\vspace{-4mm}
\end{table}

\begin{table*}[t]
  \centering
  \caption{Evaluation of the different token replacement strategies on
    the 6 datasets comprising of 3 tasks: NER, ID, DAC. Average performance computed from ten runs. The best Accuracy/F1-score in
    each class/task are in \textbf{bold} and the best text
    transformation result have asterisk (*). The replacement
    strategies use ground-truth annotations for the identification of
    sensitive tokens, i.e.\@\xspace $p=1, \varepsilon=0$.\vspace{-2mm}}
  \label{tab:transf_result}
    \centering
    \footnotesize
    \scalebox{0.90}{
    \begin{tabular}{lrrrrrr}
    \toprule
     \textbf{Replacement strategy} & \multirow{2}{20mm}[0.3em] {\textbf{VerbMobil NER F1-score}} & \multirow{2}{15mm}[0.3em] {\textbf{ATIS ID Accuracy}} & \multirow{2}{17mm}[0.3em] {\textbf{SNIPS ID Accuracy}} & \multirow{2}{20mm}[0.3em] {\textbf{en-TOD ID Accuracy}}& \multirow{2}{20mm}[0.3em] {\textbf{Restaurant DAC Accuracy}} & \multirow{2}{15mm}[0.3em] {\textbf{Taxi DAC Accuracy}} \\
     \addlinespace[1.0em]
    \midrule
      No replacement & $\mathbf{88.3 \pm 0.2}$ & $\mathbf{98.4 \pm 0.2}$ & $\mathbf{98.0 \pm 0.2}$ & $\mathbf{99.4 \pm 0.0}$ &  $\mathbf{78.9 \pm 0.1}$ & $\mathbf{90.0 \pm 0.1}$ \\ \addlinespace[0.4em]
      Redact & $0.2 \pm 0.2$ & $94.8 \pm 0.2$ & $89.7 \pm 0.8$ & $97.4 \pm 0.6$ &  $75.9 \pm 0.3$ & $88.1 \pm 0.2$ \\ \addlinespace[0.4em]
      Typed-Placeholder & $0.0 \pm 0.0$  & $95.7 \pm 0.3$ & $54.1 \pm 3.8$ & $97.2 \pm 0.7$ &  $76.5 \pm 0.2$ & $87.9 \pm 0.5$ \\ \addlinespace[0.4em]
     Named Placeholder & $13.5 \pm 1.4$  & $95.9 \pm 0.3$ & $76.2 \pm 2.9$ & $98.2 \pm 0.1$ &  $77.3 \pm 0.2$ & $89.3 \pm 0.1$ \\ \addlinespace[0.4em]
     Word-by-Word & $72.6 \pm 0.3$ & $\mathbf{98.6 \pm 0.2}^*$ & ${97.5 \pm 0.3}^*$ & ${99.2 \pm 0.1}^*$ &  $78.4 \pm 0.2$ & ${89.9 \pm 0.2}^*$ \\ \addlinespace[0.4em]
     Full Entity & ${85.9 \pm 0.3}^*$  & $\mathbf{98.5 \pm 0.2}^*$ & ${97.4 \pm 0.3}^*$ & ${99.2 \pm 0.1}^*$ & ${78.5 \pm 0.1}^*$ & ${89.9 \pm 0.1}^*$ \\\addlinespace[0.4em]
      \bottomrule
    \end{tabular}
    }
    \label{Tab:result}
    \vspace{-4mm}
\end{table*}

\noindent
An example for each of these replacement strategies is given in
Table~\ref{tab:transformation_example}. Besides discussing privacy
aspects, we have speculated on the differences of the strategies on
subsequent applications. In order to verify these considerations, we
now measure the impact of the different replacement strategies
empirically.

\section{Utility}
\label{sec:utility}

We experiment with three common NLP tasks, Named Entity Recognition
(NER), Intent Detection (ID), and Dialog Act Classification (DAC),
across six different datasets (see Table~\ref{tab:datasets}). The
variety in datasets is important since what is considered sensitive
information is typically domain-dependent. Here, we consider as
private: (1) the identity of one or both speakers, (2) organizations, such as e.g., company names, etc. (3) The locations or
addresses (4) The dates and times. This private information coincides with
typical \textit{named entities (NEs)} and \textit{slot classes} in
dialog datasets such as PER (personal names), ORG (organization), LOC
(location), DATE and TIME.

\vspace{-2mm}

\subsection{Datasets}
\label{sec:datasets}

\begin{table}[tb]
  \setlength{\tabcolsep}{0.5em}
  \renewcommand{1.3}{1.2}
  \centering
  \caption{Dataset summary for three different tasks: Named Entity
    Recognition (NER), Intent Detection (ID), and Dialog Act
    Classification (DAC)\vspace{-2mm}}
  \label{tab:transf_result}
    \centering
    \footnotesize
    \begin{tabularx}{\columnwidth}{lclcr@{\hskip .7mm}c@{\hskip .7mm}r@{\hskip .7mm}c@{\hskip .7mm}r}
    \toprule
      \textbf{Dataset} & \textbf{Task} & \textbf{Private} &
      \textbf{Classes} & \textbf{Train} & / & \textbf{Val.} & / &
      \textbf{Test} \\ \addlinespace[-0.25em]
      & & \textbf{Tokens} & & \multicolumn{5}{l}{\textbf{Sentences}} \\
    \midrule
      VerbMobil & NER & 5 NEs & 6 & 19K &\textcolor{gray}{/}& 2848 &\textcolor{gray}{/}& 5230  \\
      ATIS & ID & 21 slots & 21 & 4478 &\textcolor{gray}{/}& 500 &\textcolor{gray}{/}& 893  \\
      SNIPS & ID & 39 slots & 7 & 13K &\textcolor{gray}{/}& 700 &\textcolor{gray}{/}& 700  \\
      FB en-TOD & ID & 15 slots & 12 & 30K &\textcolor{gray}{/}& 4181 &\textcolor{gray}{/}& 8621  \\
      MS Restaurant & DAC & 21 slots & 24 & 20K &\textcolor{gray}{/}& 2936 &\textcolor{gray}{/}& 5859  \\
      MS Taxi & DAC & 10 slots & 18 & 16K &\textcolor{gray}{/}& 2273 &\textcolor{gray}{/}& 4597  \\
      \bottomrule
      \end{tabularx}
      \label{tab:datasets}
      \vspace{-4mm}
\end{table}

The \textbf{\textsc{Verbmobil}\xspace} corpus is a large collection of spontaneous
telephone conversations \cite{WeilhammerKURFS02}. In each conversation, two
speakers negotiate the details of a business meeting. The corpus
contains English, German, and Japanese conversations, however, we only
use English portion of the corpus for our experiments. The \textsc{Verbmobil}\xspace corpus does
not come pre-annotated with NE classes. About 20\% of the \textsc{Verbmobil}\xspace
corpus was thus annotated via crowd sourcing.
The remaining
80\% of the corpus was annotated automatically using
\texttt{spaCy}\xspace\footnote{\url{https://spacy.io}}
and post-corrected manually.

\noindent

The \textbf{ATIS}~\cite{hemphill_atis} corpus is a popular dataset for slot filling and
intent detection tasks in the Air Travel Information Services domain.
For the text transformation experiments, we map the
provided slot labels to the aforementioned named entity categories.

\noindent

\textbf{SNIPS} is another popular benchmark dataset for slot filling
and intent detection task by SNIPS.AI~\cite{Coucke2018SnipsVP}. The
dataset consists of seven intents from different domains  such as
\textit{``AddToPlaylist''}, \textit{``BookRestaurant''},
\textit{``GetWeather''}, \textit{``RateBook''}.

\noindent

\textbf{FB en-TOD} is a multilingual slot and intent classification
dataset recently released by
Facebook~\cite{Schuster2019CrosslingualTL}. It consists of utterances
from three languages (English, Spanish, and Thai) and three domains
(Alarm, Reminder, and Weather). In this paper, we only use the English
dataset.

\noindent

\textbf{MS Taxi} and \textbf{MS Restaurant} are two out of three dialog
challenge datasets released by Microsoft at the SLT 2018
workshop~\cite{li2018microsoft} for taxi bookings and restaurant
reservations with 19 and 29 slot types respectively and 11 dialog acts.
The number of classes for the Taxi and Restaurant datasets are 18 and
24 respectively after removing classes with less than 40 utterances.

\vspace{-2mm}

\subsection{Experiments}
\label{sec:experiments}

For all comparison experiments, we first run a baseline experiment
using the original datasets. Then, we apply the respective privacy
strategies to the training data before fine-tuning a
BERT model for token/sentence classification. We
then compare the performance of the resulting models with the baseline
with respect to the (untransformed) test set. The BERT classification
model involves \textit{fine-tuning} the pre-trained BERT embeddings on
the training data with an additional linear layer whose weights are
randomly initialized. We trained all the parameters of the model
end-to-end, including the linear layer.

For the implementation, we fine-tuned BERT on the various tasks using
the
\textit{simpletransformers\footnote{{https://github.com/ThilinaRajapakse/simpletransformers}}}
based on the \textit{transformers} library of
HuggingFace~\cite{Wolf2019HuggingFacesTS}. The hyper-parameters of the
model are 768-dimensional embedding layer (for bert-base-cased model),
batch size of 8 for NER and 16 for other tasks, maximum learning rate
of $0.00005$, maximum sequence length is 128 for NER and 64 for other
tasks. The maximum number of epochs for all experiments is 3.

\begin{figure}[tbh]
  \centering
  \vspace{-3mm}
  \includegraphics[draft=false,width=\columnwidth]{images/dp_acc_epsilon_prob.png}
  \vspace{-4mm}
  \caption{Connection between DP (privacy), Replacement probability $p$, and F1-score (performance) on \textsc{Verbmobil}\xspace.}
  \label{fig:dp_result}
   \vspace{-6mm}
\end{figure}

\subsection{Results}
\label{sec:results}

We show that the performance of the redact and word-by-word replacement strategies
can be improved by tuning the parameter $p$ described in
Section~\ref{sec:privacy}. For example, by setting $p=0.9$ (i.e.\@\xspace
$\epsilon=6.75$), we can improve the F1-score by around $4\%$ for word-by-word and $60\%$ for redact as shown
in Figure~\ref{fig:dp_result}, demonstrating the ability to control the privacy-utility trade-off. In the word-by-word replacement
experiments, we replace a NE word $t'$ by another word $t$ of the same
entity class based on their relative frequency distribution $\pi(t)$
in the corpus.

The baseline to which we compare all other experiments is simply
trained on the original training set, i.e.\@\xspace, without removing any
private information. On the test set, the resulting model yields a
prediction F1-score/Accuracy of $88.3\%$ (NER), $98.4\%$ (ATIS intent)
and $98.0\%$ (SNIPS intent), $99.4\%$ (en-TOD intent), $78.9\%$
(Restaurant DAC) and $90.0\%$ (Taxi DAC).

Table~\ref{Tab:result} shows the result for the different text
transformation strategies. Replacing private tokens using
\textit{redact}, \textit{typed placeholder} and \textit{named
  placeholder} strategies generally gave a worse result than the
word-by-word replacement. For NER, we observe a substantial drop in performance for redact
and placeholder approaches because the model overfits on the
replacement tokens which are expected to be absent in the test set. On the other
hand, the drop is minimal for intent and dialog act classification
tasks around ($2 - 4\%$) similar to the
observation in \cite{Tang2004PreservingPI}, except for the SNIPS
dataset with much larger reduction in performance of
$8 - 44\%$ depending on the placeholder strategy. This shows that
these transformation strategies are generally not suitable for
training NLU systems.

For the \textit{word-by-word} replacement, we observe a drop of 15\%
in F1-score when we replace all words labeled as named entities with
tokens of the same-type.
For NER, we find
that ``TIME'', ``ORG'' and ``DATE'' are most affected by the
word-by-word replacement in terms of drop in F1-score because many of
them are multi-word expressions. Thus, the three named entities gain
the most by full-entity replacement. On the other-hand, the drop is
very small ($<1\%$) for other intent and dialog act classification.

Table~\ref{tab:transformation_example} illustrates an example of the
full-entity replacement (e.g ``Frankfurt Airport'' is replaced by
``New York''). This approach gives the best performance out of all the
transformation strategies with only $2.4\%$ drop for NER. Interestingly,
there is no significant difference between its performance and the
baseline on the intent and dialog act classification tasks
across the datasets. In summary, the text obtained using the word-by-word or
full-entity text transformation are more suitable for training NLU
systems
while protecting the
privacy of users.

\section{Conclusion}
\label{sec:discussion}

Replacing sensitive tokens with benign alternatives is a common method
for de-identifying text documents. We prove that privacy guarantees
for a formalized version of this process can be expressed in terms of
Differential Privacy. Our approach includes two parameters, $p$ and
$\pi$ that allow different replacement strategies to be expressed as
instances of the same algorithm. The respective DP-$\epsilon$ value follows
from the choices for $p$ and $\pi$, permitting a comparison of
different replacement strategies with respect to their privacy
implications.

User privacy is juxtaposed by the performance impact that a text
transformation has on subsequent machine learning tasks. We experiment
with three different NLP tasks across six different datasets and find
that both word-by-word and full entity replacement strategies are
robust against performance drops across all examined tasks.

\todo{future work}

\section{Acknowledgments}
This research has received funding by the European Union's Horizon 2020 research and 		innovation programme under grant agreement No. 3081705 -- COMPRISE (\texttt{http://www.compriseh2020.eu/}). We thank Xiaoyu Shen, Volha Petukhova, and the anonymous reviewers for their insightful comments that helped us to improve
this paper.

\bibliographystyle{IEEEtran}

\section{Introduction}
\label{sec:introduction}

Machine learning approaches, in particular Deep Learning, dominate
many areas of Natural Language Processing (NLP). To reach peak
performance, they require large data sets to train models. It is thus
common to continuously collect user data after a model has been
deployed in order to augment existing training data. This practice raises a
clear need for protecting the privacy of the users whose data are
collected. For instance, commercial providers of voice assistants have
been criticized for recording and transcribing conversations of their
users\footnote{\url{https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio}}. But other domains are affected as well, for instance, patients'
health records in medical applications.

For text collections, one way to respect the users' privacy is to
sanitize each document through a de-identification process before
adding it to a data collection. De-identification requires to either
delete all sensitive information in a text, or to replace it with
benign surrogates. \todo{Insert citations.} Arguably, strict deletion
is the less desirable option because without an indication of where
the edit was made, texts can become impossible to understand or,
perhaps worse, change their meaning. To illustrate, consider the
following example from the medical domain, where the names of specific
medications are considered sensitive information:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\item Besides \textit{warfarin}, the patient is not taking any medication.
\vspace{-1mm}
\item Besides, the patient is not taking any medication .
\end{enumerate}
A more commonly used alternative to deletion is \textit{redaction},
where relevant text portions are blackened rather than deleted, but
this can still impact readability. For instance, when dates, times, and
locations are considered sensitive with respect to a person's
whereabouts, a sentence such as \ref{item:ex:redacted_before} would be
rather useless in a text corpus in its redacted form
\ref{item:ex:redacted_after}:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\setcounter{enumi}{2}
\item \label{item:ex:redacted_before} How about \textit{Rick's
    Caf\'{e}} around \textit{noon} on \textit{the 15th} ?
\item \label{item:ex:redacted_after} How about \censor{Rick's
    Caf\'{e}} around \censor{noon} on \censor{the 15th} ?
\end{enumerate}
This example illustrates that de-identification can imply a trade-off
between \emph{privacy} and \emph{utility}.
In order to gauge the latter, Tang et al.\@\xspace measure the impact of three
alternative methods for masking sensitive words on a subsequent
machine learning task \cite{Tang2004PreservingPI}. The methods consist
of two different ways of replacing words with other, randomly selected
words from the same category, and of a method for replacing words with
a specific category marker. Applying these strategies to the previous
example could lead e.g.\@\xspace to the sentences in \ref{item:ex:vd} and
\ref{item:ex:vcm} respectively:
\begin{enumerate}[label=(\arabic*), leftmargin=0.8cm]
\setcounter{enumi}{4}
\item \label{item:ex:vd} How about \textit{London} around \textit{4
    o'clock} on \textit{the 3rd of May} ?
\item \label{item:ex:vcm} How about \textit{$<$\footnotesize{LOCATION}$>$} around
  \textit{$<$\footnotesize{TIME}$>$} on \textit{$<$\footnotesize{DATE}$>$}?
\end{enumerate}
\vspace{-1mm}
However, Tang et al.\@\xspace~\cite{Tang2004PreservingPI} do not give any formal privacy guarantees for
their methods, making it difficult to judge the privacy-utility
trade-off.

In this paper, we fill this gap by deriving formal privacy guarantees
for general text transformation-based de-identification methods on the
basis of \emph{Differential Privacy}, a well-established framework for
quantifying privacy leakage \cite{DworkC+14}. In addition, we show the
impact of five different text transformation strategies on three
common NLP tasks, when the transformed texts are used as training data
for machine learning approaches. Unlike Tang et al.\@\xspace, we perform our
experiments on six different corpora to gain more balanced evidence.

\section{Related Work}
\label{sec:related-work}

Text de-identification, also known as sanitization, is
well established in highly sensitive domains, such as e.g.\@\xspace, for
patient health records \cite{UzunerO+07}. A large number of
de-identification methods have been suggested in that area in the
past e.g.\@\xspace
\cite{MeystreSM+10:de-identification-review,DernoncourtF+17,KhinK+18}. While such methods are important for a number of domains, we focus here on the privacy issues associated with cloud-based dialog systems, such as e.g.\@\xspace \cite{ChenH+17}, which are generally acknowledged (e.g. [8]) but have not yet received wide-spread attention.

In contrast, the necessity for protecting private information has long
been realized in the data mining community \cite{AggarwalCC+08}. Our
task is, however, quite different from data mining. For instance,
data mining transformations oftentimes pay special attention to
the preservation of certain statistical properties of the underlying
data, which is not a primary concern in our work, thus allowing us to
explore simpler approaches.

Probably the closest work to ours is \cite{Tang2004PreservingPI} where
the impact of data sanitization has been investigated on Automatic Speech Recognition (ASR) and call
classification tasks. It has been shown there that the spoken dialog
system trained on sanitized data achieves a comparable accuracy. In
contrast, we apply text replacement to Natural Language Understanding (NLU) tasks and show that some
replacement strategies can have a large destructive effect on the
performance of the model. We verify this by applying a
state-of-the-art deep learning model to train the NLU tasks by
fine-tuning BERT~\cite{Bert_jacob} embeddings on three tasks that include
named entity recognition, intent detection and dialog act
classification.

Recently, Carrell et al.\@\xspace~\cite{parrot_attack} proposed
an attack to leak sensitive information in a transformed text,
however, this attack only works on a small dataset. The attack does
not scale to large datasets because it requires the attacker to
perform annotation of private tokens, which is costly and tedious.

\section{Privacy}
\label{sec:privacy}

A general framework for protecting privacy is \textit{Differential
  Privacy} (DP) introduced in \cite{Dwork}. DP quantifies to what
extent privacy in statistical queries is preserved while extracting
useful information from a dataset and has received increasing
attention recently as a rigorous privacy methodology. In this section,
we clarify the connection between DP and text replacement methods but
first provide some technical background on general DP.

Let $\mathcal{D}$ be the set of all possible datasets for a given
domain of data points. A key concept in DP is neighboring datasets. We
call two datasets $D_1, D_2 \in \mathcal{D}$ neighboring if they are
the same except for one data point. For example, $D_1$ and $D_2$ could
be two text corpora which differ only in one single word. The intuition
behind differential privacy, as defined below, is a guarantee that a
randomized algorithm behaves similarly on similar input datasets to a
point where the output of the algorithm does not allow to infer which
dataset was used with any relevant degree of certainty. Therefore, an
attacker cannot tell whether the aforementioned data point is
contained in the algorithm's dataset or not.

\begin{definition}{(Differential Privacy).}
  A randomized algorithm $\mathcal{M}$ is $(\varepsilon, \delta)$
  private with domain $\mathcal{D}$ if \, for all measurable sets
  $S \in \text{Range}(\mathcal{M})$ and for all neighboring datasets
  $D_1$ and $D_2$ differing in at most one data point, we have
  \vspace{-1mm}
\begin{equation}
    \Pr[\mathcal{M}(D_1) \in S] \leq \exp{(\varepsilon)} \Pr[\mathcal{M}(D_2) \in S] + \delta
    \vspace{-1mm}
\end{equation}

\end{definition}

\noindent
Intuitively, a $(\varepsilon, \delta)$ differential private mechanism
guarantees that the absolute value of privacy leakage will be bounded
by $\varepsilon$ with probability at least $1-\delta$ for adjacent
datasets. The higher the value of $\varepsilon$, the higher the chance
of data re-identification

\smallskip

\vspace{-3mm}
\renewcommand{\thealgocf}{}
\begin{algorithm}
\SetAlgoLined
\KwIn{dataset $\mathcal{D}$, token replacement policy $\pi$,
  probability parameter $p$.}

\For{$t'$ in sensitive data}
{
$r \sim  U(0, 1)$
\If{$r\leq p$}
{replace $t'$  with $t \sim \pi(t|t')$}
}
 \caption{Probabilistic Text De-identification}
\end{algorithm}
\vspace{-2mm}

To define a general algorithm for de-identification,
let $\mathcal{T}$ denote the vocabulary of private tokens and consider
a token replacement policy
$\pi: \mathcal{T} \longrightarrow \mathcal{T}$, where $\pi(t|t')$ is
the probability of replacing $t'$ in the original text with $t$.
We introduce a parameter $p$ to model the probability that a token
gets replaced:

\begin{lemma}
\vspace{-1mm}
  If token replacement policy $\pi$ in the algorithm is independent
  of the token to replace, i.e.\@\xspace $\pi(t|t')=\pi(t)$, the algorithm is
  $(\varepsilon, 0)$ differentially private with:
  \vspace{-1mm}
\begin{equation}
    \varepsilon = \max_t \log\frac{1-p+p \,\pi(t)}{p \, \pi(t)}.
    \label{eqn:lemmaEqn}
    \vspace{-1mm}
\end{equation}
\end{lemma}

To prove it, we consider two neighboring datasets $D_1$ and $D_2$, which are the same except in one token. In other words, $D_2$ can be
obtained from $D_1$ by replacing a token $t_1$ in $D_1$ with $t_2$.
Using this notation, we may compute the privacy loss as follows. Let
\begin{equation}
    \varepsilon = \log \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
\end{equation}
where $\mathcal{M}(D)$ is the dataset obtained by applying the
de-identification algorithm to the original dataset $D$, and $t$ is the
observed token in the resulting text. If $t_1$ and $t_2$ are not equal
to $t$ (i.e., $t_1$ and $t_2$ are replaced by $t$), we have :
\begin{equation}
  \label{eq:case1}
     \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
    =\frac{p\, \pi(t|t_1)}{p\, \pi(t|t_2)} = 1
\end{equation}
where we have used this fact that replacement policies are independent
of the original tokens. On the other hand, if $t$ is equal to $t_1$,
we arrive at the following expression for the privacy loss:

\begin{equation}
  \label{eq:case2}
 \frac{\Pr[t\in \mathcal{M}(D_1)]}{\Pr[t\in \mathcal{M}(D_2)]}
    =\frac{1-p+p\, \pi(t)}{p\,\pi(t)}.
\end{equation}

We get the inverse of this expression in the opposite case $t=t_2$.
The overall privacy bound is given by the maximum of \eqref{eq:case1}
and \eqref{eq:case2} over the private tokens as stated in
\eqref{eqn:lemmaEqn}.
\qed

\medskip

The algorithm is a variant of \textit{randomized response} \cite{RR}
whose connection to differential privacy has been studied before (e.g.\@\xspace
\cite{DworkC+14,RR,WangY+16}), although not in the context of text
de-identification. The probability parameter $p$ gives data curators
fine-grained control over the privacy-utility trade-off: an
\textit{ideal} text replacement, corresponding to $p=1$, has zero
privacy loss ($\epsilon=0$) but in cases where the replacement noise
harms the performance of models trained on the resulting data too
much, the curator might choose to use a lower probability $p$ if reduced
privacy is deemed acceptable. Our lemma allows to quantify this effect
and compare different de-identification options.

In practice, $p=1$ cannot be achieved very easily if the sensitive
tokens are identified automatically as in e.g.\@\xspace
\cite{Tang2004PreservingPI}'s and our own experiments below. Instead,
the \textit{recall} value of the employed identification method
defines an upper bound for $p$, e.g.\@\xspace, a recall value of $0.8$
implies that an expected $20\%$ of the sensitive tokens will not be
replaced. As $p$ approaches $0$, the $\epsilon$ value approaches
infinity, meaning that no privacy is provided.

In order to interpret our result, we consider the case where an
attacker gets hold of the fully transformed data set. The level of
privacy expressed by the lemma
refer to the possibility of
reversing the replacement in order to reconstruct the source from a
transformed sentence, which is difficult when the privacy loss is small.
Context information might be helpful, but in general,
original tokens can only be guessed according to their prior
probabilities which we assume to be uniform in this paper. However, the algorithm allows for certain sentences to
appear in the output untransformed, either because of the value of the
randomized response value $r$, or when the randomly chosen replacement
token happens to be identical to the source token. The privacy
guarantee given by our lemma arises from the fact that transformed and
untransformed sentences are not obviously distinguishable. In fact,
the DP parameter, $\epsilon$ can be seen as a measure of the certainty with
which an attacker can judge whether a sentence from the output was
actually part of the source text.

Referring to a specific instance of the above algorithm, i.e.\@\xspace a fixed
choice for $p$ and $\pi$ (called a text replacement \emph{strategy}), with
tokens being either single words or multi-word expressions, we examine some straight-forward replacement strategies and the level of privacy they present in the light of our results.
\begin{description}[style=unboxed,leftmargin=0cm]
\item[\textbf{Redact}] Here, the private tokens are replaced with a
  non-word placeholder that is typically not part of the vocabulary of
  the source text e.g.\@\xspace
  {\setlength{\fboxsep}{2pt}\colorbox{lightgray}{\texttt{IIIII}}}.
  Hence, we only fall into case \eqref{eq:case1} above, implying $\epsilon=0$
   under the interpretation outlined above: an attacker can
  decide with certainty which of the tokens were part of
  the original text but cannot infer the replaced tokens.
\item[\textbf{Typed placeholder} (aka \textbf{value-class
    membership}~\cite{Tang2004PreservingPI})] This is akin to using private category markers like LOCATION as the replacement token. This is a strategy
  similar to redaction, providing the same level of privacy.
  However, it provides additional information about a replaced token's
  category and might thus be more useful than redaction for certain
  NLP tasks.
\item[\textbf{Named placeholder}] A fixed category exemplar is used to
  replace all private tokens of that
  category~\cite{pestian-etal-2007-shared_named_Placeholder}, e.g, all
  locations are replaced by ``London''. This strategy makes it
  slightly more difficult to judge which sentence was transformed and
  which was not, i.e.\@\xspace $\epsilon > 0$. But for all instances that differ
  from the exemplar, it is clear that they must have been part of the
  source.
\item[\textbf{Word-by-word replacement}] We can distinguish between
  \textit{value distortion}~\cite{Tang2004PreservingPI} if the
  replacement tokens are from an external source, and \textit{value
    dissociation}~\cite{Tang2004PreservingPI} when the surrogate
  tokens are from the same corpus. The latter keeps the distribution
  of tokens in the resulting document unchanged, which might be
  relevant for some tasks. Both variants make it hard to identify
  untransformed sentences, which is reflected in lower $\epsilon$
  values.
\item[\textbf{Full entity replacement}] Text coherence could be
  improved if source tokens were consistently replaced by the same
  surrogates. However, this case is not supported by our lemma where
  we require $\pi(t|t')=\pi(t)$. Another downside of the word-by-word
  strategy is that multi-word expressions could lead to nonsensical
  replacements, e.g.\@\xspace ``Frankfurt Airport'' could be transformed to
  ``New Francisco''. A variant is thus to replace full entities
  instead of single words. In terms of what can be captured by our lemma, this does not lead to more privacy,
  but the expected
  gain in coherence might benefit downstream tasks.
\end{description}

\begin{table}[t]
  \centering
  \caption{Examples of the replacement strategies, using color codes
    for \annoPER{PER}, \annoLOC{LOC}, \annoORG{ORG}, and \annoTIME{TIME}.
    \vspace{-2mm}}
  \label{tab:transformation_example}
  \scalebox{0.7}{

  \def1.3{1.3}
  \begin{tabularx}{0.65\textwidth}{|p{.15\textwidth}X|}
    \hline
    \textbf{Replacement strategy} & \textbf{Transformed text} \\
    \hline
    No Replacement & Hi Mister \annoPER{Miller}, the                      \annoORG{Lufthansa} flight from \annoLOC{Frankfurt Airport} to \annoLOC{Rome} is leaving by \annoTIME{six pm} \\
    \hline
    Redact & Hi Mister \annoPER{IIIII}, the                      \annoORG{IIIII} flight from \annoLOC{IIIII} to \annoLOC{IIIII}  is leaving by \annoTIME{IIIII} \\
    \hline
    Typed-Placeholder & Hi Mister \annoPER{PER }, the                      \annoORG{ORG} flight from \annoLOC{LOC} to \annoLOC{LOC} is leaving by \annoTIME{TIME} \\
    \hline
    Named-Placeholder & Hi Mister \annoPER{Smith}, the                      \annoORG{SAP} flight from \annoLOC{London} to \annoLOC{London} is leaving by \annoTIME{afternoon} \\
    \hline
    Word by word & Hi Mister \annoPER{John}, the                      \annoORG{BOSCH} flight from \annoLOC{New Boston} to \annoLOC{Berlin} is leaving by \annoTIME{eleven morning} \\
    \hline
    Full entity & Hi Mister \annoPER{John},  the                      \annoORG{BOSCH} flight from \annoLOC{New York} to \annoLOC{Berlin} is leaving by \annoTIME{twelve pm} \\
    \hline
  \end{tabularx}
}
\vspace{-4mm}
\end{table}

\begin{table*}[t]
  \centering
  \caption{Evaluation of the different token replacement strategies on
    the 6 datasets comprising of 3 tasks: NER, ID, DAC. Average performance computed from ten runs. The best Accuracy/F1-score in
    each class/task are in \textbf{bold} and the best text
    transformation result have asterisk (*). The replacement
    strategies use ground-truth annotations for the identification of
    sensitive tokens, i.e.\@\xspace $p=1, \varepsilon=0$.\vspace{-2mm}}
  \label{tab:transf_result}
    \centering
    \footnotesize
    \scalebox{0.90}{
    \begin{tabular}{lrrrrrr}
    \toprule
     \textbf{Replacement strategy} & \multirow{2}{20mm}[0.3em] {\textbf{VerbMobil NER F1-score}} & \multirow{2}{15mm}[0.3em] {\textbf{ATIS ID Accuracy}} & \multirow{2}{17mm}[0.3em] {\textbf{SNIPS ID Accuracy}} & \multirow{2}{20mm}[0.3em] {\textbf{en-TOD ID Accuracy}}& \multirow{2}{20mm}[0.3em] {\textbf{Restaurant DAC Accuracy}} & \multirow{2}{15mm}[0.3em] {\textbf{Taxi DAC Accuracy}} \\
     \addlinespace[1.0em]
    \midrule
      No replacement & $\mathbf{88.3 \pm 0.2}$ & $\mathbf{98.4 \pm 0.2}$ & $\mathbf{98.0 \pm 0.2}$ & $\mathbf{99.4 \pm 0.0}$ &  $\mathbf{78.9 \pm 0.1}$ & $\mathbf{90.0 \pm 0.1}$ \\ \addlinespace[0.4em]
      Redact & $0.2 \pm 0.2$ & $94.8 \pm 0.2$ & $89.7 \pm 0.8$ & $97.4 \pm 0.6$ &  $75.9 \pm 0.3$ & $88.1 \pm 0.2$ \\ \addlinespace[0.4em]
      Typed-Placeholder & $0.0 \pm 0.0$  & $95.7 \pm 0.3$ & $54.1 \pm 3.8$ & $97.2 \pm 0.7$ &  $76.5 \pm 0.2$ & $87.9 \pm 0.5$ \\ \addlinespace[0.4em]
     Named Placeholder & $13.5 \pm 1.4$  & $95.9 \pm 0.3$ & $76.2 \pm 2.9$ & $98.2 \pm 0.1$ &  $77.3 \pm 0.2$ & $89.3 \pm 0.1$ \\ \addlinespace[0.4em]
     Word-by-Word & $72.6 \pm 0.3$ & $\mathbf{98.6 \pm 0.2}^*$ & ${97.5 \pm 0.3}^*$ & ${99.2 \pm 0.1}^*$ &  $78.4 \pm 0.2$ & ${89.9 \pm 0.2}^*$ \\ \addlinespace[0.4em]
     Full Entity & ${85.9 \pm 0.3}^*$  & $\mathbf{98.5 \pm 0.2}^*$ & ${97.4 \pm 0.3}^*$ & ${99.2 \pm 0.1}^*$ & ${78.5 \pm 0.1}^*$ & ${89.9 \pm 0.1}^*$ \\\addlinespace[0.4em]
      \bottomrule
    \end{tabular}
    }
    \label{Tab:result}
    \vspace{-4mm}
\end{table*}

\noindent
An example for each of these replacement strategies is given in
Table~\ref{tab:transformation_example}. Besides discussing privacy
aspects, we have speculated on the differences of the strategies on
subsequent applications. In order to verify these considerations, we
now measure the impact of the different replacement strategies
empirically.

\section{Utility}
\label{sec:utility}

We experiment with three common NLP tasks, Named Entity Recognition
(NER), Intent Detection (ID), and Dialog Act Classification (DAC),
across six different datasets (see Table~\ref{tab:datasets}). The
variety in datasets is important since what is considered sensitive
information is typically domain-dependent. Here, we consider as
private: (1) the identity of one or both speakers, (2) organizations, such as e.g., company names, etc. (3) The locations or
addresses (4) The dates and times. This private information coincides with
typical \textit{named entities (NEs)} and \textit{slot classes} in
dialog datasets such as PER (personal names), ORG (organization), LOC
(location), DATE and TIME.

\vspace{-2mm}

\subsection{Datasets}
\label{sec:datasets}

\begin{table}[tb]
  \setlength{\tabcolsep}{0.5em}
  \renewcommand{1.3}{1.2}
  \centering
  \caption{Dataset summary for three different tasks: Named Entity
    Recognition (NER), Intent Detection (ID), and Dialog Act
    Classification (DAC)\vspace{-2mm}}
  \label{tab:transf_result}
    \centering
    \footnotesize
    \begin{tabularx}{\columnwidth}{lclcr@{\hskip .7mm}c@{\hskip .7mm}r@{\hskip .7mm}c@{\hskip .7mm}r}
    \toprule
      \textbf{Dataset} & \textbf{Task} & \textbf{Private} &
      \textbf{Classes} & \textbf{Train} & / & \textbf{Val.} & / &
      \textbf{Test} \\ \addlinespace[-0.25em]
      & & \textbf{Tokens} & & \multicolumn{5}{l}{\textbf{Sentences}} \\
    \midrule
      VerbMobil & NER & 5 NEs & 6 & 19K &\textcolor{gray}{/}& 2848 &\textcolor{gray}{/}& 5230  \\
      ATIS & ID & 21 slots & 21 & 4478 &\textcolor{gray}{/}& 500 &\textcolor{gray}{/}& 893  \\
      SNIPS & ID & 39 slots & 7 & 13K &\textcolor{gray}{/}& 700 &\textcolor{gray}{/}& 700  \\
      FB en-TOD & ID & 15 slots & 12 & 30K &\textcolor{gray}{/}& 4181 &\textcolor{gray}{/}& 8621  \\
      MS Restaurant & DAC & 21 slots & 24 & 20K &\textcolor{gray}{/}& 2936 &\textcolor{gray}{/}& 5859  \\
      MS Taxi & DAC & 10 slots & 18 & 16K &\textcolor{gray}{/}& 2273 &\textcolor{gray}{/}& 4597  \\
      \bottomrule
      \end{tabularx}
      \label{tab:datasets}
      \vspace{-4mm}
\end{table}

The \textbf{\textsc{Verbmobil}\xspace} corpus is a large collection of spontaneous
telephone conversations \cite{WeilhammerKURFS02}. In each conversation, two
speakers negotiate the details of a business meeting. The corpus
contains English, German, and Japanese conversations, however, we only
use English portion of the corpus for our experiments. The \textsc{Verbmobil}\xspace corpus does
not come pre-annotated with NE classes. About 20\% of the \textsc{Verbmobil}\xspace
corpus was thus annotated via crowd sourcing.
The remaining
80\% of the corpus was annotated automatically using
\texttt{spaCy}\xspace\footnote{\url{https://spacy.io}}
and post-corrected manually.

\noindent

The \textbf{ATIS}~\cite{hemphill_atis} corpus is a popular dataset for slot filling and
intent detection tasks in the Air Travel Information Services domain.
For the text transformation experiments, we map the
provided slot labels to the aforementioned named entity categories.

\noindent

\textbf{SNIPS} is another popular benchmark dataset for slot filling
and intent detection task by SNIPS.AI~\cite{Coucke2018SnipsVP}. The
dataset consists of seven intents from different domains  such as
\textit{``AddToPlaylist''}, \textit{``BookRestaurant''},
\textit{``GetWeather''}, \textit{``RateBook''}.

\noindent

\textbf{FB en-TOD} is a multilingual slot and intent classification
dataset recently released by
Facebook~\cite{Schuster2019CrosslingualTL}. It consists of utterances
from three languages (English, Spanish, and Thai) and three domains
(Alarm, Reminder, and Weather). In this paper, we only use the English
dataset.

\noindent

\textbf{MS Taxi} and \textbf{MS Restaurant} are two out of three dialog
challenge datasets released by Microsoft at the SLT 2018
workshop~\cite{li2018microsoft} for taxi bookings and restaurant
reservations with 19 and 29 slot types respectively and 11 dialog acts.
The number of classes for the Taxi and Restaurant datasets are 18 and
24 respectively after removing classes with less than 40 utterances.

\vspace{-2mm}

\subsection{Experiments}
\label{sec:experiments}

For all comparison experiments, we first run a baseline experiment
using the original datasets. Then, we apply the respective privacy
strategies to the training data before fine-tuning a
BERT model for token/sentence classification. We
then compare the performance of the resulting models with the baseline
with respect to the (untransformed) test set. The BERT classification
model involves \textit{fine-tuning} the pre-trained BERT embeddings on
the training data with an additional linear layer whose weights are
randomly initialized. We trained all the parameters of the model
end-to-end, including the linear layer.

For the implementation, we fine-tuned BERT on the various tasks using
the
\textit{simpletransformers\footnote{{https://github.com/ThilinaRajapakse/simpletransformers}}}
based on the \textit{transformers} library of
HuggingFace~\cite{Wolf2019HuggingFacesTS}. The hyper-parameters of the
model are 768-dimensional embedding layer (for bert-base-cased model),
batch size of 8 for NER and 16 for other tasks, maximum learning rate
of $0.00005$, maximum sequence length is 128 for NER and 64 for other
tasks. The maximum number of epochs for all experiments is 3.

\begin{figure}[tbh]
  \centering
  \vspace{-3mm}
  \includegraphics[draft=false,width=\columnwidth]{images/dp_acc_epsilon_prob.png}
  \vspace{-4mm}
  \caption{Connection between DP (privacy), Replacement probability $p$, and F1-score (performance) on \textsc{Verbmobil}\xspace.}
  \label{fig:dp_result}
   \vspace{-6mm}
\end{figure}

\subsection{Results}
\label{sec:results}

We show that the performance of the redact and word-by-word replacement strategies
can be improved by tuning the parameter $p$ described in
Section~\ref{sec:privacy}. For example, by setting $p=0.9$ (i.e.\@\xspace
$\epsilon=6.75$), we can improve the F1-score by around $4\%$ for word-by-word and $60\%$ for redact as shown
in Figure~\ref{fig:dp_result}, demonstrating the ability to control the privacy-utility trade-off. In the word-by-word replacement
experiments, we replace a NE word $t'$ by another word $t$ of the same
entity class based on their relative frequency distribution $\pi(t)$
in the corpus.

The baseline to which we compare all other experiments is simply
trained on the original training set, i.e.\@\xspace, without removing any
private information. On the test set, the resulting model yields a
prediction F1-score/Accuracy of $88.3\%$ (NER), $98.4\%$ (ATIS intent)
and $98.0\%$ (SNIPS intent), $99.4\%$ (en-TOD intent), $78.9\%$
(Restaurant DAC) and $90.0\%$ (Taxi DAC).

Table~\ref{Tab:result} shows the result for the different text
transformation strategies. Replacing private tokens using
\textit{redact}, \textit{typed placeholder} and \textit{named
  placeholder} strategies generally gave a worse result than the
word-by-word replacement. For NER, we observe a substantial drop in performance for redact
and placeholder approaches because the model overfits on the
replacement tokens which are expected to be absent in the test set. On the other
hand, the drop is minimal for intent and dialog act classification
tasks around ($2 - 4\%$) similar to the
observation in \cite{Tang2004PreservingPI}, except for the SNIPS
dataset with much larger reduction in performance of
$8 - 44\%$ depending on the placeholder strategy. This shows that
these transformation strategies are generally not suitable for
training NLU systems.

For the \textit{word-by-word} replacement, we observe a drop of 15\%
in F1-score when we replace all words labeled as named entities with
tokens of the same-type.
For NER, we find
that ``TIME'', ``ORG'' and ``DATE'' are most affected by the
word-by-word replacement in terms of drop in F1-score because many of
them are multi-word expressions. Thus, the three named entities gain
the most by full-entity replacement. On the other-hand, the drop is
very small ($<1\%$) for other intent and dialog act classification.

Table~\ref{tab:transformation_example} illustrates an example of the
full-entity replacement (e.g ``Frankfurt Airport'' is replaced by
``New York''). This approach gives the best performance out of all the
transformation strategies with only $2.4\%$ drop for NER. Interestingly,
there is no significant difference between its performance and the
baseline on the intent and dialog act classification tasks
across the datasets. In summary, the text obtained using the word-by-word or
full-entity text transformation are more suitable for training NLU
systems
while protecting the
privacy of users.

\section{Conclusion}
\label{sec:discussion}

Replacing sensitive tokens with benign alternatives is a common method
for de-identifying text documents. We prove that privacy guarantees
for a formalized version of this process can be expressed in terms of
Differential Privacy. Our approach includes two parameters, $p$ and
$\pi$ that allow different replacement strategies to be expressed as
instances of the same algorithm. The respective DP-$\epsilon$ value follows
from the choices for $p$ and $\pi$, permitting a comparison of
different replacement strategies with respect to their privacy
implications.

User privacy is juxtaposed by the performance impact that a text
transformation has on subsequent machine learning tasks. We experiment
with three different NLP tasks across six different datasets and find
that both word-by-word and full entity replacement strategies are
robust against performance drops across all examined tasks.

\todo{future work}

\section{Acknowledgments}
This research has received funding by the European Union's Horizon 2020 research and 		innovation programme under grant agreement No. 3081705 -- COMPRISE (\texttt{http://www.compriseh2020.eu/}). We thank Xiaoyu Shen, Volha Petukhova, and the anonymous reviewers for their insightful comments that helped us to improve
this paper.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


This is a great Bay sailer - perfect for a couple or a family. It has the great Catalina 30 interior layout - a very big boat inside for it's size, with a separate v-berth and a second double to boot. A big dinette, setee and functional galley complete the interior. Topside, it's large cockpit and well laid out deck are set up for a great time.
It is reported that the owner is moving to a smaller boat. Owner says hull is in sound condition but the bottom needs sanding.
The boat needs a new transmission and some cosmetic work.
Sleeps 2 in the V Berth, 2 in Quarter berth and 1 on Settee. The dining table converts to a double.
Galley has a 2 burner CNG gas range, double sink, ice box and Head with shower.
Sails: Sparkman & Stevens Genoa w/ CDI roller furling. Fully battened Main.


Why is card a good way to pay for purchases abroad?
When traveling, make sure to bring your Visa card. It's an easier, more convenient, and more secure way to pay for purchases abroad.
Before you use your card abroad, make sure it's been activated for overseas use, and keep your card information with you at all times.
What to do when you lose your Visa card?
If you lose your card, call your bank. They will help you secure the card, and arrange for a replacement.
Why is a Visa card better for traveling?
Before you use your card abroad, make sure it's been activated for overseas use, and keep your card information with you at all times in case anything happens.
*Cardholders must use care in protecting their card and notify their issuing financial institution immediately of any loss of unauthorized use of their card. Contact your issuer for more details relating to cardholder protection and liability.


text:
using System;
using System.Globalization;
using System.Linq;
using System.Web.Caching;
using System.Web.Mvc;
using System.Xml.Linq;
using NerdDinner.Helpers;
using NerdDinner.Models;

namespace NerdDinner.Controllers {
    public class JsonDinner {
        public int DinnerID { get; set; }
        public DateTime EventDate { get; set; }
        public string Title { get; set; }
        public double Latitude { get; set; }
        public double Longitude { get; set; }
        public string Description { get; set; }
        public int RSVPCount { get; set; }
        public string Url { get; set; }
    }

    [HandleErrorWithELMAH]
    public class SearchController : Controller {
        private readonly IDinnerRepository dinnerRepository;

        public SearchController(IDinnerRepository repository) {
            dinnerRepository = repository;
        }

        //
        // AJAX: /Search/FindByLocation?longitude=45&latitude=-90

        [AcceptVerbs(HttpVerbs.Post)]
        public ActionResult SearchByLocation(float latitude, float longitude) {
            IQueryable<Dinner> dinners = dinnerRepository.FindByLocation(latitude, longitude);

            IQueryable<JsonDinner> jsonDinners = from dinner in dinners
                                                 select JsonDinnerFromDinner(dinner);

            return Json(jsonDinners.ToList());
        }

        [AcceptVerbs(HttpVerbs.Post)]
        public ActionResult SearchByPlaceNameOrZip(string placeOrZip) {
            if (String.IsNullOrEmpty(placeOrZip)) return null;
            ;

            string url = "http://ws.geonames.org/postalCodeSearch?{0}={1}&maxRows=1&style=SHORT";
            url = String.Format(url, IsNumeric(placeOrZip) ? "postalcode" : "placename", placeOrZip);

            var result = ControllerContext.HttpContext.Cache[placeOrZip] as XDocument;
            if (result == null) {
                result = XDocument.Load(url);
                ControllerContext.HttpContext.Cache.Insert(placeOrZip, result, null, DateTime.Now.AddDays(1),
                                                           Cache.NoSlidingExpiration);
            }

            var LatLong = (from x in result.Descendants("code")
                           select new
                                      {
                                          Lat = (float)x.Element("lat"),
                                          Long = (float)x.Element("lng")
                                      }).First();

            IOrderedQueryable<Dinner> dinners = dinnerRepository.
                FindByLocation(LatLong.Lat, LatLong.Long).
                OrderByDescending(p => p.EventDate);

            return View("Results", new PaginatedList<Dinner>(dinners, 0, 20));
        }

        // IsNumeric Function
        private bool IsNumeric(object Expression) {
            // Variable to collect the Return value of the TryParse method.
            bool isNum;

            // Define variable to collect out parameter of the TryParse method. If the conversion fails, the out parameter is zero.
            double retNum;

            // The TryParse method converts a string in a specified style and culture-specific format to its double-precision floating point number equivalent.
            // The TryParse method does not generate an exception if the conversion fails. If the conversion passes, True is returned. If it does not, False is returned.
            isNum = Double.TryParse(Convert.ToString(Expression), NumberStyles.Any, NumberFormatInfo.InvariantInfo,
                                    out retNum);
            return isNum;
        }

        //
        // AJAX: /Search/GetMostPopularDinners
        // AJAX: /Search/GetMostPopularDinners?limit=5

        [AcceptVerbs(HttpVerbs.Post)]
        public ActionResult GetMostPopularDinners(int? limit) {
            IQueryable<Dinner> dinners = dinnerRepository.FindUpcomingDinners();

            // Default the limit to 40, if not supplied.
            if (!limit.HasValue)
                limit = 40;

            IQueryable<JsonDinner> mostPopularDinners = from dinner in dinners
                                                        orderby dinner.RSVPs.Count descending
                                                        select JsonDinnerFromDinner(dinner);

            return Json(mostPopularDinners.Take(limit.Value).ToList());
        }

        private JsonDinner JsonDinnerFromDinner(Dinner dinner) {
            return new JsonDinner
                       {
                           DinnerID = dinner.DinnerID,
                           EventDate = dinner.EventDate,
                           Latitude = dinner.Latitude,
                           Longitude = dinner.Longitude,
                           Title = dinner.Title,
                           Description = dinner.Description,
                           RSVPCount = dinner.RSVPs.Count,
                           //TODO: Need to mock this out for testing...
                           //Url = Url.RouteUrl("PrettyDetails", new { Id = dinner.DinnerID } )
                           Url = dinner.DinnerID.ToString()
                       };
        }
    }
}

redpajama_set_name: RedPajamaGithub


SEE THE MOST UP-TO-DATE INFORMATION ON THE STORM HERE.
A powerful nor'easter began barreling into the tri-state Friday night slightly ahead of schedule, and Storm Team 4 says it has picked up in intensity and speed, enough to become a potentially paralyzing blizzard with blasting winds, heavy snow and coastal flooding for a wider portion of the area than previously thought.
Storm Team 4 now says the heaviest snow is expected between 4 a.m. and 4 p.m. Saturday, with increased project snowfall totals for much of the area.
The storm is "bombing out," Storm Team 4 says, meaning the pressure in center of the storm is dropping rapidly, a sign of a very intense nor'easter. That center of the storm will dump 2 to 3 inches of snow per hour when it moves through the tri-state Saturday morning and into the afternoon (see forecast video above for more on bombogenesis).
Blizzard warnings are in effect for all of New York City, Long Island, large swaths of New Jersey, parts of southern Westchester and coastal Connecticut from Saturday morning until Sunday afternoon. A blizzard is defined as a wintry storm lasting longer than 3 hours with visibility below a quarter mile and winds gusting above 35 mph.
Snow started falling across southern New Jersey and the Jersey Shore Friday evening, and by about 10 p.m., flakes were falling steadily in Manhattan and other parts of New York City, moving in slightly faster than expected.
Winds will pick up overnight, and by the time most tri-staters wake up Saturday morning, several inches of snow will already be on the ground. Storm Team 4 says it's now projecting between 12 and 18 inches of snow will fall on the areas under a blizzard warning. Some isolated parts within the region could even get up to 20 inches of snow.
The lower Hudson Valley is also now expected to get about 12 to 18 inches of snow.
Areas slightly further north will get 2 to 6 inches. Above that, in the far northern reaches of New Jersey and New York's upper Hudson Valley, only trace amounts of snow are expected.
Conditions will worsen as the day progresses, with damaging winds combining with heavy snowfall making for limited visibility.
Perhaps more dangerous than the snow is the cocktail of snow, astronomical high tides and a wind-swept storm surge that could cause significant coastal flooding and beach erosion on the Jersey Shore and parts of Long Island. Waves topping 15 feet may also batter beaches built up with protective sand berms. Winds topping 60 mph could also down trees and leave homes without power.
Several towns along the Garden State's coastline asked residents to leave their homes on a voluntary basis earlier Friday; Barnegat has ordered mandatory evacuations on one street.
The region isn't expected to be hit as hard as the nation's capital, where nearly 30 inches of snow could fall, but it's not one that officials are taking lightly.
Mayor de Blasio has declared a "winter weather emergency" ahead of the storm, and has urged drivers to stay off the roads after 8 a.m. Saturday, and warned New Yorkers that vehicles blocking plows and emergency vehicles will be towed.
"Get done what you have to get done today," de Blasio said. 'Do not bring your vehicle out tomorrow."
This weekend's winter weather emergency is different from the travel ban instituted ahead of a forecast blizzard last January. De Blasio said this storm shouldn't be taken lightly, but it doesn't look like the storm will have the same potential impact as the one that forced the MTA to shut down the city's subway system last year.
"The city is fully prepared," de Blasio said.
Gov. Cuomo, meanwhile, said that the state is ready for the storm, which he said was "not the worst forecast we've ever had." He said salt spreaders and plows are ready and that 600 National Guardsmen are at the ready.
Gov. Christie, back in New Jersey from the Republican presidential campaign trail in New Hampshire, has also declared as state of emergency in New Jersey.
He's authorized NJ Transit to halt service at 2 a.m., and has ordered Motor Vehicles Commission offices closed on Saturday.
Christie and Cuomo both pleaded residents to stay off the roads.
"One of the greatest burdens becomes stuck vehicles or stranded vehicles," Cuomo said. "You don't just endanger yourself, you endanger the emergency personnel who come out to help you."


text:
What To Expect From Frank Ocean's Next Album
Israel Zuzi
Frank Ocean is known for his unique approach to R&B and Pop music making him one of the biggest faces in the genres throughout the 2010s. He released his first album Channel Orange to critical acclaim. He took a four year hiatus and then came back to music to drop his second studio album, Blonde, accompanied by his "Endless" visual album a day prior to fulfill a label contract with Def Jam before going independent. Since the released 7 droplets on streaming in the past 5 years, none have carried into the release of a full studio project soon after. But fans speculate the last 4 singles were part of an album rollout that was put on hold.
In September of 2019, Ocean opened up about where he currently is creatively in an interview with W Magazine
"I've been interested in clubs, and the many different iterations of nightlife for music and songs. And so the things I look at now have a lot to do with those scenes: Detroit, Chicago, techno, house, French electronic… "
He continues to mention how his upbringing in New Orleans helped him be closest to "New Orleans bounce" since it was the closest to the nightlife scene.
"It's so much a part of my childhood and my youth that I don't really go back to it so much as a touch point. I'm really looking forward. It's kind of a mix for me."
During October 2019, Frank Ocean hosted a series of club nights in New York City called "PrEP+," named after the HIV prevention drug of the same name. He premiered new songs that were far off from what he usually released in the past in a good way. The tracks included
A house-inspired remix of "The Weekend" by SZA covered by Ocean himself
An electro-hop version of his then-upcoming single, "Cayendo"
A rap collab with Skepta titled "Little Demon" (Single released was cancelled)
The night after his first appearance at PrEP+, he released a rap single onto streams titled "DHL" produced by German electronic music producer Boyz Noize. [ He met producer Boyz Noize in Berlin through ASAP Rocky as they were working on 'Babushka Boi', and Frank showed up mid-session.] The following month he released a club-inspired single "In My Room", produced by Sango (who remixed Cayendo).
Fans pointed out that both singles had unique artwork, they had 13 different silhouettes at the bottom. The main images used for DHL & In My Room were highlighted somewhere on the silhouette line, leading fans to believe these 2 singles were a part of his upcoming full length album titled "Look At Us We're In Love". The title was guessed based off of the header for Frank's website during the "rollout" and a logo on the PreP+ promo poster that matched the styling of the single covers.
Frank was set to perform at Coachella 2020, where it was believed he'd finally release his album soon after but the Covid-19 Pandemic prevented this. He rescheduled for 2023 anyways. Later during the year, Frank also tragically lost his younger brother. Through all these events, fans expected another push back for new Frank material coming anytime soon.
Since then he hasn't released any new material or been active on social media that much until he returned with the opening of his luxury jewelry brand, "Homer". It was also revealed through a source that he's trying to pick a label for the release of his next project. According to HITS Daily Double, Frank could be shopping a brand new album and has reportedly been meeting with and playing the new record for label executives. The list of industry players he's been in contact with remains under wraps, but HDD does point out that Ocean has been doing "his best to avoid doing business with the majors" since dropping channel ORANGE in 2012.
It's possible Frank's doing this to help market his album to a new audience as best as possible considering he's stepping away from his typical R&B/Pop Sound in the past.
Back to Online Magazine
© 2022, Zuzi. Powered by Shopify

redpajama_set_name: RedPajamaCommonCrawl


Are you wondering whether you need a rhinoplasty, chin augmentation, facelift? Do you want to know if your nose can be fixed? Can the contour of my chin be improved? Whatever question you ask when you look in the mirror, Dr. Fleming can give you the answers.
By taking a few basic photos of yourself and submitting them through our website, Dr. Fleming can provide you with all of the information you need regarding the best approach to giving you the most natural result.
Please complete the Photo Submission Form below and follow the instructions. It is very important that the photos be taken as instructed on the "Types Of Photos Required For Evaluation" page.
If you have any questions in regards to the types of photos, please feel free to contact the office at (310) 278-8823.


text:
Aston Martin is taking on the McLaren P1 GTR and Ferrari FXXKwith a track-only hypercar of its own. Called the Vulcan, it's loosely based on the road-going One-77, but with the 7.0-litre V12 engine turned up to '800bhp-plus' and an even steeper asking price of £1.8m.
The car you see here will debut on Aston's Geneva Motor Show stand next week, with the first public demonstration laps scheduled for the Le Mans 24hr race in June. Only 24 examples will be built in total, a nod towards Aston's success in 24hr endurance races around the world, and only the most committed Aston owners need apply.
The sensational new shape, dreamt up by Aston head of design Marek Reichmann, is designed to shock with its ultra-wide track, huge rear wing and extreme aero package. Cavernous air vents in the bonnet are there to cool the immense engine, while the slash from the front wheel arch along the flanks is a new idea for Aston.
However, there are elements of the new Aston Martin DB10 concept in the low-set front grille and slot-like headlights, suggesting the Vulcan isn't only about shock and awe – it points towards the next-generation DB9 and V8 Vantage, both due within the next two years.
Based around a lightly modified version of the One-77's carbon-fibre tub, the Vulcan counters the hi-tech hybrid setups used by Ferrari and McLaren with a refreshingly traditional front-engined, rear-wheel drive layout. The engine is a naturally-aspirated V12 tuned to over 800bhp, and channelling its power through an Xtrac six-speed sequential gearbox.
Designed to meet full FIA-specification, the Vulcan also gets a limited slip differential, magnesium torque tube with a carbon propeller shaft and 380mm carbon ceramic brakes discs at the front with 360mm discs at the rear. Unique 19-inch Michelin tyres are controlled by pushrod suspension with adjustable dampers, while anti-roll bars and variable traction control offers assistance to the driver if required.
Stripped-out to help reduce the overall weight to 1,350kg (150kg less than the One-77), the interior is a feast of carbon fibre, Alcantara and titanium switchgear. A full digital display sits behind a semi-circular steering wheel chopped off at the top. Aston claims it is 'almost infinitely flexible in terms of colour and trim options.'
All 24 lucky owners will also be offered the opportunity to improve their driving skills under the tutelage of Aston Martin Racing's Le Mans-winning Darren Turner.
Q&A with Aston Martin Vulcan designer Marek Reichmann
Why build a car like this and why now?
Building on what we've achieved in racing is important, it's important to make that link between the race and road cars. We've seen a trend over the last nine to twelve months that One-77's are selling for way above list price, which is a good sign, we then added in all the lessons learned through racing. You've seen the Vantage GT3, now this is the next step. Other manufacturers have shown there's a market for this sort of thing, I don't want to say that ours is more analogue, but we're certainly doing it differently.
How similar is the Vulcan to the One-77?
We used the One-77 moulds for the carbon tub, which meant we knew it was extremely rigid and it didn't need to be strength tested again. We had to modify them to make the engine mountings lower and accommodate the gearbox and roll cage. Beyond that it's a case of making sure the whole chassis is in tune with its lighter 1,350kg weight and with around 100bhp more.
There are similarities with the DB10, but how much does the Vulcan's design tell us about future Astons?
The DB10 is just for James Bond, but it does have a much lower face, like the Vulcan. Both hint at the future face of Aston Martin, but what we'll see is that the next DB9 is more gentile and the next V8 Vantage is more aggressive, we want a much wider spread of products. The new DB9 will be tech-led, too, we've never invested so much in a new product and then there's the technical partnership with Mercedes.
Since Andy Palmer took over as CEO things seem to be moving quickly, have you ever been this busy?
You're right we're extremely busy. I just had a conversation with Andy and a customer who's interested in the Vulcan, and I said the thing about Aston is the glass isn't half empty any more it's half full. Andy turned to me and said "no it's not, it's overflowing!" We had a plan in place before Andy, but his strengths are that he's a guru for product planning. He's taking the company's soul and moving it on.
Can you tell us the story behind the interior?
Firstly, it really moves things on. We've been hinting at some quite radical Astons for a while now – there was the CC100 concept, a combination of carbon and race-craft, then the mid-engined DP-100 virtual concept added tech into that equation. The Vulcan is a production version of those ideas. On the inside of course we had to cut weight, but all the touch points still feel like jewellery. We've worked with Darren Turner to make sure the layout, instruments and ergonomics were all spot on, not just for a pro driver, but amateur drivers that will buy this car, too. It looks like a spaceship that's just landed.
Are you planning to race it?
It's full FIA-spec, including the aero so it would qualify for an invitation for some races, like Bathurst. It's not a full GT3 car, but could be with some relatively simple mods – the clever part of the 7.0-litre V12 engine is it can be down-tuned from 800bhp to 500bhp relatively easily, too.
We know you're a keen racer, have you driven it yourself?
No, not yet. The first public debut on track will be at the Le Mans 24hr in June, we start testing in six to seven weeks.
2021 Range Rover Velar SVAutobiography Dynamic, 5.0 V8...
This Is The New E-Class AMG 43
Range Rover Velar SVAutobiography P550 Dynamic Edition !
Self-driving Mitsubishis Will Use Adapted Missile...
2016 Lamborghini Huracan Customed By Sunus Motorsports !
2021 Mercedes-Benz S-Class Brabus B50, 500 HP Beast !


A complete and unambiguous written contract of sale of lots, on which stood a hotel building, made no reference to hotel furniture in the building. The vendee brought an action of replevin to obtain possession of the furniture, and offered parol evidence that the transaction of sale included furniture. Held, the court was authorized to hear parol evidence of conversations, conduct, and circumstances, to enable it to determine, as a preliminary matter, whether the parties intended the writing to be limited to the single subject of real estate.
The date considered, and held, the court properly concluded the written instrument was not intended to constitute exclusive evidence of an entire transaction of sale, but was intended to relate to real estate only.
Since the writing was intended to relate to real estate only, parol proof of sale of the personal property as part of one transaction of sale was properly admitted.
There was ample evidence the hotel furniture was sold with the hotel.
The action was one of replevin to obtain possession of hotel furniture. Plaintiff recovered, and defendant appeals.
1 Plaintiff claimed ownership and right of immediate possession. Defendant denied plaintiff's claim, alleged ownership and right of possession in himself, and right of possession depended on who owned the property. Plaintiff purchased from defendant land on which stood a hotel operated by a tenant, and there was ample oral evidence that the sale included the hotel furniture owned by defendant, possession of which defendant surrendered to plaintiff with possession of the hotel. More than two years later, defendant gained occupancy of the hotel by securing an assignment of a lease, and, on being notified by plaintiff to quit, removed the furniture at night. A scrivener wrote a contract of sale of the real estate, which the parties signed. The instrument made no mention of the personal property. Defendant contended the writing alone could be looked to for evidence of the transaction, and the parol evidence by which sale of the furniture was established was inadmissible.
The writing was appropriate to evidence a sale of real estate to be consummated in the future, and was devoted to that subject. It was unambiguous, made no reference to any item of personal property, and contained nothing to indicate the parties were dealing with respect to any subject except land and matters incidental to the land described. The parties were not obliged to extend the writing beyond the single subject of land. They could leave personal property included in the transaction of sale to be transferred by bill of sale or by delivery on payment of price. St. Louis, L. & W. R. Co. v. Maddox, 18 Kan. 546. Such an arrangement would not contradict any provision of the land contract because it purported to relate to land only. Was it the intention of the parties to limit the writing to that single subject of negotiation? If so, the other subject of negotiation resting in parol could be proved by parol, and intention was a question of fact, to be determined by investigating conversations, conduct, and circumstances. When the court, not the jury, had determined in this manner the preliminary question of what the writing covered, the court was in a position to rule on whether parol evidence of sale of the furniture should be received and go to the jury.
2 The scrivener could recall nothing said to him by way of information from which he prepared the contract. The evidence relating to what was said to him was that furniture was not referred to, either as a part of the *1009 property involved in the trade, or as something reserved by defendant because not involved in the trade. Plaintiff testified he and defendant went to the scrivener's office to get the lot and block numbers, and the personal property was not mentioned because that had been talked over before. From this and other evidence, the court was authorized to conclude that the writing was complete, that it related to land only, and that the parol evidence rule did not apply; and, consequently, was authorized to permit parol proof as to what, if anything besides land, the transaction in fact embraced. The court submitted this question to the jury, under instructions not complained of if the parol evidence was competent, and, as indicated, there was abundant evidence that defendant sold and delivered the furniture with the hotel.
(1) Whether a particular subject of negotiation is embodied by the writing depends wholly upon the intent of the parties thereto. * * * The parties are not obliged to embody their transaction in a single document; yet they may, if they choose. Hence it becomes merely a question whether they have intended to do so.
Such a proposition, however, is untenable, both on principle and in practice. In practice, it is not enforced by its theoretical advocates. In theory, its fallacy is indicated by what has been already noticed (ante, section 2430)." Pages 307, 308, 309, 310.
"A deed of conveyance of real estate, which embraced a store building provided with shelving, contained the clause, 'This grant includes all the shelving in the building.' Held, that the deed is not to be taken as intended to express or constitute the exclusive evidence of the whole contract or transaction, which resulted in the giving of the deed, so as to render incompetent parol proof of a sale of personal property at the same time." Page 168, 52 N. W. 525.
34 In the present instance, the written instrument did not by itself conclusively establish whether the parties intended it should exclude every subject of sale except real estate. Parol evidence was properly received bearing upon the question of intent for the information of the court, and the conclusion of the court that the writing was intended to cover but one subject was sound. While that conclusion may not have been reached as a finality at a preliminary stage of the trial, and may have been arrived at from consideration of all the data, it was a necessary preliminary to final decision that the parol evidence *1010 rule did not apply, and to final decision to submit to the jury the question whether the furniture was sold as a part of a transaction which included sale of real estate.


From March 16, 2013. Automation of certain tasks can result in a more professional performance. Here's a demonstration of a suite of devices designed to help with this automation.
Print article This entry was posted by CWood on July 2, 2013 at 11:32 pm, and is filed under Post Filler. Follow any responses to this post through RSS 2.0. Both comments and pings are currently closed.


text:
Assam Poll: How BJP Contained Political Impact of Anti-CAA Stir
From inducting protest leaders to forming autonomous councils, BJP has cleverly tried to neutralise the CAA factor.
Amarjyoti Borah
Updated: 26 Mar 2021, 10:41 AM IST
Out of Assam's 126 Assembly seats, 47 will be voting in the first phase of polling on 27 March. In the 2016 Assembly election, the BJP-led National Democratic Alliance had won 35 out of 47 seats, sweeping Upper Assam. One seat was won by an Independent candidate, who has since joined the BJP and will now be contesting on the party's ticket.
However, Upper Assam – both the North and South banks of the Brahmaputra – had emerged as the epicentre of the anti-CAA protests in 2019.
One of the ways in which the BJP-led government in Assam has been able to douse the fire of the anti-CAA movement is by inducting many of the top leaders of the protest. However, this has come at the cost of further fragmentation of the greater Assamese community, even as the government is still fighting to douse any possible shift by the politically crucial tea tribes.
Assam Assembly Polls: When It Comes to Voting, CAA May Not Matter
The BJP needs to sweep Upper Assam as it is staring at a loss of seats in the subsequent two phases due to non-division of religious minority votes as a result of the Congress-AIUDF alliance.
This article will try to look at the different steps taken by the BJP to neutralise the CAA factor in the Assam elections and also briefly examine the question of the tea garden workers of Upper Assam.
BJP Manifesto Silent on CAA
The BJP manifesto for Assam, doesn't mention implementation of the CAA. It has also avoided mentioning the Act in its campaign, except for a few statements made by BJP national president JP Nadda and state BJP president Ranjit Dass.
"The BJP fears a backlash from the voters if it talks about CAA in its campaign, so they have avoided talking about it. The party has realised that it could lose more votes than it could possibly gain if it rakes up CAA,"
Sabita Devi, columnist and editor of Pratilipi
The BJP also denied tickets to two sitting MLAs - Rituparna Barua and Debananda Hazarika - who had opposed CAA and had expressed solidarity with the anti CAA protestors.
Its alliance partner, the AGP didn't give ticket to three MLAs who had opposed CAA - former Chief Minister Prafulla Kumar Mahanta, former ministers and senior leaders Brindaban Goswami and Utpal Dutta.
"When I spoke out against CAA, it was in the greater interest of Assam," said Goswami.
'Congress to Ensure CAA Isn't Implemented in Assam': Rahul Gandhi
Anti-CAA Leaders Inducted
However, the BJP led alliance has been able to stay on course for an electoral win by inducting prominent faces of the anti-CAA movement into the saffron party and also by setting up of more autonomous councils.
Arunjyoti Moran, a firebrand leader of the All Assam Moran Students' Union who was active during the anti CAA movement, joined the BJP in January this year. Though he didn't get a ticket to contest the election, it is believed his induction will be a value addition for the BJP.
"The BJP can solve the issues and problems in Assam and it is important to support the party," Moran told the media after joining the BJP.
This year the BJP also inducted senior All Assam Students' Union (AASU) leaders Rahul Das and Parikhit Dutta into the saffron fold - both of them were part of the anti-CAA protests in Assam.
Also brought into the BJP are about 25 actors and singers, who have a fan-following and were also faces of the anti-CAA agitation. These include actor Jatin Bora, singers Simanta Shekhar and Vidyasagar, Kallol Borthakur, Nayan Nirban, Asha Bora and Baikuntha Prince.
Jadav Das, a leader of the Scheduled Caste Students Union who had staged a naked protest against the CAA in New Delhi, also joined the party. After his induction, Das said that he was "misled and misguided".
Autonomous Councils to Ease CAA Woes
In an attempt to pacify the CAA protestors, last year the Assam government in the Assembly passed four bills to create autonomous councils for the Moran, Motok, and Koch-Rajbongshi communities in the state. The Moran Autonomous Council Bill, 2020, The Motok Autonomous Council Bill, 2020, and The Kamatapur Autonomous Council Bill, 2020 were tabled on 24 March last year.
Later the Assam Assembly also passed a Bill for creating an autonomous council for the Bodo-Kachari community living outside the Bodoland Territorial Region (BTR) that comprises four districts of Kokrajhar, Chirang, Baksa and Udalguri.
"The BJP-led government will be providing Rs 125 crore to each of the Ahom, Chutia, Moran and Motok communities for generating self-employment opportunities, infrastructure development, preservation of monuments and archaeological heritage," the Assam chief minister's office had stated after the passage of the bills.
"These councils will help the BJP regain some of the lost support from these communities and will benefit it in the electoral battle," said Devi.
Tea Tribe Factor
One of the challenges the BJP is facing in Upper Assam is the possible shift of the tea tribe votes that had played an important part in its victory in 2016.
To prevent such a shift, the BJP has inducted former Congress leader Rupesh Gowala, general secretary of the influential Assam Chah Mazdoor Sangha (ACMS), which has a strong influence over the tea tribe community.
However, tea tribe organisations and leaders have expressed dissatisfaction over the BJP government not raising the daily wage of the tea garden labourers to Rs 351 per day as was promised in the 2016 election manifesto.
The tea garden workers got their first substantial hike in 2017 when as an interim measure, the state government had raised their minimum wages to Rs 167 per day. This was following the recommendation of an advisory committee, which was formed by the Assam government. The committee, however, had suggested a daily wage of Rs 351.
"We have been hearing promises after promises but the wage hike has not been implemented," said Nabin Chandra Keot, vice president of ACMS.
The issue has become an important one for both the incumbent BJP and the main Opposition Congress, ahead of elections.
As a damage control measure, the Labour Welfare department on 23 February said that the minimum wage for tea workers should be raised from Rs 167 per day to Rs 217 per day by adding an "interim amount" of Rs 50 per day.
However, this order was immediately challenged by tea garden owners at the Gauhati High Court, which ruled that that tea garden owners in Assam will have the "liberty" to decide if they want to follow the state government's order on raising wages for workers.
After the interim order, no "coercive action" can be taken against them for not following it.
The Indian Tea Association, however, on 22 March has directed the member tea gardens in Assam to enhance the daily wage by Rs 26, as an interim measure, till the final judgement of the court is delivered.
So, be it regarding the anti-CAA protests or the wage hike demand for tea garden workers, the BJP has taken some measures to contain their impact in the elections. However, it remains to be seen whether this would be enough to guarantee as sweep of Upper Assam that's essential to the BJP's electoral prospects.
(The writer is a freelance journalist based in Assam. This is an opinion piece and the views expressed are the author's own. The Quint neither endorses nor is responsible for them. )
C-Voter: UPA To Fall Just Short in Assam; Majority for TMC in WB
Published: 26 Mar 2021, 10:39 AM IST