text:
\subsection{Main Results on Decoding Performance}
\label{subsec:benchmark}

\textbf{Comparison with off-the-shelf decoding methods.}
Table~\ref{tab:benchmark} compares the results of GPT2-medium and BART-large on six diverse NLG tasks with beam search (Beam), sampling (Sampling), and our proposed {\mbox{KID}}{} decoding algorithm. Compared with the other two commonly used decoding algorithms, knowledge-guided {\mbox{KID}}{} achieves better results for all the cases, with significant improvements (with p-value $p<0.01$) over the next-best decoding strategy (e.g., 4.1 absolute increase for BLEU-1 in ELI5 (GPT2)). We also notice that {\mbox{KID}}{} brings greater improvements to auto-regressive language models -- above 5\% improvement in 9 out of 12 metrics for GPT2-medium, in contrast to 5 out of 12 for text2text language model (BART-large). The reason could be that the reinforcement learning objective of {\mbox{KID}}{} is more similar to the MLE objective of GPT2 than the denoising objective of BART~\citep{pang2020text,guo2021text}. Compared with task-specific state-of-the-art models\footnote{Published SotA models to date (October 2021): ELI5 (RAG;~\citeyear{lewis2020retrieval}), MSMARCO (RAG;~\citeyear{lewis2020retrieval}), ROC(Knowledge-enhanced GPT2;~\citeyear{guan2020knowledge}), $\alpha$NLG (T5;~\citeyear{raffel2019exploring}), WoW (BART+DPR;~\citeyear{petroni-etal-2021-kilt}), and MuTual (Human Performance;~\citeyear{cui-etal-2020-mutual}). The information mainly comes from corresponding leaderboards.}, task-agnostic LMs armed with {\mbox{KID}}{} can beat SOTA results on three different tasks (ELI5, ROC, WoW), which is not possible when beam search and sampling decoding is used. This interesting observation demonstrates a huge potential in inference-time optimization, which we believe is worth exploring further.

\input{tables/fewshot}

\textbf{Comparison with existing knowledge-infusion methods.} Besides evaluating {\mbox{KID}}{} from a pure decoding perspective, we also compare {\mbox{KID}}{} with existing methods of integrating knowledge for knowledge-intensive NLG tasks. In addition to ELI5 and MSMARCO, we also evaluate on two extra QA tasks: PIQA and PubMedQA (discussed in $\S$\ref{sec:experiment_settings}), where answers are considered to be \textit{not} fully covered by Wiki knowledge. Besides \textbf{RAG}~\citep{lewis2020retrieval}, we consider several competitive prior methods incorporating knowledge including a) \textbf{GPT2 + Knowledge}~\citep{guan2020knowledge}, which post-trains GPT2 on triplets-converted augmentation data (e.g., $\langle$\textit{helium}, \textit{is}, \textit{gas}$\rangle \rightarrow$ \textit{``helium is gas.''}); b) \textbf{GPT2 + COMeT Embeddings}, which fuses knowledge-aware embeddings (e.g., from CoMET~\citep{bosselut2019comet}; \redit{c) \textbf{FiD-T5}~\citep{izacard2020leveraging,izacard2020distilling}, which concatenates retrieved grounding documents with context as new input;} d) \textbf{QA-GNN}~\citep{yasunaga2021qa}, which traverses a graph neural network; e) \textbf{Reflective} decoding~\citep{west-etal-2021-reflective}, which relies on forward and backward LMs to encode bi-directional context for generation.

As shown in Table~\ref{tab:few_shot} (columns with 100\% training data), {\mbox{KID}}{} outperforms all other methods in ELI5 (with 5.2 ROUGE-L points improvements over the second-best method (RAG)), and achieves competitive results requiring neither specific model architecture nor additional training to infuse knowledge. We also evaluate on a few-shot setting, where only 10\% of task training data is available to mimic domains for which ample training data is unavailable or difficult to acquire in practice. This setting also tests a method's ability to transfer to new domain and to generalize to unseen entities, concepts or events. Also shown in Table~\ref{tab:few_shot}, {\mbox{KID}}{} with a LM similar in size to the baselines (GPT2-medium) achieves best few-shot performance in all four tasks, including PIQA and PubMedQA. Our experiments find baseline methods tend to generate off-topic and hallucinatory answers
when the expected answer length is long (e.g., ELI5 and PIQA). RAG shows limited performance in few-shot scenarios, due to its static knowledge retrieved by initial context cannot be generalized to newly generated tokens. {\mbox{KID}}{}, instead, dynamically searches references for grounding, thus showing more agility in unfamiliar context. \redit{Comparison with more baselines can be found in $\S$\ref{subsec:more_baselines} of Appendix.}

\subsection{Ablation Studies for the Best Performing {\mbox{KID}}{}}
\label{subsec:ablation}

\textbf{How to choose retriever?} We experiment with replacing the default DPR document retriever of {\mbox{KID}}{}, with popular retrievers including the TF-IDF retriever from DrQA~\citep{chen2017reading}, and the Wiki retriever used in BLINK~\citep{wu2019scalable}. We also experiment with a random retriever baseline that retrieves documents randomly given the context. We choose two tasks ELI5 and WoW that provide ground-truth knowledge provenance, which are also the only two KILT tasks requiring long and abstractive generation~\citep{petroni-etal-2021-kilt}. Following KILT benchmarking metrics, we use precision@1 (Prec@1) to measure the top-1 retrieval accuracy, and ROUGE-L (R-L) to evaluate generation quality. We also consider directly measuring how much knowledge in the ground-truth evidence (provenance) appear in the actual generation. We compute knowledge Coverage (Cov) (used in \citet{guan2020knowledge}) which is the 1-gram overlap between the triplets of provenance and the generation, to measure the extent to which our generation is actually using the retrieved knowledge.

Table~\ref{tab:retriever_choice} shows the corresponding results. For reference, RAG obtains nearly equivalent performance (R-L and Cov) with random retriever and the DPR$^{\textrm{FT}}$\footnote{RAG fine-tunes the query encoder of DPR with BART, which differs the off-the-shelf DPR used in {\mbox{KID}}{}.}, which indicates its generation mainly relies on the fine-tuned BART and ignores retrieved knowledge (also observed in \citet{krishna2021hurdles}). In contrast, {\mbox{KID}}{} with the default DPR retriever outperforms all other retrievers and RAG variants in retrieval accuracy (Prec@1), knowledge coverage (Cov) and final generation quality (R-L). We observe high correlation between (Cov, R-L) and Prec@1, indicating that a good knowledge retrieval is essential for knowledge-intensive NLG tasks.

\begin{figure*}[!t]
	\begin{center}
	\mbox{
		\includegraphics[width=0.33\textwidth]{images/num_docs_fin.pdf}
		\includegraphics[width=0.33\textwidth]{images/num_hop_fin.pdf}
		\includegraphics[width=0.33\textwidth]{images/exposure_bias_fin.pdf}
	}
	\end{center}
	\vspace{-0.15in}
	\caption{Impact of hyperparameters on KID's ELI5 performance when (a) more documents are retrieved, and (b) more hops taken when querying the knowledge trie. (c) Average human ratings on different-length sequences generated by KID, sampling, beam search, and reflective decoding. KID generation has more stable quality across lengths by restraining exposure bias.}
	\label{fig:hyper}
\end{figure*}

\input{tables/retriever}

\textbf{How much knowledge do we need?}\label{subsec:hyperparameter} We also study the impact of \textit{number of documents} ($k$) we retrieve and \textrm{number of hops} ($h_{\textrm{max}}$) we use to query the knowledge trie, two factors that determine how much knowledge we use to ground the generation. As shown in Figure~\ref{fig:hyper} (a) and (b), for the example task ELI5, we find the generation performance measured by ROUGE-L does not benefit from simply more retrieved documents---an optimum $k$ is 5 through empirical observation, and similarly, $h_{\textrm{max}}=4$ brings best performance. A larger $k$ might risk retrieving less relevant Wiki documents and a larger hop $h_{\textrm{max}}$ with deeper traverse through the knowledge trie tends to  bring in off-topic knowledge. We also plot the average time consumed for generating each sentence as reference (the second $y$-axis in Figure~\ref{fig:hyper} (a) and (b)), which demonstrate the best performing $k=5$ and $h_{\textrm{max}}=4$ achieve a reasonable trade-off between generation quality and decoding speed.

In Figure~\ref{fig:hyper} (c), we quantify the exposure bias problem through human judgements. We first sample 200 ELI5 test set questions and generate answers of various lengths $\{80, 100, ..., 260\}$ (260 is the average sequence length in training set) with beam search, sampling, reflective~\citep{west-etal-2021-reflective}, and {\mbox{KID}}{}. We then ask humans to rate these generations with 7-point Likert scoring~\citep{joshi2015likert} how likely the generated text is a natural sentence. Each generation receives at least 15 ratings. We observe that both beam search and sampling methods suffer from the exposure bias problem\footnote{We use beam size 5 for beam search, and top $p=0.9$ and $k=20$ for sampling decoding, as they yield best ROUGE-L score during automatic evaluation.}, since their ratings deteriorate as the length grows. Reflective decoding exhibits similar trend since it stills relies on MLE training. {\mbox{KID}}{}, instead, dynamically infuses global knowledge with LM predictions at each step and thus can mitigate exposure bias by imitating non-local demonstrations.

\textbf{Does the size of LM matter?} We run experiments with different sizes of LMs that have the similar architecture (GPT2-medium, and GPT3 with 1.3B, and 2.7B parameters\footnote{We adopt the public implementation of GPT3---GPT-Neo (\url{github.com/EleutherAI/gpt-neo}).}). Table~\ref{tab:size_matters} shows that overall larger LMs benefits all decoding methods with {\mbox{KID}}{} consistently outperforming Beam search and sampling decoding. In addition, {\mbox{KID}}{} brings more gain for non-finetuned LMs than fine-tuned ones, potentially because the fine-tuning LM is already fit onto the new domain thus less knowledge infusion is needed. Interestingly with {\mbox{KID}}{}, vanilla GPT3-2.7B outperforms its 1.3B fine-tuned counterpart, which is especially meaningful since ever-large foundation models~\citep{bommasani2021opportunities} are expensive to fine-tune effectively on common hardware settings~\citep{liu2021gpt}.

\subsection{Human Evaluation}
\input{tables/human_eval}

We recruited 300 MTurk participants to manually examine generated outputs of several decoding algorithm in terms of relevance, factuality, \redit{and fluency}. Each participant was asked to review five sample generations without revealing their source. We used paired samples $t$-tests to examine the difference between human references and other decoding methods generation. As Table~\ref{tab:human_eval} shows, {\mbox{KID}}{} generates similar quality sequences as human references without significant differences across all three tasks, while in ELI5, beam search and reflective decoding generation are rated significantly low in both relevance and factuality, partially due to exposure bias in longer generation. \redit{There is no significant difference in grammaticality between {\mbox{KID}}{} and vanilla decoding methods. The exact questions we asked participants can be found in $\S$\ref{subsec:human_eval_details} of Appendix.}

\section{Appendix}

\setcounter{table}{0}
\renewcommand{\thetable}{A\arabic{table}}%

\setcounter{figure}{0}
\renewcommand{\thefigure}{A\arabic{figure}}%

\redit{

\subsection{Datasets Statistics}
\label{subsec:statistics}

We chose eight knowledge intensive NLG tasks to evaluate {\mbox{KID}}{}. Here we present the dataset statistics of these tasks in Table~\ref{tab:datasets_stats}. The links to these datasets can be found in $\S$\ref{sec:experiment_settings}.

\begin{table}[!h]
\centering
\caption{The dataset statistics of the eight knowledge-intensive NLG tasks we evaluate for {\mbox{KID}}{}.}
\label{tab:datasets_stats}
\resizebox{0.9\textwidth}{!}{%
\begin{tabular}{@{}lcccccccc@{}}
\toprule
Split & \cellcolor{RBRed}ELI5    & \cellcolor{RBRed}MSMARCO & \cellcolor{RBOrange}PIQA   & \cellcolor{RBOrange}PubMedQA & \cellcolor{RBGreen}ROC    & \cellcolor{RBGreen}$\alpha$NLG   & \cellcolor{RBBlue}WoW    & \cellcolor{RBBlue}MuTual \\ \midrule
Train   & 272,764 & 153,725 & 16,115 & 800      & 52,665 & 50,481 & 63,734 & 7,088   \\
Dev     & 1,507   & 12,467  & 1,838  & 100      & 1,571  & 7,252  & 3,054  & 886    \\
Test    & 600     & 12,467  & 3,000  & 100      & 4,081  & 14,313 & 2,944  & 886    \\ \bottomrule
\end{tabular}%
}
\end{table}

\subsection{Details on Knowledge Trie Construction and Query}
\label{subsec:trie_construction}

\subsubsection{Knowledge Trie Construction}
\label{subsub:construction}

In this section, we detail the process of constructing the knowledge trie and how we query for the knowledge in a dynamic fashion. In Figure~\ref{fig:knowledge_trie} (a), for a given question (say from the ELI5 dataset), the DPR retriever~\citep{karpukhin-etal-2020-dense} would retrieve $k$ documents (the effect of choosing different $k$ on performance has been discussed in $\S$\ref{subsec:ablation}; we use $k=3$ here for simplicity) from 21M 100-token Wiki documents as the grounding passages for the question. We then use co-reference resolution to replace the pronouns with their referents (colored in red), normalize the text (e.g., removing links, lower-casing, etc.), and pass them through the OpenIE~\citep{stanovsky2018supervised} to obtain knowledge triplets (the use of OpenIE can be also seen in related prior work~\citep{distiawan2019neural,wu-etal-2019-open,fan2019using}). The end nodes of the extracted triplets (i.e., the \textit{subj} and \textit{obj}) serve as the key-value pairs when they are stored in the external knowledge trie ($G_{\textrm{ext}}$), and the relations between nodes are translated to the edges. We use the stems of the tokens as the keys in $G_{\textrm{ext}}$ (e.g., ``\textit{driving}'' $\rightarrow$ ``\textit{drive}''), in order to compress duplicated nodes in a concept-centric manner.

During knowledge trie construction, besides the one-step key-value pairs from the triplets, we also consider multiple-step relations by looking for key mentions in the values of previous keys (e.g., value: ``\textit{influence of \underline{drugs}}'' $\rightarrow$ next-hop key: ``\textit{\underline{drug}}''). Such iterative procedure will end when there are no more keys appearing in the values. Specifically, we use depth-first search (DFS) to record the maximum depth of the child branch for each key node, and also memorize the corresponding values (and its next-hop key) on the branch to facilitate further query.

\subsubsection{Dynamic Querying for Knowledge}
\label{subsub:query}

Since retrieving knowledge from millions of documents is slow (even with GPU acceleration), the knowledge trie described above is constructed off-line. We pre-compute and store this knowledge trie on the disk, and build the local knowledge memory on the fly. The local knowledge memory is simply a First-in-First-out (FIFO) list which continuously stores newly generated entities ($G_{\textrm{loc}}$; initialized with entities in the input question), whose length will be limited by $w_{\textrm{max}}$ (we set $w_{\textrm{max}} = h_{\textrm{max}}$ empirically, where $h_{\textrm{max}}$ is the max query hops). In Figure~\ref{fig:knowledge_trie} (b) we show a query example with the query word ``\textit{driving}'' in the local memory. The query word is firstly converted to its stem (i.e., ``\textit{drive}''), and then its demonstrations are retrieved with the key ``\textit{drive}''.

We show the whole generation loop of {\mbox{KID}}{} in Algorithm 2. At each step of the decoding, we first collect knowledge demonstrations by querying $G_{\textrm{ext}}$ with each entity in $G_{\textrm{loc}}$ as key. The collected demonstrations (a list of tokens) will then serve as target word space to guide current step generation (has been detailed in $\S$\ref{subsec:trie_guided}). The local knowledge memory is updated only when a new entity is generated. This is because the non-entity words (such as stop words, digits, etc.) a) are rarely the keys in the $G_{\textrm{ext}}$ (which are \textit{subj} in OpenIE triplets), and b) rarely lead to meaningful next-step constraints (it's hard to answer what's a proper continuation of ``\textit{the}'').

\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{images/appendix_trie.pdf}
\caption{\redit{A sample of {\mbox{KID}}{}'s retrieved documents for the ELI5 question ``\textit{Does marijuana impair driving ability?}'' and its corresponding knowledge trie. \textbf{(a)} The top three relevant documents retrieved by DPR. We annotate the triplets end nodes (\textit{subj} and \textit{obj}) picked by OpenIE for knowledge trie construction in \textcolor{DPLightGreen}{green}. \textbf{(b)} The partially observed knowledge trie when we query ``\textit{driving}'' in the current local knowledge memory (\textcolor{DPBlueD}{blue}) by $w_{\textrm{max}}$ hops. We perform co-reference resolution to replace pronouns with actual entities in the documents, and use the stems of the tokens as the query words and keys in the external knowledge trie. The retrieved demonstrations (values in the trie; \textcolor{DPPurple}{purple}) in multiple hops will then serve as guidance for current step decoding, after split into single tokens.}}
\label{fig:knowledge_trie}
\end{figure}

Since the external knowledge trie is constructed off-line, the query time mainly depends on the number of query words in the local knowledge memory (with max length $w_{\textrm{max}}$), and the maximum number of hops we query the external knowledge trie ($h_{\textrm{max}}$) with, which is approximately $w_{\textrm{max}} * h_{\textrm{max}}$ in total. We claim this is actually constant time complexity because $h_{\textrm{max}}$ and $w_{\textrm{max}}$ are fixed hyper-parameters (normally 1-5, as discussed $\S$\ref{subsec:ablation}) and it will not scale up with sequence length or the number of grounding documents retrieved.

\begin{algorithm}[!t]
  \KwIn{constructed knowledge trie $G_{\textrm{ext}}$, local knowledge memory $G_{\textrm{loc}}$, input context $q$, max query hop $h_{\textrm{max}}$, max sequence length $L$.}
  \KwOut{generated tokens $\{x_1, x_2, ..., x_t\}$.}

    \BlankLine
    $G_{\textrm{loc}} \leftarrow$ entities in $q$\;

    \BlankLine
    \While{\textup{current sequence length} $< L$}{

    \BlankLine
    Knowledge demonstrations $D \leftarrow$ []\;

    \BlankLine
    \For{\textup{each entity $e$ in $G_{\textrm{loc}}$}}{

    \BlankLine
    Query $G_{\textup{ext}}$ with $e$ by $h_{\textrm{max}}$ hops to collect demonstrations, and store in $D$.

    }

    \BlankLine
    Use $D$ to guide current step decoding by Algorithm 1, and generate token $x_t$.

    \BlankLine
    \If{$x_t$ \textup{is entity}}{
        Append $x_t$ to $G_{\textrm{loc}}$, and slim $G_{\textrm{loc}}$ by window size $w_{\textrm{max}}$.
    }
    }

\KwRet{$\{x_1, x_2, ..., x_t\}$}
\caption{The Generation Loop of {\mbox{KID}}{}}
\end{algorithm}

\subsection{More Comparison with Improved Decoding Algorithms}
\label{subsec:more_baselines}

Besides the baselines we compare in the paper, there are other inference-time decoding methods that focus on improving generation quality in different perspectives, such as diversity~\citep{baheti2018generating}, and attributes controlling~\citep{dathathri2019plug}, and entity appearance~\citep{mao2020constrained}\footnote{We cannot find its official implementation, and thus we do not include it in the following comparison.}. We also notice some methods that are training-time optimization, such as CTRL~\citep{keskarCTRL2019}, which requires re-training language models conditioned on a set of static control words, and fusion-in-decoder (FiD)~\citep{izacard2020leveraging,izacard2020distilling}, which concatenates additional retrieved grounding sentences with input text, and fine-tunes text2text language models (e.g., T5~\citep{roberts2020much}). {\mbox{KID}}{} differs from all these methods, since 1) {\mbox{KID}}{} aims to improve knowledge awareness during decoding, though we also find {\mbox{KID}}{} could mitigate exposure bias, and 2) {\mbox{KID}}{} can function as a standalone module requiring neither specific model architecture nor re-training or fine-tuning the language model (as shown in $\S$\ref{subsec:ablation}, the experiments with vanilla GPT3-2.7B). In the following part, we compare {\mbox{KID}}{} with all above methods except CTRL since it requires costly language model retraining on each new domain and how to pick control words for dynamic knowledge seems ambiguous. We also prepare a naive version of {\mbox{KID}}{} (Little {\mbox{KID}}{}) which uses plain list instead of graph structure (the knowledge trie) to store knowledge, where we directly use all the end nodes of extracted triplets from the documents as constraints without any dynamic query. We run experiments on QA tasks as FiD was trained on, and discuss their differences on performance, deployment, and efficiency (time/space).

\subsubsection{Performance on Knowledge Awareness}
\label{subsub:more_baseline_performance}

In Table~\ref{tab:more_baselines} we present the comparison results about performance on knowledge awareness. Not surprisingly, FiD is the strongest baseline among others, as it also explicitly leverages retrieved documents to ground its generation. Compared with {\mbox{KID}}{}, we find the performance of FiD is relatively limited when the NLG tasks requires longer and creative generation. For example, ELI5, whose average reference length is around 220 tokens, and FiD tends to generate generic and repeated sequence after about 80 tokens from our observation, potentially because though FiD fuses knowledge during training, there is no scheme to guarantee the knowledge parameterized in the model can be expressed during inference. Diverse Decoding, and PPLM show mediocre performance in these tasks because no knowledge-aware object was considered, but relatively well on longer generation tasks (e.g., ELI5), which seems to demonstrate the advantage of inference-time optimization methods (as {\mbox{KID}}{}). Little {\mbox{KID}}{} is not able to dynamically query knowledge when new context is generated, and thus performs not as good as {\mbox{KID}}{}, especially in longer generation (e.g., ELI5).

\begin{table}[!h]
\centering
\caption{\redit{Compare {\mbox{KID}}{} with additional baselines. Note that only FiD~\citep{izacard2020leveraging,izacard2020distilling} is explicitly optimized for knowledge awareness. Diverse Decoding~\citep{baheti2018generating} aims to improve generation diversity by constraining decoding distribution with topic and semantic similarity. PPLM~\citep{dathathri2019plug} uses a static set of tokens as global constraints to do conditional generation. {\mbox{KID}}{} differs from all these methods as its knowledge awareness and dynamic nature. Little {\mbox{KID}}{} is a naive implementation of {\mbox{KID}}{} that uses plain list instead of trie to store knowledge.}}
\label{tab:more_baselines}
\resizebox{0.76\textwidth}{!}{%
\begin{tabular}{@{}lcccccccc@{}}
\toprule
                    & \multicolumn{2}{c}{\cellcolor{RBRed}ELI5} & \multicolumn{2}{c}{\cellcolor{RBRed}MSMARCO} & \multicolumn{2}{c}{\cellcolor{RBOrange}PIQA} & \multicolumn{2}{c}{\cellcolor{RBOrange}PubMedQA} \\ \midrule
Method              & B-1         & R-L        & B-1          & R-L          & B-1         & R-L        & B-1          & R-L           \\ \midrule
FiD-T5 (base)       & 15.3         & 11.3       & 45.3         & 47.3         & 12.9        & 11.9       & 4.0          & 4.4           \\
FiD-T5 (large)      & \underline{15.7}         & \underline{18.1}       & \underline{51.9}         & \underline{53.1}         & \underline{14.8}        & \underline{17.4}       & 4.0          & 4.5           \\
Diverse Decoding & 4.6         & 15.7       & 27.5         & 22.9         & 11.3        & 15.5       & 3.5          & 3.8           \\
PPLM                & 11.9        & 15.0       & 23.4         & 26.6         & 10.3        & 11.9       & 3.8          & 4.2           \\
Little {\mbox{KID}}{} (\textit{ref.})                & 12.1        & 14.3       & 40.2         & 46.3         & 12.2        & 13.5       & \underline{5.2}          & \underline{6.6}           \\
\textbf{Ours:} KID$_{\textrm{GPT2}}$           & \textbf{27.9}        & \textbf{26.6}       & \textbf{47.4}         & \textbf{53.9}         & \textbf{17.7}        & \textbf{18.4}       & \textbf{9.7}          & \textbf{11.5}          \\ \bottomrule
\end{tabular}%
}
\end{table}

\subsubsection{Comparison of Time and Space Consumption}
\label{subsub:time_space}

\redit{
The time and space cost of running {\mbox{KID}}{} could be a matter of concern especially when we consider real-world applications. In Table~\ref{tab:time}, we compare the time complexity of {\mbox{KID}}{} with other baselines, including vanilla beam search and sampling decoding. We provide a breakdown for different stages: retrieving knowledge (\textbf{retrieving}), training the generative model (\textbf{training}), and the actual generation (\textbf{inference}). Beam search and sampling decoding only have noticeable cost when there is sorting process filtering the candidate tokens (common in beam search and top-$p$ or top-$k$ sampling), which will often bring $O(k\log k)$ cost ($O(1)$ when pure random sampling without any sorting). Diverse Decoding, PPLM, and {\mbox{KID}}{} are similar as all are inference-time optimization (not necessarily need model fine-tuning, denoted as ``LM FT / --''), but {\mbox{KID}}{} requires additional off-line knowledge trie construction (nearly the same time cost as RAG and FiD). During inference, {\mbox{KID}}{} takes constant time for query (as discussed in $\S$\ref{subsub:query}), and several iterations policy gradient (by constant number of steps) to guide the generation. We also analyze the space cost of {\mbox{KID}}{} and other baselines (Table~\ref{tab:space}). The difference between {\mbox{KID}}{} and RAG / FiD is the external and local knowledge memory, which is relatively small (normally < 10Mb) from our observation. Other decoding methods do not have the ability to infuse knowledge, and thus do not bring additional space cost.

}
\begingroup
\setlength{\tabcolsep}{3pt}
\begin{table}[!h]
\centering
\caption{\redit{Time complexity analysis of {\mbox{KID}}{} and other baselines. We study the time consumption during knowledge retrieving, model training, and inference (i.e., generation). {\mbox{KID}}{} has comparable time efficiency as RAG and FiD, but outperforms them in knowledge infusing (discussed in the $\S$\ref{subsec:benchmark}). Note that {\mbox{KID}}{} can function as a standalone module operating only at inference time, rather than rely on a specific model fine-tuning or pre-training together (such as RAG and FiD).}}
\label{tab:time}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Method / Time} & \textbf{Retrieving}        & \textbf{Training}   & \textbf{Inference}                           \\ \midrule
Beam Search            & --                 & --          & $O(k\log k)$, $k$ = $\#$ of beams                            \\
Sampling               & --                 & --          & $O(1)$ or $O(k\log k)$, $k$ = top-$k$                     \\
RAG                    & $O(\# \textrm{docs})$, $\# \textrm{docs} \approx$ 10      & LM FT     & $\ge O(1)$                                \\
FiD-T5 (base)          & $O(\# \textrm{docs})$, $\# \textrm{docs} \approx$ 100     & LM FT     & $\ge O(1)$                                \\
FiD-T5 (large)         & $O(\# \textrm{docs})$, $\# \textrm{docs} \approx$ 100     & LM FT     & $\ge O(1)$                                \\
Diverse Decoding    & --                 & LM FT / -- & Two Neural Models Prediction + $O(1)$ \\
PPLM                   & --                 & LM FT / -- & $O(\# \textrm{control words} * \textrm{step})$                         \\
\textbf{Ours:} KID              & $O(\# \textrm{docs})$ + DFS, $\# \textrm{docs} \approx$ 5 & LM FT / -- & $O(h_{\textrm{max}}^2) \approx O(1)$                            \\ \bottomrule
\end{tabular}%
}
\end{table}
\endgroup

\begingroup
\setlength{\tabcolsep}{3pt}
\begin{table}[!h]
\centering
\caption{\redit{Memory footprint of {\mbox{KID}}{} and other baselines. Similar to RAG and FiD, {\mbox{KID}}{} requires pre-store the grounding documents on disk (and corresponding dense vectors). In addition, {\mbox{KID}}{} builds a knowledge trie off-line (less than 10Mb for all the tasks we studied), and a local knowledge memory (a limited length FIFO list), to enable knowledge infusion during generation.}}
\label{tab:space}
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{lcc}
\hline
\textbf{Method / Space} & \textbf{External Resources} / \textbf{Model}            & \textbf{Knowledge?} \\ \hline
Beam Search             & --                                     & \XSolidBrush         \\
Sampling                & --                                     & \XSolidBrush         \\
RAG                     & 13.4G Docs + 64.6G Index / BART       & \Checkmark        \\
FiD-T5 (base)           & 13.4G Docs + 64.6G Index / T5       & \Checkmark        \\
FiD-T5 (large)          & 13.4G Docs + 64.6G Index / T5       & \Checkmark        \\
Diverse Decoding     & Two Models 2.8G in Total / LM         & \XSolidBrush         \\
PPLM                    & BoW or Classifier / LM                & \XSolidBrush         \\
\textbf{Ours:} KID               & 13.4G Docs + 64.6G Index + $G_{\textrm{loc}}$ + $G_{\textrm{ext}}$ / LM & \Checkmark        \\ \hline
\end{tabular}%
}
\end{table}
\endgroup

\subsection{Sample Generations}

We list several generation samples of {\mbox{KID}}{} and RAG. In general, the generated texts of both methods are highly readable; however, we find RAG's generation tends to off-topic and not coherent with current on-going concepts flow when the generation is long (e.g., ELI5, ``\textit{smoking inside the vehicle}''), or the context is multi-fold and complicated (e.g., multi-turn dialogue WoW, ``\textit{do some exercise}'')---these are legit continuation if we only consider immediate context. {\mbox{KID}}{} is superior especially in longer generation, which seems to echo our findings that {\mbox{KID}}{} can mitigate exposure bias.

\begin{table}[!h]
\centering
\caption{Sample generation of {\mbox{KID}}{} and RAG on ELI5 (QA), ROC (Logic-centric Writing), and WoW (Dialogue) with fine-tuned language models.}
\label{tab:sample_generation_kid}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lllllllll@{}}
\toprule
\multicolumn{1}{c}{} & \multicolumn{8}{c}{ELI5}                                                                                \\ \midrule
Question             & \multicolumn{8}{l}{Does marijuana impair driving ability?}                                                                                \\ \midrule
RAG                  & \multicolumn{8}{l}{\begin{tabular}[c]{@{}l@{}}Marijuana does impair your ability to drive, but you can avoid this \\ by not being asleep together in class together and using a napkin for balance.. \\ One way to avoid discomfort would be smoking inside the vehicle. \\ That being said, smoking in public can do the same thing as smoking outside.\end{tabular}} \\ \midrule
KID                  & \multicolumn{8}{l}{\begin{tabular}[c]{@{}l@{}}Marijuana does impair your ability to drive because cannabis is a depressant drug,\\ which means it slows reaction times and information processing.\\ A person who has been using cannabis is more likely to cause a vehicle collision, \\ but may still think that they can drive safely.\end{tabular}}               \\ \midrule
                     & \multicolumn{8}{c}{ROC}                                                                                \\ \midrule
Context              & \multicolumn{8}{l}{\begin{tabular}[c]{@{}l@{}}Keith lived in an area that had occasional earthquakes.\\ But he thought keeping an first aid kit was unnecessary.\\ The next day, he cut his hand while cutting food.\\ And he had to drive himself to a hospital.\end{tabular}}                                                                                \\ \midrule
RAG                  & \multicolumn{8}{l}{Keith quickly became worried and sad.}                                                                                \\ \midrule
KID                  & \multicolumn{8}{l}{Keith eventually learned first Aid at a clinic instead of building his own kit.}                                                                                \\ \midrule
                     & \multicolumn{8}{c}{WoW}                                                                                \\ \midrule
Utterance              & \multicolumn{8}{l}{\begin{tabular}[c]{@{}l@{}}f : would you like some tea or coffee ?\\ m : no thank you . it 's very late now , that will keep me awake the whole night . \\ f : then what about some water ? \\ m : yes , please . \\ f : do n't work too late since you were not in good health .\\ you should be careful with your health .\end{tabular}}          \\ \midrule
RAG                  & \multicolumn{8}{l}{m : thank you. i would sleep earlier and do some exercise in the morning}                                                                                \\ \midrule
KID                  & \multicolumn{8}{l}{m : thank you for your concern. could i have some water please .}                                                                                \\ \bottomrule
\end{tabular}%
}
\end{table}

We show some generation samples of {\mbox{KID}}{} and RAG in Table~\ref{tab:sample_generation_kid}.

\subsection{Actual Questions we Ask Human Annotators}
\label{subsec:human_eval_details}

We asked participants about: 1) \textbf{Relevance} (i.e., \textit{``What's the relevance level between the generated text and the given context?''} Answer is from 1-\textit{totally not relevant} to 7-\textit{totally relevant}), 2) \textbf{Factuality} (i.e., \textit{``What's the factuality level of the generated text?''} Answer is from 1-\textit{totally not factual} to 7-\textit{very well factual}), and 3) \textbf{Grammaticality} (i.e., \textit{``How much the text is similar to human-generated text?''} Answer is from 1-\textit{not similar at all} to 7-\textit{very much similar}).

}

\section{Knowledge Infused Decoding}
\label{sec:approach}

We detail the implementation of {\mbox{KID}}{} in this section. As shown in Figure~\ref{fig:overview}, {\mbox{KID}}{} comprises of three steps: retrieving relevant knowledge ($\S$\ref{retrieving_kg}), constructing external and local knowledge memory ($\S$\ref{subsec:building_kg}), and guiding current step decoding under the constraint of the knowledge trie ($\S$\ref{subsec:trie_guided}).

\subsection{Retrieving Extensive Knowledge from Wikipedia}
\label{retrieving_kg}
The first step of {\mbox{KID}}{} is to retrieve several context-relevant documents to ground the following generation. We use DPR~\citep{karpukhin-etal-2020-dense} as our general-purpose knowledge retriever, which projects contexts and relevant documents to a 768-dim shared embedding space using a bi-encoder network (i.e., two independent BERTs~\citep{devlin2018bert}). Here, the documents refer to the 100-word chunks of Wikipedia passages released by RAG~\citep{lewis2020retrieval}, a total of 21M documents as our knowledge source ${\mathbb{Z}}$. We pre-load the weights from the latest checkpoint of DPR (March 2021), as it improves retrieval performance by using mined negative samples and contrastive learning, which is also suggested by \citet{jernite2020explain}. During retrieval, we perform a maximum inner-product search (MIPS) with faiss\footnote{The faiss project can be found here: \url{https://github.com/facebookresearch/faiss}.} accelerated by GPU~\citep{johnson2019billion}. Formally, we retrieve $k$ most relevant document $z_{[1,...,k]} \in {\mathbb{Z}}$ for context $x_{\textrm{context}}$ as:
\begin{equation}
        z_{[1,...,k]} = \left\{z_i\in{\mathbb{Z}}|\mathrm{topk}\, \{\textrm{BERT}(x_{\textrm{context}})^\top \cdot \textrm{BERT}(z_i)\} \right\}
\end{equation}

\noindent where $\textrm{BERT}(\cdot)$ means the vectors encoded by BERT. The number of retrieved documents $k$ is a task-specific hyper-parameter---we discuss its impact on performance in $\S$\ref{subsec:hyperparameter}.

\subsection{Constructing External and Local Knowledge Memories}
\label{subsec:building_kg}

We convert multi-document input retrieved from previous step into compressed knowledge memories, in order to 1) allow relevant knowledge to be easily identified, 2) reduce the memory footprint of the knowledge, whose size grows linearly with the number of retrieved documents.

Following design choice of previous successful methods~\citep{BosselutBC21,huang2020knowledge,fan2019using}, we adopt co-reference resolution and open information extraction (OpenIE)~\citep{stanovsky2018supervised} to convert plain text into triplet form\footnote{As an empirical trick we also remove the stop words in the documents as they seldom carry knowledge.}. For example, knowledge statement like \textit{``Iceland is a Nordic island country in the North Atlantic Ocean and it is the most sparsely populated country in Europe.''} will be converted to \textbf{subject-relation-object} triplets such as $\langle$\textit{\textbf{subj}:Iceland}, \textit{\textbf{rel}:is}, \textit{\textbf{obj}:Nordic island country}$\rangle$, $\langle$\textit{\textbf{subj}:Iceland}, \textit{\textbf{rel}:is}, \textit{\textbf{obj}:most sparsely populated country in Europe}$\rangle$, etc. To account for overlapping elements from these triplets, we use a prefix tree (namely Knowledge Trie $G_{\textrm{ext}}$) to store and organize the extracted triplets, by setting the subject in each triplet as the key in the $G_{\textrm{ext}}$. Note that unlike common character-level dictionary tries~\citep{chen2020parallel,germann2009tightly}, in $G_{\textrm{ext}}$ each triplet is stored in token unit as our goal is to efficiently query and traverse the knowledge triplets stored in it.

A tree structure encoding knowledge is appealing to knowledge intensive NLG tasks, since 1) the \textit{non-cyclic} structure helps reduce repetitions in generations, and 2) querying a prefix tree can be efficiently completed in constant time ($O(|x_{\textrm{context}}|)$) which does not involve any costly traversal on the graph~\citep{jilanguage,zhang2019grounded}, regardless of the scale of grounding knowledge (normally $|x_{\textrm{context}}| \ll |k\ \textrm{Wiki docs}|$). \redit{We also maintain a local memory $G_{\textrm{loc}}$ (a first-in-first-out list) that records all the mentioned entities in current context, to focus on concept-centric knowledge~\citep{zhou2020pre,lin2019commongen}. More details about how we construct and query these knowledge memories can be found in $\S$\ref{subsec:trie_construction} of Appendix.}

\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{images/overview_fin.pdf}
\caption{Overview of our {\mbox{KID}}{} decoding algorithm. For a given context $x_{\textrm{context}}$, we first retrieve $k$ most relevant Wikipedia documents $z_{[1,...,k]}$ with a knowledge retriever (Step 1), and then convert them into compressed knowledge trie $G_{\textrm{ext}}$ (Step 2). Meanwhile, the local memory $G_{\textrm{loc}}$ which is a first-in-first-out list will keep track of entities in current context, and in the final step (Step 3), it will continuously query the knowledge trie with max number of hops $h_\textrm{max}$. Current step LM decoding will be guided by the query results with policy gradient, to generate new token $y_i$.}
\label{fig:overview}
\end{figure}

\subsection{Knowledge Trie Constrained Decoding via Policy Gradient}
\label{subsec:trie_guided}

\paragraph{Background.} Current generative LMs are trained to maximize the probability of generating ground-truth tokens at each decoding step. Assuming ${\bm{y}}^{*}_{1:T} = \{y_1^{*}, y_2^{*}, ..., y_T^{*}\}$ is the ground-truth output sequence for a given context $x_{\textrm{context}}$, the MLE objective minimizes the following loss:
\begin{equation}
\label{eqa:mle}
    J_{\textrm{MLE}} = - \sum_{i=1}^T \log p(y_t^{*} | y_1^{*}, ..., y_{t-1}^{*}, x_{\textrm{context}})\ .
\end{equation}

In knowledge-intensive NLG tasks, however, it is reported that the MLE training goal does not explicitly model knowledge, and thus the LM often produces counterfactual generation by surface-level misguidance~\citep{zhou2020pre,petroni2019language}. Furthermore, the teacher-forcing algorithm used by MLE training leads to the exposure bias problem~\citep{ranzato2015sequence}, as the LM has access to ground truth sequence up to the next token during training, but does not have such signal during testing, which causes accumulated errors when generating longer sequence~\citep{paulus2017deep}. Both problems heavily limit the performance of popular LMs on diverse knowledge-intensive NLG tasks. One remedy is to learn a generation \textit{policy} that not only maximizes knowledge correctness but also alleviates exposure bias in longer generation, which can be made possible with policy gradient in reinforcement learning (RL).

\paragraph{Knowledge Trie Constrained Decoding.} To formulate NLG as a RL problem, we define the \textit{state} as the generated tokens before $t$ (i.e., $s_t = y_{<t}$), and the \textit{action} as the current step output token (i.e., $a_t = y_{t}$). The softmax output of the language modeling head, i.e., a categorical distribution $p_t$ over the entire vocabulary, is considered as the policy $\pi_t$ for picking token $y_t$ (action $a_t$) given the state $s_t = y_{<t}$~\citep{guo2021text,liu2020data}. Note that $p_t$ (i.e., $\pi_t$) could be from either a pre-trained LM or a LM fine-tuned with task data. While conventional sampling decoding and beam search pick next token directly from $p_t$, here we look to inject knowledge to adjust $p_t$ to guide decoding.

For current input context $x_{\textrm{context}}$ at step $t$, we first take its concept mentions in local memory $\{c_1, c_2, ..., c_m\} \subset G_{\textrm{loc}}$, and then query the knowledge trie $G_{\textrm{ext}}$ with these concepts by max hops of $h_{\textrm{max}}$. The union of queried results of all hops ${\mathbb{V}}_i = \{v_1^i, v_2^i, ..., v_n^i\}$ for $i=1,..,h_{\textrm{max}}$ then serve as knowledge demonstrations for current step generation. To put probability mass on tokens  aligning with these demonstrations, we compute the $t$-th step knowledge gain $r_t$ as the total log probability  of all retrieved tokens from $G_{\textrm{ext}}$ under decoding policy $\pi_t$ as $ r_t = \sum_{i = 1}^{h_{\textrm{max}}} \sum_{v \in {\mathbb{V}}_i} \mathbb{I}[v]^\top \cdot \log \pi_t$, where $\mathbb{I}[\cdot]$ is the indicator function that will output an one-hot vector with a 1 in the coordinate of token $v$ in the vocabulary and 0's elsewhere. With $r_t$, we define the $t$-th step reward $J_{\textrm{RL},t}$ on trajectories $\tau$ induced by $\pi_t$ as:
\begin{equation}
\label{eqa:total_reward}
   J_{\textrm{RL}, t} = \mathbb{E}_{\tau \sim \pi_t} \left( \frac{\pi^*_t(a_t | s_t)}{\pi_t(a_t | s_t)} \cdot r_t \right) - \beta \textrm{KL}(\pi_t|| \pi^*_t)\ ,
\end{equation}

\noindent where $\pi^*_t$ is the desired policy (a vector initialized from $\pi_t$) to produce tokens approaching knowledge demonstrations, and the KL penalty term $\textrm{KL}(\pi_t || \pi^*_t)$ is to avoid the updated policy drifts too much away from the original one, also known as \textit{trust region} constraint~\citep{schulman2017proximal,schulman2015trust}. Note that we use off-policy sampling to collect trajectories, with an importance weight $\pi^*_t/\pi_t$ to calibrate the knowledge gain $r_t$, which can stabilize the optimization~\citep{munos2016safe}.

\begin{minipage}[t]{0.55\textwidth}
\vspace{-0.1in}
\begin{algorithm}[H]
\label{alg}
\SetAlgoLined
\SetNoFillComment
 \For{$t=1,2,\ldots$}{
  Collect samples $(a_t|s_t)$ by vanilla policy $\pi_t$\;

  Compute reward $J_{\textrm{RL},t}$ by Eq. (\ref{eqa:total_reward})\;

  Compute updated policy
      $\pi^*_t \leftarrow \argmax_{\pi^*_t} J_{\textrm{RL},t}$
  by taking $K$ steps of SGD (via Adam)\;

  \uIf{\textup{KL}$(\pi_t||\pi^*_t) \ge 2\sigma$}{
  $\beta_{t+1} = 2\beta_t$\;
  }
  \ElseIf{\textup{KL}$(\pi_t||\pi^*_t) \le \sigma / 2$}{
  $\beta_{t+1} = \beta_t / 2$\;
  }
  Generate token $y_t$ with updated policy $\pi^*_t$;
 }
 \caption{Trie-Constrained Policy Gradient}
\end{algorithm}
\end{minipage}
\hfill
\begin{minipage}[t]{0.4\linewidth}
Algorithm ~\ref{alg} shows how we obtain the updated policy through policy gradient. We set $\beta$ dynamically to control the KL penalty within the reward function. The target divergence $\sigma$ tunes the strength of knowledge infusion---smaller $\sigma$ means less infusion while larger $\sigma$ provides more space for policy gradient (We set $\sigma$ to 0.02 across all tasks). To generate the actual token for step $t$, we pick from the updated policy $\pi^*_t$ with sampling decoding. The token should conform to the knowledge demonstrations, since its corresponding hidden states have shifted towards ${\mathbb{V}}$ due to policy gradient. We empirically choose $K=3$ for good performance in most cases.
\end{minipage}

Prior approaches have also explored to set sequence-level metrics such as BLEU~\citep{papineni-etal-2002-bleu} as the reward to directly optimize the generation quality~\citep{li2016deep,paulus2017deep}. However, many studies report such sparse reward will cause low efficiency optimization (i.e., $J_{t}=0$, $\forall t < T$)~\citep{guo2021text}. Our trie-constrained policy gradient method seemingly mitigates this problem by using an immediate reward ($J_{\textrm{RL}}^{\theta^{*}}$) at each step with reasonable approximation. Recent off-line RL for NLG work show promising results when using data itself (rather than metrics) to estimate rewards~\citep{pang2020text,jaques2020human}---our design of knowledge trie echoes their findings. The gold data distribution memorized in trie is treated as guidance to reshape each step probability distribution, and thus brings benefits on alleviating exposure bias during long generation.

\section{Conclusions and Future Work}
\label{sec:conclusion}

In this work, we proposed {\mbox{KID}}, a novel decoding algorithm which in each step of decoding dynamically fetches knowledge and guides token decoding via interaction between a local knowledge memory and a constructed external knowledge trie. Given its decoding nature, {\mbox{KID}}{} doesn't require architecture changes to existing LMs and is applicable to various generative LMs. Evaluations on eight knowledge-intensive NLG tasks demonstrated the effectiveness of {\mbox{KID}}{} and that its generations are aligned with human references well. In addition, {\mbox{KID}}{} is also found to alleviate exposure bias and maintain quality for long generations.

Future work of {\mbox{KID}}{} could study more efficient data structures to accelerate knowledge fetching and integration. One could also investigate combining {\mbox{KID}}{} with prompt based generations to further boost the performance, especially in the few-shot settings. Another direction is to further study the integration of {\mbox{KID}}{} with full size foundation models, like GPT-3 175B to understand {\mbox{KID}}'s potential.
\section*{Ethics and Reproducibility Statement}
\label{sec:ethic}

The goal of {\mbox{KID}}{} is to provide a general-purpose knowledge infusing decoding algorithm by leveraging retrieved Wikipedia documents. Still, the generation of {\mbox{KID}}{} can be affected by certain biases from the LM it is based on though those biases may be partially mitigated by the knowledge demonstrations~\citep{LIU2022103654,rae2021scaling}. Another major ethical consideration is that {\mbox{KID}}{} can mimic undesirable attributes of the target knowledge source documents that could be non-contemporary and do not represent current norms and practices---and {\mbox{KID}}{} has no scheme to diagnose these problems~\citep{lewis2020retrieval}. Furthermore, our experiments and analysis are done in English, and therefore we do not claim that our findings will generalize across all languages, although our framework has potential to be extended to other languages with necessary modifications.

For reproducibility, evaluations of {\mbox{KID}}{} and baseline methods are all conducted on public NLG data sets. We compare results from published papers and public leaderboards. Code and scripts for reproducing {\mbox{KID}}{} is available on GitHub at \url{https://github.com/microsoft/KID}.
\section{Experiments}

\subsection{Experiments Settings}
\label{sec:experiment_settings}

We consider three diverse types of knowledge-intensive tasks for evaluation (statistics see $\S$\ref{subsec:statistics}):

\textbf{Abstractive Question Answering.} We study Abstractive QA, which requires the model to \textit{generate} free-form answers to the questions. We choose long-form QA task \dpboxred{ELI5}~\citep{fan2019eli5} and \dpboxred{MSMARCO} NLG task v2.1~\citep{nguyen2016ms} as two commonsense QA tasks whose questions can be mostly answered by referring to Wikipedia passages. We also use two extra QA tasks \dpboxorange{PIQA}~\citep{bisk2020piqa} and \dpboxorange{PubMedQA}~\citep{jin2019pubmedqa} whose questions require domain-specific knowledge to answer (i.e., physical interaction knowledge for PIQA, and medical knowledge for PubMedQA). We calculate BLEU-1 and ROUGE-L scores to be able to compare directly with related work~\citep{lewis2020retrieval,krishna2021hurdles}.

\textbf{Logic-centric Writing.} We also investigate whether {\mbox{KID}}{} can benefit NLG tasks that do not have an explicit query form for certain knowledge (i.e., with specific questions, like QA tasks). We study \dpboxgreen{ROC} story ending generation~\citep{mostafazadeh2016corpus}, which requires generating a legitimate ending given a four-sentence context, and \dpboxgreen{$\alpha$NLG}~\citep{bhagavatula2019abductive} which requires generating reasonable hypothesis given two observations. We follow related work~\citep{zhou2020pre,guan2020knowledge} in using BLEU-1 and ROUGE-L as evaluation metrics.

\textbf{Dialogue Generation.} Chitchat dialogues are normally multi-turn discussions over a variety of topics or concepts, which often involve topical and factual knowledge~\citep{petroni-etal-2021-kilt}. We study two dialogue datasets that require knowledge grounding: \dpboxblue{Wizard of Wikipedia (WoW)}~\citep{dinan2018wizard}, where the speaker in the conversation must ground their utterances in Wikipedia passages, and \dpboxblue{MuTual}~\citep{cui-etal-2020-mutual}, where utterances have to be a logically-coherent continuation of the given multi-turn context. We follow existing work and the leaderboard of WoW in using F-1/ROUGE-L for WoW and MRR/ROUGE-L for MuTual evaluations.

\section{Introduction}

Pre-trained language models (LMs) have been shown to capture rich semantic and syntactic features, as demonstrated by the state-of-the-art performance on many generation tasks such as abstractive summarization~\citep{zhang2020pegasus,chu2019meansum} and dialogue generation~\citep{roller2020recipes,zhang2019dialogpt}. Their generations, however, can be quite limited in scenarios requiring \textit{knowledge}---they frequently struggle with issues such as being easily misguided by phrase co-occurrence (e.g., \textit{``Talk? Birds can \underline{talk}.''}), failing to handle negation (e.g., \textit{``The theory of relativity was \textbf{not} developed by \underline{Einstein}.''})~\citep{kassner2019negated}, and being unable to compare common-sense concepts, such as time~\citep{qin-etal-2021-timedial} and digits~\citep{talmor2020olmpics}.

To enhance the performance of LMs on knowledge-intensive NLG tasks\footnote{We define knowledge-intensive NLG tasks as those whose input context alone does not provide complete knowledge for a legitimate and plausible generation.}, prior studies have proposed to re-train LMs with knowledge-aware objectives~\citep{zhou2020pre,xiong2019pretrained,zhang2019ernie,khandelwal2019generalization} or add special architectures to encode knowledge~\citep{bosselut2019comet,logan2019barack,peters2019knowledge} from external resources (e.g., knowledge graphs such as \textsc{ConceptNet}~\citep{speer2017conceptnet} and \textsc{Atomic}~\citep{sap2019atomic}). These methods, though yielding impressive results on many downstream tasks, can be computationally expensive. More importantly, knowledge \textit{implicitly} parameterized in LM architectures is difficult to revise and expand~\citep{lewis2020retrieval}, and wrong generations are hard to diagnose due to lack of interpretation~\citep{talmor2020olmpics}, which heavily limits their real-world applications.

More recent retrieval-based models try to tackle these problems by augmenting inputs with retrieved knowledge evidence~\citep{lee2019latent,guu2020retrieval}. For example, RAG~\citep{lewis2020retrieval} leverages non-parametric memory to access extensive knowledge (in the form of unstructured documents), and jointly fine-tunes a parametric LM (i.e., BART~\citep{lewis2019bart}) to enable knowledge-aware generation. A key limitation of such methods is that they retrieve documents only once while grounding them in the input \textit{static} context, and thus cannot support the \textit{dynamic} nature of the context as new tokens are generated. The static knowledge becomes a major problem for tasks where longer and abstractive generation is expected, such as open-ended story generation~\citep{mostafazadeh2016corpus}, multi-turn dialogues~\citep{zhao2020knowledge}, and conversation summarization~\citep{gliwa-etal-2019-samsum}. Moreover, in a recent study, \citet{krishna2021hurdles} replaced the knowledge retriever in RAG with a random retriever and found little difference in the resulting performance on a long-form QA task named ELI5~\citep{fan2019eli5}, indicating the model may not be actually grounding its text generation to the retrieved documents.

To address these limitations, in this work, we present a novel decoding algorithm {\mbox{KID}}{}, aiming to better infuse knowledge into generation in a dynamic manner. Instead of solely relying on the \textit{static} knowledge retrieved at beginning, during each step of LM decoding, {\mbox{KID}}{} \textit{dynamically} searches promising continuation from retrieved knowledge, to guide the current step generation. Specifically, {\mbox{KID}}{} maintains a local knowledge memory, interacts it with a knowledge trie dynamically created from retrieved supporting documents, and updates the local memory as a knowledge-aware constraint to guide the generation. The key intuition behind {\mbox{KID}}{} is that existing LM pre-training objectives are usually defined at the token level yet do not explicitly model concept-centric knowledge~\citep{xiong2019pretrained} --- thus motivating us to reshape the probability mass at each step decoding towards the distribution of entities in knowledge.

The contribution of this work is three-fold: \textit{First}, we introduce {\mbox{KID}}{} as a model and task agnostic decoding method that integrates knowledge on the fly and can be applied to various knowledge-intensive tasks with different generative LMs. \textit{Second}, from a docoding perspective, on six knowledge-intensive NLG tasks, GPT2~\citep{radford2019language} and BART~\citep{lewis2019bart} equipped with {\mbox{KID}}{} significantly outperform conventional beam search or sampling decoding by a large margin. \textit{Third}, from a knowledge infusion perspective, unlike seven strong knowledge-infusion baselines which require either additional retraining or special architecture modifications, {\mbox{KID}}{}  leverages knowledge more effectively as a light-weight knowledge infusion solution. Additionally, in few-shot scenarios {\mbox{KID}}{} significantly improves over them, demonstrating its generalization ability in low-resource and domain shifting regimes.
\section{Limitations}
\label{sec:limitation}

\section{Related Work}
\label{sec:related}

We briefly review existing work enhancing LMs with external knowledge and representative decoding algorithms for generation.

\textbf{Enhancing Language Model with Knowledge.} Large language models implicitly encode knowledge in their parameters but with limits~\citep{petroni2019language,kassner2019negated,lin2020birds}. Several architectures and objective functions have been proposed to explicitly encode external knowledge~\citep{sun2020ernie,logan2019barack,roberts2020much,levine2019sensebert}, or to augment LM pre-training data with retrieved knowledge~\citep{lewis2020retrieval,guu2020retrieval,lee2019latent}. However, \citet{talmor2020olmpics} notes that the reasoning ability of such LMs is strongly tied to the context seen during pre-training and is thus hard to generalize to new domains. Built on LMs, {\mbox{KID}}{} incorporates extra knowledge from external resources (Wikipedia) and thus shows strong performance in knowledge-intensive NLG tasks.

\textbf{Better Decoding Algorithm.} Two common strategies dominate the decoding algorithms used by most generative models: beam search which maximizes likelihood in a local horizon (due to finite beam size), and sampling decoding (e.g., top-$k$ sampling~\citep{fan2018hierarchical,holtzman2018learning}) which relies on randomness. \citet{holtzman2019curious} find beam search often produces generic and repetitive generation, while top-$k$ sampling tends to pick unlikely tokens which creates incoherent and unrelated sequences. Existing attempts to mitigate these problems include reranking~\citep{adiwardana2020towards,shao2017generating}, adding control signals~\citep{zhang2018personalizing,xing2017topic}, and self-adaptive truncation~\citep{welleck2020consistency,peters2019sparse}. None of these decoding algorithms consider integrating knowledge in the generation process. Reflective decoding~\citep{west-etal-2021-reflective} and DeLorean~\citep{qin2020back} are two recent decoding algorithms that focus on abductive commonsense reasoning. Reflective decoding in particular has the potential to be extended to other knowledge-intensive tasks. We compare it with {\mbox{KID}}{} in our experiments.

redpajama_set_name: RedPajamaArXiv


Hello, Dubai!: Mosquitoes Ain't Got Nothing On Us!
You have to hand it to the pharmacies in Dubai. They are pretty much totally deregulated and ready for anything. Matt dropped by our local store just the other day to stock up on some travel essentials for the trip that we will be taking over Eid al-Adha (the "Festival of Sacrifice" is a religious festival celebrated in commemoration of the willingness of Ibrahim to sacrifice his son as an act of obedience to God. It happens approximately 70 days after Ramadan).
We are lucky enough to have three days off, and have decided to plan a trip to see some of the surrounding areas. For sensitivity reasons, we aren't going to disclose our location until we get there, but let's see if you can guess!
Matt returned from the pharmacy with Advil, Excedrin, Benadryl, Imodium, and two other interesting choices 'TravelGuard' capsules which contain major doses of the 'good' digestive bacterial, you know the stuff found in yogurt, which is supposed to toughen up our stomachs before we go. Oh, and Mephaquin, maybe you've heard of it? No? It's your basic anti malarial tablet. Apparently we weren't aware of the malaria threat at our desired location and as a result we are on a crash course of anti malarial drugs.
The most interesting part? In order to make sure that the drug has the desired prophylactic effect, we need to keep taking it for 4 weeks after our trip! I'll be fighting malaria until after Christmas! Hope it's all worth it! Can you guess where we are going?
I've got it - Fort Lauderdale!


I went on an outing with my children.
Our main intention was to swim and for the children to have their fair share of recreation and of my quality time. It was also a chance for me as a father, to have my rightful share of the joy of being with my children, for truly we need our children as much as they need us, if not more.
I found myself busy on that day with all sorts of little tasks – like making sandwiches, setting the picnic table, and handing out sweets. It was a beautiful day to harvest some of the fruits of happiness just by enjoying ordinary activities in total relaxation and familiarity.
That day made me think about how much we, as people, speak about happiness. Hundreds of books are published every year to address the issue from a dry, philosophical standpoint. These works strive to define happiness and reveal its connection with factors like prosperity, pleasure, and our state of mind.
There is considerable controversy about what brings about happiness and, more essentially, what it actually means to be happy. This leads to the more mundane questions of role that health, wealth, one's job, one's marriage, and being successful play in our chances for happiness.
We might fail to see that happiness is an inner state of our being, which comes into its being within ourselves, and is often connected with the most ordinary and seemingly insignificant events of our lives. It is the normal state of a person's mind when that person is enjoying an experience or an activity.
Those of us who disdain engaging in some pleasant activities, or simply fail to admit our enjoyment of them – due to our preconceived notions of what is suitable for us as adults, or as elderly people, or as people of social prominence – need to rethink some of our ideas. We should not rob ourselves of life's small but significant pleasures.
Be like a child who looks forward excitedly to taking a ride. Look forward to pleasant things. Laugh at a funny joke without first examining it to deduce whether the humor of the joke holds up under close scrutiny. Look forward to your sleep. Recognize it as being Allah's blessing and a refreshment for your body and mind. Maybe you will dream about those you love.
Try to see the beauty of a flower in bloom, the fields of crops, a flowing stream, the sparrow's song, and a child in play. Do not block these things from your sight, thinking them to be unimportant. Indeed, our sense of importance and of what carries weight with us – this is one of the greatest obstacles to happiness. This is an affliction that we impose upon ourselves. We need no enemy to do this to us.


text:
Paroxysmal dyskinesias
Arjun Tarakad MD (
Dr. Tarakad of Baylor College of Medicine has no relevant financial relationships to disclose.
Joseph Jankovic MD (
Dr. Jankovic, Director of the Parkinson's Disease Center and Movement Disorders Clinic at Baylor College of Medicine, received research and training funding from Allergan, F Hoffmann-La Roche, Medtronic Neuromodulation, Merz, Neurocrine Biosciences, Nuvelution, Revance, and Teva and consulting/advisory board honorariums from Abide, Lundbeck, Retrophin, Parexel, Teva, and Allergan.
Originally released December 29, 1993; last updated February 21, 2020; expires February 21, 2023
This article includes discussion of paroxysmal dyskinesias, paroxysmal choreoathetosis of Mount and Reback, paroxysmal kinesigenic dystonia, paroxysmal choreoathetosis precipitated by prolonged exercise, paroxysmal exertion-induced dyskinesia, paroxysmal hypnogenic dyskinesia, paroxysmal kinesigenic choreoathetosis, paroxysmal kinesigenic dyskinesia, paroxysmal nocturnal dystonia, and paroxysmal nonkinesigenic dyskinesia. The foregoing terms may include synonyms, similar disorders, variations in usage, and abbreviations.
Paroxysmal dyskinesias are a relatively rare subset of hyperkinetic movement disorders that are defined by their episodic nature. They may be categorized into paroxysmal kinesigenic dyskinesia, paroxysmal nonkinesigenic dyskinesia, and paroxysmal exertion-induced dyskinesia. Advances in the genetic causes of paroxysmal dyskinesias, particularly in the familial forms, have allowed further studies of phenotype-genotype correlation. Paroxysmal dyskinesias can also occur secondarily in the setting of other neurologic or systemic diseases. Psychogenic movement disorders, which may also be paroxysmal and abrupt in onset, should be considered in the differential diagnosis of paroxysmal dyskinesias. In this article, the authors review the expanding genetic advances and clinical characteristics of these disorders. Treatment strategies are reviewed.
• Paroxysmal dyskinesias may be categorized into paroxysmal kinesigenic dyskinesia, paroxysmal nonkinesigenic dyskinesia, and paroxysmal exertion-induced dyskinesia.
• Paroxysmal dyskinesias may be sporadic, genetic, or caused by metabolic or structural etiologies.
• Another common cause of paroxysmal dyskinesias is psychogenic movement disorders.
• Paroxysmal kinesigenic dyskinesia typically responds well to antiepileptic medications.
• Mutations in PRRT2 are associated with several childhood-onset episodic syndromes including paroxysmal kinesigenic dyskinesia, infantile convulsions, paroxysmal kinesigenic dyskinesia or infantile convulsions, and hemiplegic migraine.
• ADCY5 mutations are being increasingly recognized as a cause of paroxysmal dyskinesias.
Paroxysmal dyskinesias are a relatively rare subset of hyperkinetic movement disorders that are defined by their episodic nature. Mount and Reback were the first to use the term "paroxysmal choreoathetosis" (Mount and Reback 1940). They reported a familial form of intermittent, periodic dystonia and chorea in which the proband had infantile-onset prolonged dyskinesia induced by alcohol and other agents. The episodes were characterized by an aura of a tight sensation in the neck and abdomen and a sense of fatigue followed by involuntary flexion of the arms and extension of the legs (dystonia). The spells progressed to involuntary choreoathetosis and dysarthria despite normal consciousness. Mount and Reback called the condition "familial paroxysmal choreoathetosis." Kertesz described a group of patients who primarily had a childhood onset of movement-induced paroxysmal choreoathetosis (Kertesz 1967). He highlighted the kinesigenic component and coined the term "paroxysmal kinesigenic choreoathetosis" as a specific entity within the paroxysmal choreoathetosis syndrome. A striking feature of the kinesigenic form was the brief (seconds to minutes) duration of the episode, whereas the nonkinesigenic form was said to be longer lasting (minutes to hours). Richards and Barnett coined the term "paroxysmal dystonic choreoathetosis of Mount and Reback" to help delineate it from the more common paroxysmal kinesigenic choreoathetosis (Richards and Barnett 1968). Lance reported a family with intermediate-duration attacks that were precipitated by prolonged exercise (Lance 1977). Lance's kindred also pointed out a common feature of the nonkinesigenic phenotype that is distinctive from the kinesigenic: failure to respond to anticonvulsants other than benzodiazepines. Lance comprehensively summarized the various forms of paroxysmal movement disorders that were often labeled as paroxysmal kinesigenic choreoathetosis, including (familial) paroxysmal choreoathetosis, periodic dystonia, reflex epilepsy, movement induced seizures, conditionally responsive extrapyramidal syndrome, and hereditary kinesthetic reflex epilepsy. For a more detailed review of the history, see the review by Fahn (Fahn 1994).

redpajama_set_name: RedPajamaCommonCrawl


text:
400 millons phosphate reserves
400 millons phosphate reserves - bistrovijve.be
Jan 14, 2014· Mosaic's phosphate segment is the main revenue driver for the company. The phosphate prices fell from $533 per mt to $436 per mt in the third quarter of …
Mining: Phosphate, the green ore glimmers | The Africa ...
6 hours ago· A majority of American adults say they are living comfortably, according to an annual Federal Reserve study on financial well-being of s. But nearly a third still lack savings to handle a ...
About Bone Valley - Bay Soundings
Sheldon (1987) categorized phosphate reserves and resources according to continents and regions (Table 4). At first glance, each main continent/region has ample 'reserves' of phosphate, with the possible exception of Oceania (391 million tonnes). However, regionally, a few countries, or even one country, may dominate.
Millions of Americans are only $400 away from financial ...
Into this scenario, Arianne Phosphate of Quebec plans to build a $1.2 billion open pit mine/mill and port in Quebec with first commercial production planned for 2022. Formed in 1997 Arianne began $45 million expenditures on the deposit 200 km north of Saguenay in 2008, when phosphate …
Banaba | island, Kiribati | Britannica.com
Phosphates: Commodity Overview. Posted by: ... Currently Morocco and the Western Sahara in Africa contain 78% of the worlds phosphate reserves (about 50 billion metric tons). China, Morocco and the United States make up two thirds of global phosphate production, but China and the United States primarily use it for domestic agriculture and are ...
Florida : the new uranium producer (FGS: Special ...
Nov 26, 2017· The geopolitics of phosphate even extends to the facts, i.e. to figures about proven and expected reserves. In the same year when China imposed its export tariff, it lowered its figure for proven reserves by 3,000 million tons, quite a lot in a world total of 16,000 million tons.
'P' is for phosphate: Could urine solve a fertilizer ...
World Production of Phosphate Rock Phosphorus (P) is a vital resource for sustaining world agriculture. Reserves of phosphate rock are identified in many regions of the world. TABLE 1. World phosphate rock reserves, reserve bases, and production. Production Reserves Reserve base 1995 1996 1997 Countries million tons thousand tons
An Overview of Phosphate Mining and Reclamation in Florida
Phosphorus is an essential element for plant and animal nutrition. Most phosphorus is consumed as a principal component of nitrogen-phosphorus-potassium fertilizers used on food crops throughout the world. Phosphate rock minerals are the only significant global resources of phosphorus. The United States is the world's leading producer and consumer of phosphate rock, which is
World Phosphate Rock Reserves and Resources Executive Summary The amount of remaining phosphate rock reserves and resources worldwide has become an issue of speculation. It has been hypothesized that phosphorus (phosphate rock) production will "peak" in 2033-2034 and then production will unavoidably decrease as the reserves are depleted.
Phosphates: Commodity Overview | Geology for Investors
Global Phosphate Rock Reserves and Resources, the Future of Phosphate Fertilizer S. J. Van Kauwenbergh . Principal Scientist and Leader, Phosphate Research and Resource Initiative . IFDC . U.S. Department of Agriculture . 2014 Agricultural Outlook Forum . February 20-21, 2014
Mining - Niger - export, sector - Encyclopedia of the Nations
Apr 01, 2013· The duration of phosphate rock reserves will also be impacted by the decreasing quality of the reserves, the growing global population, increased meat and dairy consumption (which require more fertilized grain for feed), wastage along the food chain, new technologies, deposit discoveries and improvements in agricultural efficiency and the recycling of phosphorus.
Time to stop unethical phosphate trade: North Otago farmer ...
Jul 07, 2011· It takes one ton of phosphate to produce every 130 tons of grain, which is why the world mines about 170 million tons of phosphate rock every year to ship around the world and keep soils fertile. Most of the world's best phosphate reserves are …
China is Running Out of High-Grade Phosphorus. How will it ...
4 hours ago· Dairy farmer Neil Hamilton said Ravensdown and Ballance Agri-nutrients, who each import about $30 million worth of phosphate a year, had a moral duty to stop the trade.
Tunisia seeks to increase its phosphate exports by 55 pct ...
Some have stated that phosphate reserves will peak in 2033, while the International Fertilizer Development Center estimates enough world reserves to last for 300-400 years (Van Kauwenbergh et al., 2010). The History of Phosphate Mining in Florida US phosphate mining began in South Carolina and Tennessee. Starting in 1888, phosphate pebble was
Reserve bank printed out 400 MILLION $50 notes with a ...
400 millons phosphate reserves. Better utilisation of Eppawala deposit for betterment of ... - Daily News. Feb 19, 2007 ... The other prominent Phosphate production countries are Brazil, ... there is a saving of substantial foreign exchange about Rs.400 million yearly. Get Price.
Venezuela sells $400 million in gold as it struggles to ...
Jan 25, 2016· Togo may have up to 2bn tonnes of phosphate reserves, the largest in sub-Saharan Africa. In the long term, the 30-year concession is expected to yield more than $28bn in revenue, with annual export of 3m tonnes of concentrated phosphate, 500,000 tonnes of phosphoric acid and 1.3 million tonnes of fertiliser products.
Mosaic to open new Hardee phosphate mine by March - News ...
World Phosphate Rock Reserves and Resources S. Van Kauwenbergh Geologist and Principal Scientist Research and Development Division IFDC Fertilizer Outlook and Technology Conference Hosted by The Fertilizer Institute and the Fertilizer Industry Roundtable November 16-18, 2010 Savannah, GA
Phosphate Rock Cement - apeda.org.in
Banaba, also called Ocean Island, coral and phosphate formation, part of Kiribati, in the west-central Pacific Ocean. It is located 250 miles (400 km) west of the nearest Gilbert Islands and has a circumference of about 6 miles (10 km). Banaba is the location of the highest point in Kiribati, reaching 285 feet (87 metres) above sea level.
World Production of Phosphate Rock
Turning phosphate deposit into mine could generate large ...
The Future of Phosphate is in Morocco ... holds a disproportionate share of global phosphate reserves (85 percent, or 50 billion metric tons) and has committed to investing upwards of $5 billion ...
Mosaic: Betting On Phosphate - seekingalpha.com
Phosphate Rocks IntechOpen. At least 100 millions years before humankind exerted any influence on the cycles of phosphorus, it appears that phosphate reserves would last for at least 100 years . Fig. 15. 400 years . Phosphate rock prices will increase when the demand approaches the limits of supply.
Use of phosphate rocks for sustainable agriculture - fao.org
The phosphate mineral as it exists in the ground is not soluble and is hard for the plant to access. To provide the plant with the phosphate it needs, in a form it can take up through its roots, the phosphate rock is converted to phosphoric acid. Phosphate Mining Today · Flow1 · Process Water · Water Quality · Phosphate Beneficiation
Global Phosphate Rock Reserves and Resources, the Future ...
Use of phosphate rocks for sustainable agriculture - fao.org. Sheldon (1987) categorized phosphate reserves and resources according to continents and regions (Table 4). At first glance, each main continent/region has ample 'reserves' of phosphate, with the possible exception of Oceania (391 million …
World Phosphate Rock Reserves and Resources - FIRT
Economically minable deposits of phosphate rock are found in just a handful of countries. Together, Morocco, China, Algeria, Syria and South Africa control 88 percent of the world's phosphate reserves, according to USGS. Morocco alone controls about 75 percent of the world's high-grade reserves and is the second-largest producer, behind China.
400 millons phosphate reserves - pochiraju.co.in
The Reserve Bank's $20B disaster: How 400 MILLION new $50 notes were printed before a hard-to-spot typo was revealed in the fine print. New note was brought into circulation in October 2018, with ...
Sep 22, 2010· IFDC Report Indicates Adequate Phosphorus Resources Available to Meet Global Food Demands. September 22, 2010 in Press Releases. Panel participants during the release of the IFDC report "World Phosphate Reserves and Resources" included (from left to right) Steven J. Van Kauwenbergh, principal scientist and leader of IFDC's Phosphate ...
Geopolitics of phosphate in a biobased economy | Bio Based ...
'P' is for phosphate: Could urine solve a fertilizer shortage? ... The U.S.'s largest phosphate reserves are in the Western Phosphate Field in the northern Rocky Mountains. ... when global phosphate prices shot up by 700 percent from $50 to more than $400 per ton. Though phosphate prices have since fallen — hovering between about $100 and ...
IFDC Report Indicates Adequate Phosphorus Resources ...
Glucose-6-phosphate Dehydrogenase Deficiency. Glucose-6-phosphate dehydrogenase (G6PD) deficiency is an inherited -linked red cell abnormality affecting more than 400 million people across the tropics. There are over 180 different variants mainly resulting from mutations in the G6PD gene and conferring varying levels of deficiency.
pound/metric ton) of phosphate rock. The current phosphate reserve figures for Central Florida are 1,499 million short tons (1,400 million metric tons) recoverable at $15 to $20 per short ton (6). In broad terms, 224,850 short tons (204,000 metric tons) of uranium oxide are contained in the Central and South Florida phosphate reserves.
Phosphate Rock Statistics and Information - usgs.gov
12 hours ago· LONDON (Reuters) - BP is nearing the sale of its stake in a major Egyptian oil and gas company to Dubai-based Dragon Oil for over $600 million, industry and banking sources said. The sale, which ...
phosphate rock grinders
five roller fine mesh grinder above 400
hammer mills for calcium phosphate
400 mesh grinder
john 400 grindermixer screens
roll grinder 400
phosphate rock vertical roller mills indian suppliers


Thanks to Hotel Am Parkring's central inner city location in the 1st district you will quickly get to your destination by car, by underground, by bus or by tram.
Arriving at Hotel Am Parkring by car can be very easy. Just enter your data in the route planner and you will quickly get an exact route description.
If you do not want to drive to Vienna, why not take the train? By underground you will then reach your city hotel on Parkring quickly and conveniently from all major train stations.
From Hauptbahnhof Wien (Main Station) take the U1 direktion Leopoldau and change at "Stephansplatz" to the U3 direction Simmering. At the station "Stubentor" you have to walk only 200m to Hotel Am Parkring.
From Vienna Schwechat airport you will best get to your city hotel in Vienna by express train or with the City Airport Train. Take the CAT to Wien Mitte. From there it's a ca. 10-minute walk to Hotel Am Parkring. Plan about 30 minutes for the way from the airport to Hotel Am Parkring. A taxi will get you here within about 20 minutes.
Our reception team will gladly organise a limousine airport transfer (max. four people) from and to the hotel.
Book your airport transfer – the team at Hotel Am Parkring will gladly organise the transfer for you.
The team at the city hotel in Vienna wishes you a safe journey and is always available for your queries. Get further information on Hotel Am Parkring's central location and book your room at the central city hotel on Parkring online now.


The property: This Annex Victorian used to be a three-unit rental property. Now it's a spacious modern home with a third-floor loft.
The history: The seller bought the triplex in 2010 and converted it into a single-family home, complete with a basement nanny suite.
The fate: The buyers are a couple with children. Coincidentally, one of their fathers had rented out one of the units in the house before the renovation, adding an element of personal history to the sale. They loved the property's central location, and also the open layout and high ceilings.
The sale: The property was on the market for 24 days and received two offers, a few days apart. The seller accepted the second, higher offer.
The living room has a bay window and a little bit of exposed brick.
The house used to be divided into three separate apartments, but the seller turned it into a single-family dwelling. Now, the main floor is completely open.
The powder room has heated floors.
The kitchen has almost too much wine storage.
The countertops in here are marble.
The master bedroom is on the second floor.
The master ensuite has heated floors, plus a little more of that exposed brick.
The master bedroom has a door to this terrace. There's a similar terrace off the third-floor loft.


text:
Tag Archives: Metropolitan Police
Life in the UK: People arrested for drinking coffee while walking along waterfront
Posted on January 11, 2021 by DCG | 5 comments
Arrested for freely hugging people. Now you can't even sit on a bench by yourself and drink coffee. Utter madness.
British people are now being arrested for drinking coffee and walking along the beach as @BorisJohnson's new crackdown kicks in.
The flame of Liberty is fading across the Western world. pic.twitter.com/mVZkMFXbCP
— Raheem Kassam (@RaheemKassam) January 10, 2021
It's ALL ABOUT CONTROL.
Posted in Britain, CoronaVirus, COVID-19, epidemics, Health Care, Insanity, Liberals/Democrats/Left, Taxes
Tagged #itsallaboutcontrol, Boris Johnson, Metropolitan Police, Wuhan virus
Metropolitan Police arrest UK man for hugging people
Posted on January 4, 2021 by DCG | 6 comments
Notice this man didn't force himself upon anyone at all. Yet the Metropolitan Police arrest him anyway for endangering people.
Why am I being arrested?
Because you were clearly engaging with people.
They are now arresting people who hug.@Iromg @JuliaHB1
Perhaps millions of people will now make this their protest.
A mass hugathon could be the new 2021 show of defiance? pic.twitter.com/fo9pvxIKP4
— George Son of George 🤔 (@my_suhr_guitar) January 2, 2021
This is NOT the world I want to live in. The bureaucrats have WAY TOO MUCH CONTROL. And I am almost certain that at some point some one will go postal over these INSANE restrictions. I am sure of that.
Posted in Britain, CoronaVirus, COVID-19, epidemics, Fear Mongers, Health Care, Insanity, Taxes
Tagged #itsallaboutcontrol, London, Metropolitan Police, UK, Wuhan virus
Good thing they banned guns: London averaging 40 knife crimes per day
Posted on April 9, 2019 by DCG | 15 comments
London has some serious crime issues as I've blogged about before. See the following:
London Mayor Sadiq Khan has a serious crime problem: City sets record for murders as police launch 100th investigation
London stabbings: Six teenagers attacked in 90 minutes during night of violence
Guess what is responsible for this: London sees surge in knife attacks
Mayor Sadiq Khan refuses to acknowledge the real source of knife crimes in his city: criminals. Instead of getting tough on criminals the mayor focuses on political correctness and blames the lack of taxpayer dollars for this "complex" problem (of course).
From Evening Standard: The shocking extent of London's knife crime problem was revealed today as figures showed that 40 knife offences a day were reported to police over a two-year period.
A person was knifed to death on average every four days in London in 2017 and 2018, figures revealed, following a Freedom of Information request by the Standard.
The figures reveal the total number of knife offences including those involving stabbings and deaths caused by a blade for the calendar years of 2017 and 2018. Met Police dealt with 29,232 knife offences in a two years.
On average this is about 40 offences a day in London.
The total number of homicides and stabbings fell across the two-year period however. In 2018, the number of knife attacks fell by over 500. In 2017, there was 4,784 stabbings and in 2018 there was 4,246. The number of homicides also fell by 12.
Official statistics for 2019 have not been made available just yet buy Mayor of London Sadiq Khan said the decrease is encouraging. Although the number is still "too high", Mr. Khan praised London communities and the Met for their work in reducing crime.
A spokesman from the Ben Kinsella Trust, which tackles knife crime through education and campaigning, said the small reduction is not a victory yet.
A Metropolitan Police spokesman said officers are working "day and night" to identify and pursue knife criminals.
The majority of the crimes detailed in the figures were committed in Southwark with 1,594 offences occuring in that London borough The borough also had the highest number of knife-related homicides with 17 people being stabbed to death in two years. Bexley was the only borough of 33 to not have any deaths resulting from a knife in the two years.
Mr. Khan said: "Thanks to London's communities and the hard work of the Met Police, who have been targeting offenders and removing dangerous weapons from our streets, we are seeing some knife crime offences starting to fall, but it still remains too high. We still desperately need the Government to reverse its damaging cuts to the police and preventative services."
He added the level of knife crime "across the country" was still "unacceptably high".
Read the whole story here.
h/t Breitbart
Posted in crime, Liberals/Democrats/Left, Taxes
Tagged England, London, London's knife crimes, Met Policde, Metropolitan Police, Sadiq Khan
When proggies advocate for UK-style gun control laws show them this: Police struggle to stop flood of firearms into UK
Posted on December 29, 2018 by DCG | 10 comments
Obviously banning guns means nothing, absolutely nothing, to those who are determined to break the law.
From The Guardian: Police and border officials are struggling to stop a rising supply of illegal firearms being smuggled into Britain, a senior police chief has warned.
Chief constable Andy Cooke, the national police lead for serious and organised crime, said law enforcement had seen an increased supply of guns over the past year, and feared that it would continue in 2019.
The Guardian has learned that the situation is so serious that the National Crime Agency (NCA) has taken the rare step of using its legal powers to direct every single police force to step up the fight against illegal guns. The NCA has used tasking powers to direct greater intelligence about firearms to be gathered by all 43 forces in England and Wales.
Another senior law enforcement official said that "new and clean" weapons were now being used in the majority of shootings, as opposed to guns once being so difficult to obtain that they would be "rented out" to be used in multiple crimes.
Cooke, the Merseyside chief constable, told the Guardian: "We in law enforcement expect the rise in new firearms to continue. We are doing all we can. We are not in a position to stop it anytime soon. Law enforcement is more joined up now than before, but the scale of the problem is such that despite a number of excellent firearms seizures, I expect the rise in supply to be a continuing issue."
The increasing supply of guns belies problems with UK border security and innovations by organised crime gangs. Smugglers have increasingly found new ways and innovative routes to get guns past border defences.
Cooke said more are coming in from eastern Europe and the western Balkans and also component pieces are being smuggled in via the fast parcel system, hidden among a growth in packages because of the explosion of internet shopping.
Guns are being bought on the dark web and the UK is also seeing weapons designed for sports shooting being smuggled in, as well as blank firing weapons bought for just £100 being converted in the UK. Criminals have also obtained high end automatic weapons such as an AK47.
Cooke said that the dynamics of the streets of British cities had changed and that criminals were more willing to use guns: "If they bring them in people will buy them. It's a kudos thing for organised criminals."
He said said one factor was a reduction in police proactive work because of government budget cuts leading to big falls in officer numbers: "The ability of law enforcement to respond to this rise in the criminal use of firearms has been hampered by the large reduction in police officers and the resultant diminishing of proactive capability to keep these criminals on the back foot."
Simon Brough, head of firearms at the NCA, said: "The majority of guns being used are new, clean firearms … which indicates a relatively fluid supply."
He said shotguns were 40% of the total, with an increase in burglaries to try and steal them. Handguns are the next biggest category, most often smuggled in from overseas, with ferry ports such as Dover being a popular entry point into the UK for organised crime groups: "We're doing a lot to fight back against it," Brough said, adding that compared to other European countries, the availability in the UK was relatively lower.
Gun crime has been rising and the last set of official figures about recorded firearms offences, showing a 5% fall to 6,362 in the year ending June 2018, is seen as a blip against a trend of rising gun offences. Statistics released earlier in 2018 had shown gun crime up 11% in 2017/18, and in the same period, the Metropolitan police said discharges of lethal barrelled firearms rose by 23%. Compared to 2015/16, there has been a 67% increase in the capital alone.
Some in law enforcement tackling serious and organised crime believe more attention needs to be paid to that type of criminality, instead of just the intense focus on the terrorist threat. Cooke said: "The two greatest national security threats are terrorism and serious and organised crime. Nationally, we need to ensure serious and organised crime gets the same funding as the terrorist threat. More people die after getting shot by serious and organised criminals than by terrorists."
Counter-terrorism sources tell the Guardian that the most likely route for terrorists in the UK to get guns will be from criminals, not by specifically importing them. In a 2016 case, a group of Isis-inspired terrorists led by Tarik Hassane, dubbed "the surgeon", got a firearm from criminal contacts as part of a plot to stage drive by shootings.
The growing supply of guns in the UK, rising year on year according to intelligence reports, alarms those trying to stop a Paris-style marauding gun attack in the UK. They fear the more firearms are available, the easier it will be for terrorists to get them from criminals.
Cooke said: "If organised criminals have these guns, it's not a big leap of faith that they will deal them to terrorists."
Posted in 2015 Paris terrorist attacks, crime, Culture War, gun control, Liberals/Democrats/Left, Middle East, Taxes, Terrorism
Tagged AK47, Balkans, Britain, criminals do not obey gun laws, Dover, England, handguns, ISIS, Metropolitan Police, National Crime Agency, orgainzed crime in the UK, UK border security, UK gun crime, Wales
Posted on April 8, 2018 by DCG | Leave a comment
Whew, good thing London has strict gun control laws to stop the violence in their streets!
From Yahoo: Six teenagers have been stabbed within 90 minutes in another night of violence on London's streets.
A 13-year-old boy was among the victims of four separate knife attacks, which took place as protesters gathered elsewhere in the capital to demand action to prevent young people dying.
Earlier, Tottenham MP David Lammy said violence in London was the "worst I've ever seen it" and warned there was "absolutely no sign" of an end to the bloodshed following a spate of murders as drugs drive turf wars between gangs.
Crowds gathered at Hackney Central station in east London on Thursday, close to where 18-year-old Israel Ogunsola was stabbed to death the previous night.
But as demonstrators and community leaders called for peace, violence was breaking out again on the capital's streets.
A 13-year-old boy was taken to hospital with serious injuries after being stabbed near Little Ilford Park, Newham. Three other boys were arrested on suspicion of causing grievous bodily harm with intent.
Minutes later a man in his late teens was taken to hospital following a stabbing in Ealing Broadway, west London. The Metropolitan Police said it was awaiting an update on his condition.
An hour earlier, three boys were taken to hospital following a knife attack in Mile End, east London. Two were in a serious condition, Tower Hamlets police said. The third was treated for minor injuries before being arrested on suspicion of conspiracy to commit grievous bodily harm.
The night of violence had begun at 5.30pm, when a 15-year-old boy was seriously injured in a stabbing in East India Docks. His condition was stable and his injuries were not thought to be life-threatening. No arrests have been in connection to the attacks in Ealing Broadway or East India Docks.
The violence on Thursday evening came after a man in his twenties was injured in a stabbing in Walthamstow, north-east London, at around 12.50pm.
The flurry of knife attacks followed the killings of three teenagers in the capital in the first four days of this month.
Tanesha Melbourne, 17, was shot dead in a drive-by attack in Tottenham, north London, on 2 April. Minutes later, 16-year-old Amaan Shakoor died in a gun attack three miles away in Walthamstow. No arrests have been made in connection to either murder.
Two 17-year-old boys have been held on suspicion of killing 18-year-old Mr Ogunsola.
At the protest in Hackney, demonstrators of various ages huddled around the station entrance before linking hands in a wide circle nearby in solidarity for those killed.
Protest organisers Guiding A New Generation – commonly known as GANG – invited people to share their stories and plead for an end to the killings.
One speaker, who addressed the crowd under the name Scripture, said he had seen people being killed at close hand. "It's not a pretty sight," he said. "You know what, it's not a computer game. You're not gonna come back, bruv.
"These youths who are doing damage to each other are not coming back, and they're not learning their lesson neither."
Hackney Liberal Democrat mayoral candidate Pauline Pearce told the rally young people felt "disenfranchised" by their environments.
She said: "A lot of the children feel disenfranchised, they don't feel they belong, they haven't really got a meaning – they don't feel that they have that connection to society, so a lot of things go wrong for them and sadly this is the sort of retaliation that comes."
How is this possible? Over the last three years there has been a significant rise of gun crime in London
The UK is toast: London police chief says attack victims show city's "diversity"
Posted in Children, crime, Culture War, gun control, Liberals/Democrats/Left
Tagged London, Metropolitan Police, Sadiq Khan
Transsexual who battered her sleeping brother is cleared of attempted murder because she was sleepwalking
Posted on August 16, 2015 by DCG | Leave a comment
Daily Mail: A transgender police worker who battered her sleeping brother with an iron bar was cleared of attempted murder because she was sleepwalking.
Helen Doe told police she woke up in the home she shared with her brother Kenneth, 64, with an 'overwhelming hatred' for him. The 53-year-old, from Mitcham, south London, then hit M.r Doe three times on the head with a one-and-a-half foot long metal bar as he lay in bed on July 5 last year. He also received a broken finger as he tried to defend himself.
Miss Doe, who has worked with the Metropolitan Police for 36 years and was taking hormone tablets at the time of the attack, was found not guilty at Kingston-upon-Thames Crown Court. John Hilton, defending, said she was suffering 'non-insane automatism' – involuntary movements she was not responsible for.
Four expert pharmacologists agreed the drugs she was taking could have caused violent behaviour.
Paul Cavin, prosecuting, said: 'The defence will point to the conclusion Miss Doe was sleepwalking when she attacked her brother and that sleepwalking and violence was due to the drugs she was taking.' He added: 'Fortunately he did not die or suffer long-term injury at all, brain injury, however it is the prosecution case that in striking him several times with an iron bar she intended to kill him. It may be that she is not strong enough, maybe he blocked or parried the blows.'
The court heard Miss Doe, who works in the uniform division of the force, dialed 999 and told the operator at around 2.50am: 'I've had an accident. I think I've killed my brother. I've hit him with an iron bar. Just now, I've been building up to it.'
When officers entered the house the defendant was in the kitchen and told them: 'It's me. I did it, he's upstairs.'
Police went upstairs to check on the brother, who told them: 'I had an argument with her earlier and I woke up and she was hitting me with an iron bar… I don't like her friends.' He was rushed to St. George's Hospital to be treated for the three wounds, plus a fractured middle finger on his left hand.
Mr. Cavin told the jury Miss Doe spoke freely in the van on the way to Sutton Police Station, confessing: 'I wanted to do it. I wanted to kill him.'
She told police officers in an interview after the attack: 'I was in a mental state and I didn't know what I was doing. That's why I need to go to a mental place.'
'She said her friendship with her brother was not a good one and woke in the early hours with an overwhelming hatred of her brother and she makes reference to muffled voices in her head.
'She describes a history of depression for which she was taking prescribed medication. She also said she was in the process of changing gender to that of a female and was having hormone therapy.'
Doe has organised lectures for Croydon LGBT and has led the Croydon Trans Group. She has been living as a woman since 2012.
Posted in crime, Culture War, LGBT, Liberals/Democrats/Left
Tagged London, Metropolitan Police, sleepwalking, Transgender