text:
products will arrive within 10 business days. we do ship to PO Boxes and APO/FPO addresses. currently, we're not shipping to Alaska, Hawaii or Puerto Rico. view our shipping table for expedited and express rates. some products may have a small special shipping fee due to the item's weight, size and dimensions. please check the product deets for more information. you'll see your estimated total shipping and taxes listed in checkout before you place your order.
The most common causes for that are actually the coaches themselves because they only correct technique rather than involve players in various drills that would take their attention off their body and make them focus on tasks like reading the incoming ball, improving timing and rhythm, improving the feel for the strokes, hitting targets on the court, working on tactics like making opponents run, looking to wrong foot the opponent, and so on.
Most large tournaments seed players, but players may also be matched by their skill level. According to how well a person does in sanctioned play, a player is given a rating that is adjusted periodically to maintain competitive matches. For example, the United States Tennis Association administers the National Tennis Rating Program (NTRP), which rates players between 1.0 and 7.0 in 1/2 point increments. Average club players under this system would rate 3.0–4.5 while world class players would be 7.0 on this scale.
In 1968, commercial pressures and rumors of some amateurs taking money under the table led to the abandonment of this distinction, inaugurating the Open Era, in which all players could compete in all tournaments, and top players were able to make their living from tennis. With the beginning of the Open Era, the establishment of an international professional tennis circuit, and revenues from the sale of television rights, tennis's popularity has spread worldwide, and the sport has shed its middle-class English-speaking image[39] (although it is acknowledged that this stereotype still exists).[39][40]
SportMaster tennis court surfaces are technically advanced, 100% acrylic sports surfaces. Many people refer to tennis court surfaces as "tennis court paint", but SportMaster systems are designed to provide consistent speed of play, texture, and vibrant color to any indoor or outdoor facility. SportMaster tennis court surfaces are formulated to resist fading and withstand a variety of weather conditions from ice and snow to intense heat and ultra-violet rays.
Kolkmann says that asphalt courts can also develop low areas over time due to settling of the soil or base under the asphalt surface. "With a post-tensioned slab, this area can be bridged and no settling will occur. In addition, concrete courts can often be installed on unstable soils where it would be cost-prohibitive to do extensive excavating and base work to support an asphalt court," he says.
Before and during the open era, Rod Laver remains the only male player in history to have won the calendar year Grand Slam twice in 1962 and 1969 [101] and also the calendar year Professional Grand Slam in 1967.[102] More recently Björn Borg and Pete Sampras were regarded by many of their contemporaries as among the greatest ever. Andre Agassi, the first of two male players in history to have achieved a Career Golden Slam in singles tennis (followed by Rafael Nadal), has been called the best service returner in the history of the game.[103][104][105][106] He is the first man to win grand slams on all modern surfaces (previous holders of all grand slam tournaments played in an era of grass and clay only), and is regarded by a number of critics and fellow players to be among the greatest players of all time.[103][107][108] However it must be noted that both Rod Laver and Ken Rosewall also won major Pro Slam tournaments on all three surfaces (grass, clay, wood) Rosewall in 1963 and Laver in 1967.[109]
All sessions vary depending on the level of the player, the type of training program and the duration of the session. I divide them into 4 parts (order may vary as well): 1) power and control on the court: I design a plan so that the player focuses on developing his/her power and strength to be in control of his game from the moment he/she steps on the court. 2) drills 3) serves: I focus a lot on serve because it is very important for the player to build confidence on his/her serve. 4) sets: Depending on the level of the player, I like to arrange sets so they are used to pay attention to every point, and work on their conc ... View Profile
A tennis match is intended to be continuous.[65] Because stamina is a relevant factor, arbitrary delays are not permitted. In most cases, service is required to occur no more than 20 seconds after the end of the previous point.[65] This is increased to 90 seconds when the players change ends (after every odd-numbered game), and a 2-minute break is permitted between sets.[65] Other than this, breaks are permitted only when forced by events beyond the players' control, such as rain, damaged footwear, damaged racket, or the need to retrieve an errant ball. Should a player be determined to be stalling repeatedly, the chair umpire may initially give a warning followed by subsequent penalties of "point", "game", and default of the match for the player who is consistently taking longer than the allowed time limit.[66]
The track opens with Lorde questioning "Don't you think that is boring how people talk?" with her, in the words of Lester, "sweet, sultry and sour" voice, which suggests "a frictional relationship with the high life".[37] This line also summarises Pure Heroine's central theme of Lorde's observations and critiques of mainstream culture.[39][40] Lorde repeats the tongue-in-cheek word "Yeah" after each verse.[41] The refrain contains the lyrics "Baby be the class clown / I'll be the beauty queen in tears", which feature "class clown" and "beauty queen" stereotypes of high school students[25][42] and express underlying sorrow behind superficial beauty.[43][44] Follow-up lines "It's a new art form showing people how little we care / We're so happy even when we're smilin' out of fear" illustrate Lorde feigning disinterest when facing fame and adolescence.[25][42] In the second verse, Lorde wonders about her future music career, therefrom expecting inevitable nostalgia of her non-celebrity lifestyle, "But my head's filling up with the wicked games, up in flames / How can I fuck with the fun again, when I'm known?"[28][45][46][47] At the bridge, Lorde sings "I fall apart, with all my heart / And you can watch from your window", which HuffPost interpreted as a commentary on the breakdowns of teenage celebrities.[32]
Help: Scoreboard offers live tennis scores for all major ATP, WTA, Challanger and ITF tournaments together with tennis grandslams. Get ATP, WTA tennis tournament brackets, set results and match details. Follow the entire tennis season live on Scoreboard – from Hopman Cup live scores, Australian Open and Wimbledon to US Open scores, ATP Masters and WTA Championships. Scoreboard.com tennis section offers real-time WTA and ATP scores live, latest ATP results, Fed Cup and Davis Cup live tennis scores.
Your swing style is another important factor to consider when shopping for tennis racquets. If you have a fast and long swing, you are likely able to provide all the power you need and may want to consider a racquet with a mid or mid-plus head, which can help you control your shot. If you have a shorter and slower swing, an oversized racquet head may be a better choice as it can increase the power of your shot.
In August 2007 the ATP announced major changes to the tour that were introduced in 2009. The Masters Series was renamed to the "Masters 1000", the addition of the number 1000 referring to the number of ranking points earned by the winner of each tournament. Contrary to earlier plans, the number of tournaments was not reduced from nine to eight and the Monte Carlo Masters remains part of the series although, unlike the other events, it does not have a mandatory player commitment. The Hamburg Masters has been downgraded to a 500-point event. The Madrid Masters moved to May and onto clay courts, and a new tournament in Shanghai took over Madrid's former indoor October slot. As of 2011 six of the nine "1000" level tournaments are combined ATP and WTA events.[89]
The four Grand Slam tournaments are considered to be the most prestigious tennis events in the world. They are held annually and comprise, in chronological order, the Australian Open, the French Open, Wimbledon, and the US Open. Apart from the Olympic Games, Davis Cup, Fed Cup, and Hopman Cup, they are the only tournaments regulated by the International Tennis Federation (ITF).[82] The ITF's national associations, Tennis Australia (Australian Open), the Fédération Française de Tennis (French Open), the Lawn Tennis Association (Wimbledon) and the United States Tennis Association (US Open) are delegated the responsibility to organize these events.[82]
Upon completion of The Love Club, Lorde and Little quickly collaborated again, initially planning to release another EP. The pair recorded materials at Little's Golden Age Studios in Morningside and started writing "Tennis Court" in January 2013.[5] They also recorded several additional tracks and ultimately decided to work on a full-length studio album instead.[4] Little acted as the song's sole producer, using audio software Pro Tools.[6] Songwriting for "Tennis Court" was different from how Lorde usually writes songs; by and large, she would have a lyric forming before going into the studio to record. For this song, Little and Lorde first wrote the music and the beat, and the lyrics were built on the instant instrumental.[7] Speaking to Billboard in November 2013, Little appreciated Lorde's developed songwriting skills on "Tennis Court", for which the singer wrote the melody and the whole chorus, praising her as "an amazing songwriter".[8]
A break point occurs if the receiver, not the server, has a chance to win the game with the next point. Break points are of particular importance because serving is generally considered advantageous, with servers being expected to win games in which they are serving. A receiver who has one (score of 30–40 or advantage), two (score of 15–40) or three (score of love-40) consecutive chances to win the game has break point, double break point or triple break point, respectively. If the receiver does, in fact, win their break point, the game is awarded to the receiver, and the receiver is said to have converted their break point. If the receiver fails to win their break point it is called a failure to convert. Winning break points, and thus the game, is also referred to as breaking serve, as the receiver has disrupted, or broken the natural advantage of the server. If in the following game the previous server also wins a break point it is referred to as breaking back. Except where tie-breaks apply, at least one break of serve is required to win a set.
A great way of understanding this shot is by observing different professional tennis players who are good at different things. I find that you can learn tennis pretty fast by visualization. For example, Novak Djokovic, currently ranked as the number one tennis player in the world is famous for his flat forehand and backhand. Caroline Wozniacki, currently ranked as the best female tennis player is admired worldwide for her two-handed backhand. Observe such players and practice as much as you can so that you can play tennis to the best of your abilities!
The referee, who is usually located off the court, is the final authority about tennis rules. When called to the court by a player or team captain, the referee may overrule the umpire's decision if the tennis rules were violated (question of law) but may not change the umpire's decision on a question of fact. If, however, the referee is on the court during play, the referee may overrule the umpire's decision (This would only happen in Davis Cup or Fed Cup matches, not at the World Group level, when a chair umpire from a non-neutral country is in the chair).[59]
If time and finances permit, adult beginners should take a beginner tennis clinic and hit lots of balls, period. The clinic should be augmented by private lessons. After the basics are in place, find someone to hit with who can control the ball and see if he or she can practice with you. Take more clinics and hit balls … get exposure to hitting while you work on grip technique and footwork. Set a goal to get into the next level within 6-8 weeks.
This tennis racquet from Head is both affordable and very functional. It is an excellent quality entry level tennis racquet that most beginners will really find to their liking. It has a good grip that fits smaller size hands well and it is constructed out of materials that keep it stiff as you hit the ball to help generate extra power on your shots.
This level and the pro level share a lot of similarities simple because things have gotten a lot competitive in recent years. NCAA level players sometimes go pro after graduating college like John Isner. So, they accept nothing but close to perfection from young aspirants. There are NIAA colleges that offer some partial scholarships and they do not have the same rigorous requirements as the NCAA but often times they are very small colleges which are always the second choice for most students.
I focus on the student ability to improve skills in the court by having constant repetitions of drills. I also make the lesson more attractive for the students by playing with them and make them experience real game practice. I structure my lessons with short warm up continued by certain amount of drills and finally different games applied to different situations of the game.
Throughout most of tennis history, racquets were comprised of laminated wood with smaller heads of about 65 square inches. It was not until the late 1960s that steel and aluminum started to be used by professional players like Jimmy Connors. Since the late 1980s, graphite and carbon fiber are the main materials chosen for crafting racquets that are used by professional and amateur players alike. These racquet materials allow the frame of the racquet to remain lightweight but stiff enough for increased control and stability.
Explanations: Tennis livescore service on Livescore.in provides bettors and tennis fans with livescore real-time score service for all ATP and WTA events. Livescore tennis lets you stay updated and be in the games with its ultimate tennis live scores service! Get notified about live tennis scores changes by sound or coloured alerts, or just select your favourite tennis games in the Selected section. Webmasters of sports related websites who are interested in the tennis livescore service designed to match their site layout are welcome to utilize our livescore solution called inScore.
Another benefit of post-tensioning is that contractors can build larger slabs using thinner concrete sections, and they don't have to install control joints which can interfere with play. "In order to build a reinforced concrete court that has the same structural capabilities as a post-tensioned court, we would have to install so much steel and concrete that the reinforced court would actually cost more," says Kolkmann. "Also, a reinforced court needs control joints, usually at a spacing of 10 to 15 feet, including in the playing area. Eventually these joints may widen, as well as any cracks that appear."
The rackets from the Wilson Ultra series are processed in the graphite frame construction with carbon fiber, so that a power-loaded playing feeling is created. The oversize head provides good spin characteristics and the power of the club can be controlled very well via the spin. A comfortable racquet handling and a special parallel drilling technology (the strings dampen more vibrations) provide comfort.
"Tennis Court" received generally positive reviews from contemporary critics. Siân Rowe from NME complimented Lorde's "strong pop vocals".[31] Emily Yoshida from Grantland labelled it a "murkily winsome, ever-so-slightly chopped ballad",[48] while Kyle Jaeger writing for The Hollywood Reporter commended the track's lyrical content and its "catchy" melody.[49] Billboard's Jason Lipshutz was favourable towards the song's "detached attitude" and minimalist production that evoked "something intoxicating" in the music scene.[29] Sharing the same sentiment, Adam Offitzer from independent music blog Pretty Much Amazing regarded "Tennis Court" as a "[burst] of originality" in the mainstream music scene of "mindless hooks and dubstep anthems" for its minimalism and "clever" lyrics.[23]
The other type of tiebreaker Van Alen introduced is the "12-point" tiebreaker that is most familiar and widely used today. Because it ends as soon as either player or team reaches 7 points – provided that that player or team leads the other at that point by at least two points – it can actually be over in as few as 7 points. However, because the winning player or team must win by a margin of at least two points, a "12-point" tiebreaker may go beyond 12 points – sometimes well beyond. That is why Van Alen derisively likened it to a "lingering death", in contrast to the 9-point (or fewer) "sudden-death tiebreaker" that he recommended and preferred.
We may collect IP addresses and/or automatic information for purposes of system administration and to report aggregate information to our advertisers and sponsors. An IP address is a number that is automatically assigned to your computer whenever you access the Internet. Our computers identify your computer by its IP address. When you request Web pages from the Site, our computers log your IP address.
The industry has changed so much, and manufacturers are finding ways to keep you on the court. Wilson's Countervail technology helps reduce muscle fatigue to keep you on the court. Head's Graphene 360 offers a solid, dampened response at contact for more comfort. Babolat's Cortex Pure Feel features a new material throughout the hoop for better shock absorption. The list goes on.
The decision to install a top-of-the-line concrete court is just the beginning, however. You also need to evaluate your site, determine the type of playing surface you want, choose a surfacing system, and even pick out a color scheme. The next step is to find a qualified, experienced contractor who can install the court you want at a fair price. Here are some of the basics you need to know before getting in the game.
UNDER NO CIRCUMSTANCES, INCLUDING, BUT NOT LIMITED TO, NEGLIGENCE, SHALL THE USTA FAMILY OF COMPANIES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL OR CONSEQUENTIAL DAMAGES THAT RESULT FROM THE USE OF, OR THE INABILITY TO USE, ANY USTA FAMILY OF COMPANIES SITE OR MATERIALS OR FUNCTIONS ON ANY SUCH SITE, EVEN IF THE USTA FAMILY OF COMPANIES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. APPLICABLE LAW MAY NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY OR INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE ABOVE LIMITATION OR EXCLUSION MAY NOT APPLY TO YOU. IN NO EVENT SHALL OUR TOTAL LIABILITY TO YOU FOR ALL DAMAGES, LOSSES AND CAUSES OF ACTION WHETHER IN CONTRACT OR TORT (INCLUDING, BUT NOT LIMITED TO, NEGLIGENCE, OR OTHERWISE) EXCEED THE AMOUNT PAID BY YOU, IF ANY, FOR MEMBERSHIP IN THE USTA.
In order to assist litigants and attorneys using the Shreveport City Court, the court has created a Guide to Practice. This document is not meant to be a definitive statement of the law nor may it be cited to the court as support for any position. This document is merely a tool to be utilized to assist litigants and attorneys in access to the court. Legal citations are provided to assist litigants and attorneys in their quest to determine applicable law and procedure.
There is nothing that says you cannot look good when you play the game of tennis and that includes having a stylish tennis racquet that accentuates your attire. It is probably not something that you want to be a major factor in your purchasing decision but if you have narrowed your choices down to 2 or 3 similar racquets it can be used as the deciding factor.
Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return. The player who is unable to return the ball will not gain a point, while the opposite player will.
Contact us at webmaster@www.superbabyproject.com | Sitemap xml | Sitemap txt | Sitemap


text:
\section{Introduction}

\begin{table}[t!]
\resizebox{\columnwidth}{!}{
\begin{tabular}{  m{5em} |  m{7cm}  }
\hline
CNNDM Source& (CNN) \textcolor{blue}{About a quarter of a million} Australian homes and businesses have no power after \textcolor{blue}{a ``once in a decade" storm} battered Sydney and nearby areas. About 4,500 people have been isolated by flood waters as ``the roads are cut off and we won't be able to reach them for a few days,"...\\
\hline
 Bottom-up Summary& \textcolor{red}{a quarter of a million} australian homes and businesses have no power \textcolor{red}{after a decade}.\\
\hline
Corrected by SpanFact &\textcolor{orange}{about a quarter of a million} australian homes and businesses have no power \textcolor{orange}{after a ``once in a decade" storm.}\\
\hline \hline
Gigaword Source & all the \textcolor{blue}{12 victims} including \textcolor{blue}{8 killed and 4 injured} have been identified as senior high school students of the second senior high school of ruzhou city, central china's henan province, local police said friday. \\
\hline
 Pointer-Generator Summary& \textcolor{red}{12} killed, 4 injured in central china school shooting. \\
\hline
Corrected by SpanFact &\textcolor{orange}{8} killed, 4 injured in central china school shooting.\\
\hline \hline
XSum Source & st clare's \textcolor{blue}{catholic} primary school in birmingham has met with equality leaders at the city council to discuss a complaint from the pupil's family. the council is supporting the school to ensure its policies are appropriate...\\
\hline
 BertAbs Summary& a \textcolor{red}{muslim} school has been accused of breaching the equality act by refusing to wear headscarves. \\
\hline
Corrected by SpanFact &a \textcolor{orange}{catholic} school has been accused of breaching the equality act by refusing to wear headscarves.\\
\hline
\end{tabular}
}
\caption{Examples of factual error correction on different summarization datasets.  Factual
errors are marked in red. Corrections made by the proposed SpanFact models are marked in orange.}
\end{table}

Informative text summarization aims to shorten a long piece of text while preserving its main message. Existing systems can be divided into two main types: extractive and abstractive. Extractive strategies directly copy text snippets from the source to form summaries, while abstractive strategies generate summaries containing novel sentences not found in the source. Despite the fact that extractive strategies are simpler and less expensive, and can generate summaries that are more grammatically and semantically correct, abstractive strategies are becoming increasingly popular thanks to its flexibility, coherency and vocabulary diversity \citep{zhang2019pegasus}.

Recently, with the advent of Transformer-based models \citep{vaswani2017attention} pre-trained using self-supervised objectives on large text corpora \citep{devlin2018bert,radford2018gpt,lewis2019bart,raffel2019t5}, abstractive summarization models are surpassing extractive ones on automatic evaluation metrics such as ROUGE \citep{lin-2004-rouge}. However, several studies \cite{falke2019ranking,goodrich2019assessing,kryscinski2019evaluating,wang2020asking,durmus2020feqa,maynez2020faithfulness} observe that despite high ROUGE scores, system-generated abstractive summaries are often factually inconsistent with respect to the source text. Factual inconsistency is a well-known problem for conditional text generation, which requires models to generate readable text that is faithful to the input document. Consequently, sequence-to-sequence generation models need to learn to balance signals between the source for faithfulness and the learned language modeling prior for fluency \citep{kryscinski2019evaluating}. The dual objectives render abstractive summarization models highly prone to hallucinating content that is factually inconsistent with the source documents \citep{maynez2020faithfulness}.

Prior work has pushed the frontier of guaranteeing factual consistency in abstractive summarization systems. Most focus on proposing evaluation metrics that are specific to factual consistency, as multiple human evaluations have shown that ROUGE or BERTScore \citep{zhang2020BERTScore} correlates poorly with faithfulness \citep{kryscinski2019evaluating,maynez2020faithfulness}.  These evaluation models range from using fact triples \cite{goodrich2019assessing},  textual entailment predictions \citep{ falke2019ranking}, adversarially pre-trained classifiers \citep{kryscinski2019evaluating}, to question answering (QA) systems \citep{wang2020asking,durmus2020feqa}. It is worth noting that QA-based evaluation metrics show surprisingly high correlations with human judgment on factuality \citep{wang2020asking}, indicating that QA models are robust in capturing facts that can benefit summarization tasks.

On the other hand, some work focuses on model design to incorporate factual triples \citep{cao2018faithful,zhu2020boosting} or textual entailment \citep{li2018ensure,falke2019ranking} to boost factual consistency in generated summaries. Such models are efficient in boosting factual scores, but often at the expense of significantly lowering ROUGE scores of the generated summaries. This happens because the models struggle between generating pivotal content while retaining true facts, often with an eventual propensity to sacrificing informativeness for the sake of correctness of the summary. In addition, these models inherit the backbone of generative models that suffer from hallucination despite the regularization from complex knowledge graphs or text entailment signals.

In this work, we propose SpanFact, a suite of two neural-based factual correctors that improve summary factual correctness without sacrificing informativeness. To ensure the retention of semantic meaning in the original documents while keeping the syntactic structures generated by advanced summarization models, we focus on factual edits on entities only, a major source of hallucinated errors in abstractive summarization systems in practice \citep{kryscinski2019evaluating,maynez2020faithfulness}. The proposed model is inspired by the observation that fact-checking QA model is a reliable medium in assessing whether an entity should be included in a summary as a fact \citep{wang2020asking,durmus2020feqa}. To our knowledge, we are the first to adapt QA knowledge to enhance abstractive summarization. Compared to sequential generation models that incorporate complex knowledge graph and NLI mechanisms to boost factuality, our approach is lightweight and can be readily applied to any system-generated summaries without retraining the model. Empirical results on multiple summarization datasets show that the proposed approach significantly improves summarization quality over multiple factuality measures without sacrificing ROUGE scores.

Our contributions are summarized as follows.
($i$) We propose SpanFact, a new factual correction framework that focuses on correcting erroneous facts in generated summaries, generalizable to any summarization system.
($ii$) We propose two methods to solve multi-fact correction problem with single or multi-span selection in an iterative or auto-regressive manner, respectively.
($iii$) Experimental results on multiple summarization benchmarks demonstrate that our approach can significantly improve multiple factuality measurements without a huge drop on ROUGE scores.

\section{Related Work}

\begin{figure*}[t!]
    \centering
    \includegraphics[width=\textwidth]{figs/example.png}
    \caption{Training example created for the QA-span prediction model (upper right) and the auto-regressive fact correction model (bottom right).}
    \label{fig:span_selection_example}
\end{figure*}

The general neural-based encoder-decoder structure for abstractive summarization is first proposed by \citet{rush2015neural}.
Later work improves this structure with better encoders, such as LSTMs \citep{chopra2016abstractive} and GRUs \citep{nallapati2016abstractive}, that are able to capture long-range dependencies, as well as with reinforcement learning methods that directly optimize summarization evaluation scores \citep{paulus2017deep}. One drawback of the earlier neural-based summarization models is the inability to produce out-of-vocabulary words, as the model can only generate whole words based on a fixed vocabulary. \citet{see2017get} proposes a pointer-generator framework that can copy words directly from the source through a pointer network \citep{vinyals2015pointer}, in addition to the traditional sequence-to-sequence generation model.

Abstractive summarization starts to shine with the advent of self-supervised algorithms, which allow deeper and more complicated neural networks such as Transformers \citep{vaswani2017attention} to learn diverse language priors from large-scale corpora. Models such as BERT \citep{devlin2018bert}, GPT \citep{radford2018gpt} and BART \citep{lewis2019bart} have achieved new state-of-the-art performances on abstractive summarization \citep{liu2019bertabs,lewis2019bart,zhang2019pegasus,shi-etal-2019-leafnats,fabbri-etal-2019-multi}. These models often finetune pre-trained Transformers with supervised summarization datasets that contain pairs of source and summary.

However, encoder-decoder architectures widely used in abstractive summarization systems are inherently difficult to control and prone to hallucination \citep{vinyals2015neural,koehn2017six,lee2018hallucinations}, and often leads to factual inconsistency: the system-generated summary is fluent but unfaithful to the source \citep{cao2018faithful}. Studies have shown that  8\% to 30\% system-generated abstractive summaries have factual errors \citep{falke2019ranking,kryscinski2019evaluating} that cannot be discovered by ROUGE scores. Recent studies have proposed new methods to ensure factual consistency in summarization. \citet{cao2018faithful,zhu2020boosting} propose RNN-based and Transformer-based decoders that attend to both source and extracted knowledge triples, respectively. \citet{li2018ensure} propose an entailment-reward augmented maximum-likelihood training objective, and \citet{falke2019ranking} proposes to rerank beam results based on entailment scores to the source.

Our fact correction models are inherently different from these models, as we focus on post-correcting summaries generated by any model. Our models are trained with the objective of predicting masked entities identified for fact correction (Figure \ref{fig:span_selection_example}), and learn to fill in the entity masks of any system-generated summaries with single or multi-span selection mechanism (Figure \ref{fig:system}). The most similar work to ours is proposed concurrently by \citet{cao-2020-fact}, where they fine-tune a BART \citep{lewis2019bart} model on distant supervision examples and use it as a post-editing model for factual error correction.

\section{Multi-Fact Correction Models}

\begin{figure*}[ht!]
    \centering
    \includegraphics[width=\textwidth]{figs/system1.png}

    \caption{Model architecture (Left: QA-span fact correction model. Right: Auto-regressive fact correction model).}
    \label{fig:system}
\end{figure*}

In this section, we describe two models proposed for factual error correction: $(i)$ QA-span Fact Correction model, and $(ii)$
Auto-regressive Fact Correction model. As both methods rely on span selection with different masking and prediction strategies, we call them SpanFact collectively.

\subsection{Problem Formulation}
Let $(x, y)$ be a document-summary pair, where $x = (x_1, \ldots, x_M)$ is the source sequence with $M$ tokens, and $y = (y_1,\ldots, y_N )$ is the target sequence with $N$ tokens. An abstractive summarization
model aims to model the conditional likelihood $p(y|x)$, which can be factorized into a product $p(y|x) = \prod_{t=1}^{T} p(y_t|y_{1.\ldots,t-1}, x)$, where $y_{1.\ldots,t-1}$ denote the preceding tokens before position $t$. The conditional maximum-likelihood objective ideally requires summarization models to not only optimize for informativeness but also correctness. However, in reality this often fails as the models have a high propensity for leaning towards informativeness than correctness \citep{li2018ensure}.

Suppose a summarization system generates a sequence of tokens $y' =(y'_1, \ldots, y'_N)$ to form a summary. Our factual correction models aim to edit an informative-yet-incorrect summary into $y'' =(y''_1, \ldots, y''_K)$ such that
\begin{align}
   f(x,y'') > f(x,y')\,,
\end{align}
where $f$ is a metric measuring factual consistency between the source and system summary.

\subsection{Span Selection Dataset} \label{subsec:span_selection}

Our fact correction models are inspired by the \textit{span selection} task, which is often used in reading comprehension tasks such as question answering. Figure~\ref{fig:span_selection_example} shows examples of the \textit{span selection} datasets we created for training our QA-span and auto-regressive fact correction models, respectively. The \textit{query} is a reference summary masked with one or all entities,\footnote{In this work, we use SpaCy NER tagger \citep{honnibal2017spacy} to identify entities for data construction.} and the passage is the corresponding source document to be summarized. If an entity appears multiple times in the source document, we rank them based on the fuzzy string-matching scores (a variation of Levenshtein distance) between the query sentence and the source sentence containing the entity. Our models explicitly learn to predict the span of the masked entity rather than pointing to a specific token as in Pointer Network \citep{vinyals2015pointer}, because the original tokens and replaced tokens often have different lengths.

Our QA-span fact correction model iteratively mask and replace one entity at a time, while the auto-regressive model  masks  all the entities simultaneously, and replace them in an auto-regressive  fashion from left to right. Figure~\ref{fig:system} shows an overview of our models. Comparing the two models, the QA-span fact correction model works better when only a few errors exist in the draft summary, as the prediction of each mask is relatively independent of each other.
On the other hand, the auto-regressive fact correction model starts with a skeleton summary that has all the entities masked, which is often more robust when summaries contain many factual errors.

\subsection{QA-Span Fact Correction Model}
\label{subsec:iter_model}
In the iterative setting, our model aims to conduct entity correction by answering a query that contains only one mask at a time. Suppose a system summary has $T$ entities. At time step $i$, we mask the $i$-th entity and use this masked sequence as the query to our QA-span model. The prediction is placed into the masked slot in the query to generate an updated system summary to be used in the next step.

Given the source text $x$ and a masked query $q = (y'_1, \ldots, \texttt{[MASK]},\ldots y'_m)$, our iterative correction model aims to predict the answer span via modeling $p(i=\textnormal{start})$ and $p(i=\textnormal{end})$. For span selection, we use the BertForQuestionAnswering\footnote{https://github.com/huggingface/transformers} model, which adds two separate non-linear layers on top of Transformers as pointers to the start and end token position for the answer. We initialize the fact-correction model from a pre-trained BERT model \citep{devlin2018bert}, and perform finetuning with the span selection datasets we created from the summarization datasets (Figure \ref{fig:span_selection_example}).

The input to the BERT model is a concatenation of two segments: the masked query $q$ and the source $x$, separated by special delimiter markers as ($\texttt{[CLS]}, q, \texttt{[SEP]}, x$). Each token in the sequence is assigned with three embeddings: token embedding, position embedding, and segmentation embedding.\footnote{The segmentation embedding is used to distinguish the query (with two special tokens \texttt{[CLS]} and \texttt{[SEP]}) and the source in our models.} These embeddings are summed into a single vector and fed to the multi-layer Transformer model:
\begin{align}
    \tilde{{\boldsymbol h}}^l &= \textnormal{LN}({\boldsymbol h}^{l-1}+\textnormal{MHAtt}({\boldsymbol h}^{l-1}))\,, \label{eq:transformer1} \\
    {{\boldsymbol h}}^l &= \textnormal{LN}(\tilde{{\boldsymbol h}}^{l}+\textnormal{FFN}(\tilde{{\boldsymbol h}}^l))\,, \label{eq:transformer2}
\end{align}
where ${\boldsymbol h}^0$ are the input vectors, and $l$ represents the depth of stacked layers. LN and MHAtt are layer normalization and multi-head attention operations \citep{vaswani2017attention}. The top layer provides the hidden states for the input tokens with rich contextual information.
The start (s) and end (e) of the answer span are predicted as:
\begin{align}
    a^{start}_i &=p(i=s) = \frac{\textnormal{exp}(q^s_i)}{\sum_{j =0}^{H-1} \textnormal{exp}(q^s_j)}\,, \label{eq:start1} \\
    a^{end}_i &=p(i=e) = \frac{\textnormal{exp}(q^e_i)}{\sum_{j = 0}^{H-1} \textnormal{exp}(q^e_j)}\,, \label{eq:end1} \\
  q^s_i &=\textnormal{ReLU}({\boldsymbol w}^\top_{s}{\boldsymbol h}_i+b_{s})\,, \label{eq:start2} \\
    q^e_i &=\textnormal{ReLU}({\boldsymbol w}^\top_{e}{\boldsymbol h}_i+b_{e})\,, \label{eq:end2}
\end{align}
where $H$ is the number of encoder's hidden states, ${\boldsymbol w}_s,{\boldsymbol w}_e \in \mathbb{R}^d$ and $b_s,b_e \in \mathbb{R}$ are trainable parameters. The final span is selected based on the argmax of Eqn. (\ref{eq:start1}) and (\ref{eq:end1}) with the constraint of  $p_{start}<p_{end}$ and  $p_{end}-p_{start}<k$.

\subsection{Auto-regressive Fact Correction Model}
One disadvantage of the QA-style span-prediction strategy is that if the sequence contains too many factual errors, masking out one entity at a time may lead to highly erroneous skeleton summary to start with. The model might be making predictions on top of wrong entities from later in the sequence. Masking one entity at a time is essentially a greedy local method that is prone to error accumulation. To alleviate this issue, we
 propose a new sequential fact correction model to handle errors in a more global manner with beam search. Specifically, we mask out all the entities simultaneously, and use a novel \textit{auto-regressive} span-selection decoder to predict fillers for the multiple masks sequentially. By doing this, we assume dependency between the masks: the earlier predicted entities will be used as corrected context for better predictions in the later steps.

Given a source text $x = (x_1, \ldots, x_n)$ and a draft summary  $(y'_1,\ldots y'_m)$. Our model first masks out all the entities (with $T$ masks), and leaves a skeleton summary as the query  $q = (y'_1,\ldots, \texttt{[MASK]}_1,\ldots,\texttt{[MASK]}_T \ldots y'_m)$. Then, we concatenate the query $q$ with the source $x$  (similar to Section \ref{subsec:iter_model}) as inputs to the encoder. The inputs are fed into BERT to obtain contextual hidden representations.

We then \textit{select} the encoder's hidden states for the $T$ masks ${\boldsymbol h}_{y'_{mask_1}}, \ldots, {\boldsymbol h}_{y'_{mask_T}}$ as partial input to an auto-regressive Transformer-based decoder.
Unlike generation tasks that require an \texttt{[EOS]} token to indicate the end of decoding, our decoder runs $T$ steps to predict the answer spans for these $T$ masks. At step $t$, we first fuse the hidden representation ${\boldsymbol h}_{\texttt{[MASK]}_t} \in \mathbb{R}^d$ of the $t$-th \texttt{[MASK]} token and previously predicted entity representation ${\boldsymbol s}^{ent}_{t-1} \in \mathbb{R}^d$:
\begin{equation}
    \label{eq:decoder_input}
    {\boldsymbol z}_t = {\bf W} [{\boldsymbol h}_{\texttt{[MASK]}_t};{\boldsymbol s}^{ent}_{t-1}]\,,
\end{equation}
where ${\bf W} \in \mathbb{R}^{2d \times d}$, ${\boldsymbol s}^{ent}_0 = {\boldsymbol h}_{\texttt{[CLS]}}$ (the representation of \texttt{[CLS]} token), and $[;]$ denotes vector concatenation.

The input ${\boldsymbol z}_t$ is then fed to the Transformer decoder (as in Eqn. (\ref{eq:transformer1}) and (\ref{eq:transformer2}))  to generate the decoder's hidden state ${\boldsymbol h}'_t$ at time step $t$. Based on ${\boldsymbol h}'_t$, we use a two-pointer network to predict the start and end positions of the answer entity in the source (encoder's hidden states). This is achieved with cross-attention of ${\boldsymbol h}'_t$ w.r.t. the encoder's hidden states, similar to Eqn (\ref{eq:start1}) and (\ref{eq:end1}). This operation results in two distributions over the encoder's hidden states for the start and end span positions. The final prediction of the start and end positions for mask $t$ is obtained by taking the argmax\footnote{The argmax is used for selecting the start and end indexes for the answer span, and the softmax is used for computing the loss for back-propagation.} over the pointer position distributions:
\begin{align}
    p_{start} &= \arg\max(a^{start}_1,...,a^{start}_M)\,, \label{eq:argmax1} \\
    p_{end} &= \arg\max(a^{end}_1,...,a^{end}_M)\,, \label{eq:argmax2}
\end{align}
under the constraint that $p_{start}<p_{end}$ and  $p_{end}-p_{start}<k$.

Based on the start and end positions for the predicted entity, we can obtain the predicted entity representation at time step $t$ as the mean over the in-span encoder's hidden states:
\begin{equation}
    \label{eq:entity_forming}
    {\boldsymbol s}^{ent}_t = \text{Mean-Pool}(\{{\boldsymbol h}_{p_{start}}, {\boldsymbol h}_{p_{end}}\})\,,
\end{equation}
which is used as the input for the next step of decoding.
 It is worth noting that although the argmax operations in Eqn. (\ref{eq:argmax1}) and (\ref{eq:argmax2}) are non-differentiable, the model is trained based on the start and end positions of the ground-truth answer w.r.t. the start and end logits in Eqn. (\ref{eq:start1}) and (\ref{eq:end1}), which makes the gradient back-propagates to the encoder. Meanwhile, the encoder's hidden states used to compose  ${\boldsymbol s}^{ent}_i$ in Eqn. (\ref{eq:entity_forming}) also carry the gradients. During inference, beam search is used to find the best sequence of predicted spans in the source to replace the masks.

Compared to the conventional Pointer Network \citep{vinyals2015pointer,see2017get} that only points to one token at a time, our sequential span selection decoder has the flexibility to replace a mask by any number of entity tokens, which is often required in summary factual correction.
\section{Experiment}
In this section, we present our results on using SpanFact for multiple summarization datasets.

\subsection{Experimental Setup}

Training data for our fact correction models are generated as described in Section \ref{subsec:span_selection} on CNN/DailyMail \cite{hermann2015teaching}, XSum \citep{narayan2018don} and Gigaword \citep{graff2003english,rush2015neural}. The statistics of these three dataset are provided in Table \ref{tab:datasets}.
During training, if an entity does not have a corresponding span in the source, we point the answer span to the \texttt{[CLS]} token. During inference, if the answer span predicted is the \texttt{[CLS]} token, we replace back the original masked entity.

Our fact correction models are implemented via the Huggingface Transformers library \citep{Wolf2019HuggingFacesTS} in PyTorch \citep{paszke2017automatic}. We initialize all encoder models with the checkpoint of an uncased, large BERT model pre-trained on English data and SQuAD for all experiments. Both source and target texts were tokenized with BERT's sub-words tokenizer. The max sequence length is set to 512 for the encoder. We use a shallow Transformer decoder (L=2) for the auto-regressive span selection decoder, as the pre-trained BERT-large encoder is already robust for selecting right spans in the single-span selection task with only two pointers (Section \ref{subsec:iter_model}). The Transformer decoder has 1024 hidden units and the feed-forward intermediate size for all layers is 4,096.

All models were finetuned on our span prediction data for 2 epochs with batch size 12.  AdamW optimizer \citep{loshchilov2017decoupled} with $\epsilon=$1e-8 and an initial learning rate 3e-5 is used for training. Our learning rate schedule follows a linear decay scheduler with warmup=10,000. During inference, we use beam search with $b=5$ and $k=10$ (constraint for the distance between the start and end pointer). The best model checkpoints are
chosen based on performance on the validation set.  Experiments are conducted using 4 Quadro RTX 8000 GPUs with 48GB of memory.

\begin{table}[t!]
\resizebox{\columnwidth}{!}{%
\begin{tabular}{l|cccc}
\toprule
Datasets  & \# docs (train/val/test) & doc len. & summ. len. & \# mask \\ \midrule
CNN       & 90,266/1,220/1093        & 760.50          & 45.70  &     4.40      \\
DailyMail & 196,961/12,148/10,397    & 653.33          & 54.65  &    5.38       \\
XSum      & 204,045/11,332/11,334    & 431.07          & 23.26  &       2.28    \\
Gigaword  & 3,803,957/189,651/1,951  & 31.3            & 8.3    &     1.97      \\ \bottomrule
\end{tabular}
}
\caption{Comparison of summarization datasets on train/validation/test set splits, average document and summary length (numbers of words). We also report the average number of entity masks on the reference summary for each dataset.} \label{tab:datasets}
\end{table}

\subsection{Evaluation Metrics}
We use three automatic evaluation metrics to evaluate our models. The first is ROUGE \citep{lin-2004-rouge}, the standard summarization quality metric, which has high correlation with summary informativeness in the news domain \citep{kryscinski2019evaluating}.

Since ROUGE has been criticized for its poor correlation with factual consistency  \citep{kryscinski2019evaluating,wang2020asking}, we use two additional automatic metrics that specifically focus on factual consistency: FactCC \citep{kryscinski2019evaluating} and QAGS  \citep{wang2020asking}. FactCC is a pre-trained binary classifier that evaluates the factuality of a system-generated summary by predicting whether it is consistent or inconsistent w.r.t. the source. This classifier was trained on adversarial examples obtained by heuristically injecting noise into reference summaries.

\begin{table}[t!]
\centering
\resizebox{\columnwidth}{!}{%
  \begin{tabular}{ l | c|c| c c c }
    \toprule
     \multirow{2}{*}{Datasets} & QGQA & FactCC& \multicolumn{3}{c}{ROUGE} \\
    & & sent & 1 & 2 & L \\ \midrule
    Bottom-up & 70.58 & 73.66  &\textbf{41.24} & \textbf{18.70}& \textbf{38.15} \\
    Split Encoders &70.22  & 73.15 & 39.78 & 17.87 & 37.01\\
    QA-Span & \textbf{74.15} & \textbf{76.60}  & 41.13 &18.58& 38.04 \\
    Auto-regressive & 72.78& 74.42  & 41.04 &18.48& 37.95 \\
    \midrule
     BertSumAbs &  72.68& 76.76 &\textbf{ 41.67} & \textbf{19.46}& \textbf{38.79}  \\
     Split Encoders &72.13  & 76.43 & 40.21 & 18.38 & 37.87\\
    QA-Span & \textbf{74.93} & \textbf{78.69}  & 41.53 &19.28&38.65   \\
    Auto-regressive & 74.34 & 77.58 & 41.45 &19.18  & 38.57  \\

    \midrule
    BertSumExtAbs & 74.15  &79.22  & \textbf{41.87}& \textbf{19.41}&\textbf{38.94}  \\
    Split Encoders & 73.67 & 79.12 & 40.55 &    18.41&  38.45 \\
    QA-Span & \textbf{75.94} &  \textbf{80.97} & 41.75 & 19.27 &38.81  \\
    Auto-regressive &75.19  & 79.89&41.68  &19.16  & 38.74  \\

      \midrule
    TransformerAbs & 73.79  & 80.51 & \textbf{39.96} & \textbf{17.63} &\textbf{36.90} \\
    Split Encoders & 73.11& 79.54 & 38.83 &16.51 & 35.71 \\
    QA-Span & \textbf{75.70} & \textbf{82.82} & 39.87  & 17.50 & 36.80  \\
    Auto-regressive & 75.21& 81.64& 39.81 &17.40  &36.75  \\

    \bottomrule
  \end{tabular}
  }
  \caption{Factual correctness scores and ROUGE scores on CNN/DailyMail test set.}\label{table:cnndm_results}
\end{table}

In addition, very recent work proposed QA-based models for factuality evaluation \citep{wang2020asking,durmus2020feqa,maynez2020faithfulness}, and \citet{wang2020asking} showed that their evaluation models have higher correlation with human judgements on factuality when compared with FactCC \citep{kryscinski2019evaluating}. We thus include our re-implementation of a question generation and question answering model (QGQA) following \citet{wang2020asking} as an evaluation metric for factuality.\footnote{We were not able to obtain any of the QA evaluation model or code from \citet{wang2020asking,durmus2020feqa,maynez2020faithfulness} as the authors are still in the stage of making the code public. We used pre-trained UniLM model for question generation (QG) and BertForQuestionAnswering model for question answering (QA). The QG model is fine-tuned on NewsQA \citep{trischler2017newsqa} with entity-answer conditional task \citep{wang2020asking}, and the QA model is pre-trained on SQuAD 2.0 \citep{rajpurkar2018know}.}  This model generates a set of questions based on the system-generated summary, and then answers these questions using either the source or the summary to obtain two sets of answers. The answers are compared against each other using an answer-similarity metric (token-level F1), and the averaged similarity metric over all questions is used as the QGQA score. Answers generated from a highly faithful system summary should be similar to those generated from the source.

\subsection{Baselines}
We compare against the following abstractive summarization baselines. On CNNDM and XSum, we use  BertSumAbs, BertSumExtAbs and TransformerAbs \citep{liu2019bertabs}. In addition, we also compare with Bottom-up \citep{gehrmann2018bottom}. On Gigaword, we use the pointer-generator \citep{see2017get}, base and full GenParse models \citep{song2020joint} for comparison. For the factual correction baseline, we compare with the Two-encoder Pointer Generator\footnote{\url{https://github.com/darsh10/split_encoder_pointer_summarizer}} (Split Encoder) \citep{shah2020automatic}, which employs a similar setting to ours for masking entities w.r.t. the source, and uses dual encoders to copy and generate from both the source and the masked query for fact update. Compared to our span selection models that can fill in the mask with any number of tokens, their models aim to regenerate the mask query based on the source. In other words, their decoder regenerates the whole sequence token by token with a pointer-generator, which inherits the backbone of generative models that suffer from hallucination.

\begin{table}[t!]
\centering
\resizebox{\columnwidth}{!}{%
  \begin{tabular}{ l | c|c| c c c }
    \toprule
     \multirow{2}{*}{Datasets} & QGQA & FactCC& \multicolumn{3}{c}{ROUGE} \\
    & & sent& 1 & 2 & L \\
    \midrule
    BertSumAbs & 12.78 & 23.60 &\textbf{37.78} & \textbf{15.84}& \textbf{30.50} \\
     Split Encoders  & \textbf{24.65} & 24.19 & 34.22 & 13.76 & 27.86 \\
    QA-Span & 23.85 & 23.90 & 36.44 &14.56& 29.38\\
    Auto-regressive & 24.14  &  \textbf{25.08} &  36.24 & 14.37 &29.22    \\

    \midrule
    BertSumExtAbs & 13.62  & 23.12 & \textbf{38.25}& \textbf{16.16}&\textbf{30.87}  \\
    Split Encoders &\textbf{25.17} &    24.67 & 35.66 & 13.98& 27.93\\
    QA-Span & 24.52  &23.96 &  36.86 &14.82&29.70   \\
    Auto-regressive &24.96 & \textbf{25.10} &36.67 & 14.64 & 29.53  \\

     \midrule
    TransformerAbs & 7.00  & 24.15 & \textbf{29.86}& \textbf{10.05} & \textbf{23.78} \\
    Split Encoders & 11.77 & 24.78  & 28.14  & 8.65& 22.70\\
    QA-Span & 12.88& 24.44 & 29.51  & 9.67 & 23.45  \\
    Auto-regressive &\textbf{ 13.89} & \textbf{25.75} &29.45 &9.59&23.40   \\

    \bottomrule
  \end{tabular}
  }
  \caption{Factual correctness scores and ROUGE scores on XSum test set.}\label{table:xsum_results}
\end{table}
\subsection{Experimental Results}

Tables~\ref{table:cnndm_results}, \ref{table:xsum_results}, and \ref{table:gigaword_result} summarize the results on the CNN/DailyMail, XSum and Gigaword datasets, respectively. Each block in the tables compares the original summarization model's output with the corrected outputs obtained by our baseline and proposed models.

On CNN/DailyMail (Table \ref{table:cnndm_results}), our correction models significantly boost factual consistency measures (QGQA and FactCC) by large margins, with only small drops on ROUGE. This shows our models have the ability to improve the correctness of system-generated summaries without sacrificing informativeness. When comparing our two proposed models, we observe that the QA-span model performs better than the auto-regressive model.
This is expected as CNN/DailyMail's reference summaries tend to be more extractive \citep{see2017get}, and summarization models tend to make few errors per summary \citep{narayan2018don}. Thus, the iterative procedure of the QA-span model is more robust with high precision as it has more correct context from the query, with only minimum negative influence from other concurrent errors.
This is also reflected in the high scores of QGQA and FactCC across all the models we tested. Since QGQA and FactCC are based on comparing system-generated summary w.r.t. the source text, high score means high semantic similarity between system summary to the source.

\begin{table}[t!]
\centering
\resizebox{\columnwidth}{!}{%
  \begin{tabular}{ l | c|c| c c c }
    \toprule
     \multirow{2}{*}{Datasets} & QGQA & FactCC& \multicolumn{3}{c}{ROUGE} \\
    & & sent& 1 & 2 & L \\
    \midrule
    GenParse (base)      &  52.63  & 46.07 & \textbf{35.23}& \textbf{17.11} & \textbf{32.88} \\
    Split Encoders & 63.60 &  48.22 & 34.32 &  17.01 & 31.98 \\
    QA-Span  &  \textbf{66.47}  &  \textbf{52.17}   & 34.38 & 16.50 & 32.07  \\
    Auto-regressive &  64.77  &  48.95  & 33.97 & 16.08 & 31.70   \\

    \midrule
     GenParse (full)        &  55.47  & 48.44  & \textbf{36.61} &\textbf{18.85}  & \textbf{34.32} \\
    Split Encoders & 65.88& 52.11 & 35.01 & 17.54 & 32.96\\
    QA-Span  & \textbf{67.12}  & \textbf{54.59} &35.66  & 18.01  & 33.35   \\
    Auto-regressive & 66.48  & 52.18 & 35.04  & 17.27 & 32.75  \\

     \midrule
    Pointer Generator         & 45.98  & 43.62 & \textbf{34.19}  &\textbf{16.29}  & \textbf{31.81} \\
    Split Encoders & 59.46 & 48.32 & 33.11 & 15.63& 30.67 \\
    QA-Span  &  58.25  &    45.62  & 33.30 & 15.70 &  30.95  \\
    Auto-regressive  & \textbf{60.66}  & \textbf{49.82} &32.86  & 15.22 &  30.51\\
    \bottomrule
  \end{tabular}
  }
  \caption{Factual correctness scores and ROUGE scores on Gigaword test set.}\label{table:gigaword_result}
\end{table}

On XSum (Table~\ref{table:xsum_results}) and Gigaword (Table~\ref{table:gigaword_result}), both of our correction models boost factual consistency measures by large margins with a slight drop in ROUGE (-0.5 to -1.5) on average. This is still encouraging, as abstractive summarization models that use complex factual controlling components for generation often have drops of 5-10 ROUGE points \citep{zhu2020boosting}.

We also notice that the QGQA and FactCC scores of all summarization models are lower than that on CNN/DailyMail. The scores are especially low on XSum. This is likely due to the data construction protocol of XSum, where the first sentence of a source document is used as the summary and the remainder of the article is used as the source. As a result, many entities that appear in the reference summary never appear in the source, which may cause abstractive summarization models to hallucinate severely with many factual errors \citep{maynez2020faithfulness}.  As the system summaries often contain many errors, our QA-span model that relies on answering a single-mask query often has the wrong context to condition on at each step, which negatively affects the performance of this model. In contrast, the strategy of masking all the entities would provide the auto-regressive model a better query for entity replacement. We can observe in Table \ref{table:xsum_results} that the auto-regressive model performs better than the QA-span model on XSum.

\subsection{Human Evaluation}

\begin{table}[t!]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{c|cc|c}
    \toprule
         BertAbs & Better & Worse & Same \\\midrule
         QA-Span vs. original & \textbf{28.6\%} &  18.7\% & 52.7\% \\
         Auto-regressive vs. original & \textbf{31.3\%}  & 16.7\% & 52\%\\
        QA-Span vs. Auto-regressive & 26\%  & 27.3\% & 46.7\%\\
         \midrule

         TransformerAbs & Better &   Worse & Same\\\midrule
        QA-Span vs. original & \textbf{38\%}  & 11.3\% & 40.7\% \\
         Auto-regressive vs. original & \textbf{36\%}  & 19.3\% & 44.7\%\\
         QA-Span vs. Auto-regressive & 32.7\%  & 28\% & 39.3\%\\
         \midrule

         Bottom-up & Better & Worse  & Same \\\midrule
         QA-Span vs. original & \textbf{34\%} & 12\%  & 54\% \\
         Auto-regressive vs. original & \textbf{31.4\%}  & 13.3\% & 55.3\% \\
         QA-Span vs. Auto-regressive & 41.3\%  & 32\% & 26.7\%\\
         \bottomrule

    \end{tabular}
    }
    \caption{Human evaluation results on pairwise
comparison of factual correctness on 450 ($9 \times 50$) randomly sampled articles.}
    \label{tab:human_eval}
\end{table}

To provide qualitative analysis of the proposed
models, we conduct human evaluation on pairwise comparison of CNN/DailyMail summaries enhanced by different correction strategies.
We select three state-of-the-art abstractive summarization models as the backbones, and collect three sets of pairwise summaries for each setting: $(i)$ Original vs. QA-Span corrected; $(ii)$ Original vs. Auto-regressive corrected; $(iii)$ QA-Span corrected vs. Auto-regressive corrected. Nine sets of 50 randomly selected samples (total 450 samples) are labeled by AMT tuckers. For each pair (in anonymized order), three annotators from Amazon Mechanical Turk (AMT) are asked to judge which is more factually correct based on the source document.
As shown in Table \ref{tab:human_eval}, summaries from our two models are
chosen more frequently as the
factually correct one compared to the original. Between the two correction models, the preferences are comparable.

In addition, we also test our fact correction models on  the FactCC test set provided by \citet{kryscinski2019evaluating} and manually checked the outputs. Table \ref{tab:FactCC_result} shows the results of the original summaries and the summaries corrected by our models in terms of automatic fact evaluation and our manual evaluation.  Among 508 system-generated summary sentences, 62 were incorrect. The QA-span model was able to correct 18 out of 62 right, and the auto-regressive model was able to correct 16 out of 62. Among the 446 sentences that are labeled as correct by the annotators in \citet{kryscinski2019evaluating}, our two models made 3 and 4 wrong changes in the entities, respectively,\footnote{This excludes the cases where the model would change a person's full name by last name or break the fluency due to SpaCy NER errors.} while keeping most of the entities unchanged or changed with equivalent entities.

\begin{table}[t!]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{c|c|c|c}
    \toprule
         FactCC Dataset& FactCC Score & QAQG & Human Eval\\\midrule
         Before Corr.& 84.89 & 88.65 & 87.79 \\
         QA-span & \textbf{86.08} & \textbf{91.07} & \textbf{90.74} \\
         Auto-regressive & 85.96 & 90.51 & 90.35\\
         \bottomrule

    \end{tabular}
    }
    \caption{Test results on the human annotated dataset provided by FactCC \citep{kryscinski2019evaluating}.  We show the performance comparisons of the original summaries and the summaries corrected by SpanFact.}
    \label{tab:FactCC_result}
\end{table}

\section{Conclusion}
We present SpanFact, a suite of two factual correction models that use span selection mechanisms to replace one or multiple entity masks at a time. SpanFact can be used for fact correction on any abstractive summaries. Empirical results show that our models improve the factuality of summaries generated by state-of-the-art abstractive summarization systems without a huge drop on  ROUGE scores.  For future work, we plan to apply our method for other type of spans, such as noun phrases, verbs, and clauses.

\section{Appendix}

\begin{table*}[ht!]
\resizebox{\textwidth}{!}{
\begin{tabular}{  m{5em} |  m{14cm}  }
\hline
CNNDM Source& Jerusalem (CNN)The flame of remembrance burns in Jerusalem, and a song of memory haunts Valerie Braham as it never has before. This year, \textcolor{blue}{Israel}'s Memorial Day commemoration is for bereaved family members such as Braham. ``Now I truly understand everyone who has lost a loved one," Braham said. Her husband, \textcolor{blue}{Philippe Braham}, was one of 17 people killed in January's terror attacks in Paris. He was in a kosher supermarket when a gunman stormed in, killing four people, all of them Jewish. \\
\hline
 System Summary & \textcolor{red}{france}'s memorial day commemoration is for bereaved family members as braham. \textcolor{red}{valerie braham} was one of 17 people killed in january's terror attacks in paris.\\
\hline
Corrected by SpanFact & \textcolor{orange}{israel}'s memorial day commemoration is for bereaved family members as braham. \textcolor{orange}{philippe braham} was one of 17 people killed in january's terror attacks in paris.\\
\hline \hline
CNNDM Source& (CNN)If I had to describe the U.S.-Iranian relationship in one word it would be ``overmatched." ... \textcolor{blue}{America} is alienating some of our closest allies because of the Iran deal, and Iran is picking up new ones and bolstering relations with old ones who are growing more dependent because they see Iran\'s power rising...  \\
\hline
 System Summary & \textcolor{red}{iran} is alienating some of our closest allies because of the iran deal, and iran is picking up new ones.\\
\hline
Corrected by SpanFact & \textcolor{orange}{america} is alienating some of our closest allies because of the iran deal, and iran is picking up new ones.\\
\hline \hline
CNNDM Source& (CNN)A North Pacific gray whale has earned a spot in the record books after completing the longest migration of a mammal ever recorded. The whale, named Varvara, swam nearly 14,000 miles (22,500 kilometers), according to a release from \textcolor{blue}{Oregon State University}, whose scientists helped conduct the whale-tracking study. Varvara, which is Russian for ``Barbara," left her primary feeding ground off \textcolor{blue}{Russia\'s Sakhalin Island} to cross the  Pacific Ocean and down the West Coast of the United States to Baja, Mexico... \\
\hline
 System Summary & a north pacific gray whale swam nearly 14,000 miles from \textcolor{red}{oregon state university}.\\
\hline
Corrected by SpanFact & a north pacific gray whale swam nearly 14,000 miles from \textcolor{orange}{russia\'s sakhalin island}.\\
\hline \hline
CNNDM Source& Sanaa, Yemen  (CNN)Saudi airstrikes over Yemen have resumed once again, \textcolor{blue}{two days after} Saudi Arabia announced the end of its air campaign. The airstrikes Thursday targeted rebel Houthi militant positions in three parts of Sanaa, two Yemeni Defense Ministry officials said. The attacks lasted four hours. ... The Saudi-led coalition said a new initiative was underway, Operation Renewal of Hope, focused on the political process. But \textcolor{blue}{less than 24 hours later}, after rebel forces attacked a Yemeni military brigade, the airstrikes resumed, security sources in Taiz said.\\
\hline
 System Summary & the attacks lasted four hours, \textcolor{red}{two days after} rebel forces attacked yemeni military troops..\\
\hline
Corrected by SpanFact & the attacks lasted four hours, \textcolor{orange}{less than 24 hours after} rebel forces attacked yemeni military troops.\\
\hline \hline
CNNDM Source& Boston (CNN)When the bomb went off, \textcolor{blue}{Steve Woolfenden} thought he was still standing. That was because, as he lay on the ground, he was still holding the handles of his son's stroller. He pulled back the stroller's cover and saw that his son, \textcolor{blue}{Leo}, 3, was conscious but bleeding from the left side of his head. Woolfenden checked Leo for other injuries and thought, ``Let's get out of here." ...\\
\hline
 System Summary & \textcolor{red}{steve woolfenden}, 3, was conscious but bleeding from the left side of his head.\\
\hline
Corrected by SpanFact & \textcolor{orange}{leo}, 3, was conscious but bleeding from the left side of his head.\\
\hline
\end{tabular}
}

\caption{Examples of factual error correction on FactCC dataset (a human annotated  subset from CNNDM obtained by \citet{kryscinski2019evaluating}).  Factual
errors by abstractive summarization system are marked in red. Corrections made by the proposed SpanFact models are marked in orange.}
\end{table*}

\section*{Acknowledgments}

This  research  was  supported  in  part  by  Microsoft
Dynamics 365 AI Research and  the Canada CIFAR AI Chair program.
We would like to thank the reviewers for their valuable comments and special thanks to Yuwei Fang and other members of the Microsoft
Dynamics 365 AI Research team for the feedback and suggestions.

\section{Dataset Quality Analysis}

\subsection{Abstractive or Extractive?}
\yaolu{Table: Ranking of models output in terms of generation quality}. It is obvious that due to the writing style of related work, the abstractive model consistently outperformed extractive model in terms of human quality ranking.

\section{Related Work}

\subsection{Error analysis on factual errors in factCC}

redpajama_set_name: RedPajamaArXiv


For more than a century, Harley-Davidson built its iconic motorcycles here in America.
It took President Trump less than a month to change that.
Responding to Trump's idiotic, unnecessary and ultimately unwinnable trade war on our allies, the European Union retaliated by imposing $3.2 billion in tariffs on bourbon, jeans, and motorcycles. Harley-Davidson said the tariffs will add $2,200 to the cost of each of its motorcycles sold in Europe and cost the company $30 to $45 million this year.
It will also cost American jobs. Harley-Davidson responded to the EU tariffs by announcing plans to move production of motorcycles bound for Europe overseas. Harleys for sale in the EU will now be assembled in Brazil, India and Thailand.
In January, Harley-Davidson closed a plant in Kansas City, Missouri, and laid off 800 employees. As a result, Syncreon, one of its suppliers there, will lay off 207 workers in August.
If this is what Trump means by "winning," I'm just as sick of it as he predicted. How about you?


We're now a full year ― and more than 500 approval polls ― into Donald Trump's presidency. Here's what those polls show about how Americans view him so far, and what might happen in 2018.
1. Trump's approval rating started off historically bad. It's only gotten worse.
Trump marks his first anniversary as president more unpopular than he was when he took the job. That's not unusual ― many first-term presidents see their numbers drop.
What's unusual about Trump is just how little goodwill he began with. Unlike his recent predecessors, he took office with Americans equally split between approving and disapproving of his job performance, according to HuffPost Pollster's average of public polling ― and that marked a relative high point.
Within a month, Trump saw his disapproval ratings begin to rise as those who'd previously been neutral became increasingly less likely to give him the benefit of the doubt.
Trump's first months were marked by "no honeymoon period to speak of and approval ratings far worse than any president has received this early in his tenure" Gallup's Jeffrey M. Jones wrote in April, after Trump posted the worst first-quarter ratings for any president elected to his first term since World War II.
A year later, surveys are consistent in showing that a majority of the nation is unhappy with the president's performance. About 55 percent disapprove of the job he's done, per Pollster's average as of Saturday morning, while just under 41 percent approve.
That's an improvement from December, but these are still easily the worst numbers for a first-year president since pollsters first began keeping track. Trump's standing is similar to President Richard Nixon's months into the Senate hearings on the Watergate scandal. And, according to data collected by Charles Franklin, the director of the Marquette Law School poll, while individual pollsters' assessments of Trump vary, nearly all find him lagging behind his immediate predecessor, former President Barack Obama.
2. Trump's numbers are relatively stable.
In the last six months, Trump's approval rating has remained between 37 and 42 percent in Pollster's average, neither soaring nor dramatically crashing.
But it may also mean there's little room for him to rebound to a neutral or positive rating. Most Americans who disapprove of Trump say that they don't see anything positive about his record, and that there's almost nothing he can do to change their minds.
And although it's impossible to pinpoint the exact causes of the smaller approval fluctuations for Trump in 2017, there are some clues. Trump saw distinct dips in support following Republican failures to pass a health care repeal bill, while months of stories about his campaign's relationship to Russia seemingly did less to move the needle ― possibly in part because his supporters were more troubled by the former than the latter.
By contrast, his numbers ticked up during last fall's hurricane season, perhaps both because he saw some of his best ratings as president for his handling of Hurricanes Harvey and Irma, and because the weather-focused news cycle kept his name out of the headlines more than usual.
3. Opinions of Trump are deeply polarized, but he inspires more robust opposition than support.
Survey after survey conducted last year showed Americans to be immensely polarized on practically every political issue, with the clearest example being opinions of Trump himself. Opinions on many of the actions Trump has taken ― for instance, concerns about his administration's relationship with Russia ― remain more or less intractably divided along party lines. That polarization is another factor in the stability of Trump's ratings.
Americans' willingness to support a president across the aisle has shrunk dramatically in recent years. That trend accelerated markedly under Trump, who has reasonably strong ratings within his own party but remarkably little backing from the opposition.
Last spring, we noted that Republicans were 78 points likelier than Democrats to approve of Trump's first-quarter job performance. Between Presidents Dwight Eisenhower and Bill Clinton, that gap was never above 50 points.
Although it's entirely true that most of the president's voters are sticking with him, it's also true that "most" is not the same as "all." And notably, Trump's detractors are far more unified ― and vociferous ― than his supporters.
Nearly 60 percent of Trump voters strongly approve of his job performance, according to YouGov/Economist tracking data ― but nearly 90 percent of Clinton voters strongly disapprove.
Similarly, strong disapproval of the GOP's failed attempt to repeal the Affordable Care Act and its successful bid to pass tax reform has consistently outstripped the strong approval of such efforts.
4. Views of the economy are way out of whack with views of the presidency.
Throughout 2017, pollsters found Americans generally happy with the state of the economy, but less happy with Trump.
That's unusual: Economic views are traditionally among the strongest drivers of presidential approval. As FiveThirtyEight's Harry Enten noted last fall, past presidents dating back to Jimmy Carter had higher ratings on overall job approval than they did for their handling of the economy. Trump breaks that pattern in a major way.
It's perhaps understandable that the public wouldn't give Trump much credit for an economic revival that clearly predated his presidency. Nearly a year into Trump's term, voters are still more likely to say that Obama is responsible for the state of the economy.
But Trump's approval rating, Gallup found, is 9 to 14 points lower than the public's views of the economy would predict.
5. Trump's low approval rating probably matters for this year's elections.
It's still a little too soon to be talking about Trump's re-election prospects, but his unpopularity poses a more immediate problem for the GOP: The president's party tends to do worse in midterms when the president's ratings are low.
And while good economic indicators would normally be a boon to the GOP, Trump's unpopularity, and his tendency to dominate the news cycle, may overshadow them. This year's midterms are still months away, and there's plenty of time for things to change ― several recent surveys have shown the Democratic lead in congressional polling starting to taper.
But as things stand, Trump's position at the head of the party seems unlikely to do down-ballot Republican candidates many favors.
"It's hard to get any traction on anything, so far, other than the president personally," Republican pollster Bill McInturff told NBC.


text:
Obituary Current Affairs - 2020
Fact Box: Sushma Swaraj
Former Union External Affairs Minister Sushma Swaraj, passed away after suffering a cardiac arrest in New Delhi. She was 67, She had started her political career under Janata Party and was elected to the Haryana assembly in 1977 and became a state minister at age 25.
About Sushma Swaraj
She was born on 14 February 1953 at Ambala Cantonment, Haryana. Her father Hardev Sharma was a prominent Rashtriya Swayamsevak Sangh member. She was lawyer by profession and had practiced in the Supreme Court. She had started her political career as active member of Jayaprakash Narayan's anti-Emergency movement and later she became a member of Bhartiya Janata Party. She was a seven-time Member of Parliament and a three-time MLA (representing Ambala Cantonment assembly seat from 1977 to 1982 and had won for first time at age of 25).
Many firsts to her credit: (i) Youngest Cabinet Minister (age) in Haryana Government in 1977, (ii) first woman Chief Minister of Delhi in 1998 (iii) first woman spokesperson for any national political party in the country and (iv)India's first full-time woman Foreign Minister.
Important offices held: She was External Affairs Minister (from 2014 to 2019), leader of the Opposition in Lok Sabha (from 2009 to 2014), Minister for Information and Broadcasting in Atal Bihari Vajpayee Government (2000-2003), Minister of Health and Family Welfare ( anuary 2003 to May 2004), Minister of Parliamentary Affairs (January 2003 to May 2004) and 5th Chief Minister of Delhi (from October 1998 to December 1998).
Tenure as External Affairs Minister: She was second Indian woman to become External Affairs Minister, after former Prime Minister Indira Gandhi ( However, it was not full-time portfolio).
Contributions: She had played key role in resolving the Doklam standoff with China in 2017 and in India approaching International Court of Justice (ICJ) challenging Pakistan military court's death sentence to Kulbhushan Jadhav. Through her extensive use of Twitter, she had earned reputation of easily-accessible External Affairs Minister. She took personal interest in resolving minor issues like issuance of visa to the needy and helped Indian diaspora in distress through her social media outreach.
Month: Current Affairs – August, 2019 Categories: Persons in News
Tags: External Affairs Minister • first woman Chief Minister of Delhi • India's first full-time woman Foreign Minister • Obituary • Person in news
Renowned Tamil novelist Thopil Mohammed Meeran dies aged 74
The well-known Tamil writer and Sahitya Akademi awardee Thopil Mohammed Meeran passed away at the age of 74.
About Thoppil Mohamed Meeran
In his lifetime he received eight literary awards which includes Sahitya Akademi Award and Tamil Nadu Kalai Ilakkiya Perumantram Award.
He won the prestigious Sahitya Akademi Award in 1997, for his novel Saaivu Naarkkaali (The Reclining Chair), released in 1995.
Other Novels Authored: are Thuraimugam, Anju Vannam Theru, Oru Kadalora Graamaththin Kathai, and Koonan Thoppu (all between 1988 and 2011).
His Short Story Collections and extensive translation work, gave him a prominent place in literary world. Some of his short story collections includes Thangarasu, Oru Maamaramum Konjam Paravaigalum, Anbukku Muthumai Illai, Oru Kutti Theevin Varippadam, Ananthasayanam Colony, and Thoppil Mohamed Meeran Kathaigal.
Forte: Due to his mastery over both Malayalam and Tamil, he could use both languages in his novels effortlessly. This became the speciality of all his literary works i.e. his novels brilliantly narrated the fisherfolk's life with details in both Tamil and Malyalam.
Month: Current Affairs - May, 2019 Categories: Persons in News
Tags: About Thoppil Mohamed Meeran • Anju Vannam Theru • Malayalam Novelist • Obituary • Oru Kadalora Graamaththin Kathai

redpajama_set_name: RedPajamaCommonCrawl


First off I would like to say thank you to all the people who wished me well after hearing about my dad's heart attack.
I promised I would write a hub about it. Well I finally got the courage to finish it today. I wouldn't normally post a link here but I thought this time it would be okay.
If admin feels they need to snip the link I understand because rules are rules.
Dale,..Your one of the most sincere people I have met here and I can't see them deleating your link. You have followed all the rules, and asked when you wer'nt sure about the rules. This hub about your dad's heart attack and isn't about getting clicks, its about informing everyone on an update that everyone has asked for. I've been gone for a few days and I did'nt know he had one,..you all will be in my prayers.
You are one of those people that I hope to be able to chat with for a long time to come on the internet. Thanks for the kind words.
Thank you Dale and my best for your dad.
I am really the one that needs to be saying thank you over and over for so many things including the great people of HubPages.
I never do this and I'm sorry for bumping this. I just want to make sure it's around for my dad to see when we go down to his place at the end of the week. I want him to see there are great people on the internet that care.
... without even trying..."Proceed to one of your longer hubs you are proud of.Do a browser search for the word, "that".I'm busily cleaning mine up as we speak.And that is that.
What Are The Symptoms Of A Heart Attack?What are the top signs or symptoms that you might have had a heart attack? Everyone should get a doctor's advice, but what would you recommend for dealing with a possible heart attack?