\section{Introduction} \label{sec:intro}

The surface and climatic conditions of rocky planets are sensitive to the volatile budget inherited from planetary formation. Variations in the main channels of volatile delivery and reprocessing during primary accretion affect the composition and thus redox state of planetary mantles, which in turn control atmospheric speciation, and hence surface geochemistry and long-term climate \citep{2021SSRv..217...22G,Graham21}. In particular young planets, comparable to the Hadean Earth and Pre-Noachian Mars, are sensitively affected by both the delivery during primary accretion, and potential secondary volatile delivery \new{from scattering and} dynamical transfer \new{of left-over planetesimals}, after the primary accretion phase has ceased \citep{Raymond2020PlanetaryAstrobiology}. Late volatile delivery can shift the atmospheric composition during transient epochs in the direction of the impactor composition: volatile-rich, metal-poor impactors oxidize Hadean-like atmospheres, volatile-poor, iron-rich impactors reduce the atmosphere; the latter is regarded a favorable environment for subaerial prebiotic chemistry \citep{Benner2019,2020PSJ.....1...11Z,2020AsBio..20.1476F}.

The heritage and processing of volatile elements during their journey from the interstellar medium (ISM) to the atmospheres of rocky planets, however, is poorly understood. Gas and grain-surface chemistry set the speciation and dominant carriers of hydrogen, carbon, oxygen, nitrogen, and sulfur before and during the infall of the gas and dust phase onto the disk, where coagulation, fragmentation, and settling processes redistribute ice phases in the outer parts of the disk \citep{oberg2011,obergbergin2021}. The depletion of mm-sized dust grains in ALMA disks suggests that the onset of the accretion phase operates fast, and a major fraction of the initial dust mass is depleted toward later disk stages \citep{2016ApJ...828...46A}. Isotopic ratios in Solar System planetary materials suggest that a fraction of volatiles in the terrestrial planets was inherited from thermally pristine sources \citep{2014Sci...345.1590C}, but recent evidence highlights the role of local \citep{Piani+2020,2021NatAs.tmp...14G} and originally volatile-rich, thermally reprocessed materials as possible carriers of water \citep{2019NatAs...3..307L,2021Sci...371..365L}, carbon species \citep{Hirschmann21}, and moderately volatile elements \citep{2019GeCoA.260..204S} during accretion.

\new{Conversely, (small) comets and Kuiper Belt objects represent populations of icy bodies that formed in the outer Solar System, with volatile abundances closer to pre- and protostellar chemistry \citep{mumma_charnley2011, drozdovskaya2019, grundy2020}, and may have largely escaped significant thermal processing \citep{2019Icar..326...10B,Golabek21}.}

\section{Volatile processing in the disk vs. planetesimals} \label{sec:methods}

\begin{figure*}[tb]
\centering
\includegraphics[width=.90\textwidth]{fig1_rev.pdf}
\caption{\textsf{\emph{Left:} Sketch illustrating the sequence of events during which we follow the fractionation of H$_2$O, CO, and CO$_2$. \emph{Right:} \new{Volatile/rock mass ratios for:} dust near the midplane (E), dust + pebbles (F), and primordial (G) and evolved (H) planetesimals as a function of radius in our fiducial model run. In (H), the dotted lines depict the initial conditions at $t=0$.}}
\label{fig:fig2}
\end{figure*}

\begin{figure*}[tb]
\centering
\includegraphics[width=.49\textwidth]{fig_plts_grid.pdf}
\includegraphics[width=.49\textwidth]{fig_plts_time.pdf}
\caption{\textsf{Thermal evolution of planetesimals and loss of H$_2$O, CO$_2$, and CO. \emph{(Left)} Peak temperatures in the interior of planetesimals from radioactive decay of $^{26}$Al (background color), interpolated from a grid of planetesimal simulations. The y-axes show enrichment level at the time of planetesimal formation ($^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol}$) and time after CAI formation in the Solar System. The x-axis (top) gives the radius of planetesimals. Colored lines indicate the retention of $>$ 90 vol\% (dashed) or 10 vol\% (solid) of H$_2$O (blue), CO$_2$ (green), and CO (red) after peak heating inside the planetesimal. \emph{(Right)} Loss  of H$_2$O (blue), CO$_2$ (green), and CO (red) over time for different combinations of $^{26}$Al abundance and planetesimal radius.}}
\label{fig:fig1}
\end{figure*}

In this Letter, we follow the abundances of H$_2$O, CO$_2$, and CO starting from microscopic icy grains in the outer regions of protoplanetary disks, through pebbles, to primordial and ultimately evolved/processed planetesimals. Here we briefly describe the relevant processes before and after planetesimal formation, which are illustrated in Fig.~\ref{fig:fig2}A--D. Our focus is on the disk region straddling the CO iceline.

\subsection{Disk processing: dynamics and chemistry in gas and solid phase}\label{sec:disk_processing}
The volatile content of the outer parts of protoplanetary disks is largely inherited from the cold molecular cloud. Typically, the majority of the volatile carbon and oxygen is carried by H$_2$O, CO$_2$, and CO \citep[e.g.,][]{obergbergin2021}. For such a composition, the volatile content of the microscopic dust grains is then set by their location relative to major icelines, imaginary boundaries that separate regions where specific molecules are stable as ices \citep[Fig.~\ref{fig:fig2}A and][]{oberg2011}. However, chemical processing and the formation and distinct dynamical behavior of pebble-size particles is expected to alter the abundances of key volatile species both locally and on disk-wide scales \citep[][and references therein]{bosman2018,krijt2018,krijt2020,obergbergin2021}. Here, we use the results of \citet{krijt2020}, who presented numerical models merging chemical processing of CO, dust coagulation, and pebble dynamics, to describe this stage. While these models were developed with the goal of studying the inferred depletion of gas-phase CO in the upper disk regions \citep[][]{zhang2019}, they offer a unique window into the evolution of the volatile content of solids growing and moving in the disk midplane. Figure \ref{fig:fig2} shows the radial variation in the volatile content (plotted as \new{mass} content relative to the refractory dust component, which can in some cases exceed 100\%) of: dust at $t=0$ (panel E); dust + pebbles after 1~Myr (panel F) and 2~Myr (panel G). \new{For comparison, water is the dominant volatile seen in comets, with CO and CO$_2$ abundances varying between ${\approx}1-30\%$ that of H$_2$O \citep{mumma_charnley2011}. Variations in volatile composition (of these and other species) may be linked to dynamical type and formation time or location \citep[e.g.,][]{dellorusso2016,eistrup2019}.} \new{Panels F and G of Fig.~\ref{fig:fig2} highlight several effects of disk processing:} \emph{(i)} a decrease in CO and CO$_2$ in the outer disk regions because of chemical processing and transfer to CH$_4$ and CH$_3$OH ice \citep[][]{bosman2018,krijt2020}, and \emph{(ii)} an enhancement of CO around the CO iceline, resulting from the retro-diffusion of CO vapor brought in through radial drift \citep[][]{stammler2017,krijt2018}. In this formulation, the timescales for the chemical processing are determined mainly by the cosmic ray ionisation rate \citep{bosman2018}, set here to $\zeta_\mathrm{CR}=10^{-17}\mathrm{~s^{-1}}$.

\subsection{Planetesimal formation}
In recent years the streaming instability has emerged as a leading contender to overcome the various barriers to growth from dust grains to the first planetesimals in protoplanetary disks \citep[][]{2014prpl.conf..547J}.
Evidence from the asteroid and Kuiper belt \new{\citep{2017Sci...357.1026D,2019Sci...363..955S,mckinnon2020} and fluid dynamical simulations \citep{2019ApJ...885...69L}} suggest the planetesimal population to be dominated by $\sim$\new{10--100} km bodies. The composition of these bodies will closely follow the local composition of solids at the time of gravitational collapse, but the timing and location of planetesimal formation may vary considerably \citep{2017AA...602A..21S,2019ApJ...884L...5S}. The model of \citet{krijt2020} does not explicitly model the formation of solids beyond pebble sizes, and for the purpose of this work we treat the formation time $\tau_\mathrm{pf}$ and radii $R_\mathrm{p}$ of planetesimals to be free model parameters.

\subsection{Planetesimal processing: melting and degassing}\label{sec:plts_processing}
We simulate the devolatilization and degassing from planetesimals that are internally heated by the decay of the short-lived radionuclide $^{26}$Al as reviewed in \citet{Monteux2018}. We use the thermal evolution simulations from \citet{2021Sci...371..365L}, with degassing thresholds for the volatile release of H$_2$O, CO$_2$, and CO from the planetesimal interior. We \new{parameterize} planetesimal devolatilization based on laboratory experiments on volatile release from heated chondrite samples. We assume complete release of CO$_2$ at 800$^{\circ}$C (1073 K) \citep{1995Metic..30..639M} and of H$_2$O at 950$^{\circ}$C (1223 K) \citep{1999CoMP..135...18N}. Minor abundances of CO may be regenerated by smelting between graphite and ferrous silicates until the mobilization of the first silicate melts \citep{2014EPSL.390..128F} at a silicate melt fraction of $\varphi \approx$ 0.1 \citep{2019EPSL.507..154L} (${\approx}1470$ K at 1 bar), which we define as the threshold for CO release. \new{The above thresholds define upper limits on devolatilization of hydrated and carbonated rocks that can be (re-)generated by fluid-rock interactions in the host body during heating. Ice melting and degassing commence earlier and at lower temperatures, but we focus here on conservative upper limits when even left-over materials are essentially devoid of any volatile form of carbon.}

We focus on planetesimals with $\approx$ 100 km in radius, but also explore the evolution of planetesimals with 10 km radius. The distribution of initial $^{26}$Al abundances between planetary systems is expected to vary by orders of magnitude, depending on the birth star-forming environment of a given planetary system \citep{2020RSOS....701271P,2020A&A...644L...1R}. Compositional changes due to $^{26}$Al heating at the time of planetesimal formation are significant within about 3 half-lives of the initial Solar System value ($^{26}$Al$_{\Sol}$) \citep{2019NatAs...3..307L}, which is why we limit our simulations here to $^{26}$Al abundances at the time of planetesimal formation in the range of $^{26}$Al$_\mathrm{exo}$ $\in$ [0.1, 1.0] $^{26}$Al$_{\Sol}$.

Higher $^{26}$Al and larger body size lead to higher temperatures and increasing loss of H$_2$O, CO$_2$, and CO to space (Fig. \ref{fig:fig1}A). Variability for bodies larger than $\sim$100 km is dominated by changes in $^{26}$Al levels and hence system enrichment at planetesimal formation. Volatile retention in Fig. \ref{fig:fig1}A is set by the devolatilization thresholds: loss efficiency decreases from CO$_2$ to H$_2$O to CO. In the high temperature regime, differences arise from retention in the thin conductive crusts of planetesimals. In the lower parts and at lower heating rates, the different thresholds separate the devolatilization trends for the different species more visibly: 10\% loss rates (90\% retention) are offset between 0.15 to 0.3 $^{26}$Al$_{\Sol}$. Volatile loss is variable and sensitive to the planetesimal formation conditions \citep[Fig. \ref{fig:fig1}B, cf.][]{2019NatAs...3..307L,2021Sci...371..365L}. We here compare loss rates for 10 and 100 km planetesimal radii and enrichment levels of 1.0 and 0.38 $^{26}$Al$_{\Sol}$. At 1.0 $^{26}$Al$_{\Sol}$, 10 km and 100 km radii planetesimals start losing their volatiles at roughly the same time, about 0.2 to 0.4 Myr after planetesimal formation, but larger planetesimals dehydrate more efficiently. 10 km planetesimals retain pristine materials in 60 to 80 vol\% of their interiors, 100 km planetesimals retain on the order of 5 to 8 vol\%. At 0.38 $^{26}$Al$_{\Sol}$, 100 km planetesimal (dashed lines) degas later, between 0.8 to 1.5 Myr after planetesimal formation, and loss efficiency decreases. Between 20 to 40 vol\% of pristine materials can be retained.

\section{Combined trends} \label{sec:results}

\begin{figure*}[tb]
\centering
\includegraphics[width=.95\textwidth]{fig3_rev.pdf}
\caption{\textsf{CO content \new{(defined as in Fig.~\ref{fig:fig2})} of dust and pebbles in the midplane (above the dashed line) and planetesimals (below the dashed line) as a function of heliocentric distance and time for 4 different models. (A): Fiducial model, with $R_\mathrm{p}=100\mathrm{~km}$, $\tau_\mathrm{pf}=2\mathrm{~Myr}$, $\zeta_\mathrm{CR}=10^{-17}\mathrm{~s^{-1}}$, and $^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol}=0.38$. (B): Smaller planetesimals with $R_\mathrm{p}=10\mathrm{~km}$. (C):  $\zeta_\mathrm{CR}=10^{-18}\mathrm{~s^{-1}}$. (D): Earlier planetesimal formation and a higher radionuclide content with $\tau_\mathrm{pf}=1\mathrm{~Myr}$ and $^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol}=0.75$.}}
\label{fig:fig3}
\end{figure*}

We investigate how degassing from planetesimals (Sect.~\ref{sec:plts_processing}) affects the volatile evolution \new{inherited} from disk-based solid- and gas-phase chemistry and dust coagulation (Sect.~\ref{sec:disk_processing}, Fig.~\ref{fig:fig2}E-F). In our fiducial model we assume 100 km planetesimals form after 2~Myr, with $^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol}=0.38$ at the time of their formation. By feeding the midplane compositions from the disk model (at $t=2~\mathrm{Myr}$) to the planetesimal evolution calculation, we compute the evolution of the H$_2$O, CO$_2$, and CO content of the planetesimals. The results of the fiducial model are shown in Fig.~\ref{fig:fig2}H. For these parameters a significant fraction of each volatile is lost during outgassing. The final water content is fairly constant with disk location (around 4\%), while the CO$_2$ content shows a decline with orbital distance. The CO content peaks at 10\% between 30--40 au, preserving the maximum in CO ice content that was formed as a result of pebble drift. In contrast, planetesimals formed around 20 au (i.e., inside the CO iceline) contain no CO, but close to 10\% CO$_2$. Thus, the interplay of the various processes described here (chemical processing, pebble drift, and planetesimal outgassing) can substantially alter not just the individual but also the \emph{relative} abundances of major carbon and oxygen carriers.

The complete time evolution of the fiducial model is shown in Fig.~\ref{fig:fig3}A. Here, focusing exclusively on CO, we show \new{that} the volatile abundance varies in both space and time. The region above the dashed lines depicts dust and planetesimals, and the region below it exclusively planetesimals. We can clearly distinguish 3 distinct periods in time. First, chemical processing and dust evolution alter the volatile content ($t<2\mathrm{~Myr}$). Then, planetesimals form, effectively quenching the composition of solids at the time of their formation ($t\approx 2-3.5\mathrm{~Myr}$). As planetesimals heat up, however, they rapidly lose a significant fraction of their CO content (Fig.~\ref{fig:fig1}B), leaving them relatively CO poor ($t > 3.5\mathrm{~Myr}$).

Figure \ref{fig:fig3}B--D presents a parameter exploration showing the effects of varying three key parameters in our model. First, assuming planetesimals are formed smaller (10~km) enables them to avoid efficient heating, resulting in virtually no CO loss after planetesimal formation (panel B). Lowering the cosmic ray ionisation rate by a factor of 10 relative to the fiducial case reduces the amount of CO that is processed on grain surfaces, resulting in a even higher CO content just outside the CO iceline (panel C). However, if planetesimals form earlier (1~Myr) and with a higher radionuclide content ($^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol}=0.75$), there is less time for the CO peak to develop during the dust evolution phase \emph{and} planetesimals degas more readily, resulting in very low final CO contents (panel D).

\section{Volatile fractionation of planetary systems} \label{sec:discussion}

\begin{figure*}[tb]
\centering
\includegraphics[width=.99\textwidth]{fig4_rev.pdf}
\caption{\textsf{Profile of \new{CO content} of planetesimals \emph{after} degassing as a function of heliocentric distance for models without any chemical processing or pebble drift during the disk phase (black) compared to those with 3 Myr or ongoing chemistry and dust evolution (orange). Different shadings depict different levels of $(^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol})$ at the time of planetesimal formation, and panels A and B differ in the cosmic ray ionisation rate.}}
\label{fig:fig4}
\end{figure*}

In Fig.~\ref{fig:fig4} we plot the radial profile of the final planetesimal CO abundance for planetesimals that form effectively at $t=0$ (i.e., before any chemical processing or dust/pebble transport can occur in the gas-rich disk) and after $t=3~\mathrm{Myr}$ (when significant evolution has taken place). The shading indicates different levels of $(^{26}$Al$_\mathrm{exo}/^{26}$Al$_{\Sol})$ at the time of planetesimal formation, and we compare two disk models with different cosmic ray abundances. All models are for $R_\mathrm{p}=100~\mathrm{km}$. Several things stand out, in particular: \emph{(i)} in disks that experienced significant radial drift across the CO iceline, the radial profile of planetesimal CO content shows a prominent peak just exterior to the location of the CO iceline. The width of this peak here is about 10--20 au, but can vary depending on the strength of radial diffusion \citep[][]{stammler2017,krijt2018}. \new{This effect can potentially explain the existence of planetesimals or comets with anomalously high CO/H$_2$O ratios \citep{price2021,mousis2021}. More typical comets also display significant variation in CO/H$_2$O ratios in their comae, although it is not clear whether this represents variation in formation conditions or later dynamical evolution and processing \citep{dellorusso2016, altwegg2019}}.   \emph{(ii)} In disks with a higher cosmic ray rate, the CO content of planetesimals will be lower, especially if these planetesimals form after a few million years. \emph{(iii)} The $^{26}$Al content plays a major role in shaping the final abundance and fractionation of volatiles, resulting in order-of-magnitude variations for the ranges explored here.

In a broader context, our results demonstrate the necessity of considering volatile fractionation both before and immediately after planetesimal accretion when attempting to connect the volatile budget of protoplanetary disk midplanes to those of planetary-scale objects. These results illustrate that from a planetary systems perspective, the anticipated fractionation effects between individual volatiles--here specifically illustrated for the major carbon-bearing species--from disk plus planetesimal processing differ significantly from those expected from pure disk-based processing alone \citep[e.g.][]{oberg2011}. We here discuss this result in the context of planet formation theory, and relate it to potentially observable signatures in extrasolar planetary systems that connect the volatile budget of exoplanets to the specific accretion pathway of the Solar System.

\subsection{Timing and mode of volatile accretion}

Planetary growth dominated by mutual accretionary collisions between planetesimals extends well beyond the lifetime of the gaseous protoplanetary disk \citep{Raymond2020PlanetaryAstrobiology}. The direct accretion of pebbles from the ambient disk onto growing protoplanets may shift the phase of primary mass addition to earlier stages during the disk stage itself \citep{2017ASSL..445..197O}. Protoplanets accreted within the disk lifetime undergo orbital migration and inherit a substantial amount of their mass from beyond the snowline or other icelines \citep{Venturini20}. Our simulations in this work demonstrate that the major volatile carbon carriers CO and CO$_2$ are affected \new{by outgassing from planetary precursors} to a similar degree as water \citep{2019NatAs...3..307L}. Planetary growth beyond the disk stage inherits the volatile budget from such devolatilized bodies \citep{2015ApJ...804....9C} because the degassing from planetesimals operates on much shorter timescales than collisional growth after the disk phase. CO and CO$_2$ are a major ingredient for the mantle composition, surface conditions, and climate of terrestrial worlds inherited from planet formation, but the Earth is carbon-poor relative to the nominal values in the ISM \citep{2015PNAS..112.8965B} and the earliest known planetesimal bodies in the Solar System display carbon fractionation trends indicative of extensive devolatilization \citep{Hirschmann21} and carbon-rich fluid mobilization \citep{Tsuchi21}. Planetesimal degassing as illustrated in the presented simulations limits the total amount of carbon that can be delivered to nascent planets by decreasing the volatile abundances of primitive, ice-rich planetesimals to system-wide comparable levels and flattens the volatile variability in planetary building blocks (Fig.~\ref{fig:fig4}) that can contribute to the primordial atmospheric composition of protoplanets. \new{This suggests that left-over material in substantially processed planetesimal generations converges toward the abundances of refractory forms of carbon inherited from the disk \citep{Li+21}.}

In addition to volatile delivery during primary accretion, \new{planetesimal impacts during the tail-end of accretion} have been suggested as a source of volatiles for the early Earth. The bulk compositions of Earth \citep{hirschmann2016constraints} and Venus \citep{2020NatGe..13..265G}, however, suggest the nature of the late accreted material to be dry in composition and chemically reduced. Our results suggest that the level of degassing from planetesimals acts to limit the total mass of volatiles accreted from bodies that are scattered from the outside to inner orbits of planetary systems. This alters the prospects for creating transiently reducing conditions on Hadean Earth analogs \citep{Benner2019,2020PSJ.....1...11Z}, and suggests that the reducing power and carbon budget of late \new{accretion} episodes \citep{2017ApJ...843..120S} is directly tied to the geophysical and geochemical evolution of the planetesimals from the \new{combined} effects of disk chemistry and internal heating.

\subsection{Debris disks \& Kuiper belt analogs}

Another connection exists between the planetesimals formed in our simulations and left-over icy bodies in the Solar System and beyond \citep[][Sect.~4.6]{krijt2020}. Evidence suggests bodies in the Kuiper Belt formed late \citep{2019Icar..326...10B}, with little $^{26}$Al and hence minor heating \citep{Golabek21} -- similar to the end-member case in Fig.~\ref{fig:fig3}B. Given the expected variation in $^{26}$Al content from system to system, it is interesting to look at extra-solar Kuiper Belts \citep{wyatt2020}, as it is now becoming possible to constrain the volatile content of planetesimals at 10s to 100s of au in a handful of systems \citep[e.g.,][]{marino2016,kral2017}. Constraints on the (CO+CO$_2$) mass content of planetesimals range from ${<}1$ to ${\approx}35\%$ \citep{wyatt2020}, broadly consistent with the ranges we find in e.g., Figs.~\ref{fig:fig2} and \ref{fig:fig3}. If the debris disk ring location, volatile content, and planetesimal size \citep[e.g.,][]{krivov2021} can be sufficiently constrained, models such as the ones presented here can be used to infer planetesimal formation timescales and the extent of past disk processing for these systems.

\new{Finally, interstellar interlopers offer a unique window in to the volatile content of extra-solar planetesimals. In the framework presented here, the high CO abundance of 2I/Borisov \citep{cordiner2020} points to formation (just) outside the CO iceline, and only minimal chemical processing and outgassing after formation (e.g., Fig.~\ref{fig:fig4}).}

\subsection{Environmental information from short-period exoplanets}

Exoplanet surveys that will seek to detect and characterize rocky extrasolar planets \citep{2019AJ....158...83A,2021arXiv210107500L,2021arXiv210308481H} will rely on environmental clues to inform and constrain the composition and history of planetary systems. Accounted for secondary loss processes, the composition of short-period exoplanets can yield crucial information on the volatile content of the planetary system as a whole. For instance, assuming radial variation in volatile content alone, the absence of large volatile quantities in super-Earths has been suggested to result from dry formation inside the snowline \citep{2020PSJ.....1...36K}. However, as we show in this letter, formation under high abundances of short-lived radioactive isotopes can devolatilize the planetary building blocks, and hence reconcile orbital migration with volatile-poor compositions. Enhanced distribution statistics on the bulk composition of small exoplanets below the Kepler radius valley will further provide information on the diversity of formation paths, and hence volatile sources, of planets on a population level \citep{2019PNAS..116.9723Z,2021MNRAS.tmp..530R}. In addition, prolonged magma ocean phases due to runaway greenhouse climates on short periods  \citep{2015ApJ...806..216H} and on wider orbits around young stars \citep{2019AA...621A.125B} promise to reveal characteristic signatures depending on their volatile content \citep{Lichtenberg2021JGRP}.

While hot rocky exoplanets provide means to probe the detailed inventory of volatiles, sub-Neptunes and Hot Jupiters are the more tangible targets in the upcoming decade. Their atmospheric metallicity informs the distribution across systems that are dynamically altered by planetary migration and hence probe the volatile content of planetary building blocks beyond the snowline \citep{2019MNRAS.482.1485P}. Based on radial and disk chemistry variation of volatile content alone, these systems should show a wide spread in atmospheric speciation \citep{booth2017,2020MNRAS.499.2229N}. If internal processing on planetesimals, like suggested here, quenches the initially accreted volatile content, then volatile trends in gaseous exoplanets should follow system-wide trends, altering the distribution statistics from system to system.

\section{Summary} \label{sec:conclusions}

In this work, we simulated the time- and location-dependent evolution of H$_2$O, CO, and CO$_2$ from combined disk and planetesimal processing. At a single location the volatile bulk compositions of planetesimals vary by orders of magnitude over time from disk chemistry alone. Heating and degassing by planetesimals, however, lead to rapid loss of volatiles on a timescale comparable to disk-based chemical evolution and planetary accretion. Solar System-like enrichment levels of the short-lived radioactive isotope $^{26}$Al lead to relative fractionation between individual volatiles by up to two orders of magnitude, depleting planetary volatile carriers in H$_2$O, CO, and CO$_2$ down to levels of ${<}1\%$. Importantly, the relative fractions of H$_2$O, CO, and CO$_2$ can be inverted on time intervals relevant to planetary accretion.

The degree of volatile fractionation suggested in this work alters the prospects for the climatic and surface conditions of rocky planets. If planetesimal formation is rapid, carbon-bearing species fractionate on a system-wide level as a function of the internal heating in planetesimals. These effects alter the potential for early volatile delivery and during late \new{accretion} episodes, and hence shape total carbon budget, redox state, and primitive surface conditions of young rocky planets during the first few hundred million years. Our results can be tested via population statistics of short-period exoplanets, atmospheric speciation of molten exoplanets, and debris disks, and provide contextual information on the viability of Earth-like climates on temperate orbits within individual and across planetary systems.

\vspace{0.15cm}
\small{
\emph{Acknowledgments:}
We thank Gregor Golabek and an anonymous reviewer for constructive comments on the manuscript. T.L. was supported by a grant from the Simons Foundation (SCOL award No. 611576). This work benefitted from information exchange within the program `Alien Earths' (NASA Grant No. 80NSSC21K0593) for NASA's Nexus for Exoplanet System Science (NExSS) research coordination network.

\vspace{0.15cm}
\emph{Software:} \textsc{i2elvis} \citep{2007PEPI..163...83G}, \textsc{numpy} \citep{Numpy2020}, \textsc{scipy} \citep{2020SciPy-NMeth}, \textsc{pandas} \citep{pandas:2010}, \textsc{matplotlib} \citep{Hunter:2007}, \textsc{seaborn} \citep{seaborn:2018}.
}


\section*{Reproducibility}
To facilitate reproducibility of the results reported in this paper,
the code and data used are available at \url{https://github.com/WZH-NLP/Bi-FLEET}.

\input{sections/acknowledgement.tex}

\clearpage
\bibliographystyle{ACM-Reference-Format}
\balance

\section{Analysis}
\label{subsec:extensive analysis}
Now that we have answered our research questions, we take a closer the look at Bi-FLEET to analyze its performance. We examine how the \ac{C-E} relation encoder and bi-feedback scheme contribute to its performance, how the amount of target-domain data influences the performance, and how performance varies across element types.

\begin{table*}[t]
  \small

  \caption{Examples from the I2C dataset. Red and green represent incorrect and correct contract elements, respectively.}
  \label{tab:case studies}
  \begin{tabular}{l l}
    \toprule
    Models & Sentences \\
    \midrule
    \multirow{3}*{MULTI-TASK (LSTM)} & ``Party B shall return the remuneration received and pay \colorbox{red}{10\%} (\textbf{Payment Rate}) of the total remuneration \\
    ~ & as liquidated damages.'' (\textbf{Indemnity Clause})\\
    ~ &``\colorbox{red}{According to the construction area} (\textbf{O}), the unit price of the commercial housing is XXX yuan per square meter''\\
    \midrule
    \multirow{4}*{Bi-FLEET (LSTM)} & ``Party B shall return the remuneration received and pay \colorbox{green}{10\%} (\textbf{Compensation Ratio}) of  the total remuneration \\
    ~ & as liquidated damages.''  (\textbf{Indemnity Clause})\\
    ~ & ``\colorbox{green}{According to the construction area} (\textbf{Calculation Standard}), the unit price of the commercial housing is XXX yuan\\
    ~ & per square meter''\\
  \bottomrule
\end{tabular}
\end{table*}

\subsection{Ablation studies}
\label{subsection:ablation studies}
We conduct ablation studies on both cross-domain CEE and NER datasets. The results are shown in Table~\ref{tab:ablation studies}. When we only ablate the \ac{C-E} relation encoder (``- \ac{C-E} graph''), type representations $\mathbf{cla}$ and $\mathbf{ele}$ are randomly initialized. In that case, F1-scores over all of the three datasets and model variants suffer a severe drop. Since the \ac{C-E} relation encoder cannot be directly incorporated, we cannot remove the bi-feedback scheme alone. When we ablate both the \ac{C-E} relation encoder and bi-feedback scheme (``- Bi-feedback''), our models obtain a similar performance as baselines (MULTI-TASK (LSTM), MULTI-TASK+PGN and BERT). In short, both the \ac{C-E} relation encoder and the bi-feedback scheme contribute to the improvements in performance on the cross-domain CEE and NER tasks.

\begin{table}[t]
  \small
  \setlength\tabcolsep{15pt}
  \caption{Ablation studies on the BioNLP, I2C and C2I datasets.}
  \label{tab:ablation studies}

  \begin{tabular}{lccc}
    \toprule
    \multirow{2}{*}{Model} & \multicolumn{3}{c}{Datasets (F1)} \\
    \cmidrule(r){2-4}

    ~ & BioNLP  & I2C & C2I\\
    \midrule
    \ac{Bi-FLEET} (LSTM)  & \textbf{82.32}  & \textbf{68.07} & \textbf{71.44}  \\
    \quad\quad- C-E graph  & 81.78  & 66.95 & 70.03   \\
    \quad\quad- Bi-feedback  & 81.03  & 65.29 & 68.59   \\
    \midrule
    \ac{Bi-FLEET} (PGN)  & \textbf{82.46}  & \textbf{68.43} & \textbf{71.69}  \\
    \quad\quad- C-E graph  & 81.85  & 67.37 & 70.15   \\
    \quad\quad- Bi-feedback  & 81.07  & 65.58 & 68.72   \\
    \midrule
    \ac{Bi-FLEET} (BERT)  & \textbf{89.02}  & \textbf{71.92} & \textbf{74.49}  \\
    \quad\quad- C-E graph  & 88.38  & 70.58 & 73.31   \\
    \quad\quad- Bi-feedback  & 86.89  & 69.32 & 72.06   \\
  \bottomrule
\end{tabular}

\end{table}

\subsection{Influence of target-domain data}
\label{subsection: influence of targe-domain data}
Next, we study the influence of target-domain data on the I2C dataset. We compare the F1-scores of the baselines and Bi-FLEET with different amounts of target-domain data in 100 training epochs.
We keep ``Neg'' sentences unchanged and adjust the number of ``Pos'' sentences, since ``Neg'' sentences do not contain any type of elements.
See Fig.~\ref{fig:influence of target-domain data} for the results.  Initially, Bi-FLEET obtains a large improvement of more than 10\% over baselines. The gap between \ac{Bi-FLEET} and baselines becomes smaller with the increase in target-domain data. Both Bi-FLEET (LSTM) and Bi-FLEET (PGN) outperform the baselines with varying numbers of target-domain data, which demonstrates the effectiveness of our methods.

\begin{figure}
  \centering
  \includegraphics[width=0.75\linewidth]{figures/target_influence.png}
  \caption{Influence of target-domain data.}
  \label{fig:influence of target-domain data}
\end{figure}

\smallnegskip
\subsection{Fined-grained comparisons}
\label{subsection:fined-grained comparisons}
To understand the performance of Bi-FLEET at the element type level, we make fine-grained comparisons on the I2C dataset. As shown in Table~\ref{tab:fine-grained comparisons on I2C}, we analyze five element types: invoice type (IT), arbitration commission (AC), currency of payment (CP), name of subject matter (NSM), and payment period (PP). Models based on Bi-FLEET accomplish significant F1-score improvements over the baselines, demonstrating that modeling \ac{C-E} relations and uncovering the bi-feedback scheme are helpful for reducing confusion between element types.

\begin{table}[t]
  \small

  \caption{Fine-grained comparisons on the I2C dataset.}
  \label{tab:fine-grained comparisons on I2C}
  \begin{tabular}{l ccc cc}
    \toprule
    \multirow{2}{*}{Model} & \multicolumn{5}{c}{Element types (F1)} \\
    \cmidrule(r){2-6}
    ~ & IT & AC & CP & NSM & PP \\
    \midrule
    BILSTM   & 53.46  & 60.31 & 62.75  & 64.14  & 67.20  \\
    MULTI-TASK (LSTM)  & 57.92 & 63.61 & 63.80 & 65.43 & 67.39  \\
    MULTI-TASK+PGN  & 58.05 & 63.38 & 64.26 & 65.65 & 67.92 \\
    \midrule
    \ac{Bi-FLEET} (LSTM)  & 62.86 & 63.94 & 67.68 & 68.82 & 68.15  \\
    \ac{Bi-FLEET} (PGN)  & \textbf{63.74} & \textbf{64.03} & \textbf{67.94} & \textbf{69.42} & \textbf{69.10}  \\
  \bottomrule
\end{tabular}
\end{table}

\subsection{Case studies}
\label{subsection:case studies}
Table~\ref{tab:case studies} provides examples from the I2C dataset. ``10\%'' is a compensation ratio, and ``According to the construction area'' is a calculation standard. Misled by the keywords ``pay'' and ``total remuneration'', MULTI-TASK (LSTM) misjudges ``10\%'' as a payment rate. Due to the data sparsity for the calculation standard, MULTI-TASK (LSTM) is not able to recognize ``According to the construction area''. By uncovering interactions between the \acf{CC} and \acf{CEE} tasks, Bi-FLEET can correctly extract the compensation rate in the indemnity clause. Identifying invariant knowledge about element types makes it much easier to transfer long-tail element types.

\section{conclusions}
\label{sec:conclusion}
In this paper, we have investigated the new problem of cross-domain \acl{CEE}; solutions to this problem can support clause retrieval and other legal search tasks. Compared to cross-domain \acl{NER}, there are two main challenges: more fine-grained element types and larger extraction zones. To address these challenges, we have proposed a framework named Bi-FLEET that captures invariant relations between clauses and elements, and we have designed a bidirectional feedback scheme between the \ac{CEE} and \acl{CC} tasks. Experimental results on cross-domain \ac{CEE} and \ac{NER} datasets show that Bi-FLEET significantly outperforms state-of-the-art baselines, with a high degree of generalizability.

A limitation of Bi-FLEET is that it can only extract elements sentence by sentence. In future work, we intend to consider the clause-level features, and explore relations between elements.

\section{Experiments}
\label{sec:experiments}
\subsection{Research questions}
We aim to answer the following research questions:
\begin{enumerate*}[label=(RQ\arabic*),leftmargin=*,nosep]
\item Does \ac{Bi-FLEET} outperform state-of-the-art methods on the cross-domain CEE taks? (Sec.~\ref{subsection:CEE})
\item Can \ac{Bi-FLEET} be generalized to the cross-domain NER task? (Sec.~\ref{subsection:NER})
\if0
\item How do the \ac{C-E} relation encoder and bi-feedback scheme contribute to the improvements? (Sec.~\ref{subsection:ablation studies})
\item How does the amount of target-domain data influence the performances of cross-domain CEE? (Sec.~\ref{subsection: influence of targe-domain data})
\item Is \ac{Bi-FLEET} able to outperform baselines in both element type and case level? (Sec.~\ref{subsection:fined-grained comparisons} and Sec.~\ref{subsection:case studies})
\fi
\end{enumerate*}

\subsection{Datasets}
\label{subsec:datasets}
\begin{table}
  \small
  \centering
  \caption{Statistics of the CEE datasets.}
  \label{tab:CEE dataset}

  \begin{tabular}{lccll}
  \toprule
  \multirow{2}*{Dataset} & \multicolumn{2}{c}{Type} & \multirow{2}*{Size} &\multirow{2}*{Train./Valid./Test} \\
  \cmidrule{2-3}
   ~ & Element & Clause & ~ & ~ \\
  \midrule
    \multirow{3}*{Individual} & \multirow{3}*{70} & \multirow{3}*{18} & \#Pos Sentence & 13.6K/1.7K/1.7K \\
    ~ & ~ & ~ & \#Neg Sentence& 60.0K/7.5K/7.5K \\
    ~ & ~ & ~ & \#Element& 25.7K/3.3K/3.3K \\
  \midrule
  \multirow{3}*{Commercial} & \multirow{3}*{79} & \multirow{3}*{17} & \#Pos Sentence & 4.8K/0.6K/0.6K \\
  ~ & ~ & ~ & \#Neg Sentence& 19.7K/2.5K/2.5K \\
  ~ & ~ & ~ & \#Element& 8.7K/1.2K/1.1K \\
  \bottomrule
\end{tabular}

\end{table}

\begin{table}
  \small
  \centering
  \caption{Statistics of the \ac{NER} datasets.}
  \label{tab:NER dataset}

  \begin{tabular}{l lll}
  \toprule
    Dataset & Entity Type& Size & Train./Valid./Test \\
  \midrule
    \multirow{2}*{BioNLP13PC} & CHEM,CC, &\#Sentence& 2.5K/0.9K/1.7K \\
    ~ & GGP& \#Entity& 7.9K/2.7K/5.3K\\
  \midrule
  \multirow{2}*{BioNLP13CG} & CHEM,CC, &\#Sentence& 3.0K/1.0K/1.9K \\
  ~ & GGP, etc.& \#Entity& 10.8K/3.6K/6.9K\\
  \midrule
  \multirow{2}*{CoNLL-2003} & PER, LOC, &\#Sentence& 15.0K/3.5K/3.7K \\
  ~ & ORG, MISC& \#Entity& 23.5K/5.9K/5.6K\\
  \midrule
  \multirow{2}*{Broad Twitter} & PER, LOC, &\#Sentence& 6.3K/1.0K/2.0K \\
  ~ & ORG & \#Entity& 8.8K/1.7K/4.4K\\
  \midrule
  \multirow{2}*{Twitter} & PER, LOC, &\#Sentence& 4.3K/1.4K/1.5K \\
  ~ & ORG, MISC& \#Entity& 7.5K/2.5K/2.5K\\
  \bottomrule
\end{tabular}

\end{table}

For cross-domain \ac{CEE}, we collect 340 open individual Chinese contracts from the web and 1,422 business Chinese contracts from partners. For every contract, at least two annotators with a legal background were asked to label the clauses and contract elements. The whole annotation process took about 3 months. To establish the datasets, we first split the contracts sentence by sentence, and filtered out elements that appear no more than 20 times and sentences with more than 100 characters. Then, these sentences were divided into training/validation/test sets with a 8/1/1 ratio, ensuring that all the elements and clauses in the validation and test splits occur at least once in the training set. Detailed statistics are provided in Table~\ref{tab:CEE dataset}. Different from NER, some sentences in clauses may not contain any elements. Such sentences are named ``Neg'' sentences in our experiments, while sentences in which elements do exist are called ``Pos'' sentences.

For cross-domain \ac{NER}, we use five public datasets, BioNLP13PC and BioNLP13CG~\citep{DBLP:conf/bionlp/NedellecBKKOPZ13}, CoNLL-2003 English dataset~\citep{DBLP:conf/conll/SangM03}, Broad Twitter dataset~\citep{DBLP:conf/coling/DerczynskiBR16}, and Twitter dataset~\citep{DBLP:conf/acl/JiZCLN18}. Detailed statistics of these datasets are shown in Table~\ref{tab:NER dataset}.

Following \citet{DBLP:conf/acl/JiaZ20}, we reconstruct two cross-domain \ac{CEE} datasets (I2C and C2I) and three cross-domain \ac{NER} datasets (BioNLP, Broad Twitter, and Twitter) based on the seven datasets mentioned above. For the I2C (or C2I) dataset, Commercial (or Individual) is used as the target-domain dataset, and the other contract dataset is the source-domain dataset. For the BioNLP dataset, the target-domain dataset is BioNLP13CG, while the source-domain dataset is BioNLP13PC. For the Broad Twitter dataset, Broad Twitter is the target-domain and CoNLL-2003 as the source-domain dataset. For the Twitter dataset, Twitter is used as the target-domain dataset, while CoNLL-2003 is the source-domain dataset.

\subsection{Baselines}
\label{subsec:baselines}
For cross-domain \ac{CEE} and \ac{NER}, we compare Bi-FLEET with recent baselines. These methods can be categorized into three groups:
\begin{itemize}[leftmargin=*,nosep]
  \item Target-domain only methods, BILSTM and MULTI-CELL LSTM \cite{DBLP:conf/acl/JiaZ20}, are trained without source-domain information.
  \item For cross-domain settings, we conduct comparisons with the LSTM-based MULTI-TASK (LSTM)~\citep{DBLP:conf/iclr/YangSC17} and two variants, MULTI-TASK+PGN and the method of \citet{DBLP:conf/acl/JiaXZ19}, both using parameter generation networks (PGN)  to generate parameters for source and target domain LSTMs. The multi-cell compositional LSTM based method of \citet{DBLP:conf/acl/JiaZ20} is considered too.
  \item In addition, we evaluate language model (LM) pretraining based methods, including BERT~\citep{devlin2018bert}, BIOBERT~\citep{DBLP:journals/bioinformatics/LeeYKKKSK20}, and the method of Jia and Zhang (BERT)~\citep{DBLP:conf/acl/JiaZ20} that leverages the outputs of LM pretraining methods as contextualized word embeddings.
\end{itemize}
As mentioned in Sec.~\ref{subsec:context encoder}, we adopt multiple types of context encoders. Specifically, \ac{Bi-FLEET} (LSTM) uses the standard LSTM layer as the sequence encoder. \ac{Bi-FLEET} (PGN) adds an PGN to generate parameters for domain-specific LSTMs. Similar to the MULTI-TASK (LSTM) and MULTI-TASK+PGN, both \ac{Bi-FLEET} (LSTM) and \ac{Bi-FLEET} (PGN) employ the max-margin principle~\citep{DBLP:conf/naacl/GimpelS10} for loss functions. \ac{Bi-FLEET} (MULTI-CELL) and \ac{Bi-FLEET} (BERT) utilize the multi-cell compositional LSTM and BERT as context encoders.

\subsection{Evaluation metrics}
Following~\citep{DBLP:conf/conll/SangM03,sun2019toi,DBLP:conf/acl/JiaXZ19, DBLP:conf/acl/JiaZ20}, we employ entity (or element) level Precision (P), Recall (R), F1-score (F1), and Accuracy for evaluation. Precision indicates the percentage of named entities (or elements) extracted by the method that are correct. Recall represents the percentage of entities (or elements) in the datasets that are predicted by the method. To this end, an entity (or element) is correct only when the corresponding entity (or element) in the dataset is exactly matched. The F1-score is the harmonic mean of Precision and Recall. We report P, R, and F1 for the cross-domain CEE task and F1 for the cross-domain NER task. We apply Accuracy to assess the proportion of clause type predictions that is correct.

\subsection{Implementation details}
\label{subsec:implementation details}
Our parameter settings mainly follow Jia and Zhang~\citep{DBLP:conf/acl/JiaZ20}. We use the NCRF++ toolkit~\citep{DBLP:conf/acl/YangZ18} to develop our models. All word embeddings are fine-tuned in the training process, while character embeddings are randomly initialized. The batch size is set to 30, and the initial learning rates for target-domain only, cross-domain, and BERT-based methods are 0.001, 0.0015, and 3e-5 respectively. The number of the layers of the \ac{C-E} relation encoder is set to 3.

For cross-domain CEE, word embeddings of 100 dimensions are pretrained on the Baidu Encyclopedia and Weibo corpus~\citep{DBLP:conf/emnlp/Cao000L18}.
For BERT fine-tuning methods, we use the base-sized BERT~\citep{devlin2018bert} pretrained on the Chinese Wikipedia corpus.

For cross-domain NER, word embeddings are initialized with PubMed 200 dimension vectors~\citep{DBLP:conf/bionlp/ChiuCKP16} for BioNLP experiments and GloVe 100 dimension vectors~\citep{DBLP:conf/emnlp/PenningtonSM14} for others.  In cross-domain NER, ``clause'' labels do not exist. In that case, we employ the base-sized BERT~\citep{devlin2018bert} and BIOBERT~\citep{DBLP:journals/bioinformatics/LeeYKKKSK20} models to embed each sentence and then cluster [CLS] vectors by the K-Means algorithm to generate the labels of sentences in an unsupervised manner. Specifically, we cluster sentences into 20 groups and represent these groups with the clause-element relation encoder.

\section{Introduction}
Extracting information from contracts or other legal agreements is a fundamental task for businesses around the world~\citep{chalkidis2019neural,chalkidis2017deep,milosevic2004design}.
Many thousands of contracts, relevant to a large variety of transactions (such as loans, investments, or leases), are drawn up every day. These contracts usually contain legally relevant elements such as termination dates or contract parties.
Manually monitoring the legally relevant information in a large number of contracts is time-consuming, labor-intensive, and error-prone, putting a heavy burden on law firms, companies, and government agencies~\citep{sun2019toi}.
Automatic \acfi{CEE} is increasingly attracting interest~\citep{DBLP:conf/jurix/MaxwellS08}).
As Fig.~\ref{fig:CEE examples}(top) illustrates, given a clause from a contract, the goal of the \ac{CEE} task is to find the legally relevant elements (such as ``within 5 days \ldots\ fact'') in the contract. \ac{CEE} can
bring useful insights into contracts and it can facilitate downstream applications, such as relevant clause retrieval or risk assessment~\citep{DBLP:conf/emnlp/BorchmannWGKJSP20}.

Modern information extraction methods consider the \ac{CEE} task as a sequence labeling problem, classifying each word as a (part of a) type of contract element~\citep{chalkidis2019neural, chalkidis2017deep, sun2019toi}.
The \ac{CEE} task is challenging due to large variations in element mentions.
Another important challenge for this sequence labeling problem is how to transfer knowledge from one domain to another~\citep{DBLP:conf/iclr/YangSC17}.
For example, as Fig.~\ref{fig:CEE examples} shows, compared with individual contracts (top), commercial contracts (bottom) are more formal, giving more precise and complex explanations in the contract clauses.
Because of the differences in context genres and element types, transferring an element extractor from the individual domain to the commercial domain (or vice versa) is a challenging problem.

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figures/clause.png}
  \caption{Examples of contract element extractions. (Top): A clause from an individual contract. (Bottom): A clause from a commercial contract.}
  \label{fig:CEE examples}
\end{figure}

In this paper, we focus on cross-domain \acf{CEE}.
\begin{table}[t]
  \small
  \centering
  \caption{Examples of the most frequent elements in clauses from different contract domains. Percentages in brackets indicate ratios of elements appearing in clauses from the individual/commercial contracts.}
  \label{tab:examples of C-E relations}

  \begin{tabular}{ll}
  \toprule
    Clause & The most frequent elements (individual/commercial) \\
  \midrule
    Payment & Payment period (23.8\%/25.6\%) and rate (12.3\%/10.9\%) \\
  \midrule
    Deposit & Deposit rate (26.4\%/40.6\%) and amount (12.4\%/14.5\%)\\
  \midrule
    Effectiveness & Effective mode (45.5\%/31.3\%) and time (17.7\%/26.7\%) \\
  \bottomrule
\end{tabular}

\end{table}
Cross-domain \ac{CEE} is similar to the cross-domain \ac{NER} task in many ways~\citep{chalkidis2017deep, chalkidis2017extracting}. Both tasks aim to identify key information (elements or entities) from texts in different domains and have similar problem formulations (discussed in detail in Sec.~\ref{subsec:problem formulation}).
However, existing cross-domain \ac{NER} methods are not directly applicable to the cross-domain \ac{CEE} task and fail to address two important challenges:

\begin{description}[leftmargin=3.5mm,nosep]
\item[Challenge 1: Transfer of fine-grained element types.]
\sloppy A ge\-ne\-ric named entity recognizer typically extracts only several types of entities, such as persons, organizations, locations, dates or amounts.
However, in the \ac{CEE} task, there are far more types of fine-grained contract elements (e.g., more than 70 in our dataset).
For example, a typical entity recognizer may identify dates, but cannot distinguish between start, termination or other dates.
In the cross-domain setting, the presence of a larger number of fine-grained contract elements makes it more difficult to transfer the extractor.
To facilitate the transfer process, invariant knowledge about element types should be incorporated.
As Table~\ref{tab:examples of C-E relations} shows, dependencies between clause and element types are usually shared across domains.
E.g., the payment period  usually exists in the payment clauses, but not the deposit clauses, which is independent of contract domains.

\item[Challenge 2: Transfer of larger extraction zones.]
\sloppy The \ac{NER} task mainly focuses on locating entities in a single sentence. In contrast, an extractor for the \ac{CEE} task needs to find contract elements from multiple sentences in a clause.
That is, extraction zones (clauses) for contract elements are much larger than the contexts of entities.
Sentences in clauses can be organized in various ways for different domains, which brings a new challenge for the cross-domain \ac{CEE} task. To reduce the influence of sentence organizations, clause types should be decided first.
Besides, classifying clauses is of great importance for extracting contract elements.
E.g., if we know a clause is a loan clause, it is easier to figure out that the dates in the clause are  loan or repayment dates.
Hence, conducting clause classification and element extraction at the same time can enhance the inference process.
\end{description}

\noindent%
To address the challenges identified above, we propose a framework, named the \acfi{Bi-FLEET}, that captures shared knowledge about element types in two domains and interactions between contract element extraction and clause classification.
As Fig.~\ref{fig:framework} shows, \ac{Bi-FLEET} is composed of three main components:
\begin{enumerate*}[label=(\arabic*)]
\item a context encoder,
\item a clause-element relation encoder, and
\item an inference layer.
\end{enumerate*}
First, the context encoder embeds all sentences in clauses. Then, to capture invariant relations between element and clause types, in the clause-element relation encoder, a \acf{C-E} graph is constructed across domains and a hierarchical graph neural network (GNN) is implemented to encode the shared clause-element dependencies. Finally, to identify the clause types and facilitate the element extraction process, a multi-task framework with a bi-directional feedback scheme is proposed in the inference layer.
Similar to the human annotation process, both clause classification and contract element extraction tasks are conducted. To model interactions between tasks and improve the overall performance, forward and backward information is calculated using representations of contexts and types from the context and clause-element relation encoders, respectively.
To evaluate our proposed framework to extract contract elements in cross-domain CEE, we collect both individual and commercial contracts to establish a cross-domain CEE dataset.
Experimental results using both cross-domain \ac{NER} and \ac{CEE} datasets demonstrate the effectiveness of \ac{Bi-FLEET}.

The contributions of this paper can be summarized as follows:
\begin{itemize}[leftmargin=*,nosep]
    \item To the best of our knowledge, ours is the first work on the cross-domain \ac{CEE} task. Compared with cross-domain \ac{NER}, more fine-grained element types and larger extraction zones bring critical challenges for the transfer process.
    \item We propose a framework, named the \acfi{Bi-FLEET}, to capture invariant clause-element dependencies and interactions between contract element extraction and clause classification.
    \item We establish a cross-domain \ac{CEE} dataset by collecting individual contracts and commercial contracts. To the best of our knowledge, this is the first dataset for cross-domain \ac{CEE}.

    \item Experimental results show that the proposed \ac{Bi-FLEET} model achieves significant improvements over baselines in both cross-domain \ac{NER} and \ac{CEE} tasks.
\end{itemize}
\if0
\noindent%
The rest of this paper is organized as follows. In Sec.~\ref{sec:related work}, related work is reviewed.
In Sec.~\ref{sec:method}, our problem formulation and the proposed \ac{Bi-FLEET} framework are explained. In Sec.~\ref{sec:experiments} and~\ref{sec:results}, both cross-domain \ac{CEE} and \ac{NER} tasks are evaluated, and detailed analyses are given. Finally, our conclusions are formulated in Sec.~\ref{sec:conclusion}.
\fi
\section{Method}
\label{sec:method}
In this section, we describe the \ac{Bi-FLEET} framework, our proposed method for \ac{CEE}.
As illustrated in Fig.~\ref{fig:framework}, the framework has three main components: a context encoder, a clause-element relation encoder, and an inference layer.
The context encoder, which includes an input embedding layer and sequence encoder, embeds every word in a sentence from a given clause.
The clause-element relation encoder is shared by the source and target domain and calculates representations of clause and element types.
The word embeddings and type representations are input to the inference layer for clause classification and contract element extraction across domains.

Next, we formulate the cross-domain \ac{CEE} task and compare it to the cross-domain \ac{NER} task (Sec.~\ref{subsec:problem formulation}).
Then, we explain \ac{Bi-FLEET}'s context encoder (Sec.~\ref{subsec:context encoder}), the clause-element relation encoder (Sec.~\ref{subsec:C-E graph}), and the inference layer (Sec.~\ref{subsec:inference layer}).
Finally, the loss functions and training process (Sec.~\ref{subsec:traning objectives}) are presented.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/framework.pdf}
    \caption{An overview of \ac{Bi-FLEET}. \ac{Bi-FLEET} has three components: a context encoder, a clause-element relation encoder, and an inference layer.}
    \Description{The framework of our model.}
    \label{fig:framework}
\end{figure}

\smallnegskip
\subsection{Problem formulation}
\label{subsec:problem formulation}
We write $\mathbf{C} = (\mathbf{s}_{1}, \mathbf{s}_{2}, \ldots, \mathbf{s}_{n})$ for  a clause from a contract, where $\mathbf{s}_{i}$ is the $i$-th sentence in the clause $\mathbf{C}$; $\mathbf{s}_{i} = (x_{i,1}, x_{i, 2},\ldots, x_{i, m})$, where $x_{i, j}$ is the $j$-th word in sentence $\mathbf{s}_{i}$. A \emph{contract element} $\mathbf{e}$ in the clause $\mathbf{C}$ is a sequence of words in one sentence: $\mathbf{e} = \{(x_{i,start}, x_{i, start+1}$, \ldots, $x_{i, end}), l^{e}\}$, where $l^{e}$ is the type label of the element $\mathbf{e}$ (such as payment period or deposit rate). The aim of the \acfi{CEE} task is to find the element $\mathbf{e}$ in the clause $\mathbf{C}$.

For the cross-domain \ac{CEE} task, there are $N_{s}$ labeled clauses in the source domain $\mathcal{S}$, as well as $N_{t}$ labeled clauses and $N_{u}$  unlabeled clauses in the target domain $\mathcal{T}$.
Our goal is to transfer the contract element extractor to the target domain from the source domain. Specifically, the extractor is trained on the $N_{s}$ labeled clauses in the source domain and $N_{t}$ labeled clauses in the target domain to detect all the elements for the $N_{u}$ unlabeled clauses in the target domain.

A similar task is cross-domain \acf{NER}, which focuses on identifying several types of named entities in only one sentence. In contrast, cross-domain \ac{CEE} aims at extracting much more fine-grained elements in multiple sentences. As mentioned before, the transfer of larger extraction zones and more element categories brings new challenges to cross-domain \ac{CEE}.

\smallnegskip
\subsection{Context encoder}
\label{subsec:context encoder}
Similar to previous \ac{CEE} and \ac{NER} methods~\cite{chalkidis2017deep, chalkidis2019neural, DBLP:conf/iclr/YangSC17, DBLP:conf/acl/JiaZ20}, our context encoder vectorizes words in a sentence from a clause by using two components, an input embedding layer and a sequence encoder.

\subsubsection{Input embedding layer}
Given an input sentence $\mathbf{s} = \{x_{1}, x_{2}$, $x_{3}, \ldots, x_{m} \} $ from a clause $\mathbf{C}$, to capture the word-level and character-level features, each word $x_{i}$ in the sentence $\mathbf{s}$ is embedded as the concatenation of the word embedding and the output of a char-level CNN:
\begin{equation}
    \small
    \mathbf{v}_{i} = \mathbf{E}^{w}(x_{i}) \oplus {\rm CNN}(\mathbf{E}^{c}(x_{i})),
\end{equation}
where $\mathbf{E}^{w}$ denotes a shared word-level embedding lookup table, and $\mathbf{E}^{c}$ denotes a shared char-level embedding lookup table. $\rm{CNN}(\cdot)$ is a standard convolutional neural network operating on the character-level embedding sequence $\mathbf{E}^{c}(x_{i})$ for word $x_{i}$ in sentence $\mathbf{s}$. And $\oplus$ denotes the  vector concatenation operator.

\subsubsection{Sequence encoder}
To encode sentence-level features, a standard bi-directional LSTM layer~\cite{graves2005framewise} is adopted here. Given an input embedding sequence $\mathbf{v} = [\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots, \mathbf{v}_{m}]$, at each time step, the Bi-LSTM layer calculates the current hidden vectors based on memory cells. The hidden outputs of the task and domain-specific Bi-LSTM unit can be written as follows:
\begin{equation}
    \small
    \label{eq:bi-lstm}

        \overrightarrow{\mathbf{h}}^{d,t}_{i} = {\rm LSTM}(\overrightarrow{\mathbf{h}}^{d,t}_{i-1}, \mathbf{v}_{i}),\quad
        \overleftarrow{\mathbf{h}}^{d,t}_{i} = {\rm LSTM}(\overleftarrow{\mathbf{h}}^{d,t}_{i+1}, \mathbf{v}_{i}),

\end{equation}
where $\mathbf{h}^{d,t}_{i}$ denotes the output hidden vector for domain $d\in \{\mathcal{S}, \mathcal{T}\}$  and task $t\in \{CC, CEE\}$ (clause classification and contract element extraction) with the input word $x_{i}$; $\rightarrow$ and $\leftarrow$ represent the directions of the LSTM unit.
Note that a variety of methods can be adopted as the context encoder. In our experiments, BERT~\cite{devlin2018bert} is adopted as the input embedding layer to generate the contextualized word embeddings. Besides, the sequence encoders based on  the parameter generation network~\cite{DBLP:conf/acl/JiaXZ19} and multi-cell compositional LSTM~\cite{DBLP:conf/acl/JiaZ20} are evaluated as well.

\subsection{Clause-element relation encoder}
\label{subsec:C-E graph}
To model relations between element and clause classes, we establish the \acf{C-E} graph and learn representations of the clause and element types with a GNN-based \ac{C-E} relation encoder.

\subsubsection{C-E graph constructions}
Before encoding clause and element types with a hierarchical graph neural network, we construct the \ac{C-E} graph.
As mentioned above, contract elements are strongly connected with categories of clauses.
For a clause type $ct$ and element type $et$ in a given domain $d$, to identify the relation between them,  we first define the probability $p_{et, ct}$, that the elements of type $et$ exist in clauses of type $ct$, and $p_{ct, et}$, that clauses of type $ct$ contain elements of type $et$, as follows:
\begin{equation}
    \small
    \label{eq:constructions}
        p_{et, ct} = f_{et,ct}/ f_{et},\quad p_{ct, et}=f_{et,ct} / f_{ct},
\end{equation}
where, in a given domain $d$, $f_{et,ct}$ is the frequency that elements of type $et$ and clauses of type $ct$ co-occur. Here, $f_{et}$ and $f_{ct}$ are the frequencies that the elements of  type $et$ and clauses of type $ct$ appear in the given domain, respectively. Note that we only count once when multiple elements of the same types exist in one clause. When both $p_{et, ct}$ and $p_{ct, et}$ are larger than the preset threshold $\theta$, the element type $et$ and clause type $ct$ are connected in the C-E graph of the given domain $d$.
As Fig.~\ref{fig:constructions} illustrates, we first select the edges between clause and element types (red and blue nodes) to establish the \ac{C-E} graph of the source domain. And then, for the target domain, we only maintain the shared types (blue nodes) and add the domain-specific types (yellow nodes) into the original graph.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/constructions.pdf}
    \caption{\ac{C-E} graph construction. Blue nodes are the shared element and clause types, while red and yellow ones are the ``not shared'' types between the source and target domains.}
    \Description{C-E graph constructions.}
    \label{fig:constructions}
\end{figure}

\subsubsection{The GNN-based relation encoder} Given a \ac{C-E} graph $G$ in the domain $d$, we apply a hierarchical graph neural network to encode the element and clause types. Recent GNN-based frameworks follow the neighbor aggregation strategy~\cite{DBLP:conf/iclr/XuHLJ19,wang2019robust,feng2021causalgcn,feng2019graph}, updating the embeddings of the central node by iteratively aggregating the information of its neighbors.
The $l$-th layer of a GNN, composed of an aggregation function ${\rm AGGREGATE(\cdot)}$ and a combination function ${\rm COMBINE(\cdot)}$, can be presented as follows:
\begin{equation}
    \small
    \begin{split}
        a_{v}^{(l)} &= {\rm AGGREGATE}^{(l)}\left(h_{u}^{(l-1)}:u \in \mathcal{N}_v\right) \\
        h_{v}^{(l)} &= {\rm COMBINE}^{(l)}\left(h_{v}^{(l-1)}, a_{v}^{(l)}\right),
    \end{split}
\end{equation}
where $h_{v}^{l}$ is the feature vector of node $v$ at the $l$-th GNN layer and $\mathcal{N}_v$ is the set of nodes adjacent to the central node $v$.
We set two transformation matrices $\mathbf{W}_{c\rightarrow e}$ and $\mathbf{W}_{e\rightarrow c}$ for the clause-to-element and element-to-clause relations respectively, and the $l$-th layer of our proposed C-E relation encoder can be formulated:
\vspace{-1mm}
\begin{equation}
    \label{eq:relation encoder}
    \small
    \begin{split}
       \mathbf{ele}_{p}^{d, (l)}=\sigma\left(\sum_{u\in \mathcal{N}^{d,e}_{p}}{\alpha_{u}^{d, c}\mathbf{W}_{c\rightarrow e}\cdot\mathbf{cla}_{u}^{d, (l-1)}}+\mathbf{W}_{e}\cdot\mathbf{ele}_{p}^{d, (l-1)}\right)  \\
       \mathbf{cla}_{k}^{d, (l)}=\sigma\left(\sum_{v\in \mathcal{N}^{d, c}_{k}}{\alpha_{v}^{d, e}\mathbf{W}_{e\rightarrow c}\cdot\mathbf{ele}_{v}^{d, (l-1)}}+\mathbf{W}_{c}\cdot\mathbf{cla}_{k}^{d,  (l-1)}\right),
    \end{split}
\end{equation}

\noindent where, given a domain $d$, $\mathbf{ele}_{p}$ and $\mathbf{cla}_{k}$ are the representations of the element type $et_{p}$ and clause type $ct_{k}$ separately, while $\alpha_{v}^{e}$ and $\alpha_{u}^{c}$ are the trainable weights for the $v$-th element type and $u$-th clause, type respectively. Given a \ac{C-E} graph of domain $d$, $\mathcal{N}^{e}_{p}$ is the set of neighborhood clause types for $et_{p}$, and $\mathcal{N}^{c}_{k}$ is the set of neighborhood element types for $ct_{k}$. $\mathbf{W}_{e}$ and $\mathbf{W}_{c}$ denote the matrices for the  self-connection.
$\sigma(\cdot)$ is the activation function.
$\mathbf{W}_{c\rightarrow e}$, $\mathbf{W}_{e\rightarrow c}$, $\mathbf{W}_{e}$ and $\mathbf{W}_{c}$ are shared in both source and target domains.
If a clause or element types exists in both domains
, their representations are also shared across domains.

\subsection{Inference layer}
\label{subsec:inference layer}
Given the sentence-level feature and type representations of elements and clauses,  the multi-task inference layer aims to predict the clause category labels (\acf{CC}) and  extract contract elements from clauses (\acf{CEE}). To capture the interactions between the two tasks above, we design a novel bi-feedback scheme in the inference layer. In this section, we demonstrate the bi-feedback scheme first and then explain how to extract elements with forward information and verify the clause classification results using backward information.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{figures/bifeedback.pdf}
    \caption{The inference layer with the bi-feedback scheme.  }

    \Description{bi-feedback scheme.}
    \label{fig:bifeedback}
\end{figure}

\subsubsection{The bi-feedback scheme}
Given a clause from the contract, a human annotator usually first decides the clause type and then identifies the elements in the clause. In that case, the classification of clauses often facilitates the element extraction process, while the identification of elements can also verify the clause type predictions. Based on this observation, as shown in Fig.~\ref{fig:bifeedback}, we design a multi-task framework with the bi-feedback scheme.
Given a training instance for task $t\in \{CC, CEE\}$, assuming that the result distribution vector $\mathbf{res}^{t}$ is predicted, we combine the result distribution vector with the clause or element representations , and obtain the latent type vector $\mathbf{ltv}^{t}$ for $\mathbf{res}^{t}$ as follows:
\begin{equation}
    \small
    \label{eq:latent class}

        \mathbf{ltv}^{CC} = \sum_{k}{res^{CC}_{k}\cdot \mathbf{cla}_{k}},\quad
        \mathbf{ltv}^{CEE} = \sum_{p}{res^{CEE}_{p}\cdot \mathbf{ele}_{p}},

\end{equation}
where, given a task $t$, $res_{k}$  denotes the $k$-th type of the results. $\mathbf{cla}_{k}$ and $\mathbf{ele}_{p}$ are the representations of the $k$-th clause type or $p$-th element type.
Then, we calculate the forward information $\mathbf{fw}$ and backward information $\mathbf{bw}$ based on the latent type vector $\mathbf{ltv}^{t}$:
\begin{equation}
    \centering
    \small
    \label{eq:forward and backward}

        \mathbf{fw} =\sigma(\mathbf{W}_{f}\cdot \mathbf{ltv}^{CC}),\quad
        \mathbf{bw} =\sigma(\mathbf{W}_{b}\cdot \mathbf{ltv}^{CEE}),

\end{equation}
where $\mathbf{W}_{f}$ and $\mathbf{W}_{b}$ are the transformation matrices. Note that we set different transformation matrices and biases for source and target domains. In the rest of this section, we explain how to predict the result distribution $\mathbf{res}^{t}$ and obtain the final predictions, $\mathbf{\hat{y}}^{CC}$ and $\mathbf{\hat{y}}^{CEE}$, using forward and backward information. In our multi-task framework, $\mathbf{fw}$ and $\mathbf{bw}$ serve as gates to filter out the noise and reduce prediction mistakes.

\subsubsection{Clause classification (CC)}
To classify the clause that the given sentence $\mathbf{s}$ belongs to, given a domain $d$, we use the sentence-level feature $\mathbf{h}=[\overrightarrow{\mathbf{h}}_{1}\oplus\overleftarrow{\mathbf{h}}_{1}, \overrightarrow{\mathbf{h}}_{2}\oplus\overleftarrow{\mathbf{h}}_{2}, \ldots ,\overrightarrow{\mathbf{h}}_{m}\oplus\overleftarrow{\mathbf{h}}_{m}]$ from the Bi-LSTM. The $k$-th element of the result distribution vector $\mathbf{res}^{CC}$ is predicted by
\begin{equation}
    \centering
    \small
    \label{eq: res CC}
    res^{CC}_{k}=\sigma(\mathbf{w}_{CC}\cdot[\mathbf{\hat{h}}\oplus \mathbf{cla}_{k}] + b_{CC}),
\end{equation}
where $\mathbf{\hat{h}} = \sum_{i}{\alpha_{i}\cdot \mathbf{h}_{i}}$ and $\sum_{i}{\alpha_{i}}=1$. $\mathbf{\hat{h}}$ represents the overall feature of the given sentence and $\alpha_{i}$ denotes the trainable attention weight for word $x_{i}$. Considering the different label sets across domains, the parameters, $\mathbf{w}_{CC}$ and $b_{CC}$, are not shared. Based on the result distribution vector $\mathbf{res}^{CC}$, forward information $\mathbf{fw}_{s}$ for sentence $\mathbf{s}$ can be calculated by Eq.~\ref{eq:latent class} and~\ref{eq:forward and backward}.
The final prediction $\mathbf{\hat{y}}^{CC}$ can be verified by the backward information as follows:
\begin{equation}
    \label{eq:y CC}
    \small
    \mathbf{\hat{y}}^{CC}= {\rm softmax}(\mathbf{bw}_{1}\otimes\mathbf{bw}_{2}\otimes \cdots \otimes\mathbf{bw}_{m}\otimes \mathbf{res}^{CC}),
\end{equation}
where $\mathbf{bw}_{i}$ denotes the backward information for word $x_{i}$ in sentence $\mathbf{s}$ (from the contract element extraction task). ${\rm softmax}(\cdot)$ is the softmax function. $\otimes$ is the element-wise product operator.

\subsubsection{The contract element extraction}
Similar to the \ac{NER} task, standard CRFs~\citep{DBLP:conf/acl/MaH16} are adopted in the inference layer for \ac{CEE}.
Given a sentence $\mathbf{s}$ from the contract clause, the sentence-level feature $\mathbf{h}=[\overrightarrow{\mathbf{h}}_{1}\oplus\overleftarrow{\mathbf{h}}_{1}, \overrightarrow{\mathbf{h}}_{2}\oplus\overleftarrow{\mathbf{h}}_{2}, \ldots ,\overrightarrow{\mathbf{h}}_{m}\oplus\overleftarrow{\mathbf{h}}_{m}]$, where $\mathbf{h}_{i}=\overrightarrow{\mathbf{h}}_{i}\oplus\overleftarrow{\mathbf{h}}_{i}$, can be obtained from the context encoder. We merge $\mathbf{fw}_{s}$ (from the clause classification task for sentence $\mathbf{s}$) with each hidden vector in $\mathbf{h}$:
\begin{equation}
    \small
    \label{eq:h FD}
    \mbox{}\hspace*{-1mm}
    \begin{split}
    &\mathbf{h}^{fw}= {}\\
    &[{\rm norm}(\mathbf{h}_{1}\otimes\mathbf{fw}_{s}), {\rm norm}(\mathbf{h}_{2}\otimes\mathbf{fw}_{s}), \ldots ,{\rm norm}(\mathbf{h}_{m}\otimes\mathbf{fw}_{s})],
    \hspace*{-1mm}\mbox{}
    \end{split}
\end{equation}
where ${\rm norm}(\cdot)$ is the normalization function. $\otimes$ is the element-wise product operator. Then, the output probability $p(\mathbf{\hat{y}}^{CEE}|\mathbf{s})$ of the final prediction sequence $\mathbf{\hat{y}}^{CEE} = [l_{1}, l_{2}, \ldots, l_{m}]$ over the input sentence $\mathbf{s}$ can be calculated by:
\begin{equation}
    \small
    \label{eq: CRF}
    p(\mathbf{\hat{y}}^{CEE}|\mathbf{s})=\frac{{\rm exp}\{\sum_{i}(\mathbf{w}_{CRF}^{l_{i}}\cdot \mathbf{h}^{fw}_{i}+b_{CRF}^{{l}_{i-1}, l_{i}})\}}{\sum_{\mathbf{\hat{y}}'}{\rm exp}\{\sum_{i}(\mathbf{w}_{CRF}^{l'_{i}}\cdot \mathbf{h}^{fw}_{i}+b_{CRF}^{l'_{i-1}, l'_{i}})\}},
\end{equation}
where $\mathbf{\hat{y}}'$ denotes an arbitary label sequence; $\mathbf{w}_{CRF}^{l_{i}}$ is the model parameter for $l_{i}$, and $b_{CRF}^{l_{i-1}, l_{i}}$ is the bias for $l_{i-1}$ and $l_{i}$. As Fig.~\ref{fig:framework} shows, we use CRF($s$) and CRF($t$) for the different label sets in the source and target domains respectively.

To effectively search the highest scored label sequence, the first-order Viterbi algorithm is used. We consider the result distribution vector $\mathbf{res}^{CEE}_{i}$ for the word $x_{i}$ in the given sentence $s$ as a one-hot vector.
$res^{CEE}_{i, p} = 1$ only when the word $x_{i}$ is classified as the $p$ th element type. Then, the backward information $\mathbf{bw}_{i}$ for word $x_{i}$ is obtained by Eq.~\ref{eq:latent class} and~\ref{eq:forward and backward}.

\subsection{Joint training}
\label{subsec:traning objectives}
Given the labeled dataset $\mathcal{D}=\{(\mathbf{s}_{n},\mathbf{y}^{CC}_{n},\mathbf{y}^{CEE}_{n})\}_{n=1}^{N}$, loss functions for the clause classification task ($\mathcal{L}_{CC}$) and the contract element extraction task ($\mathcal{L}_{CEE}$) are defined over the training dataset.
Specifically, for the clause classification task, the cross-entropy loss function is used as follows:
\begin{equation}
    \small
    \label{eq: loss CC}
    \mathcal{L}_{CC} = -\frac{1}{N}\sum_{n=1}^{N}{\mathbf{y}^{CC}_{n}{\rm log}(p(\mathbf{\hat{y}}^{CC}_{n}))}.
\end{equation}
For the contract element extraction task, the sentence-level negative log-likelihood loss is adopted for the training process:
\begin{equation}
    \small
    \label{eq: loss CEE}
    \mathcal{L}_{CEE} = -\frac{1}{N}\sum_{n=1}^{N}{{\rm log}(p(\mathbf{y}^{CEE}_{n}|\mathbf{s}_{n}))}.
\end{equation}
And the overall loss is defined as follows:
\begin{equation}
    \small
    \label{eq: loss}
    \mathcal{L} = \sum_{d\in\{\mathcal{S}, \mathcal{T}\}}{\lambda^{d}(\mathcal{L}^{d}_{CEE}+\lambda^{t}\mathcal{L}^{d}_{CC})} + \frac{\lambda}{2}\|\Theta \|^{2},
\end{equation}
where $\lambda^{d}$ is the domain weight and $\lambda^{t}$ is the task weight. $\lambda$ is the regularization weight for the parameter set $\Theta$.

As Algorithm~\ref{algorithm:multi-task training} shows, we design a cross-domain and multi-task training process. Before training, the C-E graphs are constructed (line 1). In each training time step, we encode the sentence-level feature $\mathbf{H}^{d,t}$ (line 4) and type representations, $\mathbf{ele}^{d}$ and  $\mathbf{cla}^{d}$ (line 5). Then, the result distribution $\mathbf{res}^{d, CC}$ for clause classification is predicted (line 6) and forward information $\mathbf{fw}^{d}$ is computed based on the result distribution (line 7). For the contract element extraction, the element labels $\mathbf{\hat{y}}^{CEE}$ are calculated with forward information (line 8). Next, the backward information $\mathbf{bw}^{d}$ is computed (line 9) and the final predictions $\mathbf{\hat{y}}^{CC}$ for clause classification are verified (line 10). Finally,  the overall loss function $\mathcal{L}$ is obtained (line 11) and the model parameters are jointly updated (line 13).
\begin{algorithm}[t]
    \caption{Joint Training.}
    \label{algorithm:multi-task training}
    \begin{algorithmic}[1]
    \Require Training data $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$; parameter sets for the context encoder $\theta_{cont}$, C-E relation encoder $\theta_{C-E}$ (including $\mathbf{W}_{c\rightarrow e}$, $\mathbf{W}_{e\rightarrow c}$, $\mathbf{W}_{c}$, $\mathbf{W}_{e}$, $\alpha^{e}$, and $\alpha^{c}$), and inference layer $\theta_\mathit{infer}$ (including $\mathbf{W}_{f}$, $\mathbf{W}_{b}$, $\mathbf{w}_{CC}$, $b_{CC}$, $\mathbf{w}_{CRF}$, and $b_{CRF}$).
    \Ensure Target-domain model;
    \State Contruct the C-E graphs $G^{\mathcal{S}}$ and $G^{\mathcal{T}}$ for both domains (Eq.~\ref{eq:constructions});
    \While{Training process not terminated}
    \For{$d$ in $\{\mathcal{S}, \mathcal{T}\}$}
    \State Compute sentence-level feature $\mathbf{h}^{d,t}\leftarrow {\rm BiLSTM(\mathbf{v})}$

    \mbox{}\hfill(Eq.~\ref{eq:bi-lstm});
    \State Calculate class representations $\mathbf{ele}^{d}$, $\mathbf{cla}^{d}$ (Eq.~\ref{eq:relation encoder});
    \State Predict clause labels $\mathbf{res}^{d, CC}$ with $\mathbf{h}^{d,CC}$ and $\mathbf{cla}^{d}$ (Eq.~\ref{eq: res CC});
    \State Compute the forward information $\mathbf{fw}^{d}$ with $\mathbf{res}^{d, CC}$

    \mbox{}\hfill(Eq.~\ref{eq:forward and backward});
    \State Predict element labels $\mathbf{\hat{y}}^{CEE}$ with $\mathbf{h}^{d,CEE}$ and $\mathbf{fw}^{d}$;

    \mbox{}\hfill(Eq.~\ref{eq:h FD}--\ref{eq: CRF});
    \State Compute the backward information $\mathbf{bw}^{d}$ with $\mathbf{res}^{d, CEE}$

    \mbox{}\hfill(Eq.~\ref{eq:forward and backward});
    \State Verify the final prediction $\mathbf{\hat{y}}^{CC}$ using $\mathbf{bw}^{d}$ (Eq.~\ref{eq:y CC});
    \State Compute the overall loss function $\mathcal{L}$ (Eq.~\ref{eq: loss CC}--\ref{eq: loss});
    \EndFor
 \State Update $\theta_{cont}$, $\theta_{C-E}$, and $\theta_\mathit{infer}$ based on $\mathcal{L}$.
 \EndWhile
    \end{algorithmic}
    \end{algorithm}
\section{Related work}
\label{sec:related work}
We survey related work along three dimensions:
\begin{enumerate*}
\item legal information retrieval and extraction,
\item contract element extraction, and
\item cross-domain named entity recognition.
\end{enumerate*}

\subsection{Legal information retrieval and extraction}
The digitization of legal documents has given rise to the development of legal information retrieval systems~\citep{DBLP:conf/jurix/MaxwellS08, DBLP:conf/coling/KienNBTNP20}. Numerous challenges for legal information retrieval and extraction have been presented~\citep{DBLP:journals/jd/Chu11,DBLP:conf/icail/KanoKGS17}. Traditional legal search systems are keyword-based, relying heavily on the professional knowledge of end-users. Recently, to alleviate the dependence on user's expertise and improve retrieval effectiveness, a significant amount of effort~\citep{do2017legal, perera2018legal, DBLP:journals/ail/TranNTS20, DBLP:conf/coling/KienNBTNP20} has been devoted to the automatic classification of legal documents and queries. The extraction of key legal concepts can also facilitate the retrieval process~\citep{DBLP:conf/icail/TranNS19, DBLP:conf/emnlp/BorchmannWGKJSP20}.

To the best of our knowledge, there is no previous work on legal information retrieval and extraction that focuses on cross-domain \acf{CEE}.

\subsection{Contract element extraction}
The goal of \ac{CEE} is to recognize essential legal elements, such as execution date, jurisdiction, and amount, in legal documents~\citep{curtotti2010corpus,indukuri2010mining}.
Early \ac{CEE} methods are mainly rule-based or traditional statistical methods.
\citet{chalkidis2017extracting} introduce 11 contract element types and extract contract elements based on Logistic Regression and SVM with hand-crafted features.
\citet{garcia2017cliel} design a system named CLIEL for extracting core information from commercial law documents. Specifically, CLIEL identifies five types of contract elements using rule-based layout detection. \citet{azzopardi2016integrating} propose a hybrid approach based on regular expressions and provide a contract editing tool for lawyers.

Recent \ac{CEE} methods are developed with deep learning and formulate the \ac{CEE} task as sequence labeling.
\citet{chalkidis2017deep} explore deep learning methods for the \ac{CEE} task and employ a BiLSTM without manually written rules.
\citet{sun2019toi} define seven semantic-specific clause categories and introduce a TOI pooling layer for the nested elements.
\citet{chalkidis2019neural} revisit the \ac{CEE} task and explore how sequence encoders, CRF layers, and input representations affect the extractors.

Existing \ac{CEE} approaches cannot be applied to the cross-domain scenario because of differences in context genres and element types.

\subsection{Cross-domain named entity recognition}
The cross-domain \ac{NER} task aims to identify named entities in a target domain using the shared knowledge from a source domain.
Recently, methods based on deep neural networks have been proposed for cross-domain \ac{NER}.
\citet{pan2013transfer} design transfer joint embeddings for this task.
\citet{DBLP:conf/emnlp/QuFZHB16} model the correlation between source and target entity types with a two-layer neural network.
To investigate the transferability of model components, \citet{DBLP:conf/iclr/YangSC17} present three sharing architectures for different transfer learning scenarios.
Moreover, \citet{DBLP:conf/acl/JiaXZ19} propose a parameter generation network and incorporate the language modeling (LM) task to deal with zero-shot learning settings.
Unlike the above parameter-sharing frameworks, parameter transfer approaches first initialize the target model on the source domain \ac{NER} or LM~\citep{DBLP:conf/lrec/LeeDS18, sachan2018effective}, and then fine-tune the original model on the labeled \ac{NER} data from the target domain.
\citet{DBLP:conf/emnlp/LinL18} add three neural adaptation layers (word adaptation, sentence adaptation, and output adaptation) to an existing \ac{NER} method.
\citet{DBLP:conf/acl/JiaZ20} transfer entity type level knowledge using a multi-cell compositional LSTM structure and model each entity type using a separate cell state. For a thorough review of other techniques,  please refer to~\citep{li2020survey}.

We mainly focus on cross-domain \acf{CEE}. To the best of our knowledge, ours is the first study to concentrate on this task. The most closely related task is the cross-domain \ac{NER} task. However, cross-domain contract element extractors need to address adaptations of larger extraction zones and more fine-grained element types.
In our proposed model \ac{Bi-FLEET}, to handle the transfer of fine-grained element types, invariant relations between clauses and elements are captured by constructing a \ac{C-E} graph across domains (see below). In addition, a multi-task framework with a bi-directional feedback scheme is designed to reduce the impact of more diverse contexts in larger extraction zones.

\section{Experimental Results}
\label{sec:results}

Prior to addressing our research questions RQ1 and RQ2, we examine the training process of Bi-FLEET. In particular, we examine whether cross-domain \acf{CEE} and \acf{CC} conflict with each other during the training process.
We use the I2C dataset for this purpose. In Fig.~\ref{fig:development}, the learning curves of cross-domain CEE and CC against the training epochs are shown. MULTI-TASK (LSTM) and MULTI-TASK+PGN are the cross-domain baselines. For comparison, we also show the learning curves of \ac{Bi-FLEET} (LSTM) and \ac{Bi-FLEET} (PGN). We see that:
\begin{itemize}[leftmargin=*,nosep]
  \item F1-scores of all models increase continuously before 60 training epochs, and reach a stable level when the number of training epoch rises to 100.
  \item Cross-domain CEE and CC share the same improvement trend, and there is no conflict between the two tasks during the training process, indicating that both tasks can adopt the same form of feature space.
\end{itemize}
\begin{figure}
  \centering
  \includegraphics[width=0.75\linewidth]{figures/develop_experiment.png}
  \caption{Learning curves on the I2C dataset.}
  \label{fig:development}
\end{figure}

\subsection{Cross-domain CEE}
\label{subsection:CEE}
\begin{table}[t]
  \small
  \setlength\tabcolsep{3.5pt}
  \caption{Results on two cross-domain CEE datasets. Significant improvements against the best baseline with the same context encoder are marked with $\ast$ (t-test, $p < 0.05$).}
  \label{tab:cross-domain CEE results}
  \begin{tabular}{l ccc ccc}
    \toprule
    \multirow{2}{*}{Model} & \multicolumn{3}{c}{I2C} & \multicolumn{3}{c}{C2I}\\
    \cmidrule(r){2-4}
    \cmidrule(r){5-7}
    ~ & P & R & F1 & P & R & F1 \\
  \midrule
    BILSTM (Random)  & 60.35  & 64.02 & 62.13  & 63.65  & 68.32 & 65.90 \\
    BILSTM  & 61.29  & 65.18 & 63.18  & 64.92  & 69.87 & 67.30 \\
    \midrule
    MULTI-TASK (Random) & 62.14 & 65.89 & 63.96 & 65.30 & 70.36 & 67.73 \\
    MULTI-TASK (LSTM)  & 63.21 & 67.50 & 65.28 & 65.86 & 71.61 & 68.62 \\
    MULTI-TASK+PGN  & 63.67 & 67.77 & 65.66 & 65.99 & 71.91 & 68.83 \\
    \citet{DBLP:conf/acl/JiaXZ19}  & 62.57 & 66.96 & 64.69 & 65.45 & 70.84 & 68.04 \\
    \midrule
    BERT-BASE  & 67.57 & 71.25 & 69.36 & 68.72 & 75.70 & 72.04 \\
    \midrule
    \ac{Bi-FLEET} (LSTM) & $66.07$\rlap{$^{\ast}$} & $70.20$\rlap{$^{\ast}$} & $68.07$\rlap{$^{\ast}$} & $67.98$\rlap{$^{\ast}$} & $75.26$\rlap{$^{\ast}$} & $71.44$\rlap{$^{\ast}$}\\
    \ac{Bi-FLEET} (PGN)  & $66.53$\rlap{$^{\ast}$} & $70.45$\rlap{$^{\ast}$} & $68.43$\rlap{$^{\ast}$} & $68.31$\rlap{$^{\ast}$} & $75.37$\rlap{$^{\ast}$} & $71.69$\rlap{$^{\ast}$} \\
    \ac{Bi-FLEET} (BERT) & $\mathbf{70.17}$\rlap{$^{\ast}$} & $\mathbf{73.75}$\rlap{$^{\ast}$} & $\mathbf{71.92}$\rlap{$^{\ast}$} & $\mathbf{70.71}$\rlap{$^{\ast}$} & $\mathbf{78.69}$\rlap{$^{\ast}$} & $\mathbf{74.49}$\rlap{$^{\ast}$} \\
  \bottomrule
\end{tabular}
\end{table}

We turn to RQ1. Table~\ref{tab:cross-domain CEE results} shows the experimental outcomes for the cross-domain CEE on the I2C and C2I datasets. To investigate the influence of pretrained word embeddings, we set up two more baseline models, BILSTM (Random) and MULTI-TASK (Random). In our experiments, the multi-cell compositional LSTM based models~\citep{DBLP:conf/acl/JiaZ20} overfit very easily, since their parameter counts increase drastically with element types; for this reason we do not report the results of multi-cell compositional LSTM based methods.
Based on Table~\ref{tab:cross-domain CEE results}, we can draw the following conclusions:

\begin{itemize}[leftmargin=*,nosep]
  \item Cross-domain CEE is challenging, and for most baselines the F1-score is substantially less than 0.7.
  In contrast, our proposed framework, Bi-FLEET, can effectively transfer contract element extractors from one domain to another.

  \item The proposed model \ac{Bi-FLEET} significantly outperforms the baselines. Compared to the baseline models with the same context encoder, \ac{Bi-FLEET} (LSTM), \ac{Bi-FLEET} (PGN) and \ac{Bi-FLEET} (BERT) can obtain a consistent increase in P, R, and F1-scores, respectively. That is, integrated with the C-E relation encoder and bi-feedback scheme  improves the performance across the board.
  \item In comparison with target-domain only methods, all of the cross-domain methods achieve significant improvements, indicating the importance of source-domain information. Compared to methods with randomly initialized word embeddings, including BILSTM (Random) and MULTI-TASK (Random), BILSTM and MULTI-TASK (LSTM) that use pretrained word embeddings are able to obtain large performance enhancements. Combining with a pretrained LM is also helpful for the cross-domain task, since BERT-based methods perform best among all methods.
\end{itemize}

\noindent%
In summary, the C-E relation encoder and bi-feedback scheme enhance the performance. Encoding relations between elements and clauses, as well as capturing interactions between clause classification and contract element extraction, are both beneficial for the cross-domain CEE problem.

\negskip
\subsection{Cross-domain NER}
\label{subsection:NER}
\begin{table}[t]
  \small
  \caption{Results on three cross-domain NER datasets. Significant improvements against the best baseline with the same context encoder are marked with $\ast$ (t-test, $p < 0.05$).}
  \label{tab:cross-domain NER results}

  \begin{tabular}{lccc}
    \toprule
    \multirow{2}{*}{Model} & \multicolumn{3}{c}{Datasets (F1)} \\
    \cmidrule(r){2-4}

    ~ & BioNLP & Broad Twitter & Twitter \\
    \midrule
    BILSTM  & 79.24  & 72.98 & 77.18 \\
    MULTI-CELL LSTM  & 78.76 &72.54 & 77.05 \\
    \midrule
    MULTI-TASK (LSTM)  & 81.06 &73.84 & 79.55 \\
    MULTI-TASK+PGN  & 81.17 & 73.70 & 80.07 \\
    \citet{DBLP:conf/acl/JiaXZ19}  & 79.86 & -- & -- \\
    \citet{DBLP:conf/acl/JiaZ20}  & 83.12 & 74.82 & 81.37 \\
    \midrule
    BERT-BASE  & -- & 77.28 & 83.77 \\
    BIOBERT-BASE  & 85.72 & -- & --\\
    \citet{DBLP:conf/acl/JiaZ20} (BERT)  & 86.96 & 78.43 & 85.80 \\
    \midrule
    \ac{Bi-FLEET} (LSTM) & $82.32$\rlap{$^{\ast}$} & $74.91$\rlap{$^{\ast}$} & $80.78$\rlap{$^{\ast}$}\\
    \ac{Bi-FLEET} (PGN)  & $82.46$\rlap{$^{\ast}$} & $75.11$\rlap{$^{\ast}$} & $81.35$\rlap{$^{\ast}$}\\
    \ac{Bi-FLEET} (MULTI-CELL) & $84.76$\rlap{$^{\ast}$} & $76.02$\rlap{$^{\ast}$} & $82.24$\rlap{$^{\ast}$}\\
    \ac{Bi-FLEET} (BERT) & $\mathbf{89.02}$\rlap{$^{\ast}$} & $\mathbf{80.71}$\rlap{$^{\ast}$} & $\mathbf{87.73}$\rlap{$^{\ast}$}\\
  \bottomrule
\end{tabular}

\end{table}
To investigate the generality of \ac{Bi-FLEET}, we turn to RQ2 and report on experiments on the cross-domain NER datasets. F1-scores are adopted here to evaluate the overall performance. As baselines, we use the results reported in \citep{DBLP:conf/acl/JiaZ20}. Based on the results in Table~\ref{tab:cross-domain NER results}, we arrive at the following conclusions:

\begin{itemize}[leftmargin=*,nosep]
  \item Even if labels are generated in an unsupervised manner, Bi-FLEET can still effectively transfer named entity recognizers.
  \item Bi-FLEET outperforms state-of-the-art cross-domain NER methods. Lacking source-domain information, target-domain only methods can hardly handle the cross-domain NER task. Compared to cross-domain and BERT-based NER methods, methods based on the Bi-FLEET framework attain the highest F1-scores on all three datasets.
\end{itemize}

\noindent%
In summary, the Bi-FLEET framework cannot only effectively address the cross-domain \ac{CEE} problem, but also achieves state-of-the-art results on the cross-domain \ac{NER} task.