text:
\section{Appendix}
\subsection{Full Set of Rules}
\label{full_set_rules}
Predicates:
\begin{itemize}
    \item INTERSECTIONUNDER(object, class, image) - Known predicate (observation). Describes if the area under the object is of a particular class in the image. The classes are 'r' - road, 'pw' - pedestrian walk, 'gr' - ground. During rule weight optimization obtained from ground truth semantic segmentation, during inference obtained from the predictions of a network.
    \item INTERSECTIONBEHIND(object, class, image) - Known predicate (observation). Describes if the area behind the object is of particular class in the image. The class is 'bckgr' - combined buildings and nature. During rule weight optimization obtained from ground truth semantic segmentation, during inference obtained from the predictions of a network.
    \item DISTANCE(object1, object2, image) - Known predicate (observation). Describes if two objects are close to each other in the image. Inferred from the RGB input and corresponding depth map.
    \item HASCOLOR(object, color, image) - Known predicate (observation). Describes if the object has the color in it. The colors are 'white', 'blue', 'yellow', 'red'. Inferred from the RGB input.
    \item ISTYPE(object, class, image) - Unknown predicate (target). Defines the class of the object. Class can be one of 'ped' (pedestrian), 'tr\_sign' (traffic sign), 'tr\_signal' (traffic signal)
\end{itemize}

\begin{small}

\begin{verbatim}
0.768: INTERSECTIONUNDER(O1, 'r', I) >> ISTYPE(O1, 'ped', I) ^2

0.957: INTERSECTIONUNDER(O1, 'pw', I) >> ISTYPE(O1, 'ped', I) ^2

0.789: INTERSECTIONUNDER(O1, 'gr', I) >> ISTYPE(O1, 'ped', I) ^2

3.2E-5: INTERSECTIONBEHIND(O1, 'bckgr', I) >> ISTYPE(O1, 'ped', I) ^2

0.234: ~INTERSECTIONBEHIND(O1, 'bckgr', I) & ~INTERSECTIONUNDER(O1, 'pw', I)
& ~INTERSECTIONUNDER(O1, 'gr', I) & ~INTERSECTIONUNDER(O1, 'r', I)
>> ~ISTYPE(O1, 'ped', I) ^2

0.372: (O1
>> ISTYPE(O1, 'ped', I) ^2

0.278: (O1
& ISTYPE(O3, 'tr_sign', I) >> ISTYPE(O1, 'ped', I) ^2

0.002: (O1
& ISTYPE(O4, 'tr_signal', I) & DISTANCE(O2, O4, I) >> ISTYPE(O1, 'ped', I) ^2

0.899: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I) & INTERSECTIONUNDER(O1, 'r', I)
>> ISTYPE(O1, 'ped', I) ^2

0.246: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I) & INTERSECTIONUNDER(O1, 'pw', I)
>> ISTYPE(O1, 'ped', I) ^2

0.512: INTERSECTIONUNDER(O1, 'gr', I) & ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I)
>> ISTYPE(O1, 'ped', I) ^2

0.213: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I)
& INTERSECTIONBEHIND(O1, 'bckgr', I) >> ISTYPE(O1, 'ped', I) ^2

1.4E-4: DISTANCE(O1, O3, I) & ISTYPE(O3, 'tr_sign', I) >> ISTYPE(O1, 'ped', I) ^2

0.317: ISTYPE(O4, 'tr_signal', I) & DISTANCE(O1, O4, I) >> ISTYPE(O1, 'ped', I) ^2

0.542: HASCOLOR(O3, 'white', I) & HASCOLOR(O3, 'blue', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.561: HASCOLOR(O3, 'yellow', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.841: HASCOLOR(O3, 'white', I) & HASCOLOR(O3, 'blue', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.181: HASCOLOR(O3, 'red', I) & HASCOLOR(O3, 'white', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.709: ~HASCOLOR(O3, 'white', I) & ~HASCOLOR(O3, 'blue', I)
& ~HASCOLOR(O3, 'yellow', I) & ~HASCOLOR(O3, 'red', I) >> ~ISTYPE(O3, 'tr_sign', I) ^2

0.476: ~INTERSECTIONBEHIND(O3, 'bckgr', I) >> ~ISTYPE(O3, 'tr_sign', I) ^2

1.0 * ISTYPE(O, +Type, I) = 1.0 .
\end{verbatim}
\end{small}

\pagebreak
\subsection{Evaluation Plots}
\label{eval_plots}

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/ped_wrong.png}
        \caption{Scores distribution for the objects "pedestrians".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_sign_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_sign_wrong.png}
        \caption{Scores distribution for the objects "traffic sign".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_signal_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_signal_wrong.png}
        \caption{Scores distribution for the objects "traffic signal".}
    \end{subfigure}
    \caption{The scores of the network compared to the scores of the rules on the test dataset. It should be noted that the objects distribution is unbalanced in the test set: amount of objects "pedestrian" is the smallest, followed by "traffic signal". The largest number of objects are "traffic sign". The network can not distingusih between traffic signs or signals and we thus only report the (joined) result as ``tr\_sign'' for the wrong classes.}
    \label{fig:brightness5_preds}
\end{figure}

\pagebreak
\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/ped_wrong.png}
        \caption{Scores distribution for the objects "pedestrians".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_sign_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_sign_wrong.png}
        \caption{Scores distribution for the objects "traffic sign".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_signal_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_signal_wrong.png}
        \caption{Scores distribution for the objects "traffic signal".}
    \end{subfigure}
    \caption{The scores of the network compared to the scores of the rules on the corrupted test dataset. The corruption selected here is "brightness" with the severity $5$. One can directly see that the network scores quality drops substantially while rules are nearly not affected by the corruption.}
    \label{fig:brightness5_preds2}
\end{figure}

\pagebreak
\subsection{Corruption Plots}
\label{all_corr_plots}

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``brightness'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``fog'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``frost'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[d]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``snow'' corruption.}
    \end{subfigure}
    \caption{\small{Different type of corruptions with increasing severity. One can notice that corruptions can affect the rules scores: so brightness and snow add white color to the image, with that the rules for traffic signs are becoming more confident--based on the hasColor predicate. Another aspect is that amount of 'Pedestrian' objects is very little (~20 compared to ~2000 of 'Traffic sign') and because of this network mostly fails to distinguish them.}}
    \label{fig:all_severity_corruptions}
\end{figure}

\section{Approach}
The goal of our research is to find ways in which high-level domain knowledge can improve and robustify the predictions of deep learning models in the context of autonomous driving.
While multiple lines of research discussed in \secref{sec:related_work} make use of domain knowledge exclusively for building a model, it is our aim to combine the data-driven approach and high-level knowledge during inference.
\par
For various traffic scenes, humans can typically use common and empirical knowledge about behaviour and features of the objects to infer their type and in-scene relations, even if generalizing to unseen scenarios.
Considering different features and interrelations of objects, this human common knowledge can partially be encoded as a set of simple rules, some of which are showcased in \figref{fig:psl-motivation}.
The way a human would reason about such a traffic scene cannot be based on hard Boolean logic, though, in which rules hold either always or never.
For example, an object having a zebra walk underneath and standing close to a traffic sign has a high probability to be a human, as depicted in \figref{fig:psl-motivation}.
But there is a chance that this could also be something else, e.g.,\  a dog or some obstacle in a construction site.
Therefore, to integrate such high-level human knowledge in the same ``uncertain'' manner into a neural network model, we need to define logical rules with relaxed truth values.
Structuring the rules upon the identification and classification of objects aligns with the task of semantic segmentation that lends itself well to finding image regions with a common semantic, which we treat as object proposals.
We selected PSL as a framework for knowledge representation for two reasons:
First, it allows us to flexibly use soft truth values that are suitable to define both strict and loose rules, and, second, it can be fine-tuned on data, hence, distilling high-level knowledge directly via the given annotations.
\par
We first introduce PSL in \secref{sec:PSL} and briefly describe the input requirements for a PSL program.
Based on this, \secref{sec:DerivingRules} elaborates on how we define object entities and rules in our domain of interest followed by a description of the approach to combine the PSL rules with semantic segmentation.

\subsection{Probabilistic Soft Logic}
\label{sec:PSL}

Probabilistic Soft Logic (PSL)~\citep{Bach2017HLMRFsPSL,kimmig2012short} is a programming framework for probabilistic reasoning in domains with relational structure.
Typical applications of PSL are collective classification (of interlinked objects in structured data), entity resolution (finding a mapping from multiple interrelated references to the true set of underlying entities) and link prediction (predicting whether a relationship exists between two entities).
PSL makes use of Hinge-Loss Markov Random Fields (HL-MRF), a graphical model, and applies these to structured machine learning problems.
Graphical models define a type of loss function that represents a probability score for each possible value configuration respecting the graph-encoded dependencies among the variables.
For HL-MRFs~\citep{Bach2013HingeLoss,Bach2017HLMRFsPSL} the continuous loss functions are defined using the hinge loss, which allows for efficient convex optimization techniques during inference.
\par
The HL-MRFs are defined using declarative rules, whose importance can be indicated by optional non-negative \textit{weights}.
These rules are formulated using first order logic.
In contrast to the binary truth values \emph{true} and \emph{false} of Boolean algebra, PSL uses continuous soft truth values in the interval $[0,1]$.
\begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{psl-rule-example}
    \caption{PSL rules have a conjuncted body and a single literal head. \emph{Atoms} (e.g.,\  \texttt{IntersectionUnder(O1, 'r')} are the basic elements that are used for reasoning.
    An atom is a combination of \emph{predicates} (here: \texttt{IntersectionUnder$\backslash$2}) and a sequence of variables and constants equal to the predicate's arity.
    A \emph{constant} indicates a concrete element (e.g.,\  ``\texttt{ped}'' is the concrete object class pedestrian) and a \emph{variable} denotes a placeholder later to be substituted by concrete elements.
    Here, the predicate \texttt{IntersectionUnder} has an arity of two, thus, the atom consists of the variables \texttt{O1} indicating an object plus the constant ``\texttt{r}'' indicating the underground type is the road.
    The identifier \texttt{O1} will be substituted for all available objects during inference while the constant ``\texttt{r}'' will stay the same.
    The \texttt{O1} identifier is used in three atoms indicating that during inference, each object-1-variable will be substituted by the same constant (same holds for object-2-variable \texttt{O2}, respectively).
    The exponent \texttt{\^}\texttt{2} at the end of the rule indicates that the quadratic loss is chosen. This example rule can be interpreted as \emph{``if an object 1 is placed on a sidewalk and in close distance to another object 2, which is a traffic signal, there is a probability that this object 1 is a pedestrian''}.}
    \label{fig:psl-rule-example}
\end{figure*}
For a detailed explanation of the elements of a PSL rule, see the example depicted in \figref{fig:psl-rule-example}.
\par
Conceptually, we require that the ``satisfaction'' of rules can be measured in a soft-Boolean way if all ``variables'' have a defined value. In this case, we can aggregate the set of all rules to a global score using individual weights for each rule.
Given a set of ``ground truth'' observations, the weights can be fine-tuned such that the importance of each rule agrees with the ground truth observations.
The resulting fixed set of weights then allows us to perform inference over incomplete observations by choosing the most probable value for unknown variables.

In detail, this means each atom is mapped to a soft truth value -- before inference for all known variables, and during inference for all unknown variables
To compute the continuous truth value of the whole rule, values on atom-level are combined using the \emph{Lukasiewicz t-norm}~\citep{klir1995fuzzy} as a relaxation for the logical conjunction, disjunction, and negation.
Each rule's \emph{satisfaction} is calculated indicating to what degree that rule is fulfilled given the current configuration of values.
A rule is satisfied if the head is assigned at least the same truth value as the body, and a fully satisfied rule has a value of $1$.

All possible variable configurations for the rules are combined to a probability distribution, in which those that satisfy more ground rules are more probable.
If a weight is assigned to a rule, this indicates the importance of this rule to be satisfied in relation to the other rules and their corresponding weights.
Rules that are unweighted induce hard constraints which require the rule to be met at all times.
During inference, the most probable value configuration for unknown predicates is found given evidence (via known predicates).
Given ground truth, the weights of rules can also be learned with maximum-likelihood estimation.
Thus, the importance of each rule in the rule set can be learned from the data itself.
\par
Apart from these rules, a PSL program consists of a database that defines the elements of the domain of interest as well as their relations in the form of predicates.
The database is built up from known predicates (i.e.,\  atoms that are completely observed) and unknown predicates (i.e.,\  atoms that are unobserved and must be inferred) that are part of one of the three following sets:
(i) Necessary for all known predicates is the set of \textit{observations}. These are variable combinations with an associated soft truth value in the form of actual extracted and known data for each.
(ii) Necessary for all unknown predicates is the set of \textit{targets} that is a list of all atoms (combinations of variables) that should be inferred, therefore without any associated truth values.
(iii) For weight learning a set of \textit{truths} with actual known data (likewise observations) to be used as ground truth for all unknown predicates is necessary.
Inference in PSL is done based on the first two sets, while evaluation of the inferred predicates can be done with the test set of the \textit{truths}.

There are two types of rules that are supported in PSL: logical and arithmetic rules.
We use logical rules to encode dependencies between atoms as for example depicted in the rule in \figref{fig:psl-rule-example}.
Additionally, we need to imply an arithmetic constraint to restrict the substitution for variables further.
In our setting, we want to reason about the class of an object in the scene that can only be one of three classes, constraiing the corresponding truth values to sum up to $1$.

\subsection{Deriving Rules from Domain Knowledge}
\label{sec:DerivingRules}
\par
The overall goal of employing PSL to camera-based perception in autonomous driving is to identify parts of common knowledge about the traffic scene that can be readily extracted and represent it in a form that facilitates integration into the model performing inference.
Based on the input requirements for a PSL program as defined above, we first need a database, i.e.,\  objects that we want to reason about as well as their relationships encoded within chosen predicates.
Then, we define a set of rules that carries additional relational knowledge about the domain of interest that is not yet covered by the annotated data and thus can be deployed to support and validate the perception task.
When defining an initial set of rules, one therefore has to carefully choose the objects they link up: it should be simple to both describe their behavior and to extract them from raw data.

Here, we choose pedestrians, traffic signs and traffic signals as our three types of objects of interest, for which we want to infer the class.
To identify these objects in a logical setting, one would need to fall back to low-level sensor interpretation.
Typically, autonomous vehicles have diverse sensors from which objects, their attributes (e.g.,\  colors, shapes) and their surroundings (e.g.,\  area underneath, background) and their relationships (e.g.,\  distance to other objects) can be derived.
\par
For our case study, however, we solely focus on the aspect of how external knowledge can be meaningfully integrated into an existing prediction workflow.
We therefore purposefully deploy an oracle to extract the object entities thereby neglecting additional challenges resulting from, e.g.,\ ,  unreliable sensor readings and object extraction.
Our solution is to employ the connected components on ground truth segmentation masks using additional smoothing to avoid fractured objects due to occlusion.
Additionally, we emulate a depth sensor by propagating the RGB inputs to a publicly available\footnote{\url{https://github.com/nianticlabs/monodepth2}} version of MonoDepth2~\citep{godard2019digging}.
This ``oracle sensor'' information already allows for creating a meaningful set of objects as basis for the rules.

Subsequently, the rules describing the objects have to be defined.
To this end, we develop an initial set of rules, for which we refine the weights later on.
As we want to reason about the objects of our three selected classes, we define our target predicate \texttt{isType(Obj,Class)}.
This is used to later infer a soft truth value for each object and the possible three classes, resulting in one class being more likely than the others.
We then base the logical reasoning for an object class on the semantic features, on the surroundings of the objects, and on their relations.
Surroundings of the objects are described with the observations (known predicates) \texttt{intersectionUnder(Obj,Class)} and \texttt{intersectionBehind(Obj,Class)}.
For example, to infer the class of an object to be ``traffic sign'' \texttt{intersectionUnder} should have a high likelihood for the classes ``pedestrian walk'' or ``street''.
We integrate color as a semantic feature as it is especially important for traffic signs and traffic signals with the predicate \texttt{hasColor(Obj,Color).}
Rules we defined using the color predicate are for example inspired by the officially specified color combinations of common German traffic signs.
The relations between objects are specified as a distance value based on depth readings with the known predicate \texttt{distance(Obj1,Obj2)}.
The defined rules for our setup can be found in appendix~\ref{full_set_rules}.
\par
As a final step, the knowledge gained from the set of data via probabilistic rules has to be combined with the predictions from the neural network, in this case the semantic segmentation masks.
For our conceptual demonstration we follow a simplified approach integrating scores obtained from the network as prior assumptions within the rule based framework.
\begin{figure}[h]
    \centering
    \includegraphics[width=\columnwidth]{images/psl_approach_diagram.png}
    \caption{Overall flow of information: Oracles are serving as a source of information for objects and depth maps and also observations for weights learning; predictions produced by the neural network are combined with PSL rules via priors.}
    \label{fig:approach_diagram}
\end{figure}
\par
Overall, the practical approach can be described as follows (see \figref{fig:approach_diagram}):
\begin{enumerate}
    \item Identify objects to reason about and extract them from sensor data (oracle using ground truth semantic segmentation);
    \item Construct the set of rules using expert and common knowledge;
    \item Prepare the required information from lower level sensors for the rules inference (depth, colors);
    \item Learn the weights of the rules using a training set of recorded camera frames;
    \item Perform inference with the neural network and with the rules on the test set of frames for validation, possibly using network outputs as prior.
\end{enumerate}

\section{Discussion and Conclusion}

We propose a way to reason about high level scene properties, i.e.,\  the type or class of objects, in terms of Probabilistic Soft Logic (PSL).
Taking advantage of human knowledge, we draw information from various domains, ranging from  sensor input such as color to scene properties such as the relations and distances among object.
This approach  enables us to  include additional information from data-driven model predictions, e.g.,\  as prior assumption, and improve their prediction reliability via reasoning.
Our initial evaluation in a simplified setup features a scenario from autonomous driving and indicates that the approach is viable.
Even in tests with highly adverse input augmentations it is possible to robustify the predictions of the employed semantic segmentation network.

Within the PSL framework reasoning is based on rules, i.e.,\  human understandable concepts regarding the interpretation and meaning of object relations or properties.
Besides the possibility to guide the rule creation by human expert knowledge this offers strong interpretability, the set of rules could, e.g.,\ , be audited with respect to consistency.
The relevance of each rule can be adjusted individually, where the necessary weights are either deliberately chosen or learned from a training data set.
The latter case, as demonstrated in our evaluation, can serve to validate the applicability of given rules.
This places PSL in-between human designed and data driven approaches, making it an especially suited candidate to incorporate common knowledge into fundamentally data driven applications.

This work showcases the potential benefits such common knowledge can have for the reliability of a machine learning framework.
At this early stage we, however, neglected the cumbersome step to obtain the entities the PSL framework reasons about.
In practice this information has to be aggregated from various, most likely not fully reliable, sources, for instance stemming from additional sensors (e.g.,\  LIDAR) or alternative processing approaches.
While a probabilistic framework seems appropriate to deal with further uncertain input we restrict this demonstration to only a single ML model providing the semantic segmentation mask.
The entities and their further attributes are instead given by an oracle using the ground truth annotation.
This allows us to test the approach without the necessity to instantiate a full framework.
Another important step is the involvement of human experts for rules formulation. The presented rules are formulated with the common knowledge of the driving situations and environment, but setup specific rules, as well as the rules that are tuning a particular aspect of recognition need multiple iterations over the rules construction process.

With respect to the incorporation of machine learning models we currently only use the provided prediction.
Given the probabilistic nature of both these outputs and the PSL framework the inclusion of prediction uncertainty in terms of, e.g.,\ , calibrated softmax scores could improve the reliability further.
Adding further models as discussed above would additionally require a larger set of rules.
While the used set is sufficient for demonstration purposes enlarging it to cover more attributes seems to be a straightforward way to extend our work.
Specific follow up questions could address a more systematic approach to rule building, which investigates the interplay between the set coverage, reflected by the amount and variety of rules, and its achieved performance, reflected in learned weights and task performance, more closely.
For the case of autonomous driving the incorporation of temporal consistency, requiring, e.g.,\ , that a pedestrian on two consecutive frames should not unexpectedly vanish, could add new aspects to the discussion, possibly including reasoning on the trajectories and intentions of road users.

Especially for domains such as automated driving we believe that the inclusion of human expertise into predictions is a valuable addition to safety and reliability.
Often humans are capable to infer relevant information from context.
For instance a sign indicating the potential presence of children could, by formulation of an appropriate rule, increase the likelihood that a discovered entity is inferred as child.
This inclusion of external knowledge might additionally help counterbalance the rareness of safety relevant situations, which would otherwise make it difficult to learn relevant correlations purely from data.
Deliberately fixing weights or the inclusion of specific rules as above might therefore increase the safety of vulnerable road users (VRUs) without affecting the training and optimization of the underlying machine learning models.

\section{Discussion and Conclusion}

In this work, we propose a way to formulate, evaluate, and take advantage of human knowledge from various domains, such as knowledge about colors or relations between objects, common-sense, and scene understanding.
This enables us to combine model predictions and logical reasoning to improve the reliability of data-driven models.
\par
Our initial evaluation in a simplified setup features a scenario from autonomous driving and indicates that the approach is viable to robustify network predictions.
The importance weights for this initial rule set is optimized with application-relevant data.
Having obtained this optimized rule set, we show that it is possible to validate the output of trained neural networks for semantic segmentation in highly adverse scenarios.
The fact that PSL rules are expressed in a simple and understandable syntax and, on the other hand, weighted by given data endows the system with straightforward layer of interpretability in which the validity of certain predicates can be observed after PSL inference.
\par
In our view, the main caveats that remain to be addressed in future work are high-level object selection and the principled construction of a meaningful set of the rules:
First, in order to focus this work on the aptitude of PSL knowledge integration, we resorted to defining object entities using ground truth data.
Whether it will be possible to use potentialy noisy data from different sensors (e.g.,\  LIDAR) to generate object proposals of sufficient quality remains to be examined.
Second, the rules from our experiments have been selected in a thoughtful but not systematic approach.
This procedure has to be investigated in order to decide if starting from and optimizing over a in some sense complete set of initial rules will result in better outcomes.
\par
There are multiple ways to follow up on the research presented in the paper.
The most straightforward way would be to enlarge the initial set of possible predicates to cover more attributes of the objects, e.g.,\  reasoning about shapes of traffic signs or the width-height-proportion or pedestrians.
When involving more object classes, more complex relations can be represented, e.g.,\  recognizing and treating small pedestrians as children and, thus, attributing unusual and possibly erratic behavior.
Focusing more on the way of knowledge integration, a more elaborate way of incorporating the knowledge from rules into the network could be introduced by taking into account uncertainty measures of the neural network predictions reflecting on what part of the prediction validation by structured knowledge is needed most.
Another direction is to widen reasoning from one frame to a sequences, using the temporally continuous nature of driving scenery.
\par
Especially in domains such as automated driving we also picture the possibility of incorporating empirical knowledge.
With only minor changes to the initial setup, our set of rules could be extended to cover more of such practical knowledge that humans gained through experience in road traffic situations.
For example, rules reflecting the higher possibility for the appearance of pedestrians around bus stops, train stations or in traffic-calmed areas, could be based on the identification of the corresponding traffic signs.
Furthermore, rules could be reweighted to allow for prioritization of vulnerable road user (VRU) classes for added safety.

\section{Evaluation}
The model that we selected for semantic segmentation is ERFNet~\citep{romera2017erfnet} whose lightweight encoder-decoder architecture and limited performance make it straightforward to observe the benefit of knowledge injection via PSL.
We trained ERFNet on the A2D2 public data set \citep{geyer2020a2d2}, which features a large set of environmental scenes for cars on the road.
The data set was separated into independent subsets (by the sequences recorded) for training and testing.
The labels in the data set were simplified to $13$ classes via grouping.
The network was trained using the Adam optimizer with decaying learning rate for $200$ epochs.
The network achieved an mIoU of $0.59$ after training--in particular $0.26$ for 'Pedestrian' class and $0.63$ for 'Traffic sign' and 'Traffic signal'.

Rules selected for the evaluation first have to be ranked with weights.
In our evaluation we obtained the weights using the learning procedure, in which the weights are assigned to the rules based on the training set with ground truth annotations.
The observations for weight learning are extracted from the training images (predicate \texttt{hasColor}), their depth information (predicate \texttt{distance}) and the ground truth semantic segmentation (predicates \texttt{intersectionBehind} and \texttt{intersectionUnder}) using in each case the object entities from the oracle.
The full list of rules we defined along with the learned weights can be found in appendix~\ref{full_set_rules}.
This step of weight learning serves to validate our initially proposed rules discarding those which turned out to be inadequate given the data.
In these cases, the weights assigned were in the order of $10^{-5}$.
One such rule was for example increasing the probability of an object to be a pedestrian if it is close to an object that is a traffic sign.

An independent test set of images is used both for evaluation of the network prediction and rule inference.
The connected components selected as objects for soft rule-based reasoning are compared against the network predictions for the same region.
These regions are produced using the oracle.
We refer to the ratio of correctly classified pixels as the network score for this object.
Note that for weight learning we used information from the ground truth while for inference network predictions are employed.

To evaluate the robustness of the rule predictions we employed image corruptions as described by~\citet{michaelis2019dragon}.
As the most natural we selected four, namely brightness, fog, frost and snow.
Each of them can be applied with five levels of severity, of which $1$ is the weakest (see \figref{fig:corrupt_image} for examples).
The evaluation then follows the same route -- the network infers predictions on the corrupted images and the scores for the objects are computed.
Likewise, rules are inferred on corrupted images and their respective predicted semantic segmentation on them.
The oracle as further source of information is left unaffected by the applied corruptions.
\begin{figure}[th]
    \centering
    \begin{subfigure}{\columnwidth}
        \centering
        \includegraphics[width=0.46\columnwidth]{images/corruptions/20180807145028_camera_frontcenter_000012330.png}%
        \hfill
        \includegraphics[width=0.46\columnwidth]{images/corruptions/brightness_20180807145028_camera_frontcenter_000012330.png}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}{\columnwidth}
        \centering
        \includegraphics[width=0.46\columnwidth]{images/corruptions/fog_20180807145028_camera_frontcenter_000012330.png}%
        \hfill
        \includegraphics[width=0.46\columnwidth]{images/corruptions/frost_20180807145028_camera_frontcenter_000012330.png}
    \end{subfigure}
    \vskip\baselineskip
    \caption{Example of corruptions: original image (top left), brightness, frost and fog corruption (clockwise).}
    \label{fig:corrupt_image}
\end{figure}
The results of the predictions are depicted in \figref{fig:histograms}.
We display the scores for the pedestrian objects, either denoting the correct or incorrect class, in the case of the original images and images corrupted with brightness.
The histograms for the other two object classes is shown in the appendix~\ref{eval_plots}.
One can observe that the scores produced by the rules are nearly not affected--judging by the distribution of histogram peaks and the average score displayed in the legend--while network scores are very sensitive to the corruption of the data.

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{\columnwidth}
        \centering
        \includegraphics[width=0.49\columnwidth]{images/clean_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.49\columnwidth]{images/clean_run/ped_wrong.png}
        \caption{Scores distribution for the objects ``pedestrians'' on the original images.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{\columnwidth}
        \centering
        \includegraphics[width=0.49\columnwidth]{images/brightness5_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.49\columnwidth]{images/brightness5_run/ped_wrong.png}
        \caption{Scores distribution for the objects ``pedestrians'' on the corrupted (brightness, severity 5) images.}
    \end{subfigure}
    \vskip\baselineskip
    \caption{The scores of the network compared to the scores of the rules on the test dataset for pedestrian objects. It should be noted that the distribution of objects is unbalanced in the test set: the number of objects ``pedestrian'' is the smallest, followed by ``traffic signal''. The most frequent class of objects is ``traffic sign''. The network was not trained to distinguish between traffic signs or signals and we thus only report the (joined) result as ``tr\_sign'' for the wrong classes.}
    \label{fig:histograms}
\end{figure}
The final evaluation step is to combine network predictions and rules inference.
We used the scores produced by the network as a prior for the \texttt{isType} predicate in the PSL rules.
\figref{fig:corruption_vs_score} reflects the change in the scores assigned to the objects under increasing severity of corruption.
The scores for the objects of class ``traffic signal'' are presented here, the other classes and corruptions can be seen in the appendix~\ref{all_corr_plots}.
One can observe that the severity of the corruption affects the scores produced by a network while the inference with the rules deteriorates only slightly.
Finally, the robustification of the network scores is also stable and shows better results than network scores on their own.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\columnwidth]{images/severity_corruption/frost_comparison_tr_signal.png}
    \caption{The development of average scores for the class ``traffic signal'' with  increasing severity of the ``frost'' corruption.}
    \label{fig:corruption_vs_score}
\end{figure}
\section{Introduction}
\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{images/psl-motivation-traffic-scene.png}
    \caption{Most humans approach the task of understanding and reasoning about traffic scenes using relations between objects. Given that all of them have been identified, common knowledge --- describing both behaviour and features, e.g.,\  relative position --- can straightforwardly be delineated in terms of simple rules. Some of the rules that we utilize in this paper are exemplified in the above figure.}
    \label{fig:psl-motivation}
\end{figure}

Deep neural networks are an integral part of autonomous driving as they achieve unmatched performance in a number of tasks, in particular image-based environment perception.
Still, they are prone to random or targeted disturbances, which renders robustness a crucial requirement for safety-critical applications.
These disturbances might be of a natural source -- like overexposure, fog, snow -- or of an artificial and purposefully malicious source -- like adversarial attacks.
At the same time, wrong predictions in such tasks are most of the time blatantly obvious for human viewers that use multiple intuitive rules based on their knowledge and experience to reveal what is happening in a given scene.
Neural approaches, in their current form, aim to implicitly learn these rules from data, which makes it necessary to acquire and annotate large amounts of inputs and furthermore introduces the danger of adopting spurious correlations.
The integration of scene knowledge into the training or inference of neural networks seems a viable approach, however, this knowledge ranges from considerations of scene geometry, common knowledge, and rules of human behavior, rendering it a difficult problem \citep{vonrueden2020informed}.
There are not only a number of competing approaches to perform the integration of knowledge itself but also multiple ways of representing the knowledge.
\par
In this paper, we study a way to represent knowledge in form of relations delineated as logic rules between identified objects in a traffic scene (see \figref{fig:psl-motivation}) and examine how these relations can be used to robustify data-driven environment perception.
To this end, we adapt the Probabilistic Soft Logic (PSL) framework~\citep{kimmig2012short} to define and learn relations between objects in the surrounding scene.
We demonstrate the results of initial experiments aimed at quantifying its effect on the robustness of a recent small-scale semantic segmentation neural network.

\section{Related Work}
\label{sec:related_work}
The importance of high-level knowledge injection for image-related tasks is widely accepted~\citep{aditya2019integrating}.
Utilizing background and common-sense knowledge about objects or regions, which goes beyond data annotations, can benefit many computer vision tasks, such as object recognition or semantic segmentation.
Mostly due to its use in robotics and autonomous driving vehicles, semantic segmentation, as one form of low-level scene representation, is currently one of the actively developing fields in image recognition.
\par
In their survey~\citet{siam2017deep} show the development of semantic segmentation from feature-based approaches to the end-to-end deep learning networks.
Among other techniques, they point out and discuss an increase in accuracy achieved by including structural knowledge, which is abundant in the particular task of autonomous driving.
To facilitate knowledge inclusion, according to~\citet{siam2017deep}, many researchers apply conditional random fields (CRF) additionally to deep convolutional networks for classification to improve the accuracy of the predictions.
There are also approaches that are based only on an ontology or logical reasoning for semantic segmentation.
For example, \citet{zand2016ontology} propose to deploy Dirichlet mixture models based on super-pixel segmentation using knowledge from a fixed, user-defined ontology after which prediction is performed using conditional random fields.
Analogously, \citet{leon2020big} use fuzzy logic to perform segmentation and \citet{dasiopoulou2005knowledge} use a high-level semantic ontology for object detection in videos.
\par
In our work, we propose to employ the PSL framework as an expressive and interpretable way to reason about the environment of the autonomous vehicle.
To the best of our knowledge it is the first attempt to combine deep-learning-based semantic segmentation with probabilistic logical reasoning about the environment.
Nevertheless, the idea of employing knowledge via logic rules in autonomous driving tasks is not new.
In fact, \citet{kanaujia2015markov, kardacs2013learning, kembhavi2010did} use Markov logic networks for recognizing complex events in video data.

Related work in the same area of application as ours is by \citet{souza2011probabilistic}, in which the authors use logic reasoning to support a lane detection system with high-level knowledge.
They employ the output of their segmentation method for further reasoning and controlling the movement of the vehicle (or alerting the driver).
Similar to the PSL framework used in our work, these authors adopt Markov logic networks to employ high-level knowledge in the form of logical hard and soft rules.
However, the object-based rules are only used to classify events after recognizing objects (e.g.,\  person, car) within image frames.
In contrast, we employ soft object-based rules to robustify the prediction results of a semantic segmentation neural network for the objects in the image frames itself.
Thus, our goal is to construct rules that analogously would support semantic segmentation in cases when the network itself is compromised (e.g.,\  by an image distortion) in order to prevent completely missing critical objects such as pedestrians, traffic signs or traffic signals.

\section{Appendix}
\subsection{Full Set of Rules}
\label{full_set_rules}
Predicates:
\begin{itemize}
    \item INTERSECTIONUNDER(object, class, image) - Known predicate (observation). Describes if the area under the object is of a particular class in the image. The classes are 'r' - road, 'pw' - pedestrian walk, 'gr' - ground. During rule weight optimization obtained from ground truth semantic segmentation, during inference obtained from the predictions of a network.
    \item INTERSECTIONBEHIND(object, class, image) - Known predicate (observation). Describes if the area behind the object is of particular class in the image. The class is 'bckgr' - combined buildings and nature. During rule weight optimization obtained from ground truth semantic segmentation, during inference obtained from the predictions of a network.
    \item DISTANCE(object1, object2, image) - Known predicate (observation). Describes if two objects are close to each other in the image. Inferred from the RGB input and corresponding depth map.
    \item HASCOLOR(object, color, image) - Known predicate (observation). Describes if the object has the color in it. The colors are 'white', 'blue', 'yellow', 'red'. Inferred from the RGB input.
    \item ISTYPE(object, class, image) - Unknown predicate (target). Defines the class of the object. Class can be one of 'ped' (pedestrian), 'tr\_sign' (traffic sign), 'tr\_signal' (traffic signal)
\end{itemize}

\begin{small}

\begin{verbatim}
0.768: INTERSECTIONUNDER(O1, 'r', I) >> ISTYPE(O1, 'ped', I) ^2

0.957: INTERSECTIONUNDER(O1, 'pw', I) >> ISTYPE(O1, 'ped', I) ^2

0.789: INTERSECTIONUNDER(O1, 'gr', I) >> ISTYPE(O1, 'ped', I) ^2

3.2E-5: INTERSECTIONBEHIND(O1, 'bckgr', I) >> ISTYPE(O1, 'ped', I) ^2

0.234: ~INTERSECTIONBEHIND(O1, 'bckgr', I) & ~INTERSECTIONUNDER(O1, 'pw', I)
& ~INTERSECTIONUNDER(O1, 'gr', I) & ~INTERSECTIONUNDER(O1, 'r', I)
>> ~ISTYPE(O1, 'ped', I) ^2

0.372: (O1
>> ISTYPE(O1, 'ped', I) ^2

0.278: (O1
& ISTYPE(O3, 'tr_sign', I) >> ISTYPE(O1, 'ped', I) ^2

0.002: (O1
& ISTYPE(O4, 'tr_signal', I) & DISTANCE(O2, O4, I) >> ISTYPE(O1, 'ped', I) ^2

0.899: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I) & INTERSECTIONUNDER(O1, 'r', I)
>> ISTYPE(O1, 'ped', I) ^2

0.246: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I) & INTERSECTIONUNDER(O1, 'pw', I)
>> ISTYPE(O1, 'ped', I) ^2

0.512: INTERSECTIONUNDER(O1, 'gr', I) & ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I)
>> ISTYPE(O1, 'ped', I) ^2

0.213: ISTYPE(O3, 'tr_signal', I) & DISTANCE(O1, O3, I)
& INTERSECTIONBEHIND(O1, 'bckgr', I) >> ISTYPE(O1, 'ped', I) ^2

1.4E-4: DISTANCE(O1, O3, I) & ISTYPE(O3, 'tr_sign', I) >> ISTYPE(O1, 'ped', I) ^2

0.317: ISTYPE(O4, 'tr_signal', I) & DISTANCE(O1, O4, I) >> ISTYPE(O1, 'ped', I) ^2

0.542: HASCOLOR(O3, 'white', I) & HASCOLOR(O3, 'blue', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.561: HASCOLOR(O3, 'yellow', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.841: HASCOLOR(O3, 'white', I) & HASCOLOR(O3, 'blue', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.181: HASCOLOR(O3, 'red', I) & HASCOLOR(O3, 'white', I) >> ISTYPE(O3, 'tr_sign', I) ^2

0.709: ~HASCOLOR(O3, 'white', I) & ~HASCOLOR(O3, 'blue', I)
& ~HASCOLOR(O3, 'yellow', I) & ~HASCOLOR(O3, 'red', I) >> ~ISTYPE(O3, 'tr_sign', I) ^2

0.476: ~INTERSECTIONBEHIND(O3, 'bckgr', I) >> ~ISTYPE(O3, 'tr_sign', I) ^2

1.0 * ISTYPE(O, +Type, I) = 1.0 .
\end{verbatim}
\end{small}

\pagebreak
\subsection{Evaluation Plots}
\label{eval_plots}

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/ped_wrong.png}
        \caption{Scores distribution for the objects "pedestrians".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_sign_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_sign_wrong.png}
        \caption{Scores distribution for the objects "traffic sign".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_signal_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/clean_run/tr_signal_wrong.png}
        \caption{Scores distribution for the objects "traffic signal".}
    \end{subfigure}
    \caption{The scores of the network compared to the scores of the rules on the test dataset. It should be noted that the objects distribution is unbalanced in the test set: amount of objects "pedestrian" is the smallest, followed by "traffic signal". The largest number of objects are "traffic sign". The network can not distingusih between traffic signs or signals and we thus only report the (joined) result as ``tr\_sign'' for the wrong classes.}
    \label{fig:brightness5_preds}
\end{figure}

\pagebreak
\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/ped_wrong.png}
        \caption{Scores distribution for the objects "pedestrians".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_sign_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_sign_wrong.png}
        \caption{Scores distribution for the objects "traffic sign".}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{0.8\textwidth}
        \centering
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_signal_right.png}%
        \hfill
        \includegraphics[width=0.45\textwidth]{images/brightness5_run/tr_signal_wrong.png}
        \caption{Scores distribution for the objects "traffic signal".}
    \end{subfigure}
    \caption{The scores of the network compared to the scores of the rules on the corrupted test dataset. The corruption selected here is "brightness" with the severity $5$. One can directly see that the network scores quality drops substantially while rules are nearly not affected by the corruption.}
    \label{fig:brightness5_preds2}
\end{figure}

\pagebreak
\subsection{Corruption Plots}
\label{all_corr_plots}

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/brightness_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``brightness'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/fog_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``fog'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[c]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/frost_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``frost'' corruption.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[d]{\textwidth}
        \centering
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_ped.png}%
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_tr_sign.png}
        \hfill
        \includegraphics[width=0.3\textwidth]{images/severity_corruption/snow_comparison_tr_signal.png}
        \caption{Comparison of the scores under increasing severity of ``snow'' corruption.}
    \end{subfigure}
    \caption{\small{Different type of corruptions with increasing severity. One can notice that corruptions can affect the rules scores: so brightness and snow add white color to the image, with that the rules for traffic signs are becoming more confident--based on the hasColor predicate. Another aspect is that amount of 'Pedestrian' objects is very little (~20 compared to ~2000 of 'Traffic sign') and because of this network mostly fails to distinguish them.}}
    \label{fig:all_severity_corruptions}
\end{figure}

\section{Approach}
The goal of our research is to find ways in which high-level domain knowledge can improve and robustify the predictions of deep learning models in the context of autonomous driving.
While multiple lines of research discussed in \secref{sec:related_work} make use of domain knowledge exclusively for building a model, it is our aim to combine the data-driven approach and high-level knowledge during inference.
\par
For various traffic scenes, humans can typically use common and empirical knowledge about behaviour and features of the objects to infer their type and in-scene relations, even if generalizing to unseen scenarios.
Considering different features and interrelations of objects, this human common knowledge can partially be encoded as a set of simple rules, some of which are showcased in \figref{fig:psl-motivation}.
The way a human would reason about such a traffic scene cannot be based on hard Boolean logic, though, in which rules hold either always or never.
For example, an object having a zebra walk underneath and standing close to a traffic sign has a high probability to be a human, as depicted in \figref{fig:psl-motivation}.
But there is a chance that this could also be something else, e.g.,\  a dog or some obstacle in a construction site.
Therefore, to integrate such high-level human knowledge in the same ``uncertain'' manner into a neural network model, we need to define logical rules with relaxed truth values.
Structuring the rules upon the identification and classification of objects aligns with the task of semantic segmentation that lends itself well to finding image regions with a common semantic, which we treat as object proposals.
We selected PSL as a framework for knowledge representation for two reasons:
First, it allows us to flexibly use soft truth values that are suitable to define both strict and loose rules, and, second, it can be fine-tuned on data, hence, distilling high-level knowledge directly via the given annotations.
\par
We first introduce PSL in \secref{sec:PSL} and briefly describe the input requirements for a PSL program.
Based on this, \secref{sec:DerivingRules} elaborates on how we define object entities and rules in our domain of interest followed by a description of the approach to combine the PSL rules with semantic segmentation.

\subsection{Probabilistic Soft Logic}
\label{sec:PSL}

Probabilistic Soft Logic (PSL)~\citep{Bach2017HLMRFsPSL,kimmig2012short} is a programming framework for probabilistic reasoning in domains with relational structure.
Typical applications of PSL are collective classification (of interlinked objects in structured data), entity resolution (finding a mapping from multiple interrelated references to the true set of underlying entities) and link prediction (predicting whether a relationship exists between two entities).
PSL makes use of Hinge-Loss Markov Random Fields (HL-MRF), a graphical model, and applies these to structured machine learning problems.
Graphical models define a type of loss function that represents a probability score for each possible value configuration respecting the graph-encoded dependencies among the variables.
For HL-MRFs~\citep{Bach2013HingeLoss,Bach2017HLMRFsPSL} the continuous loss functions are defined using the hinge loss, which allows for efficient convex optimization techniques during inference.
\par
The HL-MRFs are defined using declarative rules, whose importance can be indicated by optional non-negative \textit{weights}.
These rules are formulated using first order logic.
In contrast to the binary truth values \emph{true} and \emph{false} of Boolean algebra, PSL uses continuous soft truth values in the interval $[0,1]$.
\begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{psl-rule-example}
    \caption{PSL rules have a conjuncted body and a single literal head. \emph{Atoms} (e.g.,\  \texttt{IntersectionUnder(O1, 'r')} are the basic elements that are used for reasoning.
    An atom is a combination of \emph{predicates} (here: \texttt{IntersectionUnder$\backslash$2}) and a sequence of variables and constants equal to the predicate's arity.
    A \emph{constant} indicates a concrete element (e.g.,\  ``\texttt{ped}'' is the concrete object class pedestrian) and a \emph{variable} denotes a placeholder later to be substituted by concrete elements.
    Here, the predicate \texttt{IntersectionUnder} has an arity of two, thus, the atom consists of the variables \texttt{O1} indicating an object plus the constant ``\texttt{r}'' indicating the underground type is the road.
    The identifier \texttt{O1} will be substituted for all available objects during inference while the constant ``\texttt{r}'' will stay the same.
    The \texttt{O1} identifier is used in three atoms indicating that during inference, each object-1-variable will be substituted by the same constant (same holds for object-2-variable \texttt{O2}, respectively).
    The exponent \texttt{\^}\texttt{2} at the end of the rule indicates that the quadratic loss is chosen. This example rule can be interpreted as \emph{``if an object 1 is placed on a sidewalk and in close distance to another object 2, which is a traffic signal, there is a probability that this object 1 is a pedestrian''}.}
    \label{fig:psl-rule-example}
\end{figure*}
For a detailed explanation of the elements of a PSL rule, see the example depicted in \figref{fig:psl-rule-example}.
\par
Conceptually, we require that the ``satisfaction'' of rules can be measured in a soft-Boolean way if all ``variables'' have a defined value. In this case, we can aggregate the set of all rules to a global score using individual weights for each rule.
Given a set of ``ground truth'' observations, the weights can be fine-tuned such that the importance of each rule agrees with the ground truth observations.
The resulting fixed set of weights then allows us to perform inference over incomplete observations by choosing the most probable value for unknown variables.

In detail, this means each atom is mapped to a soft truth value -- before inference for all known variables, and during inference for all unknown variables
To compute the continuous truth value of the whole rule, values on atom-level are combined using the \emph{Lukasiewicz t-norm}~\citep{klir1995fuzzy} as a relaxation for the logical conjunction, disjunction, and negation.
Each rule's \emph{satisfaction} is calculated indicating to what degree that rule is fulfilled given the current configuration of values.
A rule is satisfied if the head is assigned at least the same truth value as the body, and a fully satisfied rule has a value of $1$.

All possible variable configurations for the rules are combined to a probability distribution, in which those that satisfy more ground rules are more probable.
If a weight is assigned to a rule, this indicates the importance of this rule to be satisfied in relation to the other rules and their corresponding weights.
Rules that are unweighted induce hard constraints which require the rule to be met at all times.
During inference, the most probable value configuration for unknown predicates is found given evidence (via known predicates).
Given ground truth, the weights of rules can also be learned with maximum-likelihood estimation.
Thus, the importance of each rule in the rule set can be learned from the data itself.
\par
Apart from these rules, a PSL program consists of a database that defines the elements of the domain of interest as well as their relations in the form of predicates.
The database is built up from known predicates (i.e.,\  atoms that are completely observed) and unknown predicates (i.e.,\  atoms that are unobserved and must be inferred) that are part of one of the three following sets:
(i) Necessary for all known predicates is the set of \textit{observations}. These are variable combinations with an associated soft truth value in the form of actual extracted and known data for each.
(ii) Necessary for all unknown predicates is the set of \textit{targets} that is a list of all atoms (combinations of variables) that should be inferred, therefore without any associated truth values.
(iii) For weight learning a set of \textit{truths} with actual known data (likewise observations) to be used as ground truth for all unknown predicates is necessary.
Inference in PSL is done based on the first two sets, while evaluation of the inferred predicates can be done with the test set of the \textit{truths}.

There are two types of rules that are supported in PSL: logical and arithmetic rules.
We use logical rules to encode dependencies between atoms as for example depicted in the rule in \figref{fig:psl-rule-example}.
Additionally, we need to imply an arithmetic constraint to restrict the substitution for variables further.
In our setting, we want to reason about the class of an object in the scene that can only be one of three classes, constraiing the corresponding truth values to sum up to $1$.

\subsection{Deriving Rules from Domain Knowledge}
\label{sec:DerivingRules}
\par
The overall goal of employing PSL to camera-based perception in autonomous driving is to identify parts of common knowledge about the traffic scene that can be readily extracted and represent it in a form that facilitates integration into the model performing inference.
Based on the input requirements for a PSL program as defined above, we first need a database, i.e.,\  objects that we want to reason about as well as their relationships encoded within chosen predicates.
Then, we define a set of rules that carries additional relational knowledge about the domain of interest that is not yet covered by the annotated data and thus can be deployed to support and validate the perception task.
When defining an initial set of rules, one therefore has to carefully choose the objects they link up: it should be simple to both describe their behavior and to extract them from raw data.

Here, we choose pedestrians, traffic signs and traffic signals as our three types of objects of interest, for which we want to infer the class.
To identify these objects in a logical setting, one would need to fall back to low-level sensor interpretation.
Typically, autonomous vehicles have diverse sensors from which objects, their attributes (e.g.,\  colors, shapes) and their surroundings (e.g.,\  area underneath, background) and their relationships (e.g.,\  distance to other objects) can be derived.
\par
For our case study, however, we solely focus on the aspect of how external knowledge can be meaningfully integrated into an existing prediction workflow.
We therefore purposefully deploy an oracle to extract the object entities thereby neglecting additional challenges resulting from, e.g.,\ ,  unreliable sensor readings and object extraction.
Our solution is to employ the connected components on ground truth segmentation masks using additional smoothing to avoid fractured objects due to occlusion.
Additionally, we emulate a depth sensor by propagating the RGB inputs to a publicly available\footnote{\url{https://github.com/nianticlabs/monodepth2}} version of MonoDepth2~\citep{godard2019digging}.
This ``oracle sensor'' information already allows for creating a meaningful set of objects as basis for the rules.

Subsequently, the rules describing the objects have to be defined.
To this end, we develop an initial set of rules, for which we refine the weights later on.
As we want to reason about the objects of our three selected classes, we define our target predicate \texttt{isType(Obj,Class)}.
This is used to later infer a soft truth value for each object and the possible three classes, resulting in one class being more likely than the others.
We then base the logical reasoning for an object class on the semantic features, on the surroundings of the objects, and on their relations.
Surroundings of the objects are described with the observations (known predicates) \texttt{intersectionUnder(Obj,Class)} and \texttt{intersectionBehind(Obj,Class)}.
For example, to infer the class of an object to be ``traffic sign'' \texttt{intersectionUnder} should have a high likelihood for the classes ``pedestrian walk'' or ``street''.
We integrate color as a semantic feature as it is especially important for traffic signs and traffic signals with the predicate \texttt{hasColor(Obj,Color).}
Rules we defined using the color predicate are for example inspired by the officially specified color combinations of common German traffic signs.
The relations between objects are specified as a distance value based on depth readings with the known predicate \texttt{distance(Obj1,Obj2)}.
The defined rules for our setup can be found in appendix~\ref{full_set_rules}.
\par
As a final step, the knowledge gained from the set of data via probabilistic rules has to be combined with the predictions from the neural network, in this case the semantic segmentation masks.
For our conceptual demonstration we follow a simplified approach integrating scores obtained from the network as prior assumptions within the rule based framework.
\begin{figure}[h]
    \centering
    \includegraphics[width=\columnwidth]{images/psl_approach_diagram.png}
    \caption{Overall flow of information: Oracles are serving as a source of information for objects and depth maps and also observations for weights learning; predictions produced by the neural network are combined with PSL rules via priors.}
    \label{fig:approach_diagram}
\end{figure}
\par
Overall, the practical approach can be described as follows (see \figref{fig:approach_diagram}):
\begin{enumerate}
    \item Identify objects to reason about and extract them from sensor data (oracle using ground truth semantic segmentation);
    \item Construct the set of rules using expert and common knowledge;
    \item Prepare the required information from lower level sensors for the rules inference (depth, colors);
    \item Learn the weights of the rules using a training set of recorded camera frames;
    \item Perform inference with the neural network and with the rules on the test set of frames for validation, possibly using network outputs as prior.
\end{enumerate}

\section{Discussion and Conclusion}

We propose a way to reason about high level scene properties, i.e.,\  the type or class of objects, in terms of Probabilistic Soft Logic (PSL).
Taking advantage of human knowledge, we draw information from various domains, ranging from  sensor input such as color to scene properties such as the relations and distances among object.
This approach  enables us to  include additional information from data-driven model predictions, e.g.,\  as prior assumption, and improve their prediction reliability via reasoning.
Our initial evaluation in a simplified setup features a scenario from autonomous driving and indicates that the approach is viable.
Even in tests with highly adverse input augmentations it is possible to robustify the predictions of the employed semantic segmentation network.

Within the PSL framework reasoning is based on rules, i.e.,\  human understandable concepts regarding the interpretation and meaning of object relations or properties.
Besides the possibility to guide the rule creation by human expert knowledge this offers strong interpretability, the set of rules could, e.g.,\ , be audited with respect to consistency.
The relevance of each rule can be adjusted individually, where the necessary weights are either deliberately chosen or learned from a training data set.
The latter case, as demonstrated in our evaluation, can serve to validate the applicability of given rules.
This places PSL in-between human designed and data driven approaches, making it an especially suited candidate to incorporate common knowledge into fundamentally data driven applications.

This work showcases the potential benefits such common knowledge can have for the reliability of a machine learning framework.
At this early stage we, however, neglected the cumbersome step to obtain the entities the PSL framework reasons about.
In practice this information has to be aggregated from various, most likely not fully reliable, sources, for instance stemming from additional sensors (e.g.,\  LIDAR) or alternative processing approaches.
While a probabilistic framework seems appropriate to deal with further uncertain input we restrict this demonstration to only a single ML model providing the semantic segmentation mask.
The entities and their further attributes are instead given by an oracle using the ground truth annotation.
This allows us to test the approach without the necessity to instantiate a full framework.
Another important step is the involvement of human experts for rules formulation. The presented rules are formulated with the common knowledge of the driving situations and environment, but setup specific rules, as well as the rules that are tuning a particular aspect of recognition need multiple iterations over the rules construction process.

With respect to the incorporation of machine learning models we currently only use the provided prediction.
Given the probabilistic nature of both these outputs and the PSL framework the inclusion of prediction uncertainty in terms of, e.g.,\ , calibrated softmax scores could improve the reliability further.
Adding further models as discussed above would additionally require a larger set of rules.
While the used set is sufficient for demonstration purposes enlarging it to cover more attributes seems to be a straightforward way to extend our work.
Specific follow up questions could address a more systematic approach to rule building, which investigates the interplay between the set coverage, reflected by the amount and variety of rules, and its achieved performance, reflected in learned weights and task performance, more closely.
For the case of autonomous driving the incorporation of temporal consistency, requiring, e.g.,\ , that a pedestrian on two consecutive frames should not unexpectedly vanish, could add new aspects to the discussion, possibly including reasoning on the trajectories and intentions of road users.

Especially for domains such as automated driving we believe that the inclusion of human expertise into predictions is a valuable addition to safety and reliability.
Often humans are capable to infer relevant information from context.
For instance a sign indicating the potential presence of children could, by formulation of an appropriate rule, increase the likelihood that a discovered entity is inferred as child.
This inclusion of external knowledge might additionally help counterbalance the rareness of safety relevant situations, which would otherwise make it difficult to learn relevant correlations purely from data.
Deliberately fixing weights or the inclusion of specific rules as above might therefore increase the safety of vulnerable road users (VRUs) without affecting the training and optimization of the underlying machine learning models.

\section{Discussion and Conclusion}

In this work, we propose a way to formulate, evaluate, and take advantage of human knowledge from various domains, such as knowledge about colors or relations between objects, common-sense, and scene understanding.
This enables us to combine model predictions and logical reasoning to improve the reliability of data-driven models.
\par
Our initial evaluation in a simplified setup features a scenario from autonomous driving and indicates that the approach is viable to robustify network predictions.
The importance weights for this initial rule set is optimized with application-relevant data.
Having obtained this optimized rule set, we show that it is possible to validate the output of trained neural networks for semantic segmentation in highly adverse scenarios.
The fact that PSL rules are expressed in a simple and understandable syntax and, on the other hand, weighted by given data endows the system with straightforward layer of interpretability in which the validity of certain predicates can be observed after PSL inference.
\par
In our view, the main caveats that remain to be addressed in future work are high-level object selection and the principled construction of a meaningful set of the rules:
First, in order to focus this work on the aptitude of PSL knowledge integration, we resorted to defining object entities using ground truth data.
Whether it will be possible to use potentialy noisy data from different sensors (e.g.,\  LIDAR) to generate object proposals of sufficient quality remains to be examined.
Second, the rules from our experiments have been selected in a thoughtful but not systematic approach.
This procedure has to be investigated in order to decide if starting from and optimizing over a in some sense complete set of initial rules will result in better outcomes.
\par
There are multiple ways to follow up on the research presented in the paper.
The most straightforward way would be to enlarge the initial set of possible predicates to cover more attributes of the objects, e.g.,\  reasoning about shapes of traffic signs or the width-height-proportion or pedestrians.
When involving more object classes, more complex relations can be represented, e.g.,\  recognizing and treating small pedestrians as children and, thus, attributing unusual and possibly erratic behavior.
Focusing more on the way of knowledge integration, a more elaborate way of incorporating the knowledge from rules into the network could be introduced by taking into account uncertainty measures of the neural network predictions reflecting on what part of the prediction validation by structured knowledge is needed most.
Another direction is to widen reasoning from one frame to a sequences, using the temporally continuous nature of driving scenery.
\par
Especially in domains such as automated driving we also picture the possibility of incorporating empirical knowledge.
With only minor changes to the initial setup, our set of rules could be extended to cover more of such practical knowledge that humans gained through experience in road traffic situations.
For example, rules reflecting the higher possibility for the appearance of pedestrians around bus stops, train stations or in traffic-calmed areas, could be based on the identification of the corresponding traffic signs.
Furthermore, rules could be reweighted to allow for prioritization of vulnerable road user (VRU) classes for added safety.

\section{Evaluation}
The model that we selected for semantic segmentation is ERFNet~\citep{romera2017erfnet} whose lightweight encoder-decoder architecture and limited performance make it straightforward to observe the benefit of knowledge injection via PSL.
We trained ERFNet on the A2D2 public data set \citep{geyer2020a2d2}, which features a large set of environmental scenes for cars on the road.
The data set was separated into independent subsets (by the sequences recorded) for training and testing.
The labels in the data set were simplified to $13$ classes via grouping.
The network was trained using the Adam optimizer with decaying learning rate for $200$ epochs.
The network achieved an mIoU of $0.59$ after training--in particular $0.26$ for 'Pedestrian' class and $0.63$ for 'Traffic sign' and 'Traffic signal'.

Rules selected for the evaluation first have to be ranked with weights.
In our evaluation we obtained the weights using the learning procedure, in which the weights are assigned to the rules based on the training set with ground truth annotations.
The observations for weight learning are extracted from the training images (predicate \texttt{hasColor}), their depth information (predicate \texttt{distance}) and the ground truth semantic segmentation (predicates \texttt{intersectionBehind} and \texttt{intersectionUnder}) using in each case the object entities from the oracle.
The full list of rules we defined along with the learned weights can be found in appendix~\ref{full_set_rules}.
This step of weight learning serves to validate our initially proposed rules discarding those which turned out to be inadequate given the data.
In these cases, the weights assigned were in the order of $10^{-5}$.
One such rule was for example increasing the probability of an object to be a pedestrian if it is close to an object that is a traffic sign.

An independent test set of images is used both for evaluation of the network prediction and rule inference.
The connected components selected as objects for soft rule-based reasoning are compared against the network predictions for the same region.
These regions are produced using the oracle.
We refer to the ratio of correctly classified pixels as the network score for this object.
Note that for weight learning we used information from the ground truth while for inference network predictions are employed.

To evaluate the robustness of the rule predictions we employed image corruptions as described by~\citet{michaelis2019dragon}.
As the most natural we selected four, namely brightness, fog, frost and snow.
Each of them can be applied with five levels of severity, of which $1$ is the weakest (see \figref{fig:corrupt_image} for examples).
The evaluation then follows the same route -- the network infers predictions on the corrupted images and the scores for the objects are computed.
Likewise, rules are inferred on corrupted images and their respective predicted semantic segmentation on them.
The oracle as further source of information is left unaffected by the applied corruptions.
\begin{figure}[th]
    \centering
    \begin{subfigure}{\columnwidth}
        \centering
        \includegraphics[width=0.46\columnwidth]{images/corruptions/20180807145028_camera_frontcenter_000012330.png}%
        \hfill
        \includegraphics[width=0.46\columnwidth]{images/corruptions/brightness_20180807145028_camera_frontcenter_000012330.png}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}{\columnwidth}
        \centering
        \includegraphics[width=0.46\columnwidth]{images/corruptions/fog_20180807145028_camera_frontcenter_000012330.png}%
        \hfill
        \includegraphics[width=0.46\columnwidth]{images/corruptions/frost_20180807145028_camera_frontcenter_000012330.png}
    \end{subfigure}
    \vskip\baselineskip
    \caption{Example of corruptions: original image (top left), brightness, frost and fog corruption (clockwise).}
    \label{fig:corrupt_image}
\end{figure}
The results of the predictions are depicted in \figref{fig:histograms}.
We display the scores for the pedestrian objects, either denoting the correct or incorrect class, in the case of the original images and images corrupted with brightness.
The histograms for the other two object classes is shown in the appendix~\ref{eval_plots}.
One can observe that the scores produced by the rules are nearly not affected--judging by the distribution of histogram peaks and the average score displayed in the legend--while network scores are very sensitive to the corruption of the data.

\begin{figure}[h]
    \centering
    \begin{subfigure}[a]{\columnwidth}
        \centering
        \includegraphics[width=0.49\columnwidth]{images/clean_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.49\columnwidth]{images/clean_run/ped_wrong.png}
        \caption{Scores distribution for the objects ``pedestrians'' on the original images.}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{\columnwidth}
        \centering
        \includegraphics[width=0.49\columnwidth]{images/brightness5_run/ped_right.png}%
        \hfill
        \includegraphics[width=0.49\columnwidth]{images/brightness5_run/ped_wrong.png}
        \caption{Scores distribution for the objects ``pedestrians'' on the corrupted (brightness, severity 5) images.}
    \end{subfigure}
    \vskip\baselineskip
    \caption{The scores of the network compared to the scores of the rules on the test dataset for pedestrian objects. It should be noted that the distribution of objects is unbalanced in the test set: the number of objects ``pedestrian'' is the smallest, followed by ``traffic signal''. The most frequent class of objects is ``traffic sign''. The network was not trained to distinguish between traffic signs or signals and we thus only report the (joined) result as ``tr\_sign'' for the wrong classes.}
    \label{fig:histograms}
\end{figure}
The final evaluation step is to combine network predictions and rules inference.
We used the scores produced by the network as a prior for the \texttt{isType} predicate in the PSL rules.
\figref{fig:corruption_vs_score} reflects the change in the scores assigned to the objects under increasing severity of corruption.
The scores for the objects of class ``traffic signal'' are presented here, the other classes and corruptions can be seen in the appendix~\ref{all_corr_plots}.
One can observe that the severity of the corruption affects the scores produced by a network while the inference with the rules deteriorates only slightly.
Finally, the robustification of the network scores is also stable and shows better results than network scores on their own.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\columnwidth]{images/severity_corruption/frost_comparison_tr_signal.png}
    \caption{The development of average scores for the class ``traffic signal'' with  increasing severity of the ``frost'' corruption.}
    \label{fig:corruption_vs_score}
\end{figure}
\section{Introduction}
\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{images/psl-motivation-traffic-scene.png}
    \caption{Most humans approach the task of understanding and reasoning about traffic scenes using relations between objects. Given that all of them have been identified, common knowledge --- describing both behaviour and features, e.g.,\  relative position --- can straightforwardly be delineated in terms of simple rules. Some of the rules that we utilize in this paper are exemplified in the above figure.}
    \label{fig:psl-motivation}
\end{figure}

Deep neural networks are an integral part of autonomous driving as they achieve unmatched performance in a number of tasks, in particular image-based environment perception.
Still, they are prone to random or targeted disturbances, which renders robustness a crucial requirement for safety-critical applications.
These disturbances might be of a natural source -- like overexposure, fog, snow -- or of an artificial and purposefully malicious source -- like adversarial attacks.
At the same time, wrong predictions in such tasks are most of the time blatantly obvious for human viewers that use multiple intuitive rules based on their knowledge and experience to reveal what is happening in a given scene.
Neural approaches, in their current form, aim to implicitly learn these rules from data, which makes it necessary to acquire and annotate large amounts of inputs and furthermore introduces the danger of adopting spurious correlations.
The integration of scene knowledge into the training or inference of neural networks seems a viable approach, however, this knowledge ranges from considerations of scene geometry, common knowledge, and rules of human behavior, rendering it a difficult problem \citep{vonrueden2020informed}.
There are not only a number of competing approaches to perform the integration of knowledge itself but also multiple ways of representing the knowledge.
\par
In this paper, we study a way to represent knowledge in form of relations delineated as logic rules between identified objects in a traffic scene (see \figref{fig:psl-motivation}) and examine how these relations can be used to robustify data-driven environment perception.
To this end, we adapt the Probabilistic Soft Logic (PSL) framework~\citep{kimmig2012short} to define and learn relations between objects in the surrounding scene.
We demonstrate the results of initial experiments aimed at quantifying its effect on the robustness of a recent small-scale semantic segmentation neural network.

\section{Related Work}
\label{sec:related_work}
The importance of high-level knowledge injection for image-related tasks is widely accepted~\citep{aditya2019integrating}.
Utilizing background and common-sense knowledge about objects or regions, which goes beyond data annotations, can benefit many computer vision tasks, such as object recognition or semantic segmentation.
Mostly due to its use in robotics and autonomous driving vehicles, semantic segmentation, as one form of low-level scene representation, is currently one of the actively developing fields in image recognition.
\par
In their survey~\citet{siam2017deep} show the development of semantic segmentation from feature-based approaches to the end-to-end deep learning networks.
Among other techniques, they point out and discuss an increase in accuracy achieved by including structural knowledge, which is abundant in the particular task of autonomous driving.
To facilitate knowledge inclusion, according to~\citet{siam2017deep}, many researchers apply conditional random fields (CRF) additionally to deep convolutional networks for classification to improve the accuracy of the predictions.
There are also approaches that are based only on an ontology or logical reasoning for semantic segmentation.
For example, \citet{zand2016ontology} propose to deploy Dirichlet mixture models based on super-pixel segmentation using knowledge from a fixed, user-defined ontology after which prediction is performed using conditional random fields.
Analogously, \citet{leon2020big} use fuzzy logic to perform segmentation and \citet{dasiopoulou2005knowledge} use a high-level semantic ontology for object detection in videos.
\par
In our work, we propose to employ the PSL framework as an expressive and interpretable way to reason about the environment of the autonomous vehicle.
To the best of our knowledge it is the first attempt to combine deep-learning-based semantic segmentation with probabilistic logical reasoning about the environment.
Nevertheless, the idea of employing knowledge via logic rules in autonomous driving tasks is not new.
In fact, \citet{kanaujia2015markov, kardacs2013learning, kembhavi2010did} use Markov logic networks for recognizing complex events in video data.

Related work in the same area of application as ours is by \citet{souza2011probabilistic}, in which the authors use logic reasoning to support a lane detection system with high-level knowledge.
They employ the output of their segmentation method for further reasoning and controlling the movement of the vehicle (or alerting the driver).
Similar to the PSL framework used in our work, these authors adopt Markov logic networks to employ high-level knowledge in the form of logical hard and soft rules.
However, the object-based rules are only used to classify events after recognizing objects (e.g.,\  person, car) within image frames.
In contrast, we employ soft object-based rules to robustify the prediction results of a semantic segmentation neural network for the objects in the image frames itself.
Thus, our goal is to construct rules that analogously would support semantic segmentation in cases when the network itself is compromised (e.g.,\  by an image distortion) in order to prevent completely missing critical objects such as pedestrians, traffic signs or traffic signals.

redpajama_set_name: RedPajamaArXiv


While working on the inner pages of my website I came across a number of broken external links. I realize that many SEOs advocate against broken links. However, changing these links would mean I would have to change the entire inner pages of my website. And there is hardly any time or resource at my hands to spend on the correction and maintenance of these links. Is it essential that I repair these links immediately or can I postpone the task to further notice? I would really appreciate your opinion on this matter as it would help me create a more optimized website. Please feel free to share your ideas.


Our mission is to create and promote opportunities for YOU to experience all the arts of Ketchikan! Check out our educational opportunities and upcoming events. Want to volunteer your talents? There's an opportunity to match your skills. Learn more and get involved.


text:
'Whodunit' or 'Whowuzit'
Post on October 20, 2006 by Daphne Mason
Have you been wondering how you might acknowledge, as opposed to celebrate, Halloween this year? You need to search no farther than the Waterworks Theatre. The present play Murderer is most appropriate for Halloween because it is a macabre treat filled with tricks. Another Shaffer takes a mundane love triangle and introduces twists and turns which hardly can be absorbed before the next shock occurs.
Norman Bartholomew, played by Chris Brochon, is a student of murder. He has studied the reasons, techniques, and results of infamous murderers and has reached a personal philosophical analysis of the heinous. Brochon is masterful in his command of this character and his person conflict of fantasy versus reality. He approaches this fantasy with relish-or was it mustard. Brochon definitely has the lion's share of lines and he demonstrates his uncanny capability to assume a variety of British accents. His facial expressions and body language are most convincing and crucial to the movement of the plot. Sarah Gayle gives a creditable performance as Norman's young mistress Millie Sykes. She is consumed by frustration with Norman who doesn't seem to be concluding his marriage fast enough to suit her The third point of the triangle, Norman's wife Elizabeth, is played by Elizabeth Whiley. Whiley is cool, disinterested, and disdainful of Norman and his fling. Brochon becomes a different Norman with her. Is he the braggadocio of his research or is he spineless and ineffectual? Round out the cast is Jordan Whiley. While is, as usual, excellent in developing his character. Sgt. Stenning may be a provincial officer but he is insulted by Norman's crass approach to murder and his insensitivity to Sykes's position.
New to Waterworks is a stage with seats on three sides. This seating is most appropriate for this play because the close proximity of the audience to the stage makes the eerie atmosphere more pronounced. Director Dudley Suave and his huge crew have again created an intricate and fascinating set in a relatively small space. The lighting, and sometimes the lack thereof, is quite effective-again, adding to the overall atmosphere. The background music in the beginning of the play is crucial to the suspense generated. Joe Brown, stage manager and lighting designer; Stephanie Trovato, light board operator; and Rick Neller, sound creator, are to be commended for jobs well done. Convincing production are often the result of the hard work backstage.
Murderer is a fun and challenging show fit for "the season." When you go, you might want to sit next so someone who doesn't mind having his/her hand squeezed unmercifully for the first half hour! Because of the theme, concerns, and incidentally spicy language, the play is recommended for a mature audience. Murderer will be performed October 20, 21, 27 and 28 at 8 p.m.

redpajama_set_name: RedPajamaCommonCrawl


Twelve companies have joined Drop it so far, a campaign to encourage people to use tap water.
Pouya Parsafar was already questioning his company's use of bottled water when he came across an awareness campaign that promoted switching to tap water.
He seized the opportunity and for five months, his IT company, Enterprise Systems Trading, has used a filtering system.
Before the switch, the company would go through 12 five-gallon bottles per month and 2,000 half-litre ones, said Mr Parsafar, managing director of the company, which has 30 employees.


text:
April 11, 2017 by Press Features comments (0)
Book: How We Speak to One Another: An Essay Daily Reader
Minneapolis, MN: Coffee House Press, March 14, 2017 (328 pp; $20.00 paperback, $19.99 eBook), ISBN 9781566894579
Editors: Ander Monson and Craig Reinbold
Ander Monson wears many hats: teacher, author, and editor. He's associate professor of English at the University of Arizona in Tucson and director of the Creative Writing Program there. And he just won a Guggenheim Fellowship.
His own writing includes fiction, poetry, and essays: The Available World, Letter to a Future Lover, Neck Deep and Other Predicaments, Other Electricities, Vacationland, and Vanishing Point. Monson is editor of the literary journal DIAGRAM and head of the affiliated New Michigan Press, which has been publishing broadsides and chapbooks since 1999. DIAGRAM and New Michigan Press run an annual contest for a chapbook of poetry, fiction, creative nonfiction, or hybrid-genre work. This year's deadline is nearing: April 28. Monson is also founder of the Essay Daily website, a conversational space about the essay. "We mostly publish critical/creative engagements with essays (text and visual), Q&As, and reviews of essays, books, collections, or journals."
Craig Reinbold is former Essay Daily managing editor and continues to contribute regularly. He has also been assistant editor at Terrain.org: A Journal of the Built + Natural Environments. His writing has appeared in a number of literary journals. He currently works in the emergency department of a hospital in the Milwaukee area, where he is studying to become a nurse.
Publisher: Coffee House Press
Allan Kornblum founded Coffee House Press in 1984 and served as editor and publisher for twenty-seven years. A poet, he published in a variety of small magazines during the seventies and eighties, when he was also editor, designer, printer, and publisher at Toothpaste Press. Kornblum, who died in 2014, received a 1997 American Book Award recognizing his work at both presses.
Current Coffee House publisher Chris Fischbach has been with the press for over twenty years, beginning as a letterpress intern and becoming head in 2011. He discussed his publishing philosophy in a 2014 VQR article.
Coffee House publishes "literary novels, full-length short story collections, poetry, creative nonfiction, book-length essays and essay collections, and the occasional memoir." Submissions, currently closed, open again September 1 and do not require an agent.
"Without language thought is a vague, uncharted nebula."
—Ferdinand de Saussure
Course in General Linguistics (p. 112)
In this collection of essays, forty-seven writers speak to other essayists (some living, some not) in conversations employing both signs and symbols. One essay appears as a black-and-white graphic comic, while another is a series of paintings—some in color. Occasionally we encounter photos among the words. One page is totally black. From time to time, an essayist reaches out to draw the reader into the chat as well.
Editor Ander Monson notes some essays have already been published elsewhere, while others appear here for the first time. He calls Essay Daily "a place to aggregate our thinking," and defines essay as "thinking in action."
Topics run the gamut: language identity, turtles, subway reading, old age, trout, queer trans bodies, drowning, smart phones, baseball, trail rides, failure, professional wrestling, wrens, fool's gold. Techniques span memoir, the long lyric essay, metaphor, elision, maps, photography, memento mori, email, simile, trauma narrative, syntax, endings. Length ranges from one page to fourteen.
The essayists themselves are an array of professors, editors, poets, writers, workshop directors, investigative journalists, playwrights, translators, radio scriptwriters, and horse wranglers from around the globe: Arizona, Prague, Utah, Alaska, Singapore, Massachusetts, New York, Melbourne, Austin, Memphis, Wyoming, Iowa, Oregon, Chicago, Alabama, Minnesota, Omaha, Wichita, Ohio, San Francisco, Colorado, Philadelphia, and South Carolina. They have names like Marcia Aldrich, Robert Atwan, Rigoberto González, Phillip Lopate, Pam Houston, Patrick Madden, Bonnie J. Rough, John D'Agata, César Díaz.
The essayists dialogue with a wide variety of folks who also essay: Joan Didion, H. L. Mencken, James Agee, Ilan Stavans, Rebecca Solnit, Julian Barnes, Sarah Bakewell, Tom Junod, Claudia Rankine, Ansel Adams, Leslie Jamison, Michel de Montaigne, Donald Hall, Wendell Berry, Alison Bechdel, Jim Bouton, Cynthia Ozick, David Shoemaker, Anne Carson, Max Beerbohm, E. B. White, Louis C.K., Charles Lamb, Roland Barthes, Virginia Woolf, Percy Bysshe Shelley, Norman Mailer, Robert Frost, Michel Foucault, Gerard Manley Hopkins, Walt Whitman, Shirley Jackson, Ann Beattie, Robert Louis Stevenson.
Elegant turns of phrase abound, more frequently in some essays than in others.
Semaphore flaggers line up on the cover, each waving a letter of the alphabet, yet the translated message is a jumble of letters, with the final icon signaling "Error." Inside, though, a six-semaphore image decodes easily into "ESSAYS." Semaphores, designed in the late eighteenth century and used by Napoleon's army, segue into semiotics—the study of signs and symbols and their use or interpretation. That, of course, is precisely what's happening in this volume. Essayists parse how we communicate: nonverbal versus verbal, voiceless versus voiced, or signed versus written. Even the cover's four-flag "Error" flourish highlights Jacques Lacan's idea that errors can suddenly transform into truths and that truth arises from misunderstanding.
Wallace Stevens looked at a blackbird thirteen ways through poetry. These writers look at the essay forty-seven ways through both words and images, following thoughts wherever they may lead. And there, Monson notes, "sometimes something interesting happens." In How We Speak to One Another, many interesting thoughts definitely do happen.
Interview: CM Taylor
Video: Horizon Lines