text:
\section{Introduction}
\label{intro}

Continuous deployment (CD) is the process of rapidly deploying software or services automatically to end-users~\citep{me:agile:cd2015}. The use of infrastructure as code (IaC) scripts is essential to implement an automated deployment pipeline, which facilitates continuous deployment~\citep{Humble:2010:CD}. Companies such as Netflix~\footnote{https://www.netflix.com/}, Ambit Energy~\footnote{https://www.ambitenergy.com/}, and Wikimedia Commons~\footnote{https://commons.wikimedia.org/wiki/Main\_Page}, use IaC scripts to automatically manage their software dependencies and construct automated deployment pipelines~\citep{parnin:adages}~\citep{ambit:pup}. The use of IaC scripts helps organizations to increase their deployment frequency. For example, Ambit Energy uses IaC scripts to increase their deployment frequency by a factor of 1,200~\citep{ambit:pup}.

Defects in IaC scripts can have serious consequences as companies, such as Wikimedia, use these scripts to provision their servers and ensure availability of services~\footnote{https://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-servers-are-configured/}. IaC scripts are susceptible to human errors~\citep{parnin:adages} and bad coding practices~\citep{cito:fse2015:cloud}, which make scripts susceptible to defects~\citep{JiangAdamsMSR2015}~\citep{parnin:adages}. Any defect in a script can propagate at scale, leading to wide-scale service discrepencies. For example in January 2017, execution of a defective IaC script erased home directories of around 270 users in cloud instances maintained by Wikimedia~\footnote{https://wikitech.wikimedia.org/wiki/Incident\_documentation/20170118-Labs}. Prior research studies~\citep{JiangAdamsMSR2015}~\citep{parnin:adages} and the above-mentioned Wikimedia incident motivate us to systematically study the defects that occur in IaC scripts.

Categorization of defects can guide organizations on how to improve their development process. Chillarege et al.~\citep{odc:original} proposed the orthogonal defect classification (ODC) technique which included a set of defect categories. According to Chillarege et al.~\citep{odc:original}, each of these defect categories map to a certain activity of the development process which can be improved. Since the introduction of ODC in 1992, companies such as IBM~\citep{butcher:ibm:odc}, Cisco~\footnote{https://www.stickyminds.com/sites/default/files/presentation/file/2013/t12.pdf}, and Comverse~\citep{comverse:odc:ram} have successfully used ODC to categorize defects. Such categorization of defects help practitioners to identify process improvement opportunities for software development. For example, upon adoption of ODC practitioners from IBM~\citep{butcher:ibm:odc} reported ``\textit{The teams have been able to look at their own data objectively and quickly identify actions to improve their processes and ultimately their product. The actions taken have been reasonable, not requiring huge investments in time, money, or effort}". A systematic categorization of defects in IaC scripts can help in understanding the nature of IaC defects and in providing actionable recommendations for practitioners to mitigate defects in IaC scripts.

\textit{The goal of this paper is to help practitioners improve their development process of infrastructure as code (IaC) scripts by categorizing the defects in IaC scripts based upon a qualitative analysis of commit messages and issue report descriptions.}

We investigate the following research questions:

\begin{description}[leftmargin=*]
\item{\textbf{\textit{RQ1: How frequently do defects occur in infrastructure as code scripts?}}}
\item{\textbf{\textit{RQ2: By categorizing defects using the defect type attribute of orthogonal defect classification, what process improvement recommendations can we make for infrastructure as code development?}}}
\item{\textbf{\textit{RQ3: What are the differences between infrastructure as code (IaC) and non-IaC software process improvement activities, as determined by their defect category distribution reported in the literature?}}}
\end{description}

We use open source datasets from four organizations, Mirantis, Mozilla, Openstack, and Wikimedia Commons, to answer the three research questions. We collect 20, 2, 61, and 11 repositories respectively from Mirantis, Mozilla, Openstack, and Wikimedia Commons. We use 1021, 3074, 7808, and 972 commits that map to 165, 580, 1383, and 296 IaC scripts, collected from Mirantis, Mozilla, Openstack, and Wikimedia Commons, respectively. With the help of 89 raters, we apply qualitative analysis to categorize the defects that occur in the IaC scripts using the defect type attribute of ODC~\citep{odc:original}. We compare the distribution of defect categories found in the IaC scripts with the categories for 26 non-IaC software systems, as reported in prior research studies, which used the defect type attribute of ODC and were collected from IEEEXplore~\footnote{https://ieeexplore.ieee.org/Xplore/home.jsp}, ACM Digital Library~\footnote{https://dl.acm.org/}, ScienceDirect~\footnote{https://www.sciencedirect.com/}, and SpringerLink~\footnote{https://link.springer.com/}.

We list our contributions as following:
\begin{itemize}[leftmargin=*]
\item{A categorization of defects that occur in IaC scripts;}
\item{A comparison of the distribution of IaC defect categories to that of non-IaC software found in prior academic literature; and}
\item{A set of curated datasets where a mapping of defect categories and IaC scripts are provided.}
\end{itemize}

We organize the rest of the paper as following: Section~\ref{bg-rel} provides background information and prior research work relevant to our paper. Section~\ref{meth} describes our methodology conducted for this paper. We use Section~\ref{case-studies} to describe our datasets. We present our findings in Section~\ref{results}. We discuss our findings in Section~\ref{discussion}. We list the limitations of our paper in Section~\ref{limitations}. Finally, we conclude the paper in Section~\ref{conclusion}.

\section{Background and Related Work}
\label{bg-rel}

In this section, we provide background on IaC scripts and briefly describe related academic research.

\subsection{Background}
\label{bg}

IaC is the practice of automatically defining and managing network and system configurations, and infrastructure through source code~\citep{Humble:2010:CD}. Companies widely use commercial tools such as Puppet, to implement the practice of IaC~\citep{Humble:2010:CD}~\citep{JiangAdamsMSR2015}~\citep{ShambaughRehearsal2016}. We use Puppet scripts to construct our dataset because Puppet is considered one of the most popular tools for configuration management~\citep{JiangAdamsMSR2015}~\citep{ShambaughRehearsal2016}, and has been used by companies since 2005~\citep{propuppet:book}. Typical entities of Puppet include modules and manifests~\citep{puppet-doc}. A module is a collection of manifests. Manifests are written as scripts that use a .pp extension.

Puppet provides the utility `class' that can be used as a placeholder for the specified variables and attributes, which are used to specify configuration values. For attributes, configuration values are specified using the `$=>$' sign. For variables, configuration values are provided using the `=' sign. Similar to general purpose programming languages, code constructs such as functions/methods, comments, and conditional statements are also available for Puppet scripts. For better understanding, we provide a sample Puppet script with annotations in Figure~\ref{fig-bg-pp}.

\begin{figure}[t]
\centering
\includegraphics[scale=0.80]{bg-pp.pdf}
\caption{Annotation of an example Puppet script.}
\label{fig-bg-pp}
\end{figure}

\subsection{Related Work}
\label{related}

Our paper is related to empirical studies that have focused on IaC technologies. Sharma et al.~\citep{SharmaPuppet2016} investigated smells in IaC scripts and proposed 13 implementation and 11 design configuration smells. Hanappi et al.~\citep{Hanappi:2016:pupp:converge} investigated how convergence of Puppet scripts can be automatically tested, and proposed an automated model-based test framework. Jiang and Adams~\citep{JiangAdamsMSR2015} investigated the co-evolution of IaC scripts and other software artifacts, such as build files and source code. They reported IaC scripts to experience frequent churn. In a recent work, Rahman and Williams~\citep{me:icst2018:iac} characterized defective IaC scripts using text mining and created prediction models using text feature metrics. Bent et al.~\citep{Bent:Saner2018:Puppet} proposed and validated nine metrics to detect maintainability issues in IaC scripts. In another work, Rahman et al.~\citep{Rahman:RCOSE18} investigated the questions that practitioners ask on Stack Overflow to identify the potential challenges practitioners face while working with Puppet. We analyze the defects that occur in Puppet scripts and categorize them using the defect type attribute of ODC for the purpose of providing suggestions for process improvement.

Our paper is also related to prior research studies that have categorized defects of software systems using ODC and non-ODC techniques.  Chillarege et al.~\citep{odc:original} introduced the ODC technique in 1992. They proposed eight orthogonal `defect type attributes'. Since then, researchers and practitioners have used the defect type attribute of ODC to categorize defects that occur in software. Duraes and Madeira~\citep{Duraes:Madeira2006} studied 668 faults from 12 software systems and reported that 43.4\% of the 668 defects were algorithm defects, and 21.9\% of the defects were assignment-related defects. Fonseca et al.~\citep{Fonseca2008} used ODC to categorize security defects that appear in web applications. They collected 655 security defects from six PHP web applications and reported 85.3\% of the security defects belong to the algorithm category. Zheng et al.~\citep{Zheng:Williams:2006} applied ODC on telecom-based software systems and observed an average of 35.4\% of defects belong to the algorithm category. Lutz and Mikluski~\citep{Lutz:TSE:2004} studied defect reports from seven missions of NASA and observed functional defects to be the most frequent category of 199 reported defects. Christmasson and Chillarege~\citep{Chris:1996} studied 408 IBM OS faults extracted and reported 37.7\% and 19.1\% of these faults to belong to the algorithm and assignment categories, respectively. Basso et al.~\citep{basso} studied defects from six Java-based software namely, Azureus, FreeMind, Jedit, Phex, Struts, and Tomcat, and observed the most frequent category to be algorithm defects. Cinque et al.~\citep{Cinque2014} analyzed logs from an industrial system that belong to the air traffic control system and reported that 58.9\% of the 3,159 defects were classified as algorithm defects.

The above-mentioned studies highlight the research community's interest in systematically categorizing the defects in different software systems. These studies focus on non-IaC software which highlights the lack of studies that investigate defect categories in IaC, and motivate us further to investigate defect categories of IaC scripts. Categorization of IaC-related defects can help practitioners provide actionable recommendations on how to mitigate IaC-related defects and improve the quality of their developed IaC scripts.

\section{Categorization Methodology}
\label{meth}
In this section, we first provide definitions related to our research study and then provide the details of the methodology we used to categorize the IaC defects.

\begin{itemize}
\item{\textbf{Defect}: An imperfection that needs to be replaced or repaired~\citep{ieee:def}.}
\item{\textbf{Defect-related commit}: A commit whose message indicates that an action was taken related to a defect.}
\item{\textbf{Defective script}: An IaC script which is listed in a defect-related commit.}
\item{\textbf{Neutral script}: An IaC script which is not listed in any defect-related commit.}
\end{itemize}

\subsection{Dataset Construction}
\label{meth-dataset}
Our methodology of dataset construction involves two steps: repository collection (Section~\ref{repo-collect}) and commit message processing (Section~\ref{commit-collect}).

\subsubsection{Repository Collection}
\label{repo-collect}
We use open source repositories to construct our datasets. An open source repository contains valuable information about the development process of an open source project, but the frequency of commits might be small indicating the repository's dormant status~\citep{MunaiahCuration2017}. This observation motivates us to apply the following selection criteria to identify repositories for mining:

\begin{itemize}
\item{\textbf{Criteria-1}: The repository must be available for download.}
\item{\textbf{Criteria-2}: At least 11\% of the files belonging to the repository must be IaC scripts. Jiang and Adams~\citep{JiangAdamsMSR2015} reported that in open source repositories IaC scripts co-exist with other types of files, such as source code and Makefiles files. They observed a median of 11\% of the files to be IaC scripts. By using a cutoff of 11\%, we collect a set of repositories that contains a sufficient amount of IaC scripts for analysis.}
\item{\textbf{Criteria-3}: The repository must have at least two commits per month. Munaiah et al.~\citep{MunaiahCuration2017} used the threshold of at least two commits per month to identify repositories which include a lower amount of development activity.}
\end{itemize}

\subsubsection{Commit Message Processing}
\label{commit-collect}

We use two artifacts from version control systems of the selected repositories from Section~\ref{repo-collect}, to construct our datasets: (i) commits that indicate modification of IaC scripts; and (ii) issue reports that are linked with the commits. We use commits because commits contain information on how and why a file was changed. Commits can also include links to issue reports. We use issue report summaries because they can give us more insights on why IaC scripts were changed to supplement what is found in commit messages. We collect commits and other relevant information in the following manner:
\begin{itemize}
\item{First, we extract commits that were used to modify at least one IaC script. A commit lists the changes made on one or multiple files~\citep{maletic:commit:icpc2008}.}
\item{Second, we extract the message of the commit identified from the previous step. A commit includes a message, commonly referred to as a commit message. The commit messages indicate why the changes were made to the corresponding files~\citep{maletic:commit:icpc2008}.}
\item{Third, if the commit message included a unique identifier that maps the commit to an issue in the issue tracking system, we extract the identifier and use that identifier to extract the summary of the issue. We use regular expressions to extract the issue identifier. We use the corresponding issue tracking API to extract the summary of the issue; and}
\item{Fourth, we combine the commit message with any existing issue summary to construct the message for analysis. We refer to the combined message as `extended commit message (XCM)' throughout the rest of the paper. We use the extracted XCMs to categorize defects, as described in Section~\ref{categ}.}
\end{itemize}

\subsection{Categorization of Infrastructure as Code (IaC) Script Defects}
\label{categ}

We use the defect type attribute of ODC to categorize defects. We select the ODC defect type attribute because ODC uses semantic information collected from the software system to categorize defects~\citep{odc:original}. According to the ODC defect type attribute, a defect can belong to one of the eight categories: algorithm (AL), assignment (AS), build/package/merge (B), checking (C), documentation (D), function (F), interface (I), and timing/serialization (T).

The collected XCMs derived from commits and issue report descriptions might correspond to feature enhancement or performing maintenance tasks, which are not related to defects. As an XCM might not correspond to a defect, we added a `No defect (N)' category. An example XCM that belongs to the `No defect' category is ``add example for cobbler", as shown in Table~\ref{table-xcm-example}. Furthermore, a XCM might not to belong to any of the eight categories included in the ODC defect type attribute. Hence, we introduced the `Other (O)' category. An example XCM that belongs to the `Other' category is ``minor fixes", as shown in Table~\ref{table-xcm-example}.

We categorize the XCMs into one of 10 categories. In the case of the eight ODC categories, we follow the criteria provided by Chillarege et al.~\citep{odc:original} and used two of our own criteria for categories `No defect', and `Other'. The criteria for each of the 10 categories are described in Table~\ref{table-odc}.

In Table~\ref{table-odc}, the `Process Improvement Activity Suggested By ODC' column corresponds to one or more software development activities which can be improved based on the defect categories determined by ODC. For example, according to ODC, algorithm-related defects can be reduced by increasing activities that are related to coding, code inspection, unit testing, and function testing.

\begin{table*}[]
\centering
\caption{Determining Defect Categories Based on ODC adapted from~\citep{odc:original}}
\label{table-odc}
{\footnotesize
\begin{tabular}{ p{2.8cm}  p{5.0cm}  p{2.8cm}}
\hline
\textbf{Category} & \textbf{Definition} & \textbf{Process Improvement Activity}  \\
\hline
Algorithm (AL) & Indicates efficiency or correctness problems that affect task and can be fixed by re-implementing an algorithm or local data structure. & Coding, Code Inspection, Unit Testing, Function Testing \\
\hline
Assignment (AS) & Indicates changes in a few lines of code.  & Code Inspection, Unit Testing \\
\hline
Build/Package/Merge (B) & Indicates defects due to mistakes in change management, library systems, or version control system. & Coding, Low Level Design  \\
\hline
Checking (C) & Indicates defects related to data validation and value checking. & Coding, Code Inspection, Unit Testing  \\
\hline
Documentation (D) & Indicates defects that affect publications and maintenance notes. & Coding, Publication \\
\hline
Function (F) & Indicates defects that affect significant capabilities. & Design, High Level Design Inspection, Function Testing \\
\hline
Interface (I) & Indicates defects in interacting with other components, modules, or control blocks. & Coding, System Testing \\
\hline
No Defect (N) & Indicates no defects. & Not Applicable \\
\hline
Other (O) & Indicates a defect that do not belong to the categories: AL, AS, B, C, D, F, I, N, and T. & Not Applicable \\
\hline
Timing/Serialization (T) & Indicates errors that involve real time resources and shared resources. & Low Level Design, Low Level Design Inspection, System Testing \\
\hline
\end{tabular}
}
\end{table*}

We perform qualitative analysis on the collected XCMs to determine the category to which a commit belongs. We had raters with software engineering experience apply the 10 categories stated in Table~\ref{table-odc} on the collected XCMs. We record the amount of time they took to perform the categorization. After applying qualitative analysis based on the 10 defect categories, we find a mapping between each XCM and a category. We list an example XCM that belongs to each of the 10 defect categories in Table~\ref{table-xcm-example}.

We conduct the qualitative analysis in the following manner:

\begin{itemize}
\item{\textbf{Categorization Phase}: We randomly distribute the XCMs so that each XCM is reviewed by at least two raters to mitigate the subjectivity introduced by a single rater. Each rater determines the category of an XCM using the 10 categories presented in Table~\ref{table-odc}. We provide raters with an electronic handbook on IaC~\citep{puppet-doc}, the IEEE Anomaly Classification publication~\citep{ieee:def}, and the ODC publication~\citep{odc:original}. We do not provide any time constraint for the raters to categorize the defects. We record the agreement level amongst raters using two techniques: (a) by counting the XCMs for which the raters had the same rating; and (b) by computing the Cohen's Kappa score~\citep{cohens:kappa}.}
\item{\textbf{Resolution Phase}: When raters disagree on the identified category, we use a resolver's opinion to resolve the disagreements. The first author of the paper is the resolver and is not involved in the categorization phase.}
\item{\textbf{Practitioner Agreement}: To evaluate the ratings in the categorization and the resolution phase, we randomly select 50 XCMs for each dataset. We contact practitioners who authored the commit message via e-mails. We ask the practitioners if they agree to our categorization of XCMs. High agreement between the raters' categorization and practitioner's feedback is an indication of how well the raters performed. The percentage of XCMs to which practitioners agreed upon are recorded and the Cohen's Kappa score is computed.}
\end{itemize}

From the qualitative analysis we determine which commits are defect-related. We use the defect-related commits to identify defective scripts.

\begin{table*}[]
\centering
\caption{Example of Extended Commit Messages (XCMs) for Defect Categories}
\label{table-xcm-example}
\tiny{
\begin{tabular}{ p{1.75cm}  p{2.0cm}  p{2.0cm}  p{2.0cm}  p{2.0cm} }
\hline
\textbf{Category} & \textbf{Mirantis} & \textbf{Mozilla} & \textbf{Openstack} & \textbf{Wikimedia} \\
\hline
\textbf{Algorithm} & fixing deeper hash merging for firewall & bug 869897: make watch\_devices.sh logs no longer infinitely growing; my thought is logrotate.d but open to other choices here & fix middleware order of proxy pipeline and add missing modules  this patch fixes the order of the middlewares defined in the swift proxy server pipeline & nginx service should be stopped and disabled when nginx is absent \\
\hline
\textbf{Assignment} & fix syntax errors  this commit removes a couple of extraneous command introduced by copy/past errors & bug 867593 use correct regex syntax & resolved syntax error in collection & fix missing slash in puppet file url \\
\hline
\textbf{\makecell{Build/\\Package/\\Merge}} & params should have been imported; it was only working in tests by accident & bug 774638-concat::setup should depend on diffutils & fix db\_sync dependencies: this patch adds dependencies between the cinder-api and cinder-backup services to ensure that db\_sync is run before the services are up.  change-id: i7005 & fix varnish apt dependencies  these are required for the build it does. also; remove unnecessary package requires that are covered by require\_package \\
\hline
\textbf{Checking} & ensure we have iso/ directory in jenkins workspace  we share iso folder for storing fuel isos which are used for fuel-library tests & bug 1118354: ensure deploystudio user uid is $>$500 & fix check on threads & fix hadoop-hdfs-zkfc-init exec unless condition \$zookeeper\_hosts\_str was improperly set; since it was created using a non-existent local var '@zoookeeper\_hosts' \\
\hline
\textbf{Documentation} & fixed comments on pull. & bug 1253309 - followup fix to review comments & fix up doc string for workers variable change-id:ie886 & fix hadoop.pp documentation default \\
\hline
\textbf{Function} & fix for jenkins swarm slave variables & bug 1292523-puppet fails to set root password on buildduty-tools server & make class rally work  class rally is created initially by tool & fix ve restbase reverse proxy config  move the restbase domain and `v1' url routing bits into the apache config rather than the ve config. \\
\hline
\textbf{Interface} & fix for iso build broken dependencies & bug 859799-puppetagain buildbot masters won't reconfig because of missing sftp subsystem & update all missing parameters in all manifests & fix file location for interfaces \\
\hline
\textbf{No Defect} & added readme & merge bug 1178324 from default & add example for cobbler & new packages for the server \\
\hline
\textbf{Other} & fuel-stats nginx fix & summary: bug 797946: minor fixups; r=dividehex & minor fixes  & fix nginx configpackageservice ordering \\
\hline
\textbf{\makecell{Timing/\\Serialization}} & fix /etc/timezone file  add newline at end of file /etc/timezone & bug 838203-test\_alerts.html times out on ubuntu 12.04 vm & fix minimal available memory check change-id:iaad0 & fix hhvm library usage race condition  ensure that the hhvm lib `current' symlink is created before setting \/usr\/bin\/php to point to \/usr\/bin\/hhvm instead of \/usr\/bin\/php5. previously there was a potential for race conditions due to resource ordering rules \\
\hline
\end{tabular}
}
\end{table*}

\subsection{\textbf{RQ1: How frequently do defects occur in infrastructure as code scripts?}}
\label{rq1}

We answer RQ1 by quantifying defect density and defects per month. We calculate defect density by counting defects that appear per 1000 lines (KLOC) of IaC script, similar to prior work~\citep{battin:software:2001}~\citep{mohagheghi:ICSE2004}~\citep{hatton:software1997}. We use Equation~\ref{equ-dd-kloc} to calculate defect density ($DD_{KLOC}$). $DD_{KLOC}$ gives an assessment of how frequently defects occur in IaC scripts. We select this measure of defect density, as this measure has been used as an industry standard to (i) establish a baseline for defect frequency; and (ii) helps to assess the quality of the software~\citep{harlan:cleanroom}~\citep{McConnell:CodeComplete}.

\begin{equation}\label{equ-dd-kloc}
\begin{aligned}
DD_{KLOC} = \\ \frac{\text{number of defects for all IaC scripts}}{\bigg(\frac{\text{count of lines for all IaC scripts}}{1000}\bigg)}
\end{aligned}
\end{equation}

Defects per month provides an overview on how frequently defects appear with the evolution of time. We compute the proportion of defects that occur every month to calculate defects per month. We use the metric `Defects per Month' and calculate this metric using Equation~\ref{equ-overall-temp-freq}:

\begin{equation}\label{equ-overall-temp-freq}
\begin{aligned}
\text{Defects per Month ($m$)} = \\ \frac{\text{number of defects in month $m$}}{\text{total commits in month $m$}} \times 100
\end{aligned}
\end{equation}

To observe the trends for defects per month, we apply the Cox-Stuart test~\citep{CoxStuart}. The Cox Stuart test is a statistical test that compares the earlier data points to the later data points in a time series to determine whether or not the trend observant from the time series data is increasing or decreasing with statistical significance. We apply the following steps:

\begin{itemize}[leftmargin=*]
\item{if Cox-Stuart test output shows an increasing trend with a p-value $< 0.05$, we determine the temporal trend to be `increasing'.}
\item{if Cox-Stuart test output shows a  decreasing trend with a p-value $< 0.05$, we determine the temporal trend to be `increasing'.}
\item{ if we cannot determine if the temporal trend as `increasing' or `decreasing', then we determine the temporal trend to be `consistent'.}
\end{itemize}

\subsection{\textbf{RQ2: By categorizing defects using the defect type attribute of orthogonal defect classification, what process improvement recommendations can we make for infrastructure as code development?}}
\label{rq2}

We answer RQ2 using the categorization achieved through qualitative analysis and by reporting the count of defects that belong to each defect category. We use the metric Defect Count for Category $x$ ($DCC$) calculated using Equation~\ref{equ-dmc}.

\begin{equation}\label{equ-dmc}
\begin{aligned}
\text{\small{Defect Count for Category $x$ ($DCC$)}} = \\ \bigg(\frac{\text{\small{count of defects that belong to category $x$}}}{\text{\small{total count of defects}}} \bigg) \\ \times 100
\end{aligned}
\end{equation}

Answers to RQ2 will give us an overview on the distribution of defect categories for IaC scripts, which we use to determine what process improvement opportunities can be recommended for IaC development. In Table~\ref{table-odc} we have provided a mapping between defect categorization and the corresponding process improvement activity, as suggested by Chillarege et al.~\citep{odc:original}.

\subsection{\textbf{RQ3: What are the differences between infrastructure as code (IaC) and non-IaC software process improvement activities, as determined by their defect category distribution reported in the literature?}}
\label{rq4}

We answer RQ3 by identifying prior research that have used the defect type attribute of ODC to categorize defects in other systems such as safety critical systems~\citep{Lutz:TSE:2004}, and operating systems~\citep{Chris:1996}. We collect necessary publications based on the following selection criteria:
\begin{itemize}
\item{\textbf{Step-1}: The publication must cite Chillarege et al.~\citep{odc:original}'s ODC publication, be indexed by ACM Digital Library or IEEE Xplore or SpringerLink or ScienceDirect, and have been published on or after the year 2000. By selecting publications on or after the year 2000 we assume to obtain defect categories of software systems that are relevant and comparable against modern software systems such as IaC.}

\item{\textbf{Step-2}: The publication must use the defect type attribute of ODC in its original form to categorize defects of a software system. We exclude publications that cite Chillarege et al.~\citep{odc:original}'s paper as related work and not use the ODC defect type attribute for categorization. We also exclude publications that modify the ODC defect type attribute to form more defect categories, and use the modified version of ODC to categorize defects.}

\item{\textbf{Step-3}: The publication must explicitly report the software systems they studied with a distribution of defects across the ODC defect type categories and the total count of bugs/defects/faults for each software system. Along with defects, we consider bugs and faults, as in prior work researchers have used the terms bugs~\citep{Thung:Lo:ISSRE2012} and faults interchangeably with defects~\citep{pecchia:issre2012}.}

\end{itemize}

Our answer to RQ3 provides a list of software systems with distribution of defects categorized using the defect type attribute of ODC. For each software system, we report the main programming language it was built and the reference publication.

\section{Datasets}
\label{case-studies}

We construct datasets using Puppet scripts from open source repositories maintained by four organizations: Mirantis, Mozilla, Openstack, and Wikimedia Commons. We select Puppet because it is considered as one of the most popular tools to implement IaC~\citep{JiangAdamsMSR2015}~\citep{ShambaughRehearsal2016}, and has been used by organizations since 2005~\citep{propuppet:book}. Mirantis is an organization that focuses on the development and support of cloud services such as OpenStack~\footnote{https://www.mirantis.com/}. Mozilla is an open source software community that develops, uses, and supports Mozilla products such as Mozilla Firefox~\footnote{https://www.mozilla.org/en-US/}. Openstack foundation is an open-source software platform for cloud computing where virtual servers and other resources are made available to customers~\footnote{https://www.openstack.org/}. Wikimedia Foundation is a non-profit organization that develops and distributes free educational content~\footnote{https://wikimediafoundation.org/}.

\subsection{Repository Collection}

We apply the three selection criteria presented in Section~\ref{repo-collect} to identify the repositories that we use for analysis. We describe how many of the repositories satisfied each of the three criteria in Table~\ref{table-criteria-dataset}. Each row corresponds to the count of repositories that satisfy each criteria. For example, 26 repositories satisfy Criteria-1, for Mirantis. Altogether, we obtain 94 repositories to extract Puppet scripts from.

\begin{table}[]
\centering
\caption{Filtering Criteria to Construct Defect Datasets}
\label{table-criteria-dataset}
{\footnotesize
\begin{tabular}{ p{1.5cm}  p{1.35cm} p{1.35cm} p{1.35cm} p{1.30cm} }
\hline
\textbf{Criteria}   & \textbf{Dataset} \\
\hline
                    & Mirantis & Mozilla & Openstack & Wikimedia \\
\hline
\textbf{Criteria-1} & 26 & 1,594 & 1,253 & 1,638 \\
\textbf{Criteria-2} & 20 & 2     & 61 & 11 \\
\textbf{Criteria-3} & 20 & 2     & 61 & 11 \\
\hline
\textbf{Final}      & 20 & 2     & 61 & 11 \\
\hline
\end{tabular}
}
\end{table}

\subsection{Commit Message Processing}

We report summary statistics on the collected repositories in Table~\ref{table-defect-dataset}. According to Table~\ref{table-defect-dataset}, for Mirantis we collect 165 Puppet scripts that map to 1,021 commits. Of these 1,021 commits, 82 commits include identifiers for bug reports. The number of lines of code for these 165 Puppet scripts is 17,564.

\begin{table*}[]
\centering
\caption{Summary Attributes of Defect Datasets}
\label{table-defect-dataset}
{\footnotesize
\begin{tabular}{ p{2.65cm} p{1.70cm} p{1.70cm} p{1.70cm} p{1.70cm} }
\hline
\textbf{Properties}  & \textbf{Dataset} \\
                     & Mirantis & Mozilla & Openstack & Wikimedia \\
\hline
\textbf{Puppet Scripts} & 165 & 580  & 1,383 & 296 \\
\hline
\textbf{Puppet Code Size (LOC)} & 17,564 & 30,272 & 122,083 & 17,439 \\
\hline
\textbf{Defective Puppet Scripts} & 91 of 165, 55.1\% & 259 of 580, 44.6\%  & 810 of 1383, 58.5\% & 161 of 296, 54.4\% \\
\hline
\textbf{Commits with Puppet Scripts} & 1,021  & 3,074  & 7,808  & 972  \\
\hline
\textbf{Commits with Report IDs} & 82  of 1021, 8.0\% & 2764 of 3074, 89.9\% & 2252 of 7808, 28.8\% & 210 of 972, 21.6\% \\
\hline
\textbf{Defect-related Commits} &  344 of 1021, 33.7\% & 558 of 3074, 18.1\%  & 1987 of 7808, 25.4\% & 298 of 972, 30.6\% \\
\hline
\end{tabular}
}
\end{table*}

\subsection{Determining Categories of Defects}
\label{res-study-dataset}
We use 89 raters to categorize the XCMs, using the following phases:

\begin{itemize}
\item{
\textbf{Categorization Phase}:
\begin{itemize}
\item{\textbf{Mirantis}: We recruit students in a graduate course related to software engineering via e-mail. The number of students in the class was 58, and 32 students agreed to participate. We follow Internal Review Board (IRB) protocol, IRB\#12130, in recruitment of students and assignment of defect categorization tasks. We randomly distribute the 1,021 XCMs amongst the students such that each XCM is rated by at least two students. The average professional experience of the 32 students in software engineering is 1.9 years. On average, each student took 2.1 hours.}
\item{\textbf{Mozilla}: The second and third author of the paper, separately apply qualitative analysis on 3,074 XCMs. The second and third author, respectively, have a professional experience of three and two years in software engineering. The second and third author, respectively, took 37.0 and 51.2 hours to complete the categorization. }
\item{\textbf{Openstack}: The third and fourth author of the paper, separately, apply qualitative analysis on 7,808 XCMs from Openstack repositories. The third and fourth author, respectively, have a professional experience of two and one years in software engineering. The third and fourth author completed the categorization of the 7,808 XCMs respectively, in 80.0 and 130.0 hours.}
\item{\textbf{Wikimedia}: 54 graduate students recruited from the `Software Security' course are the raters. We randomly distribute the 972 XCMs amongst the students such that each XCM is rated by at least two students. According to our distribution, 140 XCMs are assigned to each student. The average professional experience of the 54 students in software engineering is 2.3 years. On average, each student took 2.1 hours to categorize the 140 XCMs. }
\end{itemize}
}
\item{
\textbf{Resolution Phase}:
\begin{itemize}
\item{\textbf{Mirantis}: Of the 1,021 XCMs, we observe agreement for 509 XCMs and disagreement for 512 XCMs, with a Cohen's Kappa score of 0.21. Based on Cohen's Kappa score, the agreement level is `fair'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Mozilla}: Of the 3,074 XCMs, we observe agreement for 1,308 XCMs and disagreement for 1,766 XCMs, with a Cohen's Kappa score of 0.22. Based on Cohen's Kappa score, the agreement level is `fair'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Openstack}: Of the 7,808 XCMs, we observe agreement for 3,188 XCMs, and disagreements for 4,620 XCMs. The Cohen's Kappa score was 0.21. Based on Cohen's Kappa score, the agreement level is `fair'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Wikimedia}: Of the 972 XCMs, we observe agreement for 415 XCMs, and disagreements for 557 XCMs, with a Cohen's Kappa score of 0.23. Based on Cohen's Kappa score, the agreement level is `fair'~\citep{Landis:Koch:Kappa:Range}.}
\end{itemize}

The first author of the paper was the resolver, and resolved disagreements for all four datasets. In case of disagreements the resolver's categorization is considered as final.

We observe that the raters agreement level to be `fair' for all four datasets. One possible explanation can be that the raters agreed on whether an XCM is defect-related, but disagreed on which of the 10 defect category of the defect is related to. For defect categorization, fair or poor agreement amongst raters however, is not uncommon. Henningsson et al.~\citep{Henningsson:ISESE2004} also reported a low agreement amongst raters.

\textbf{Practitioner Agreement}: We report the agreement level between the raters' and the practitioners' categorization for randomly selected 50 XCMs as following:
\begin{itemize}
\item{\textbf{Mirantis}: We contact three practitioners and all of them respond. We observe a 89.0\% agreement with a Cohen's Kappa score of 0.8. Based on Cohen's Kappa score, the agreement level is `substantial'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Mozilla}:   We contact six practitioners and all of them respond. We observe a 94.0\% agreement with a Cohen's Kappa score of 0.9. Based on Cohen's Kappa score, the agreement level is `almost perfect'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Openstack}: We contact 10 practitioners and all of them respond. We observe a 92.0\% agreement with a Cohen's Kappa score of 0.8. Based on Cohen's Kappa score, the agreement level is `substantial'~\citep{Landis:Koch:Kappa:Range}.}
\item{\textbf{Wikimedia}: We contact seven practitioners and all of them respond. We observe a 98.0\% agreement with a Cohen's Kappa score of 0.9. Based on Cohen's Kappa score, the agreement level is `almost perfect'~\citep{Landis:Koch:Kappa:Range}.}
\end{itemize}
}
\end{itemize}

We observe that the agreement between ours and the practitioners' categorization varies from 0.8 to 0.9, which is higher than that of the agreement between the raters in the Categorization Phase. One possible explanation can be related to how the resolver resolved the disagreements. The first author of the paper has industry experience in writing IaC scripts, which may help to determine categorizations that are consistent with practitioners. Another possible explanation can be related to the sample provided to the practitioners. The provided sample, even though randomly selected, may include commit messages whose categorization are relatively easy to agree upon.

\textbf{Dataset Availability}\label{dataset-availability}: The constructed datasets used for empirical analysis are available as online~\footnote{https://doi.org/10.6084/m9.figshare.6465215}.

\section{Results}
\label{results}
In this section, we provide empirical findings by answering the three research questions:

\subsection{\textbf{RQ1: How frequently do defects occur in infrastructure as code scripts?}}
\label{res-rq1}

The defect density, measured in $DD_{KLOC}$  is respectively, 27.6, 18.4, 16.2, and 17.1 per 1000 LOC, respectively, for Mirantis, Mozilla, Openstack, and Wikimedia. We observe defect densities to vary from organization to another, which is consistent with prior research. For example, for a Fortran-based satellite planning software system, Basili and Perricone~\citep{Basili:Perricone:1984} have reported defect density to vary from 6.4 to 16.0, for every 1000 LOC. Mohagheghi et al.~\citep{mohagheghi:ICSE2004} have reported defect density to vary between 0.7 and 3.7 per KLOC for a telecom software system written in C, Erlang, and Java.

We report the defects per month values for the four datasets in Figure~\ref{fig-res-overall-temporal}. In Figure~\ref{fig-res-overall-temporal}, we apply smoothing to obtain visual trends. The x-axis and y-axis, respectively, presents the months and the defects per month values. According to our Cox-Stuart test results, as shown in Table~\ref{table-res-overall-temporal}, all trends are consistent. The p-values obtained from the Cox-Stuart test output for Mirantis, Mozilla, Openstack, and Wikimedia are respectively, 0.22, 0.23, 0.42, and 0.13. Our findings indicate that overall, frequency of defects do not significantly change across time for IaC scripts.

\begin{figure*}
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{mir_temporal_overall.pdf}
 \label{fig-res-overall-temporal-mir}
}
\\
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{moz_temporal_overall.pdf}
 \label{fig-res-overall-temporal-moz}
}
\\
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{ost_temporal_overall.pdf}
 \label{fig-res-overall-temporal-ost}
}
\\
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{wik_temporal_overall.pdf}
 \label{fig-res-overall-temporal-wik}
}
\caption{Defects per month for all four datasets. Figures~\ref{fig-res-overall-temporal-mir}, ~\ref{fig-res-overall-temporal-moz}, ~\ref{fig-res-overall-temporal-ost}, and~\ref{fig-res-overall-temporal-wik} respectively presents defects per month for Mirantis, Mozilla, Openstack, and Wikimedia with smoothing. Overall, defect-related commits exhibit consistent trends over time.}
\label{fig-res-overall-temporal}
\end{figure*}

\begin{table}
\centering
\caption{Cox-Stuart Test Results for Defects per Month}
\begin{tabular}{ c c c c c }
\hline
\textbf{Output} & \textbf{Mirantis} & \textbf{Mozilla} & \textbf{Openstack} & \textbf{Wikimedia} \\
\hline
Trend           & Increasing & Decreasing & Decreasing & Decreasing \\
$p-value$       & 0.22 & 0.23 & 0.42 & 0.13 \\
\hline
\end{tabular}
\label{table-res-overall-temporal}
\end{table}

We use our findings related to defects per month to draw parallels with hardware and non-IaC software systems. For hardware systems researchers have reported the `bathtub curve' trend, which states when initially hardware systems are put into service, the frequency of defects is high~\citep{smith1993reliability}. As time progresses, defect frequency decreases and remains constant during the `adult period'. However, after the adult period, defect frequency becomes high again, as hardware systems enter the `wear out period'~\citep{smith1993reliability}. For IaC defects we do not observe such temporal trend.

In non-IaC software systems, initially defect frequency is high, which gradually decreases, and eventually becomes low as time progresses~\citep{hartz1996:reliability}. Eventually, all software systems enter the `obsolescence period' where defect frequency remains consistent, as no significant upgrades or changes to the software is made~\citep{hartz1996:reliability}. For Wikimedia, we observe a similar trend to that of non-IaC software systems: initially defect frequency remains high, but decreases as time progresses. However, this observation does not generalize for the other three datasets. Also, according to the Cox-Stuart test, the visible trends are not statistically validated. One possible explanation can be attributed to how organizations resolve existing defects.  For example, while fixing a certain set of defects, practitioners may be inadvertently introducing a new set of defects, which results in an overall constant trend.

\begin{shaded}
\noindent{The defect density is 27.6, 18.4, 16.2, and 17.1 defects per KLOC respectively for Mirantis, Mozilla, Openstack, and Wikimedia. For all four datasets, we observe IaC defects to follow a consistent temporal trend.}
\end{shaded}

\subsection{\textbf{RQ2: By categorizing defects using the defect type attribute of orthogonal defect classification, what process improvement recommendations can we make for infrastructure as code development?}}
\label{res-rq2}

We answer RQ2 by first presenting the values for defect count per category (DCC) that belong to each defect category mentioned in Table~\ref{table-odc}. In Figure~\ref{fig-res-rq2-dcc}, we report the DCC values for the four datasets. In Figure~\ref{fig-res-rq2-dcc}, the x-axis presents the nine defect categories, whereas, the y-axis presents DCC values for each category. We observe the dominant defect category to be assignment. As shown in Figure~\ref{fig-res-rq2-dcc}, Assignment-related defects account for 49.3\%, 36.5\%, 57.6\%, and 62.7\% of the defects, for Mirantis, Mozilla, Openstack, and Wikimedia, respectively.

\begin{figure*}[htbp]
\centering
\includegraphics[scale=0.70]{res-rq2-dcc.pdf}
\caption{Defect count per category (DCC) that belong to each defect category Algorithm (AL), Assignment (AS), Build/Package/Merge (B), Checking (C), Documentation (D), Function (F), Interface (I), Other (O), and Timing/Serialization (T). Defects categorized as `Assignment' is the dominant category.}
\label{fig-res-rq2-dcc}
\end{figure*}

One possible explanation can be related to how practitioners utilize IaC scripts. IaC scripts are used to manage configurations and deployment infrastructure automatically~\citep{Humble:2010:CD}. For example, practitioners use IaC to provision cloud instances, such as Amazon Web Services~\citep{cito:fse2015:cloud}, or manage dependencies of software~\citep{Humble:2010:CD}. When assigning configurations of library dependencies or provisioning cloud instances practitioners might be inadvertently introducing defects. Fixing these defects involve few lines of code in IaC scripts, and these defects fall in the assignment category. Correcting syntax issues also involve a few lines of code and also belongs to the assignment category. Another possible explanation can be related to the declarative nature of Puppet scripts. Puppet provides syntax to declare and assign configurations. While assigning these configuration values practitioners may be inadvertently introducing defects.

As shown in Figure~\ref{fig-res-rq2-dcc}, for category Other, defect count per category is 12.5\%, 9.4\%, 6.7\%, and 0.8\%, for Mirantis, Mozilla, Openstack, and Wikimedia, respectively. This category includes XCMs that correspond to a defect but the rater was not able to identify the category of the defect. One possible explanation can be attributed to the lack of information content provided in the messages. Practitioners may not strictly adhere to the practice of writing detailed commit messages, which eventually leads to commit messages that do not provide enough information for defect categorization. For example, the commit message `minor puppetagain fixes', implies that a practitioner performed a fix-related action on an IaC script, but what category of defect was fixed remains unknown. We observe organization-based guidelines to exist on how to write better commit messages for Mozilla~\footnote{https://developer.mozilla.org/en-US/docs/Mozilla/Developer\_guide/}, Openstack~\footnote{https://wiki.openstack.org/wiki/GitCommitMessages}, and Wikimedia~\footnote{https://www.mediawiki.org/wiki/Gerrit/Commit\_message\_guidelines}. Another possible explanation can be attributed to the lack of context inherent in commit messages. The commit messages provide a summary of the changes being made, but that might not be enough to determine the defect category. Let us consider two examples in this regard, provided in Figures~\ref{fig-res-other-exa1} and~\ref{fig-res-other-exa2}. Figures~\ref{fig-res-other-exa1} and~\ref{fig-res-other-exa2} respectively presents two XCMs categorized as `Other', and obtained respectively from Mozilla and Openstack. In Figure~\ref{fig-res-other-exa2}, we observe that in the commit, a newline is being added for printing purposes, which is not captured in the commit message `summary: bug 746824: minor fixes'. From Figure~\ref{fig-res-other-exa2}, we observe that in the commit, the `stdlib::safe\_package' is being replaced by the `package' syntax, which is not captured by the corresponding commit message `minor fixes'.

\begin{figure*}
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{fig-res-other-exa1.pdf}
 \label{fig-res-other-exa1}
}\\
\subfloat[]{
 \includegraphics[width=0.95\textwidth]{fig-res-other-exa2.pdf}
 \label{fig-res-other-exa2}
}
\caption{Code changes in commits categorized as the `Other' category. Figures~\ref{fig-res-other-exa1} and~\ref{fig-res-other-exa2} respectively presents the code changes for two commits marked as `Other' obtained from Mozilla and Openstack.}
\label{fig-res-other}
\end{figure*}

\textbf{Recommendations for IaC Development}: Our findings indicate that assignment-related defects are the most dominant category of defects. According to Chillarege et al.~\citep{odc:original}, if assignment-related defects are not discovered with code inspection and unit tests earlier, these defects can continue to grow at latter stages of development. Based on our findings, we recommend organizations to allocate more code inspection and unit testing efforts.

\begin{shaded}
\noindent{Assignment is the most frequently occurring defect category for all four datasets: Mirantis, Mozilla, Openstack, and Wikimedia. For Mirantis, Mozilla, Openstack, and Wikimedia, respectively 49.3\%, 36.5\%, 57.6\%, and 62.7\%, of the defects belong to the category, assignment. Based on our findings, we recommend practitioners to allocate more efforts on code inspection and unit testing.}
\end{shaded}

\subsection{\textbf{RQ3: What are the differences between infrastructure as code (IaC) and non-IaC software process improvement activities, as determined by their defect category distribution reported in the literature?}}
\label{res-rq4}

We identify 26 software systems using the three steps outlined in Section~\ref{rq4}:
\begin{itemize}
\item{\textbf{Step-1}: As of August 11, 2018, 818 publications indexed by ACM Digital Library or IEEE Xplore or SpringerLink or ScienceDirect, cited the original ODC publication~\citep{odc:original}. Of the 818 publications, 674 publications were published on or after 2000.}
\item{\textbf{Step-2}: Of these 674 publications, 16 applied the defect type attribute of ODC in its original form to categorize defects for software systems.}
\item{\textbf{Step-3}: Of these 16 publications, 7 publications explicitly mentioned the total count of defects and provided a distribution of defect categories. We use these seven publications to determine the defect categorization of 26 software systems.}
\end{itemize}

In Table~\ref{table-rq4-res}, we present the categorization of defects for these 26 software systems. The `System' column reports the studied software system followed by the publication reference in which the findings were reported. The `Count' column reports the total count of defects that were studied for the software system. The `Lang.' column presents the programming language using which the system is developed. The dominant defect category for each software system is highlighted in bold.

From Table~\ref{table-rq4-res}, we observe that for four of the 26 software systems, 40\% or more of the defect categories belonged to the assignment category. For 15 of the 26 software systems, algorithm-related defects are dominant. We also observe documentation and timing-related defects to rarely occur in the previously-studied software systems. In contrast to IaC scripts, assignment-related defects are not prevalent: assignment-related defects were the dominant category for 3 of the 26 software systems. We observe IaC scripts to have a different defect category distribution than non-IaC software system written in general purpose programming languages.

The differences in defect category distribution may yield different set of guidelines for software process improvement. For the 15 software systems where algorithm-related defects are dominant, based on ODC, software process improvement efforts can be focused on the following activities: coding, code inspection, unit testing, and functional testing. In case of IaC scripts, as previously discussed, software process improvement efforts can be focused on code inspection and unit testing.

\begin{shaded}
\noindent{Defects categorized as assignment are more prevalent amongst IaC scripts compared to that of non-IaC systems. Our findings suggest that process improvement activities will be different for IaC development compared to that of non-IaC software development.}
\end{shaded}

\begin{table*}[]
\captionsetup{justification=centering}
\caption{Defect Categories of Previously Studied Software Systems. Columns AL, AS, B, C, D, F, I, and T respectively correspond to the categories Algorithm, Assignment, Build/Package/Merge, Checking, Documentation, Function, Interface, and Timing/serialization.}
\label{table-rq4-res}
{\tiny
\begin{tabular}{ p{2.2cm}  p{0.5cm}  p{0.5cm}  p{0.5cm}  p{0.5cm} p{0.5cm} p{0.5cm} p{0.5cm}  p{0.5cm} p{0.5cm} p{0.5cm} p{0.5cm} p{0.5cm} }
\hline
\textbf{System} & \textbf{Lang.} & \textbf{Count} & \textbf{AL(\%)} & \textbf{AS(\%)} & \textbf{B(\%)} & \textbf{C(\%)} & \textbf{D(\%)} & \textbf{F(\%)} & \textbf{I(\%)} & \textbf{T(\%)} \\
\hline
Bourne Again Shell (BASH)~\citep{Cotroneo:TestType:JSS2013} & C & 2 & 0.0 & \textbf{100.0} & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
\hline
ZSNES-Emulator for x86~\citep{Cotroneo:TestType:JSS2013} & C, C++ &  3 & 33.3 & \textbf{66.7} & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
\hline
Pdftohtml-Pdf to html converter~\citep{Cotroneo:TestType:JSS2013} & Java &  20 & 40.0 & \textbf{55.0} & 0.0 & 5.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
\hline
Firebird-Relational DBMS~\citep{Cotroneo:TestType:JSS2013} & C++ &  2 & 0.0 & \textbf{50.0} & 0.0 & \textbf{50.0} & 0.0 & 0.0 & 0.0 & 0.0 \\
\hline
Air flight application~\citep{lyu:issre2003} & C &  426 & 19.0 & 31.9 & 0.0 & 14.0 & 0.0 & \textbf{33.8} & 1.1 & 0.0 \\
\hline
Apache web server~\citep{pecchia:issre2012} & C & 1,101 & \textbf{47.6} & 26.4 & 0.0 & 12.8 & 0.0 & 0.0 & 12.9 & 0.0 \\
\hline
Joe-Tex editor~\citep{Cotroneo:TestType:JSS2013} & C &  78 & 15.3 & 25.6 & 0.0 & \textbf{44.8} & 0.0 & 0.0 & 14.1 & 0.0 \\
\hline
Middleware system for air traffic control~\citep{Cinque2014} & C & 3,159 & \textbf{58.9} & 24.5 & 0.0 & 1.7 & 0.0 & 0.0 & 14.8 & 0.0 \\
\hline
ScummVM-Interpreter for adventure engines~\citep{Cotroneo:TestType:JSS2013} & C++ &  74 & \textbf{56.7} & 24.3 & 0.0 & 8.1 & 0.0 & 6.7 & 4.0 & 0.0 \\
\hline
Linux kernel~\citep{Cotroneo:TestType:JSS2013} & C &  93 & \textbf{33.3} & 22.5 & 0.0 & 25.8 & 0.0 & 12.9 & 5.3 & 0.0 \\
\hline
Vim-Linux editor~\citep{Cotroneo:TestType:JSS2013} & C &  249 & \textbf{44.5} & 21.2 & 0.0 & 22.4 & 0.0 & 5.2 & 6.4 & 0.0 \\
\hline
MySQL DBMS~\citep{pecchia:issre2012} & C, C++ &  15,102 & \textbf{52.9} & 20.5 & 0.0 & 15.3 & 0.0 & 0.0 & 11.2 & 0.0 \\
\hline
CDEX-CD digital audio data extractor~\citep{Cotroneo:TestType:JSS2013} & C, C++, Python &  11 & 9.0 & 18.1 & 0.0 & 18.1 & 0.0 & 0.0 & \textbf{54.5} & 0.0 \\
\hline
Struts~\citep{basso} & Java &  99 & \textbf{48.4} & 18.1 & 0.0 & 9.0 & 0.0 & 4.0 & 20.2 & 0.0 \\
\hline
Safety critical system for NASA spacecraft~\citep{Lutz:TSE:2004} & Java & 199 & 17.0 & 15.5 & 2.0 & 0.0 & 2.5 & \textbf{29.1} & 5.0 & 10.5 \\
\hline
Azureus~\citep{basso} & Java & 125 & \textbf{36.8} & 15.2 & 0.0 & 11.2 & 0.0 & 28.8 & 8.0 & 0.0 \\
\hline
Phex~\citep{basso} & Java & 20 & \textbf{60.0} & 15.0 & 0.0 & 5.0 & 0.0 & 10.0 & 10.0 & 0.0 \\
\hline
TAO Open DDS~\citep{pecchia:issre2012} & Java & 1,184 & \textbf{61.4} & 14.4 & 0.0 & 11.7 & 0.0 & 0.0 & 12.4 & 0.0 \\
\hline
JEdit~\citep{basso} & Java & 71 & \textbf{36.6} & 14.0 & 0.0 & 11.2 & 0.0 & 25.3 & 12.6 & 0.0 \\
\hline
Tomcat~\citep{basso} & Java & 169 & \textbf{57.9} & 12.4 & 0.0 & 13.6 & 0.0 & 2.3 & 13.6 & 0.0 \\
\hline
FreeCvi-Strategy game~\citep{Cotroneo:TestType:JSS2013} & Java & 53 & \textbf{52.8} & 11.3 & 0.0 & 13.2 & 0.0 & 15.0 & 7.5 & 0.0 \\
\hline
FreeMind~\citep{basso} & Java & 90 & \textbf{46.6} & 11.1 & 0.0 & 2.2 & 0.0 & 28.8 & 11.1 & 0.0 \\
\hline
MinGW-Minimalist GNU for Windows~\citep{Cotroneo:TestType:JSS2013} & C & 60 & \textbf{46.6} & 10.0 & 0.0 & 38.3 & 0.0 & 0.0 & 5.0 & 0.0 \\
\hline
Java Enterprise Framework~\citep{Gupta:EMSE2009} & Java & 223 & 3.1 & 16.7 & \textbf{36.0} & 3.1 & 0.0 & 21.2 & 19.9 & 0.0 \\
\hline
Digital Cargo Files~\citep{Gupta:EMSE2009} & Java & 438 & 3.3 & 12.4 & 31.0 & 10.4 & 0.5 & \textbf{31.3} & 9.9 & 1.2 \\
\hline
Shipment and Allocation~\citep{Gupta:EMSE2009} & Java & 649 & 22.8 & 6.7 & 18.7 & 10.9 & 0.5 & \textbf{29.8} & 9.3 & 1.6 \\
\hline
\hline
Mirantis [This paper] & Puppet & 344 & 6.5 & \textbf{49.3} & 6.7 & 1.9 & 7.5 & 6.4 & 12.5 & 2.6 \\
\hline
Mozilla [This paper] & Puppet & 558 & 7.7 & \textbf{36.5} & 6.4 & 17.0 & 2.3 & 10.0 & 1.9 & 8.4 \\
\hline
Openstack [This paper] & Puppet & 1987 & 5.9 & \textbf{57.5} & 8.6 & 6.7 & 2.6 & 2.4 & 2.9 & 6.5 \\
\hline
Wikimedia [This paper] & Puppet & 298 & 3.3 & \textbf{62.7} & 4.7 & 12.0 & 4.3 & 4.3 & 2.6 & 5.1 \\
\hline
\hline
\end{tabular}
}
\end{table*}

\section{Discussion}
\label{discussion}
In this section, we discuss our findings with possible implications:

\subsection{Implications for Process Improvement}
One finding our paper is the prevalence of assignment-related defects in IaC scripts. Software teams can use this finding to improve their process in two possible ways: first, they can use the practice of code review for developing IaC scripts. Code reviews can be conducted using automated tools and/or team members' manual reviews. For example, through code reviews software teams can pinpoint the correct value of configurations at development stage. Automated code review tools such as linters can also help in detecting and fixing syntax issues of IaC scripts at the development stage. Typically, IaC scripts are used by organizations that have implemented CD, and for these organizations, Kim et al.~\citep{GeneKim:DevOps2016} recommends manual peer review methods such as pair programming to improve code quality.

Second, software teams might benefit from unit testing of IaC scripts to reduce defects related to configuration assignment. We have observed from Section~\ref{res-rq2} that IaC-related defects are mostly related to the assignment category which includes improper assignment of configuration values and syntax errors. Practitioners can test if correct configuration values are assigned by writing unit tests for components of IaC scripts. In this manner, instead of catching defects at run-time that might lead to real-world consequences e.g. the problem reported by Wikimedia Commons~\footnote{https://wikitech.wikimedia.org/wiki/Incident documentation/20170118-Labs}, software teams might be able to catch defects in IaC scripts at the development stages with the help of testing.

\subsection{Future Research}
Our findings have the potential to facilitate further research in the area of IaC defects. In Section~\ref{results} we have observed the process differences that occur between organizations, and future research can systematically investigate if there are process differences in IaC development and why they exist. We have applied a qualitative process to categorize defects using the defect type attribute of ODC. We acknowledge that our process is manual and labor-intensive. We advocate for future research that can look into how the process of ODC can be automated. In future researchers can also investigate why the observed defects occur, and provide a causal analysis of IaC defects.

\section{Threats To Validity}
\label{limitations}
We describe the threats to validity of our paper as following:

\begin{itemize}

\item{\textbf{Conclusion Validity}: Our approach is based on qualitative analysis, where raters categorized XCMs, and assigned defect categories. We acknowledge that the process is susceptible human judgment, and the raters' experience can bias the categories assigned. The accompanying human subjectivity can influence the distribution of the defect category for IaC scripts of interest. We mitigated this threat by assigning multiple raters for the same set of XCMs. Next, we used a resolver, who resolved the disagreements. Further, we cross-checked our categorization with practitioners who authored the XCMs, and observed `substantial' to `almost perfect' agreement.
}

\item{\textbf{Internal Validity}: We have used a combination of commit messages and issue report descriptions to determine if an IaC script is associated with a defect. We acknowledge that these messages might not have given the full context for the raters. Other sources of information such as practitioner input, and code changes that take place in each commit could have provided the raters better context to categorize the XCMs.

In our paper, we have used a defect-related commit message as a defect. We acknowledge that in a defect-related commit message more than one defects can be fixed, but may not be expressed in the commit message and/or accompanied issue report description.

We acknowledge that we have not used the trigger attribute of ODC, as our data sources do not include any information from which we can infer trigger attributes of ODC such as, `design conformance', `side effects', and `concurrency'.
}

\item{\textbf{Construct validity}: Our process of using human raters to determine defect categories can be limiting, as the process is susceptible to mono-method bias, where subjective judgment of raters can influence the findings. We mitigated this threat by using multiple raters.

Also, for Mirantis and Wikimedia, we used graduate students who performed the categorization as part of their class work. Students who participated in the categorization process can be subject to evaluation apprehension, i.e. consciously or sub-consciously relating their performance with the grades they would achieve for the course. We mitigated this threat by clearly explaining to the students that their performance in the categorization process would not affect their grades.

The raters involved in the categorization process had professional experience in software engineering for at two years on average. Their experience in software engineering may make the raters curious about the expected outcomes of the categorization process, which may effect the distribution of the categorization process. Furthermore, the resolver also has professional experience in software engineering and IaC script development, which could influence the outcome of the defect category distribution.
}

\item{\textbf{External Validity}: Our scripts are collected from the OSS domain, and not from proprietary sources. Our findings are subject to external validity, as our findings may not be generalizable.

We construct our datasets using Puppet, which is a declarative language. Our findings may not generalize for IaC scripts that use an imperative form of language. We hope to mitigate this limitation by analyzing IaC scripts written using imperative form of languages.
}

\end{itemize}

\section{Conclusion}
\label{conclusion}
Use of IaC scripts is crucial for the automated maintenance of software delivery and deployment infrastructure. Similar to software source code, IaC scripts churn frequently, and can include defects which can have serious consequences. IaC defect categorization can help organizations to identify opportunities to improve IaC development. We have conducted an empirical analysis using four datasets from four organizations namely, Mirantis, Mozilla, Openstack, and Wikimedia Commons. With 89 raters, we apply the defect type attribute of the orthogonal defect classification (ODC) methodology to categorize the defects. For all four datasets, we have observed that majority of the defects are related to assignment i.e. defects related to syntax errors and erroneous configuration assignments. We have observed compared to IaC scripts, assignment-related defects occur less frequently in non-IaC software. We also have observed the defect density is 27.6, 18.4, 16.2, and 17.1 defects per KLOC respectively for Mirantis, Mozilla, Openstack, and Wikimedia. For all four datasets, we observe IaC defects to follow a consistent temporal trend. We hope our findings will facilitate further research in IaC defect analysis.

\begin{acknowledgements}
The Science of Security Lablet at the North Carolina State University supported this research study. We thank the Realsearch research group members for their useful feedback. We also thank the practitioners who answered our questions.
\end{acknowledgements}

\bibliographystyle{spbasic}


Hey there, this Apricot Pancakes recipe is sure to get you hooked to it! It is always prepared with a focus on apricot. The European Apricot Pancakes is a delight to serve and enjoy. It would be a sin to not try this sinfully delicious Apricot Pancakes recipe.
This quick simple apricot pancake recipe can be made in minutes. Firstly sift the flour in a small bowl, add the large egg and mix with a fork.
Add the milk and keep stirring until nice and smooth, then add the chopped apricots and stir well.
In a small frying pan add the knob of butter and melt. Once pan is hot add the apricot pancake mixture (your desired amount), cooked on each side for around 2-3 minutes until nice and golden on each side.
Serve once cooked. These are also tasty with maple syrup on top should you wish.


text:
Piecemeal DRM-Free Efforts Aren't Going To Unseat Apple's Online Music Dominance
$150 Million Sure Can Inject Some Life Into A Pointless Standards Battle Over Next Generation DVDs
ES&S Gave California E-Voting Machines That Weren't Certified
(Mis)Uses of Technology
from the and-the-hits-keep-coming dept
Wed, Aug 22nd 2007 6:55am — Mike Masnick
Well, the hits keep for Election Systems & Software (ES&S). The company was already facing some controversy over the fact that its e-voting machines time stamped the ballots in a way that could reveal how voters voted, and now California's Secretary of State has discovered that the company gave the state e-voting machines for the last election that were not certified. California had certified one model of ES&S machines, but the company sold a different model to five counties. The state is now looking to fine ES&S. Of course, ES&S is also the company that originally refused to hand over its source code to the state, but eventually did so along with a threatening letter about how it would hold the Secretary of State personally responsible for any trade secrets that leaked. Trade secrets like how their machines accidentally were found to count votes in triplicate? ES&S, by the way, is also the company whose machines were used in the Florida election where many votes went missing, and it was later discovered that the company had passed around memos about their buggy software that could have caused the problem. It's also the company that had an employee come here to Techdirt and post a bunch of angry comments about how e-voting machines were perfectly safe and went through more than enough testing. Apparently not in California. Update: As pointed out in the comments, the company also was just caught failing to disclose foreign manufacturing partners to federal agencies, as it's required to do. ES&S's response on being caught? Shrugged it off as an oversight that they would fix. It appears that the company doesn't seem to worry about getting punished for all its mistakes.
Filed Under: california, e-voting
Companies: es&s
Federal Court Strikes Down California's Ban On 'Offensive' License Plates
California Court Says Wiretap Target Should Have Access To Wiretap Documents
California Assemblywoman Celebrates The RIAA Giving Her A Gold Record For Exempting Musicians From Her Terrible AB5 Law
Voting Machine Vendors, Election Officials Continue To Look Ridiculous, As Kids Hack Voting Machines In Minutes
Biggest Voting Machine Maker Admits -- Ooops -- That It Installed Remote Access Software After First Denying It
Anonymous Coward, 22 Aug 2007 @ 6:22pm
It appears that the company doesn't seem to worry about getting punished for all its mistakes.
Especially since the rewards seem to be greater that the punishment.
If Intellectual Property Is Neither Intellectual, Nor Property, What Is It?
An Economic Explanation For Why DRM Cannot Open Up New Business Model Opportunities
Advertising Is Content; Content Is Advertising
Free Access To Academic Papers For Everyone In India: Government Proposes 'One Nation, One Subscription' Approach As Part Of Major Shift To Openness
12:00 Funniest/Most Insightful Comments Of The Week At Techdirt (12)

redpajama_set_name: RedPajamaCommonCrawl


text:
Debórah Dwork is an American historian, specializing in the history of the Holocaust. She is the Founding Director of the Strassler Center for Holocaust and Genocide Studies and  formerly served as the Rose Professor of Holocaust History at Clark University in Worcester, Massachusetts.

Education and career
Dwork earned a B.A. from Princeton University in 1975, an M.P.H. from Yale University in 1978, and a Ph.D. from University College London in 1984. After postdoctoral studies at the Smithsonian Institution, she joined the faculty of the University of Michigan in 1984, and moved to the Yale Child Study Center at Yale University in 1989. She took her current position as Rose Professor at Clark University in 1996. She has held fellowships  from the Guggenheim Foundation, the American Council of Learned Societies, and the Woodrow Wilson International Center for Scholars, and has served as the Shapiro Senior Scholar-in-Residence at the United States Holocaust Memorial Museum and as a visiting scholar at Rutgers University.

Dwork is the founding Director of the Strassler Center for Holocaust and Genocide Studies at Clark University, and a delegate to the International Holocaust Remembrance Alliance.

Academic work
In her first book, War is Good for Babies and Other Young Children (1987), Dwork examined questions about the family, the role of women, and the concept of children's rights in the context of the development of the modern welfare system.

Dwork  moved from the history of childhood as a social construct to the history of children as subjects and actors.  Her historical analysis used children's experiences as a lens through which to view all of society.  In her Children With A Star (1991), she presented the daily lives of young people caught in the net of Nazisim. The book became the subject of a documentary of the same name by the Canadian Broadcasting Corporation.

Auschwitz, 1270 to the Present (1996), co-authored with Robert Jan van Pelt, demonstrated the connection between industrial killing and the daily functions of a society that believed it was involved in constructive activity. The book uses architectural evidence to understand Auschwitz. The  book received the National Jewish Book Award and the Spiro Kostof Award.

Dwork's book Voices and Views: A History of the Holocaust (2002)  is an edited, annotated, and illustrated collection used by the national Holocaust education program of the Jewish Foundation for the Righteous.
Dwork and van Pelt also collaborated on Holocaust: A History, which  discusses the place of the Holocaust in the overall history of Europe, from the Middle Ages to the middle of the twentieth century.  It explores how the different occupation regimes shaped the local populations' ability to respond to the genocide enacted outside their windows.  It  integrates the history of World War II and the history of the Holocaust.

In Flight from the Reich (2009), Dwork and van Pelt turned their attention to the question of refugee Jews from 1933  through the postwar period.

Dwork edited and annotated The Terezin Album of Mariaka Zadikow (2008), a posie album collected by a Jewish inmate as the Germans pushed forward with deportations from Theresienstadt. In A Boy in Terezin: The Private Diary of Pavel Weiner, April 1944 – April 1945 (2011, ), she returned to the experiences of children as an important source for contemporaneous accounts of Jewish life under Nazi persecution.

Personal life
Dwork is the daughter of mathematician Bernard Dwork, and sister of computer scientist Cynthia Dwork.

Film Credits
Dwork has served as the historian of record on and off film in feature-length and TV documentaries. These include director Rick Trank's "Against the Tide" (2008) and "Unlikely Heroes" (2003), and the Ken Burns/Artemis Joukowsky documentary, "Defying the Nazis" (2016).  Television documentaries include "Hiding in Plain Sight" (CBS, 2009) and "Misha Defonseca and her Hoax Memoir," (RTBF, Belgian National TV, 2008).

Bibliography
 Dwork, Debórah (2012). A Boy in Terezin: The Private Diary of Pavel Weiner, April 1944 – April 1945. Evanston: Northwestern University Press. .
 Dwork, Debórah; van Pelt, Robert (2009). Flight from the Reich: Refugee Jews, 1933–1946. New York: W.W. Norton. . Translations: Dutch (Elmar); French (Calmann-Lévy).
 Dwork, Debórah (2008). The Terezin Album of Marianka Zadikow. Chicago: University of Chicago Press. .
 Dwork, Debórah; van Pelt, Robert (2008). Auschwitz. New York: W.W. Norton & Co. . First published as: Auschwitz 1270 to the Present. New York: Norton. . Translations: Czech (Argo); Dutch (Boom); German (Pendo); Polish (Swiat Ksiazki).
 Dwork, Debórah; van Pelt, Robert (2002). Holocaust: A History. New York: Norton. . Translations: Dutch (Boom); Portuguese (Imago); Spanish (EDAF).
 Dwork, Debórah (2002). Voices and Views: A History of the Holocaust. New York: Jewish Foundation for the Righteous. .
 Dwork, Debórah (1991). Children With A Star: Jewish Youth in Nazi Europe. New Haven: Yale University Press. . Translations: Dutch (Boom); German (Beck); Italian (Marsilio); Japanese (Sogen Sha).
 Dwork, Debórah (1987). War Is Good for Babies and Other Young Children: A History of the Infant and Child Welfare Movement in England 1898–1918. London; New York: Tavistock Publications. .

References

External links
Debórah Dwork Academic homepage.
Debórah Dwork Personal homepage.
Strassler Center for Holocaust and Genocide Studies Homepage.

Year of birth missing (living people)
Living people
University of Michigan faculty
Yale University faculty
Clark University faculty
Historians of Nazism
Historians of the Holocaust
Jewish American historians
Scholars of antisemitism
21st-century American historians
American women historians
Social constructionism
21st-century American women

redpajama_set_name: RedPajamaWikipedia


75 mi. S.W. Chicago. I-80 W. to Marseilles Exit 97, S. to Rt. 6, E. 11 blocks to Wilson St., N. to the entrance. In the Illinois River Valley – 35 minutes W. of Joliet. Nestled in the lush Illinois River valley on 400+ wooded, hilly acres. One mile to Illinois River (free boat ramp, fishing, etc) and the historical I and M Canal (biking, hiking, snowmobiling). Explore the wonders along this picturesque and unique area. 17 minutes to Starved Rock State Park, 45 minutes to Chicagoland Speedway and Rt. 66 Speedway. Only 1 mile to Cliff's Insane Terrain off road park and Zip Chicago, for the area's most exciting zip line activity. We offer planned activities, swimming pool, fishing pond with sandy beach, hiking trails, hayrides, basketball, baseball, volleyball, tennis and playgrounds.


text:
Zhang, C & Lukose, D 1997, Preface.
Antoniou, G & Williams, M 1997, 'Reasoning With Incomplete And Changing Information: The Cin Project', Information Sciences, vol. 99, no. 1-2, pp. 83-99.
Most information systems are faced with incomplete information, even for simple database applications; therefore, they must make plausible conjectures in order to operate in a satisfactory way. A simple example is the closed world assumption, which is us
Blumenstein, M & Verma, B 1997, 'A segmentation algorithm used in conjunction with Artificial Neural Networks for the recognition of real-world postal addresses', Journal of Intelligent Systems, vol. 7, no. 3-4, pp. 191-203.
Artificial Neural Networks (ANNs) have been successfully applied for pattern recognition, speech recognition, control and other real world problems. This paper presents a method for segmentation of printed and difficult handwritten postal addresses. The segmentation algorithm is used to prepare raw training data for use with an Artificial Neural Network. The C programming language, the SP2 supercomputer and a SUN workstation were used for the experiments. The algorithm has been successfully tested on real world handwritten postal addresses. Some experimental results are presented in this paper.
Catchpoole, D, Lam, WWK, Valler, D, Temple, IK, Joyce, JA, Reik, W, Schofield, PN & Maher, ER 1997, 'Epigenetic modification and uniparental inheritance of H19 in Beckwith-Wiedemann syndrome', Journal of Medical Genetics, vol. 34, no. 5, pp. 353-359.
Beckwith-Wiedemann syndrome (BWS) is a congenital overgrowth syndrome associated with a characteristic pattern of visceromegaly and predisposition to childhood tumours. BWS is a genetically heterogeneous disorder; most cases are sporadic but approximately 15% are familial and a small number of BWS patients have cytogenetic abnormalities involving chromosome 11p15. Genomic imprinting effects have been implicated in familial and non-familial BWS. We have investigated the molecular pathology of 106 sporadic BWS cases; 17% (14/83) of informative cases had uniparental disomy (UPD) for chromosome 11p15.5. In each case UPD appeared to result from a postzygotic event resulting in mosaicism for segmental paternal isodisomy. The critical region for isodisomy was refined to a 25 cM interval between D11S861 and D11S2071 which contained the IGF2, H19, and p57(KIP2) genes. In three cases isodisomy for 11q markers was detected but this did not extend further than 11q13-q21 suggesting that complete chromosome 11 disomy may not produce a BWS phenotype. The allele specific methylation status of the H19 gene was investigated in 80 sporadic BWS cases. All 13 cases with UPD tested displayed hypermethylation consistent with an excess of paternal H19 alleles. In addition, five of 63 (8%) cases with normal biparental inheritance had H19 hypermethylation consistent with an 'imprinting centre' mutation (ICM) or 'imprinting error' (IE) lesion. The phenotype of patients with putative ICM/IE mutations was variable and overlapped with that of non-UPD sporadic BWS cases with normal H19 methylation. However, exomphalos was significantly (p < 0.05) more common in the latter group. These findings may indicate differential effects on the expression of imprinted genes in chromosome 11p15 according to the precise molecular pathology. Analysis of H19 methylation is useful for the diagnosis of both UPD or altered imprinting in BWS and shows that a variety of molecular mechanisms may cause relaxation o...
Gopalsamy, K & He, X 1997, 'Oscillations And Convergence In An Almost Periodic Competition System', Acta Applicandae Mathematicae, vol. 46, no. 3, pp. 247-266.
Sufficient conditions are derived for the existence of a globally attractive almost periodic solution of a competition system modelled by the nonautonomous Lotka-Volterra delay differential equations dN(1)(t)/dt = N-1(t)[r(1)(t) - a(11)(t)N-1(t-r(t)) - a
He, X & Gopalsamy, K 1997, 'Persistence, Attractivity, And Delay In Facultative Mutualism', Journal Of Mathematical Analysis And Applications, vol. 215, no. 1, pp. 154-173.
Relations between persistence and ultimate boundedness of the solutions of the Lotka-Volterra system dx(t)/dt = x(t)[r(1) - a(11)x(t - tau) + a(22)y(t - tau)] dy(t)/dt = y(t)[r(2) + a(21)x(t - tau) - a(22)y(t - tau)] modelling ''facultative mutualism'' w
Jay, CB 1997, 'Covariant types', Theoretical Computer Science, vol. 185, pp. 237-258.
Jay, CB 1997, 'Finite Objects in a Locos', Journal of Pure and Applied Algebra, vol. 116, pp. 169-83.
Jay, CB & Steckler, PA 1997, 'The Functional Imperative: Shape!'.
Joyce, JA, Lam, WK, Catchpoole, DJ, Jenks, P, Reik, W, Maher, ER & Schofield, PN 1997, 'Imprinting of IGF2 and H19: Lack of reciprocity in sporadic Beckwith-Wiedemann syndrome', Human Molecular Genetics, vol. 6, no. 9, pp. 1543-1548.
Genomic imprinting is a novel form of control of gene expression in which the transcription of each allele of an imprinted gene is dependent on the sex of the gamete from which it was derived; to date > 15 genes have been demonstrated to show imprinting. The maintenance of a normal imprinting pattern in many loci has been shown to be essential for normal development and adult life. Many tumours, and some developmental disorders, exhibit loss of imprinting (LOI) in key genes such as insulin-like growth factor 2 (IGF2) which often results in hyperplasia and is associated with cancer. The mechanism by which the genomic imprint is first established, then maintained, is not understood. However, in the case of IGF2, the expression of a neighbouring gene, H19, has been suggested to influence its transcription by competition for a common enhancer, thereby generating a mutually exclusive and allele-specific pattern of gene expression. Associated changes in CpG methylation in discrete areas of both genes have been implicated in maintenance of the imprint. We have examined the allele-specific expression of IGF2 and H19 in fibroblasts derived from patients with sporadic Beckwith-Wiedemann syndrome (BWS), a fetal overgrowth syndrome associated with an imprinted locus on 11p15.5. We report that the majority of karyotypically normal patients show LOI of IGF2 with biallelic expression. In a proportion of these patients, loss of IGF2 imprinting was associated with complete suppression of H19 expression, as predicted by the enhancer competition model. However, in a significant number of cases, IGF2 showed biallelic expression even though H19 expression and methylation status were normal. This indicates that there must be an alternative H19-independent pathway by which allele-specific IGF2 expression is established or maintained.
Lin, CJ & Lin, CT 1997, 'An ART-based fuzzy adaptive learning control network', IEEE TRANSACTIONS ON FUZZY SYSTEMS, vol. 5, no. 4, pp. 477-496.
Lin, CT & Juang, CF 1997, 'An adaptive neural fuzzy filter and its applications', IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART B-CYBERNETICS, vol. 27, no. 4, pp. 635-656.
Luo, X, Zhang, C & Cai, J 1997, 'The weighting issue in fuzzy logic', Informatica (Ljubljana), vol. 21, no. 2, pp. 255-262.
The objective of this paper is to establish a class of models for solving the weight problem in fuzzy logic. First, some constraints that weighted fuzzy logic should satisfy are given based on Triangular norms and conorms. Then, based on these constraints, a class of models (referred to as relative weighted models) are established for handling weights in fuzzy logic. These models are novel in three aspects: (1) they include non-weighted models as their special cases, (2) the weighted conjunction and weighted disjunction can be distinguished from each other, and (3) the information from all sub-propositions can be sufficiently considered. In addition, this paper proposes principles for selecting proper weighted models.
Polshakov, VI, Williams, MA, Gargaro, AR, Frenkiel, TA, Westley, BR, Chadwick, MP, May, FEB & Feeney, J 1997, 'High-resolution solution structure of human pNR-2/pS2: A single trefoil motif protein', Journal of Molecular Biology, vol. 267, no. 2, pp. 418-432.
pNR-2/pS2 is a 60 residue extracellular protein, which was originally discovered in human breast cancer cells, and subsequently found in other tumours and normal gastric epithelial cells. We have determined the three-dimensional solution structure of a C58S mutant of human pNR-2/pS2 using 639 distance and 137 torsion angle constraints obtained from analysis of multidimensional NMR spectra. A series of simulated annealing calculations resulted in the unambiguous determination of the protein's disulphide bonding pattern and produced a family of 19 structures consistent with the constraints. The peptide contains a single 'trefoil' sequence motif, a region of about 40 residues with a characteristic sequence pattern, which has been found, either singly or as a repeat, in about a dozen extracellular proteins. The trefoil domain contains three disulphide bonds, whose 1-5, 2-4 and 3-6 cysteine pairings form the structure into three closely packed loops with only a small amount of secondary structure, which consists of a short α-helix packed against a two-stranded antiparallel β-sheet. The structure of the domain is very similar to those of the two trefoil domains that occur in porcine spasmolytic polypeptide (PSP), the only member of the trefoil family whose three-dimensional structure has been previously determined. Outside the trefoil domain, which forms the compact 'head' of the molecule, the N and C-terminal strands are closely associated, forming an extended 'tail', which has some β-sheet character for part of its length and which becomes more disordered towards the termini as indicated by 15N(1H) NOEs. We have considered the structural implications of the possible formation of a native C58-C58 disulphide-bonded homodimer. Comparison of the surface features of pNR-2/pS2 and PSP, and consideration of the sequences of the other human trefoil domains in the light of these structures, illuminates the possible role of specific residues in ligand/receptor binding.
Ravert, P, Williams, M & Fosbinder, DM 1997, 'The Interpersonal Competence Instrument for Nurses', Western Journal of Nursing Research, vol. 19, no. 6, pp. 781-791.
A new tool, the Interpersonal Competence Instrument for Nurses, was evaluated for content validity and readability. Evaluators consisted of a panel of 10 nursing experts. The instrument measures four categories of the patient-nurse interaction: translating, getting to know you, establishing trust, and going the extra mile. The content validity indexes (CVI) for 14 of 15 behaviors within the four categories ranged from .8 to 1.0 and are considered to have content validity. The CVI for the 15th behavior, clicking, was calculated as .7, and, thus, was modified. The CVI for the entire instrument of 111 items was determined to be .84. Readability analyzed with the SMOG formula established a grade level of 8.09. Additional psychometric testing of the tool is in progress. Internal consistency reliability is being evaluated through the use of coefficient alpha and item analysis. Construct validity is being estimated through the experimental approach. The internal structure of the instrument is being assessed through factor analysis. Presuming that this instrument continues to demonstrate validity and reliability with future testing, it will facilitate the evaluation of nurse-patient interaction and promote focused education for nurses in the career development continuum.
Williams, MA 1997, 'Fostering collegiality.', Nursing management, vol. 28, no. 6, p. 66.
Yang, H & Zhang, C 1997, 'Application Of MAS In Implementing Rational Ip Routers On The Priced Internet', Multi-agent Systems, vol. 1286, pp. 166-180.
Zhang, M & Zhang, C 1997, 'Methodologies Of Solution Synthesis In Distributed Expert Systems', Multi-agent Systems, vol. 1286, pp. 137-151.
Jay, CB 1997, 'Separating shape from data', Category theory and computer science: 7th international conference, CTCS'97, Santa Margherita Ligure, Italy, September 1997 Proceedings, Springer Verlag, pp. 47-48.
Jay, CB & Sekanina, M 1997, 'Shape checking of array programs', Computing: the Australasian Theory Seminar, Proceedings, 1997, pp. 113-121.
Jay, CB, Cole, MI, Sekanina, M & Steckler, PA 1997, 'A Monadic Calculus for Parallel Costing of a Functional Language of Arrays', Euro-Par'97 Parallel Processing, Springer, pp. 650-661.
Juang, CF & Lin, CT 1997, 'Recurrent self-organizing neural fuzzy inference network', IEEE International Conference on Fuzzy Systems, pp. 1369-1374.
A Recurrent Self-Organizing Neural Fuzzy Inference Network (RSONFIN) is proposed in this paper. The RSONFIN is constructed from a series of dynamic fuzzy rules. The temporal relations embedded in the network are built by adding some feedback connections representing the memory elements to a feedforward neural fuzzy network. Each weight as well as node in the RSONFIN has its own meaning and represents a special element in a fuzzy rule. There are no hidden nodes (i.e., no membership functions and fuzzy rules) initially in the RSONFIN. They are created on-line via concurrent structure identification (the construction of dynamic fuzzy if-then rules) and parameter identification (the tuning of the free parameters of membership functions). The structure learning together with the parameter learning forms a fast learning algorithm for building a small, yet powerful, dynamic neural fuzzy network. Simulations on temporal problems are done finally.
Palsberg, J & Jay, CB 1997, 'The Essence of the Visitor Pattern', University of Technology, Sydney.
Williams, MA 1997, 'Anytime belief revision', IJCAI International Joint Conference on Artificial Intelligence, pp. 74-79.
Belief Revision is a ubiquitous process underlying many forms of intelligent behaviour. The AGM paradigm is a powerful framework for modeling and implementing belief revision systems based on the principle of Minimal Change; it provides a rich and rigorous foundation for computer-based belief revision architectures. Maxi-adjustment is a belief revision strategy for theory bases that can be implemented using a standard theorem prover, and one that has been used successfully for several applications. In this paper we provide an anytime decision procedure for maxi-adjustments, and study its complexity. Furthermore, we outline a set of guidelines that serve as a protomethodology for building belief revision systems employing a maxi-adjustment. The algorithm is under development in the belief revision module of the CIN Project.
Williams, MA 1997, 'Belief revision as database update', Proceedings - Intelligent Information Systems, IIS 1997, pp. 410-414.
© 1997 IEEE. Belief revision is a fundamental process that underlies numerous forms of intelligent behaviour. Intelligent information systems must be adept at modifying their beliefs in a rational, coherent and consistent fashion. We show that iterated belief revision strategies can be implemented using standard relational database technology. We illustrate the main ideas using Spohn's (1988) notion of conditionalization. For the purpose of motivation we focus our attention on a problem in market research, namely modeling changes to consumer preferences. The key idea is that possible product profiles can be represented as tuples in a relational database, and the consumer's preference for products is captured using an ordinal ranking over the tuples. Using this representation iterated belief revision strategies can be implemented simply and efficiently. In particular, belief revision is performed using database transactions that modify this ranking.
Zhang, C & Zhang, S 1997, 'Time framework for temporal databases', IEEE Region 10 Annual International Conference, Proceedings/TENCON, pp. 215-218.
There are two different strategies for incorporating a temporal dimension into the relational models in the literature. One is the first-normal-form(1NF) model; and the other is the non-first-normal-form(N1NF) model. Both of them are required to support an ability of handling data with temporal element (or temporal set, or gap-interval). In order to support these temporal operations for temporal databases, some basic concepts and a calculus form between temporal elements are established in this paper.
Zhang, S & Zhang, C 1997, 'Query languages for 1NF temporal databases', IEEE Region 10 Annual International Conference, Proceedings/TENCON, pp. 211-214.
There are two factors that should be considered in defining or designing a temporal database. One is that the uncertain time `NOW' should be noticed in the relational algebraic operators extended for incorporating a temporal dimension. The other is that the temporal-normal-form is closure under operators extended. In this paper, we establish gap-interval-based 1NF temporal tuple calculus language and the corresponding temporal relational algebra. The query languages are given some considerations to above two factors.
Jay, CB 1997, 'FiSh v 0.4'.


text:
Valdemar o Waldemar de Brandenburg dit el Gran -en alemany: Waldemar der Große- (~1280 - 14 d'agost de 1319, Bärwalde) va ser marcgravi de Brandenburg de la branca de Brandenburg-Stendal, l'últim de la casa d'Ascània. Era fill del marcgravi Conrad de Brandenburg de la branca de Brandenburg-Stendal i de Constança, filla de Przemysl I de Gran Polònia. Va governar junt amb el seu cosí Enric II de Brandenburg.

Biografia
Valdemar va ser co-regent des de la mort del seu pare vers 1304, i va assolir el 1309 la tutoria del seu cosí Enric II marcgravi de Brandenburg-Stendal.

Pel tractat de Soldin (1309) Waldemar va renunciar a les seves pretensions a Pomerèlia i Danzig a favor de l'Orde Teutònic per a un pagament de 10.000 marcs de plata. Les castellanies de Stolpe i Schlawe van quedar per Brandenburg. A aquestos districte però, foren cedits, juntament amb Rügenwalde, al duc Wartislaw IV de Pomerània-Wolgast el 1317.

El 1316 va donar suport a Stralsund contra Dinamarca i això va provocar una gran coalició formada per Dinamarca, Suècia, Polònia, Mecklenburg i la Casa de Welf contra ell. Va mantenir la seva posició, però al final va haver de rendir Stargard-Neubrandenburg a Mecklemburg. El 1319 Waldemar va adquirir Zuellichau i Schwiebus de Brandenburg.

Waldemar va ser gairebé l'últim membre governant de la línia dels Ascanis de Brandenburg. El 1309 es va casar amb Agnès de Brandenburg (1296-1334), filla del marcgravi Herman III. Aquest matrimoni no va tenir fills.

Amb la mort de Waldemar el 1319, Enric II el va succeir però va morir el 1320 i amb ell va acabar la casa Ascània, i Brandenburg va passar a l'Imperi. El 1323 l'emperador Lluís IV de Wittelsbach el va cedir al seu fill Lluís V de Wittelsbach.

El 1348, un impostor anomenat "Fals Waldemar" va al·legar que era Waldemar, tornat de peregrinació a Terra Santa, després que algú havia estat enterrat en el seu lloc. Ràpidament va guanyar suport a causa de les ciutats, i a causa de les rivalitats polítiques, de l'Emperador que el va arribar a investir i el va reconèixer durant dos anys; després "l'últim Ascani" va passar-se la resta de la vida a la cort a Dessau.

Bibliografia
 Wilhelm von Sommerfeld: Waldemar (Markgraf von Brandenburg) a: Allgemeine Deutsche Biographie (ADB), Duncker & Humblot, Leipzig 1896

Marcgravis de Brandenburg
Morts a Polònia


How is firing under-performing federal employees a bad thing?
Trump announces big changes for Federal government employees, ending automatic pay hikes that kick in "irrespective of performance," changing retirement benefits and making it easier to fire bad employees and reward good ones. So Trump wants to fire people who aren't doing a good job or pulling their weight? Of course the union thinks this is a bad idea.
As a society, we've gotten so that we turn a blind eye on anything that's happening unless we're concerned about being sued!
Telltale internet message may have foreshadowed Parkland, Florida school shooting massacre. So why didn't anyone notice? Or do anything?
Discipline is dead in our nation's public school system. Kids think they can do whatever they want. We are seeing a country of kids being raised to only respect their fellow man while facing the threat of legal retribution.
Social media blocks crowdfunding site for new movie "Roe v. Wade" on how abortion became legal in U.S. Why? I thought Liberals were all about CHOICE!


The Infamous Stringdusters are an acoustic/bluegrass band. The band emerged in 2007 with the album Fork in the Road on Sugar Hill Records. The band's current line-up features Andy Hall (Dobro), Andy Falco (guitar), Chris Pandolfi (banjo), Jeremy Garrett (fiddle), and Travis Book (double bass). The band has become known for a complex, distinctive, and groove-friendly sound along with a bluegrass theme.
The Infamous Stringdusters won three awards at the International Bluegrass Music Association Awards Ceremony in October 2007: Emerging Artist of the Year, Album of the Year for Fork in the Road (in a tie with J.D. Crowe & the New South's album Lefty's Old Guitar), and Song of the Year for the album's title cut. The band was also nominated for 2011 Entertainer of the Year by the International Bluegrass Music Association.
Their "Magic No. 9" (from Things That Fly) was nominated for a 2011 Grammy award for "Best Country Instrumental".
Five-piece bluegrass outfit The Infamous Stringdusters released their Undercover Vol. 2 EP this past October, featuring songs by The Allman Brothers Band, My Morning Jacket, Marvin Gaye and even Daft Punk.
But on Laws of Gravity, the Infamous Stringdusters return to their trademark style after innovating a bit on their previous record Ladies and Gentlemen; apparently they weren't wrong when they wrote a song titled "A Hard Life Makes a Good Song".
It was just the other day that Andy Falco and his bandmates in The Infamous Stringdusters learned their latest album, "Laws of Gravity," had been nominated for a Grammy for Best Bluegrass Album. As anyone with children knows, that doesn't impress a 5-year-old.
"Jesse Cobb departs the 'Dusters"
"The Infamous Stringdusters: New Album, Spring Tour 2012"
"The Infamous Stringdusters "Silver Sky""