text:
\section{Appendix}

\clearpage
\input{tex/NIPS-supplement}

\end{document}
\section{Introduction}
A protein consists of poly-peptide chains, each of the chains being embedded sequences of amino acids, wach with specific 3D structure. By a chemical reaction, amino acids become linked to form these poly-peptide chain. A residue is a transformed amino acid as it forms the poly-peptide chain. Each residue consists of \(N-H, C_\alpha, C=O\), which are part of the backbone of the protein, and  a side chain \(R\). The side chain \(R\) is different for different amino acids and quite often thought to be attached to the $C_\alpha$ with different rotamer orientations.. The \textbf{primary structure }of protein is the 1-D sequence of
residues. However, to be functional, protein fold into a certain metastable 3D configurations that minimizes their  potential energy which is a combination of hydrogen bonding, electrostatics and the solvation terms capturing their interactions with the ionic watery environment at various PH levels. These metastable 3D configurations of the protein gives the  tertiary and quaternary structure of the protein.

Atoms in a single residue are conceptualized to be linked by various bond edges (single, double covalent and even ionic). The bond angle between these bond edges, the length of these edges and the torsion angles called \textbf{dihedral} angles around each of the four consecutive atoms along the backbone chains will specify the  structure of a protein. Side-chain conformations (SCP) are the problem of determining the rotamer angle of the side chain residues in the protein structure, and are a critical component of protein prediction. The SCP problem is non-convex and equally NP-hard as is the full structure prediction problem \cite{A97,PW02}.  It cannot even be approximated in polynomial time, i.e. no PTAS (polynomial time approximation scheme) is possible  \cite{CKS04}. Many SCP algorithms with various techniques ranging from combinatorial algorithm to deep learning solutions have been developed and implemented in the past three decades.

\textbf{Classical methods}: Given a fixed (low -energy) structure embedding of the protein sequence forming a  protein backbone chain in 3D, and a discretized rotamer library of feasible side chain orientations, SCP can be formulated as a combinatorial search problem for the optimal sequence of rotamer orientation selections along the backbone that optimizes the total  energy function of the side chain packed backbone chain. Many of the classical methods \cite{CKS04,D02,DC97,DK93,DK94,LWRR00,MDL97,MIS87,PR87,TEHL91,X05,XB06} without a deep learning framework
rely on this combinatorial formulation and also the discretized rotamer libraries, obtained from  a statistical clustering of observed side-chain conformations in known protein crystal structures. The overal protein potential energy function or Hamiltonian typically
includes a term for the backbone conformation, one for each side chain rotamer assignment in the protein, and one term for every pair of chosen rotamer orientations.
More complex energy terms such potential energy functions combined with complex solvation free energy terms \cite{LG02,MBCS01,XH01} are sometimes used.  The Dead End Elimination (DEE) theorem \cite{DMHL92} is often used to prune the number of rotamers to be considered for finding a global energy minima. Many algorithms based on this theorem can find the global minima provided they are able to converge \cite{DMHL92,DML97,G94,GM99,KSM+95,LD93,LMD95,LH01,MDL97,PSDM00,VGM00}, while some algorithms based on other techniques such as Markov Chain Monte-Carlo
simulation \cite{HS91}, cyclical search \cite{DK93,XH01}, spatial restraint
satisfaction \cite{SB93}, and approximation algorithms based on semi-definite programming
\cite{CKS04,KCS05,XB06} have reasonable running times but do not guarantee a global optima. Other used methods include A* search \cite{LL98}, simulated annealing \cite{LS91},
mean-field optimization \cite{L94}, maximum edge-weighted cliques \cite{BTSA05},
and integer linear programming \cite{AKLM00,EZE01}.

\textbf{Deep learning framework}: Earlier work utilizing deep learning \cite{CNN4SCP, SIDEpro} relies heavily on rotamer libraries , to handle side-chain prediction. SIDEPro, for example, use a family of ANNs to learn energy function. The ANNs' weights are optimized based on the posterior probability of the rotamers, which is calculated by the given prior probability in the rotamer libraries.

Recently, many end-to-end deep learning models that rely on sequence homology rather than rotamer libraries have been proposed for protein structure prediction problem. AlQuraishi \cite{RGN} uses a RNN to encode a protein sequence and then supervised learning with \textbf{ProteinNet} dataset \cite{ProteinNet} to predict the 3D protein structure. AlphaFold \cite{AF} uses deep two-dimensional dilated convolutional residual network and trains on Protein Data Bank (PDB) to predict distances between $C_{\beta}$ atoms, and use this information to infer the 3D structure. AlphaFold2 \cite{AF2} further augments AlphaFold1 with an attention-based network to predict backbone dihedrals and sidechain torsion angles which are used to reconstruct the atomic coordinates. Ingraham et al. \cite{Ingraham:2019} instead uses a deep network model to learn a stochastic energy function, a simulator based on unrolled Langenvin dynamics enhanced by stabilization techniques, and also an atomic imputation network to predict the final structure given sequence information.

\textbf{Molecular Reinforcement learning framework:} There are not many works that use reinforcement learning for both side-chain prediction and protein folding. Most of these works rely on some discrete models with either discrete action or state or both. For instance, Jafari et al. \cite{Jafari2020-lo} considered the lattice-based model called the hydrophobic-polar (HP) model \cite{Lau1989-fd}. The state receives only two values $1$ (hydrophilic) or $-1$ (hydrophobic) while the action can only be either up, down, left or right. \cite{Wu2019-uz} also consider the same problem also with a discrete formulation.

 We  use a reinforcement learning method and take a continuous optimal control approach instead of all prior works.
The main contributions of our paper are:
\begin{itemize}
    \item we develop a reinforcement learning framework in a continuous setting and based on a new stochastic parameterized Hamiltonian version of the Pontryagin maximum principle (PMP) for the protein side-chain and folding problems
    \item special parameter settings of our stochastic Hamiltonian dynamics, through the velocity adjoint variables can capture full stochastic Langevin dynamics of prior methods
    \item our approach allows for the use of different models of protein energetics and force-fields (gradients)
    \item we provide flexible optimal trajectory (path) assignment solutions to the rotamer angles for side-chain packing, and the dihedral angles of the backbone chains for folding
    \item our neural networks are trained using continuous accelerated Nesterov gradient descent
\end{itemize}

The rest of our paper is organized as follows. Section $2.1$ describes the protein side-chain packing and folding problem formulations. Section $2.2$ restates the  soft-actor critic RL algorithm but for our optimally controlled generalized Hamiltonian coordinates with the adjoint velocity variables encapsulating the control via the Pontryagin Maximum Principle. This RL algorithm is specialized to the protein  side-chain packing and folding solutions detailed in Section $3$. Section $2.4$ details the theoretical underpinnings of our continuous stochastic reinforcement learning setting. Section $2.4.1$ first states an appropriate  formulation of an optimally controlled  Hamiltonian dynamics with a customized cost (reward) function. Section $2.4.2$ extends this to the stochastic optimal control setting and through use of  Pontryagin maximum principle maps  the stochastic continuous optimal control problem into a parameterixed stochastic Hamiltonian dynamics setting. From use of the theory of  first order adjoint processes of stochastic differential equations and the result of \cite{Peng1990-gc} we are able to show that our RL formulation and the resulting optimal trajectories encapsulated the optimal Langevin dynamics trajectories of \cite{Ingraham:2019}. This reduction as only a special case of our RL formulation is proved in Section $2.4.3$  Finally in Section $2.4.4$ we show that the continuous  RL solution of our optimal controlled Hamiltonian dynamics recovers the continuous Nesterov's accelerated gradient descent dynamics.  The proofs of these results are given in the appendix.

\section{Broader Impact}:
We don't envision any broad  ethical issues surrounding our largely mathematical learning technique. We would of course welcome any issues the reviewers might raise, and promise to address them.
\section{Experiments}
\subsection{State and Action Space Representation}
The state $s_t$ in the protein folding problem is a vector consisting of all torsion angles $(d_i)_{i=1}^M$. For the side chain problems, this only includes the set of all side chain angles. Each action $a_t$ is represented as a torsion angle perturbation vector $\dot{d}$ of length $M$, where $M$ is the number of torsion angles that the agent can change. Each time step the agent performs the action by multiplying $\dot{d}$ by a step size and adding it to the current torsion angle vector represented by length $M$ vector $d$. The new state at time $t+1$ can simply be updated as:
\begin{equation}
s_{t+1} = s_t + \epsilon a_t \text{ for some small }\epsilon
\end{equation}

In the protein side-chain packing problem, we represent the protein's configuration as a molecular graph embedded in Cartesian space where each node represents an atom and each edge represents a covalent bond. Let $\mathcal{G} = (X, A)$ be the embedded molecular graph with node feature matrix $X$ of size $N_{atom} \times N_{features}$ and bond adjacency matrix $A$ of size $N_{atoms} \times N_{atoms}$. Each atom is represented by a feature vector in $X$ consisting of its Cartesian coordinates, a one-hot encoding of its element name (i.e. C, N, O, H, or S), its van-der Waal radius, the atomic charge, a binary number indicating if the atom can be an acceptor, a binary number indicating if the atom can be a donor, and a binary number indicating if the atom is on the backbone. This protein's configuration is updated in each time step $t$ based on the collection of angles in state $s_t$.

All atomic chemical features were retrieved from the protein modeling software Rosetta and all energies were calculated using Rosetta's default scoring function \cite{Alford2017}. Finally, we define the reward as in algorithm 1 for the soft actor-critic.

\subsection{Deep Learning Architecture}
For our neural network, we create a Soft-Actor Critic (SAC) architecture as formulated in \cite{Haarnoja:2018} with automatic temperature tuning using optimization of the dual Lagrangian. Since our state space is in the form of a graph, we use a graph convolutional neural network with the same architecture as MolGan \cite{decao2018molgan} to create a valid state embedding. Actions are processed using standard trained multilayer perceptrons.

\begin{table}[]
\centering
\begin{tabular}{cc}
\hline
\multicolumn{1}{|c|}{Atomic Feature} & \multicolumn{1}{c|}{Dimension} \\ \hline
\multicolumn{1}{|c|}{x, y, z}        & \multicolumn{1}{c|}{3}          \\ \hline
\multicolumn{1}{|c|}{Element}        & \multicolumn{1}{c|}{5}          \\ \hline
\multicolumn{1}{|c|}{LJ Radius}      & \multicolumn{1}{c|}{1}          \\ \hline
\multicolumn{1}{|c|}{Atomic Charge}  & \multicolumn{1}{c|}{1}          \\ \hline
\multicolumn{1}{|c|}{Is Acceptor}    & \multicolumn{1}{c|}{1}          \\ \hline
\multicolumn{1}{|c|}{Is Donor}       & \multicolumn{1}{c|}{1}          \\ \hline
\multicolumn{1}{|c|}{Is Backbone}    & \multicolumn{1}{c|}{1}          \\ \hline
\multicolumn{1}{l}{}                 & \multicolumn{1}{l}{}
\end{tabular}
\caption{The feature attributes specified for each atom an their dimensions}
\end{table}
\begin{table}[]
\centering
\begin{tabular}{cc}
\hline
\multicolumn{1}{|c|}{\textbf{Hyperparameter}}          & \multicolumn{1}{c|}{\textbf{Value}} \\ \hline
\multicolumn{1}{|c|}{Discount Rate}           & \multicolumn{1}{c|}{1}       \\ \hline
\multicolumn{1}{|c|}{Discount Rate Threshold} & \multicolumn{1}{c|}{100}     \\ \hline
\multicolumn{1}{|c|}{Max Episode Time Step}   & \multicolumn{1}{c|}{300}     \\ \hline
\multicolumn{1}{|c|}{Max Total Time Steps (T)}    & \multicolumn{1}{c|}{100000}  \\ \hline
\multicolumn{1}{|c|}{Step Size}               & \multicolumn{1}{c|}{0.5}     \\ \hline
\multicolumn{1}{|c|}{Alpha}                   & \multicolumn{1}{c|}{0.2}     \\ \hline
\multicolumn{1}{|c|}{Learning Rate}           & \multicolumn{1}{c|}{0.0003}  \\ \hline
\multicolumn{1}{l}{}                          & \multicolumn{1}{l}{}
\end{tabular}
\end{table}

\subsection{Folding and Sidechain Packing of Dipeptides}

We first trained our agent on optimizing the 4 backbone dihedral angles of dialanine to achieve the lowest energy. In each epoch we resampled the backbone dihedrals using Dunbrack Lab's Ramanchandran distributions \cite{Dunbrack2010}. The lowest energy conformation that was found had a Rosetta energy of 1.28 using the default scoring function. Although the agent was able to achieve high cumulative reward, there is a high variance in the final energy. This is due to the Brownian motion term in the final optimal Langevin dynamics. When the energy was high (i.e. > 12), the agent generally found a conformation with lower energy. However, for low initial starting energies, it proved difficult for the agent to find the optimal configuration as it often prefers exploration more than exploitation. This is again due to the constant Brownian motion terms in the agent's dynamics (equation \ref{Langevin_dynamics}). \\

\begin{figure}[!htb]
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{figures/finaluntitled00229.jpg}
\endminipage\hfill
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{figures/finaluntitled00249.jpg}
\endminipage\hfill
\minipage{0.32\textwidth}%
  \includegraphics[width=\linewidth]{figures/finaluntitled00299.jpg}
\endminipage
\caption{Snapshots from One Episode of Dialanine Folding}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=.80\linewidth]{figures/AverageTestChangeEnergy.jpg}
    \caption{Average Change in Energy from Initial to Final Conformation at Test Time}
    \label{fig:avg_change}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=.80\linewidth]{figures/AverageTestCumulativeReward.jpg}
    \caption{The Average Cumulative Reward at Evaluation Time}
    \label{fig:avg_rew}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=.80\linewidth]{figures/init_change.png}
    \caption{The Change in Energy at Training Time}
    \label{fig:train_change}
\end{figure}
\section{Protein Side Chain Packing and Folding}

\subsection{Problem Statement}
The primary learning problem is to predict the final folded structure from the primary structure (and possibly using protein atomistic data from databases such as PDB).
By stacking all 3D coordinate $x_i$ of $i$th atom in the protein for $i \in \overline{1, N}$, we obtain the state coordinate $x$. Denote the rotamer angles of the $i^{th}$ side-chain by $d^r_i$ and the dihedral angles along the protein backbone $N-C_{\alpha}, C_{\alpha}-C'$ and $C'-N$, as $d^b_i$ . These dihedral angles are often denoted by $(\phi_j, \psi_j, \omega_j)$. We assume that the bond lengths and the bond angles are rigid so that only the $d^r_i$ and $d^b_i$ angles collectively determine the final geometric shape of the folded protein.  For the side-chain packing problem, the $d^b_i$ angles are fixed and only the $d^r_i$ angles are movable.

As in \cite{Ingraham:2019}, we  define the protein internal coordinates as $z_i$. Assume that the bond angle, bond length are $a_i, b_i$ respectively. We denote the $d_i$ vector to capture all the movable configuration angles, namely $d_i = d^r_i$ for the side chain packing problem, and {$d_i$} = ${d^b_i} \cup {d^r_i}$   This yields the equation
\begin{align}
x_i &= x_{i-1}  \nonumber\\
&+ b_i \begin{bmatrix}u_{i-1} & n_{i-1} \times u_{i-1}  & n_{i-1}\\ \end{bmatrix} \begin{bmatrix} \cos(\pi-a_i) \\ \sin(\pi-a_i)\cos(d_i) \\ \sin(\pi-a_i)\sin d_i \end{bmatrix}
\end{align}
The side chain packing (and folding) problem is (are) to find the optimal set of side-chain rotamer angles (and additionally the backbone dihedral angles)  corresponding to the optimal configuration $c = (c_i)_{i=1}^N$ that minimize the potential energy:
\begin{equation}
c^* = \argmin_c U(c)
\end{equation}

The potential energy include bonded energy, which is caused by interactions between atoms that are chemically bonded to one another, and non-bonded energy for atoms not chemically bonded:
\begin{align}
U &= U_{total} = U_{bonded} + U_{non-bonded} \nonumber\\
&=\bigg(\sum_{bonds} K_r(r-r_0)^2 + \sum_{angles} K_{\theta}(\theta-\theta_0) \nonumber\\
&+ \sum_{rotamers} f(rotamer) \bigg) \nonumber\\
&+ \sum_{dihedrals} f(dihedral) \bigg) \nonumber\\
&+ \left(\sum_{i < j} \frac{q_iq_j}{\epsilon r_{ij}} + \sum_{i < j} \left(\frac{A_{ij}}{r_{ij}^{12}} - \frac{B_{ij}}{r_{ij}^6}\right) \right)
\end{align}

Here the first three sums belong to bounded energy while the fourth term is due to charged (electrostatic) interactions and modeled by Coulomb's law and the fifth term is due to van der Waals interactions and modeled by Lennard-Jones potential.

Instead of only finding the optimal $c^*$, we wish to find the optimal trajectory from any initial configuration $c^0$ to the final optimal configuration $c^*$.

\subsection{Soft actor-critic}
The reinforcement learning objective is the expected sum of rewards $\sum_t\E_{(s_t, a_t) \sim \rho_{\phi}}[r(s_t, a_t)]$. The goal is to learn the optimal stochastic policy $\pi(a_t|s_t)$ that maximizes the objective. For soft-actor critic, the objective is augmented with an entropy term $\mathcal{H}$:
\begin{equation}
\pi^* = \argmax_{\pi} \sum_t \E_{(s_t, a_t) \sim \rho_{\pi}}[r(s_t, a_t) + \alpha \mathcal{H}(\pi(.|s_t))]
\end{equation}
The soft Q-function parameters can be trained by minimizing the soft Bellman residual:
\begin{align}
J_Q(\phi) &= \E_{(s_t, a_t) \sim \mathcal{D}}\bigg[\frac{1}{2}\bigg(Q_{\phi}(s_t, a_t) - (r(s_t, a_t) \nonumber \\
&+ \E_{s_{t+1} \sim \rho}[V_{\bar{\phi}}(s_{t+1})])\bigg)^2 \bigg]
\end{align}
This can be optimized with stochastic gradient:
\begin{align}
\hat{\nabla}_{\phi}J_{Q}(\phi) = \nabla_{\phi}Q_{\phi}(a_t, s_t)\Bigg(Q_{\phi}(s_t, a_t)-(r(s_t, a_t) \nonumber \\
+ (Q_{\bar{\theta}}(s_{t+1}, a_{t+1}) - \alpha \log(\pi_{\theta}(a_{t+1}|s_{t+1})))\Bigg)
\end{align}
We reparameterize the policy by a neural network:
\begin{equation}
a_t = f_{\theta}(\epsilon_t; s_t)
\end{equation}
where $\epsilon_t$ is input noise vector, sampled from a fixed distribution.

We also optimize the policy's parameter by minimizing:
\begin{align}
J_{\pi}(\theta) = \E_{s_t \sim \mathcal{D}, \epsilon_t \sim \mathcal{N}}\Bigg[\alpha \log \pi_{\theta}(f_{\theta}(\epsilon_t; s_t)|s_t) \nonumber \\
- Q_{\phi}(s_t, f_{\theta}(\epsilon_t; s_t))\Bigg]
\end{align}
We can approximate the gradient of $J_{\pi}$ by
\begin{align}
\hat{\nabla}_{\theta} J_{\pi}(\theta) = \nabla_{\theta}\alpha \log(\pi_{\theta}(a_t|s_t)) + (\nabla_{a_t} \alpha\log(\pi_{\theta}(a_t|s_t)) \nonumber\\
- \nabla_{a_t}Q_{\phi}(s_t, a_t))\nabla_{\theta}f_{\theta}(\epsilon_t, s_t)
\end{align}

\subsection{Soft actor-critic method with mutual-information}
Similar to soft actor-critic, Grau-Moya et al. \cite{grau-moya2018soft} want to maximize the cumulative reward with an additional entropy regularization term. However, in this work, Grau-Moya et al. adds a prior $\rho$ on the set of actions and jointly optimizes policy $\pi$ and the prior $\rho$ to optimize the following objective function:
\begin{equation}
\max_{\pi, \rho}\E_\pi\left[\sum_{t=0}^{\infty} \gamma^t\left(r(s_t, a_t)-\frac{1}{\beta} \log \frac{\pi(a_t|s_t)}{\rho(a_t)} \right)\right]
\end{equation}

\begin{algorithm}
\caption{Soft Actor-Critic for Stochastic Pontryagin}\label{SAC}
    \begin{algorithmic}[1]
    \State  \textbf{Input}: initial policy parameter $\theta$, Q-function parameter $\phi_1, \phi_2$, empty replay buffer $\mathcal{D}$, velocity (adjoint) step size $\epsilon$, maximum number of time steps $T$, hyperparameter $\gamma$, and number of iterations $I_{max}$
    \State Set target parameters to $Q$-function parameters $\bar{\phi}_i \gets \phi_i$ for $i \in \set{1, 2}$
    \Repeat
        \State Observe state $s_t=(d_j)_{j=1}^M$ and configuration $c_t=(X_t,A)$, where $X_t$ is the new feature node vector that corresponds to $(d_j)_{j=1}^M$
        \State Select action $a_t = (\dot{d}_j)_{j=1}^M$ so that $a_t \sim \pi_{\theta}(.|s_t)$
        \State Execute $a_t$ to obtain new $s_{t+1} = (d_j^{(t+1)})_{j=1}^M$: $d_j^{(t+1)} \gets d_j^{(t)} + \epsilon \dot{d}_j$
        \State Calculate coordinates $(x^{(t)}_i)_{i=1}^N$ and update new feature vector $X_{t+1}$
        \State Calculate potential  energy $U_t$ using the octree method for non-bonded neighbor interactions
        \State Set $r(s_t, a_t) \gets e^{\gamma t/T}[(\sum_{j=1}^M \dot{d}_j^2)/2-E_t]$
        \State Update replay buffer $\mathcal{D} \gets \mathcal{D} \cup \set{(s_t, a_t, r(s_t, a_t), s_{t+1}}$
        \State If $s_{t+1}$ is terminal, reset the $(x_i)_{i=1}^N$ by using the rotamer library (and sampled Ramachandran densities).
        \If {update step}
            \State Randomly sample batch $\mathcal{B}$ from $\mathcal{D}$.
            \State Compute targets for $Q$-functions and update $Q$-function using stochastic gradient descent on $\mathcal{B}$.
            \State Update policy using stochastic gradient descent on $\mathcal{B}$
            \State Update target network $\bar{\phi_i} \gets \tau \phi_i + (1-\tau)\bar{\phi}_i$ for $i \in \set{1, 2}$
        \EndIf
    \Until{the number of iterations is more than $I_{max}$}

    \end{algorithmic}
\end{algorithm}

\subsection{Continuous-time reinforcement learning}
The soft actor-critic method is formulated in discrete-time setting. However, we will now introduce continuous version of reinforcement learning and from here give the motivation for our choices of reward function as well as the general Markov decision process formulation for the protein folding problem.

\subsubsection{Deterministic setting}
\begin{lem}\label{deterministic_control}
Assume that we're given the optimal control problem of finding optimal control $u$ that minimizes:
\begin{equation}
V(s, t) = \min_a \int_0^{T} l(s_t, a_t)ds
\end{equation}
subject to
\begin{align}
ds_t &= a_tdt \nonumber\\
s(0) &= s \nonumber
\end{align}
Then with the choice of $l(s, a) = e^{\gamma t}(U(s) - a^2/2)$, the optimal trajectory $s_t$ has the form:
\begin{align}
ds_t &= a_tdt \\
da_t &= -\gamma a_t dt -\nabla U(s_t) dt
\end{align}
\end{lem}
Note that The cumulative loss $\sum_{t=0}^T l(s_t, a_t)$ can be consider as the discretization of the integral $\int_0^T l(s_t, a_t)dt$. The reward can simply be chosen as: $r(s_t, a_t) = -l(s_t, a_t)$.

\subsubsection{Stochastic Pontryagin maximum principle}
Suppose the stochastic control problem is to minimize the cost functional
\begin{equation}\label{control_prob}
J(a(.)) = \E \left[\int_0^T l(s_t, a_t)dt\right] + \E[h(T)]
\end{equation}
subject to
\begin{align}
ds_t &= g(s_t, a_t)dt + \sigma(s_t, a_t)dW_t\\
s(0) &= s_0
\end{align}
where $s_t$ is the state variable at time $t$ with values in $\mathbb{R}^n$, $a(.)$ is the control variable with values in a subset of $\mathbb{R}^n$, and $W_t$ is the Brownian motion in $\mathbb{R}^n$. Note that this stochastic control problem is the continuous version of the usual reinforcement learning problem formulation.

\begin{thm} (See \cite{Peng1990-gc})\label{stochastic_pontryagin}
Let the first-order adjoint process $(p(.), K(.))$ be the unique process that satisfies the stochastic differential equation:
\begin{align}
-dp_t &= \bigg(g_s(s_t, a_t)^T p_t + \sum_{j=1}^d (\sigma_s^{j}(s_t, a_t))^T K_j(t) \nonumber\\
&+ l_s(s_t, a_t)\bigg)dt - K_tdW_t\\
p_t &= h_s(s(T))
\end{align}
We also consider the Hamiltonian:
\begin{equation}\label{Hamiltonian}
H(s, a, p, K) = l(s, a) + (p, g(s, a)) + \sum_{j=1}^d (K_j, \sigma^j(s, a))
\end{equation}
Given that all function $g$ and $\sigma$ are continuously differentiable with respect to $a$, and the control domain is convex, then for the optimal solution $(s^*(.), a^*(.))$ of the control problem \ref{control_prob}, we have:
\begin{equation}
(H_a(s^*(t), a^*(t), p^*(t), K^*(t)), a-a^*(t)) \geq 0\ \forall s \in U
\end{equation}
almost everywhere, almost surely
\end{thm}

\subsubsection{Stochastic setting}
\begin{corollary}\label{stochastic_control}
By applying theorem \ref{stochastic_pontryagin} to the special case $\sigma \equiv 0$, $g(s, a) = a$, and the running loss $l(s, a) = \Gamma(t)(U(s) - a^2/2)$, and the reward $r(s, a) = -l(s, a)$ for the potential energy $U$, we obtain the following optimal trajectory $x$:
\begin{align}
ds_t &= a_tdt\\
da_t &= -\frac{\Gamma'(t)}{\Gamma(t)} a_tdt - \nabla U(s_t) dt - K dW_t
\end{align}
for some constant $K$. In particular, for $\Gamma(t) = e^{\gamma t}$ for some constant $\gamma$, we obtain the optimal dynamics:
\begin{align}
ds_t &= a_tdt\\
da_t &= -\gamma a_tdt - \nabla U(s_t) dt - K dW_t \label{Langevin_dynamics}
\end{align}
\end{corollary}

This optimal trajectory follows the Langevin dynamics \cite{Ingraham:2019}, a stochastic version of the dynamics considered in the previous section. This result motivates us to choose the reward function $r(s, a) = -l(s, a) = e^{\gamma t}(a^2/2-U(s))$ with backward discounted rate for the side chain prediction problem.

Note that the constant $K$ here is defined as the diffusion part of the process $-p_t$:
\begin{align}
-dp_t &= l_s(s_t, a_t)dt - K_tdW_t\\
p_t &= h_s(y(T))
\end{align}

\subsubsection{Connection to Nesterov's accelerated gradient method}
Su et al. \cite{Su2015-pe} show that the continuous version of Nesterov's accelerated gradient method for minimizing a function $U$ is the following ODE:
\begin{equation}
\ddot{s} + \frac{3}{t}\dot{s} + \nabla U(s) = 0
\end{equation}

Note that for the choice of $\Gamma(t) = t^3$, $\Gamma'(t)/\Gamma(t)$ is exactly $3/t$. Hence, by corollary \ref{stochastic_control}, for the Lagrangian (or regret) function $r(s, a) = t^3(a^2/2-U(s))$, we can recover the continuous Nesterov's accelerated gradient descent dynamics with Brownian motion noise added. Thus, we can improve the algorithm \ref{SAC} by setting $r(s_t, a_t)$ to $(t/T)^3[(\sum_{j=1}^M\dot{d}_j^2)/2 - E_t]$.
\section{Introduction}
\label{sec:intro}
A protein is composed of amino acids linked together by peptide bonds.
Each amino acid consists of backbone atoms and side chains; the backbone
is a sequence of carbon, nitrogen or oxygen atoms, whereas each side-chain
(also called residue) can be one of a number of functional groups
(see for e.g. left Figure \ref{fi:int-coords}).  The
side-chains of an amino acid determine its functionality and chemical
properties (see for e.g. right Figure \ref{fi:int-coords}).

Side chains play an important role in the flexibility of a protein.
Being more flexible than the backbone, they assume different relative
orientations to produce varying conformations of the protein. The
probability that the protein is in any one of these conformations
is not only dictated by the energy of the conformation but also
depends on the presence of ligands, if any.  When two molecules dock,
for example, the proteins at the molecular interface undergo a
reorientation of side chains to yield a  configuration that minimizes
the potential energy.  Side chain configurations are also influenced by
the surrounding solvent, as solvation energies differ; furthermore, there
may be multiple side chain configurations that are `correct' in the sense
of yielding a low potential energy.  The problem of finding preferential
side chain configurations is called the {\em side chain packing}
or {\em side chain positioning} (SCP) problem.

Clustering the various known conformations for a given side chain (or residue) shows
that only a few energetically preferred states called {\em rotamers} are
predominantly present in nature.  Ponder and Richards showed that
the variation in the cluster was much lower than previously
thought \cite{PR87}.  Backbone independent \cite{D02,TEHL91}
and backbone dependent rotamer libraries \cite{D02,DC97,DK93,DK94}
have been identified and used to model partially flexible macromolecules
in simulations \cite{CFK97}.

The SCP problem is non-convex and is also
NP-hard \cite{A97,PW02}.  It cannot even be approximated in polynomial
time \cite{CKS04}.  Many SCP algorithms have been developed and implemented
in the past two decades.  Most of them
\cite{CKS04,D02,DC97,DK93,DK94,LWRR00,MDL97,MIS87,PR87,TEHL91,X05,XB06}
use rotamer libraries. Given a fixed backbone, the sequence of a protein,
and a rotamer library, SCP can be formulated as a combinatorial search
problem that optimizes an energy function. The energy function typically
includes a term for the backbone, one for each side chain in the protein
(describing the preference of a given rotamer over others for a given side
chain, e.g., log probabilities of backbone-dependent rotamer populations
\cite{D99}), and one term for every pair of chosen rotamers.  More complex
energy terms such as molecular mechanics potential energy combined with
complex solvation free energy terms \cite{LG02,MBCS01,XH01} are sometimes used.
The hardness of the SCP problem results from the difficulty in handling
the pairwise energy terms. Since the SCP problem is NP-hard many heuristic
and some approximation algorithms have been developed for solving it.
The Dead End Elimination (DEE) theorem \cite{DMHL92} is often used to prune
the number of rotamers to be considered for finding a global energy minima.
Many algorithms based on this theorem can find the global minima provided
they are able to converge \cite{DMHL92,DML97,G94,GM99,KSM+95,LD93,LMD95,LH01,MDL97,PSDM00,VGM00},
while some algorithms based on other techniques such as Monte-Carlo
simulation \cite{HS91}, cyclical search \cite{DK93,XH01}, spatial restraint
satisfaction \cite{SB93}, and approximation algorithms based on semi-definite programming
\cite{CKS04,KCS05,XB06} have reasonable running times but do not guarantee a global optima.
Other used methods include A* search \cite{LL98}, simulated annealing \cite{LS91},
mean-field optimization \cite{L94}, maximum edge-weighted cliques \cite{BTSA05},
and integer linear programming \cite{AKLM00,EZE01}.

Two of the most efficient software package for solving the SCP problem are
SCWRL3.0 ({\em{Side Chains With a Rotamer Library}}) \cite{BCD97,CSD03,D99} and
TreePack ({\em{Tree-Decomposition Based Side-Chains Packing}}) \cite{X05,XB06},
and both of them are based on graph theory, however both are non-polynomial time solutions in the worst case. SCWRL3.0 represents the
interaction among various side chains as an undirected graph called
the residue interaction graph and then decomposes the graph into small
biconnected components. At most one articulation side chain exists between
any two biconnected components, and if one can fix the rotamer for this
side chain then rotamers can be assigned to the two components independently
of each other. Thus the original problem is decomposed into a set of smaller
subproblems each of which is then solved independently.
TreePack uses a novel tree decomposition technique for decomposing
the residue interaction graph into even smaller components than SCWRL3.0.
The algorithm returns a globally optimal solution in $\Oh{(N + M) (n_{rot})^{{tw} + 1} }$
time where $N$ is the number of side chains, $n_{rot}$ is the average number of
rotamers per side chain, $M$ is the number of interacting side chain pairs, and
tw $(= \Oh{ N^{2 \over 3}\log{N} }$ is the tree width of the residue interaction graph.

 \bibliographystyle{acm}

\section{Proofs of Lemma and Corollary}
\subsection{Proof of lemma \ref{deterministic_control}}
\begin{proof}
By Pontryagin's maximum principle, for the Hamiltonian $H(s, a, p, t) = p_t^T f(s_t, a_t) + l(s_t, a_t)$, we obtain the Hamiltonian system for the optimal trajectory $(s, a) = (s^*, a^*)$ (for simplicity we ignore the star notation):
\begin{align}
ds &= H_p(s(t), a(t), p(t), t) dt\\
dp &= -H_s(s(t), a(t), p(t), t) dt
\end{align}
Moreover, $H_a(s(t), v, p(t), t)$ is minimized at $v = a(t)$ so that $H_a(s(t), a(t), p(t), t) = 0$. Then for the dynamic function $f(s, a) = a$, and for the Lagrangian $l(x, u) = e^{\gamma t}(U(s) - a^2/2)$, we get:
\begin{equation}
p(t) + l_a(s_t, a_t) = 0
\end{equation}
Therefore, $p(t) = -l_a(s_t, a_t) = e^{\gamma t}a_t$. Plugging back into the dynamics of $p$, we get:
\begin{equation}
e^{\gamma t}(da + \gamma adt) = -l_s(s_t, a_t)dt = -e^{\gamma t}\nabla U(s_t)dt
\end{equation}
This yields the desired dynamics for \ref{deterministic_control}
\end{proof}

\subsection{Proof of corollary \ref{stochastic_control}}
\begin{proof}
By a similar proof to that of \ref{deterministic_control}, we obtain the optimal trajectory $s_t$ with the following dynamics:
\begin{align}
ds_t &= a_t dt\\
da_t &= -\gamma a_tdt - \nabla U(s_t)dt - e^{-\gamma t} K_t dW_t
\end{align}
Because of the uniqueness of the solution to the adjoint stochastic differential equations \ref{stochastic_control}, $e^{-\gamma t} K_t$ will be unique and must be the same as the constant in the Langevin dynamics because of the existence of the solution to the Langevin dynamics. Thus, this gives $e^{-\gamma t} K_t = K$ as desired.
\end{proof}

\subsection{Further Enhancement}
The Brownian motion term in equation \ref{Langevin_dynamics} can be adjusted to better balance exploration and exploitation. To make this adjustment, we derived another formula based on our more general framework with stochastic Pontryagin maximum principle:
\begin{align}
ds_t &= a_tdt + \exp(-U(s_t))dW_t\\
da_t &= -\gamma a_tdt - \nabla U(s_t)dt
\end{align}
We still choose the same Lagrangian (or regret) function $l(s, a) = e^{\gamma t}(U(s) - a^2/2)$. For this formulation, the function $K$ in the Hamiltonian in equation \ref{Hamiltonian} will become $0$. The function $u$ is now deterministic rather than a stochastic process as before. The proof is the same as that of the corollary \ref{stochastic_control}. With this formulation, when the energy is large, the molecular agent dynamical process focuses more on exploitation and helps reduce the variance of the trajectory.

\section{Trialanine Experiments}

In addition to dialanine, we also tested our agent on the finding the minimal protein conformation for trialanine (i.e. three consecutively bonded alanines). Although the agent converged to a low energy conformation for trialanine, it did not converge to the global energy minimum. Moreover, thete

\begin{figure}[!htb]
    \minipage{0.48\textwidth}
      \includegraphics[width=\linewidth]{figures/tralanine_training.png}
    \endminipage\hfill
    \minipage{0.48\textwidth}
      \includegraphics[width=\linewidth]{figures/trialanine_eval_energy.png}
    \endminipage
\caption{Initial vs final Energy plotted for each evaluation episode of trialanine}
\end{figure}

\section*{Checklist}

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \answerYes{}
  \item Did you describe the limitations of your work?
   \answerYes{See section \ref{sec:conclusion}}
  \item Did you discuss any potential negative societal impacts of your work?
   \answerNo{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerYes{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerYes{}
	\item Did you include complete proofs of all theoretical results?
    \answerYes{}
\end{enumerate}

\item If you ran experiments...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerYes{The code and data will be stated in supplementary materials}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerYes{See section \ref{sec:experiment}}
	\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerNo{}
	\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerYes{See section \ref{sec:experiment}}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerYes{See section \ref{sec:experiment} for software and prior code-base citation.}
  \item Did you mention the license of the assets?
    \answerYes{The license of the assets lies in cited paper.}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerNo{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerNo{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerNA{The data we are using are public data.}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerNA{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerNA{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerNA{}
\end{enumerate}

\end{enumerate}

\section{Introduction}
\label{sec:intro}
A protein is composed of amino acids linked together by peptide bonds.
Each amino acid consists of backbone atoms and side chains; the backbone
is a sequence of carbon, nitrogen or oxygen atoms, whereas each side-chain
(also called residue) can be one of a number of functional groups
(see for e.g. left Figure \ref{fi:int-coords}).  The
side-chains of an amino acid determine its functionality and chemical
properties (see for e.g. right Figure \ref{fi:int-coords}).

Side chains play an important role in the flexibility of a protein.
Being more flexible than the backbone, they assume different relative
orientations to produce varying conformations of the protein. The
probability that the protein is in any one of these conformations
is not only dictated by the energy of the conformation but also
depends on the presence of ligands, if any.  When two molecules dock,
for example, the proteins at the molecular interface undergo a
reorientation of side chains to yield a  configuration that minimizes
the potential energy.  Side chain configurations are also influenced by
the surrounding solvent, as solvation energies differ; furthermore, there
may be multiple side chain configurations that are `correct' in the sense
of yielding a low potential energy.  The problem of finding preferential
side chain configurations is called the {\em side chain packing}
or {\em side chain positioning} (SCP) problem.

Clustering the various known conformations for a given side chain (or residue) shows
that only a few energetically preferred states called {\em rotamers} are
predominantly present in nature.  Ponder and Richards showed that
the variation in the cluster was much lower than previously
thought \cite{PR87}.  Backbone independent \cite{D02,TEHL91}
and backbone dependent rotamer libraries \cite{D02,DC97,DK93,DK94}
have been identified and used to model partially flexible macromolecules
in simulations \cite{CFK97}.

\begin{figure}[h]
\centerline{ { \epsfxsize=4.5in \epsfysize=2.5in \epsfclipon \epsffile[100 310 690 660]{../figures1/internal.eps}} }
\centerline{ { \epsfxsize=2.5in \epsfysize=3.2in \epsfclipon \epsffile[220 240 570 730]{../figures1/lysine.eps}} }
\label{fi:int-coords}
\caption{(Left) A polypeptide chain with backbone and side-chains. Variation in dihedral angles result in various conformations of the protein.
(Right) A Lysine side-chain with side-chain dihedrals ($\chi_{1,1}$, $\chi_{1,2}$, $\chi_{1,3}$, $\chi_{1,4}$).
The side-chain assumes various conformations based on the values of the dihedral angles.
}
\end{figure}

The SCP problem is non-convex and is also
NP-hard \cite{A97,PW02}.  It cannot even be approximated in polynomial
time \cite{CKS04}.  Many SCP algorithms have been developed and implemented
in the past two decades.  Most of them
\cite{CKS04,D02,DC97,DK93,DK94,LWRR00,MDL97,MIS87,PR87,TEHL91,X05,XB06}
use rotamer libraries. Given a fixed backbone, the sequence of a protein,
and a rotamer library, SCP can be formulated as a combinatorial search
problem that optimizes an energy function. The energy function typically
includes a term for the backbone, one for each side chain in the protein
(describing the preference of a given rotamer over others for a given side
chain, e.g., log probabilities of backbone-dependent rotamer populations
\cite{D99}), and one term for every pair of chosen rotamers.  More complex
energy terms such as molecular mechanics potential energy combined with
complex solvation free energy terms \cite{LG02,MBCS01,XH01} are sometimes used.
The hardness of the SCP problem results from the difficulty in handling
the pairwise energy terms. Since the SCP problem is NP-hard many heuristic
and some approximation algorithms have been developed for solving it.
The Dead End Elimination (DEE) theorem \cite{DMHL92} is often used to prune
the number of rotamers to be considered for finding a global energy minima.
Many algorithms based on this theorem can find the global minima provided
they are able to converge \cite{DMHL92,DML97,G94,GM99,KSM+95,LD93,LMD95,LH01,MDL97,PSDM00,VGM00},
while some algorithms based on other techniques such as Monte-Carlo
simulation \cite{HS91}, cyclical search \cite{DK93,XH01}, spatial restraint
satisfaction \cite{SB93}, and approximation algorithms based on semi-definite programming
\cite{CKS04,KCS05,XB06} have reasonable running times but do not guarantee a global optima.
Other used methods include A* search \cite{LL98}, simulated annealing \cite{LS91},
mean-field optimization \cite{L94}, maximum edge-weighted cliques \cite{BTSA05},
and integer linear programming \cite{AKLM00,EZE01}.

Two of the most efficient software package for solving the SCP problem are
SCWRL3.0 ({\em{Side Chains With a Rotamer Library}}) \cite{BCD97,CSD03,D99} and
TreePack ({\em{Tree-Decomposition Based Side-Chains Packing}}) \cite{X05,XB06},
and both of them are based on graph theory, however both are non-polynomial time solutions in the worst case. SCWRL3.0 represents the
interaction among various side chains as an undirected graph called
the residue interaction graph and then decomposes the graph into small
biconnected components. At most one articulation side chain exists between
any two biconnected components, and if one can fix the rotamer for this
side chain then rotamers can be assigned to the two components independently
of each other. Thus the original problem is decomposed into a set of smaller
subproblems each of which is then solved independently.
TreePack uses a novel tree decomposition technique for decomposing
the residue interaction graph into even smaller components than SCWRL3.0.
The algorithm returns a globally optimal solution in $\Oh{(N + M) (n_{rot})^{{tw} + 1} }$
time where $N$ is the number of side chains, $n_{rot}$ is the average number of
rotamers per side chain, $M$ is the number of interacting side chain pairs, and
tw $(= \Oh{ N^{2 \over 3}\log{N} }$ is the tree width of the residue interaction graph.

In this paper, we explore a new approach to solving the SCP problem using a graph-theoretic
analogy which we show here, reduces the problem to solving a min-cost max-flow problem
with unit edge capacities.  We have used this technique to solve the SCP problem in
polynomial time for a simplified pairwise energy function which demonstrates the
potential of using this new approach for solving more difficult variants.

\section{Problem Formulation}
\label{sec:prob}

We consider the following formulation of the SCP problem
as used in \cite{DMHL92,CKS04}. We are given a fixed protein
backbone, a set of possible candidate rotamers $\{r_{i,p}\}$ for
each residue position $i$ on the backbone, and the following energy
function which we want to minimize:

$$\xi = E_{0} + \sum_{i}{E\left( r_{i, p} \right)} + \sum_{i < j}{E\left( r_{i, p}, r_{j, q} \right)},$$

\noindent
where $E_{0}$ is the self-energy of the backbone,
$E\left( r_{i, p} \right)$ is the energy due to the
interaction between the backbone and the rotamer
$r_{i, p}$ chosen for position $i$ as well as the
intrinsic energy of $r_{i, p}$, and $E\left( r_{i, p}, r_{j, q} \right)$
is the interaction energy between rotamers $r_{i, p}$
and $r_{j, q}$ chosen for positions $i$ and $j$,
respectively.

\para{Simplified SCP}
One can simplify the SCP problem by using the
following simplified definition of $E\left( r_{i, p}, r_{j, q} \right)$
instead of the general interaction energy function as in \cite{DMHL92,CKS04}.

\begin{equation}\label{eq:pairE}
  E\left( r_{i, p}, r_{j, q} \right) = \left\{ \begin{array}{ll}
                   +\infty & \textrm{if $r_{i, p}$ and $r_{j, q}$ have steric collision,}\\
                   0 & \textrm{otherwise.}
                   \end{array} \right.
\end{equation}

\noindent
Therefore, a collision-free configuration of the side-chains
will result in a finite value of $\xi$, and
minimizing $\xi$ will produce a collision-free
configuration, if one exists, with the minimum
$E_{0} + \sum_{i}{E\left( r_{i, p} \right)}$ value.

We also make the following two assumptions about $E( r_{i,p}, r_{j,q} )$:

\begin{itemize}

\item[] {\bf 1) Reflexivity:} $E( r_{i,p}, r_{j,q} ) = E( r_{j,q}, r_{i,p} ), ~i \neq j$, and

\item[] {\bf 2) Transitivity:} $(E( r_{i,p}, r_{j,q} ) = +\infty )
    ~\wedge~ (E( r_{j,q}, r_{k,s} ) = +\infty ) \Rightarrow (E( r_{i,p}, r_{k,s} ) = +\infty ), ~i \neq j \neq k$

\end{itemize}

The transitivity assumption is a simplifying assumption on the collision
property of the rotamers which says that for any three distinct
side chains $i$, $j$, and $k$, if rotamer $r_{i,p}$ collides with
rotamer $r_{j,q}$ and $r_{j,q}$ collides with $r_{k,s}$ then $r_{i,p}$
also collides with $r_{k,s}$.

\section{Our Approach}
\label{sec:approach}

We will reformulate SCP as a problem of finding
a set of weighted vertex disjoint paths through a directed graph,
and use a min-cost max-flow algorithm to solve it.
We will first show how to obtain a polynomial time algorithm
for the simplified SCP problem introduced in Section \ref{sec:prob}
(equation \ref{eq:pairE}), and then explain how to cast the
most general version of the problem as a min-cost max-flow
problem.

\subsection{A Polynomial Time Solution to Simplified SCP}~\\
\label{ssec:poly-scp}

\vspace{-0.3cm}
\noindent
Suppose each side-chain $i \in [1, N]$ is associated with a set of
rotamers $R_{i} = \{ r_{i,p} \}$. Suppose the average number
of rotamers per side-chain is $n_{rot}$. Thus there are a total
of $n_{rot} \times N$ rotamers. We are also given a set $C$ of
rotamer sets, and $S \in C$ implies that the rotamers in $S$
collide sterically. We call $C$ the collision set.
Let $|C| = M$.

 \begin{figure}[t!]
 \centerline{ { \epsfysize=3.5in \epsffile{../figures1/simplified-scp.eps} }}
 \caption{The figure shows the construction of a collision digraph for simplified SCP. All paths originate
 at the source vertex $s$ and terminate at the target vertex $t$. For each residue $i$, there is a
 vertex $v_{i}$ in the residue layer, and for each rotamer $r_{i, p}$ there is a vertex $v_{i, p}$ in the
 rotamer layer. For each rotamer set $S_{k}$ in the collision set $C$ there is a
 vertex $x_{k}$ in the collision layer. For each $r_{i, p} \in S_{k}$ there is
 an edge from $v_{i, p}$ to $x_{k}$. There is an edge of weight $E( r_{i, p} )$ from each $v_{i}$ to
 each $v_{i, p}$. All remaining edges have zero weight.}
 \label{fi:simplified-scp}
 \end{figure}

 \subsubsection{The Collision Graph $\mathscr{G}$.}
 \label{sssec:collision}
 %
We now construct a collision graph $\mathscr{G}$ (Figure \ref{fi:simplified-scp}) as follows.
The graph is directed, and has a source vertex $s$ from which all paths originate and a
target vertex $t$ at which all paths terminate. There are three layers in G:
{\em residue layer}, {\em rotamer layer} and {\em collision layer}. All edges in $\mathscr{G}$
have weight (i.e., cost) zero unless specified otherwise. We have a vertex $v_{i}$ for each
residue $i \in [1, N]$ in the residue layer, and an edge connects $s$ to $v_{i}$.
For each rotamer $r_{i,p} \in R_{i}$ we introduce a vertex $v_{i,p}$ in the rotamer layer,
and connect $v_{i}$ to $v_{i,p}$ by an edge of weight (i.e., cost) $E(r_{i,p})$.
The collision layer contains a vertex $x_k$, $k \in [1, M]$, for each rotamer set
$S_{k} \in C$. For each $r_{i,p} \in S_{k}$, there is an edge from $r_{i,p}$ to $x_{k}$.
All vertices in the rotamer layer not connected to vertices in the collision layer
as well as all vertices in the collision layer are connected to the target vertex $t$.
Any set $P$ of paths going from $s$ to $t$ through different residue and collision layer
vertices corresponds to a collision-free set $Q$ of rotamers with at most one rotamer
per side chain. A rotamer $r_{i,p}$ belongs to $Q$ provided a path in $P$ passes through
$v_{i,p}$. Our goal is to find a set $P$ of maximum cardinality ($\leq N$) with minimum
total edge weight.

\subsubsection{Transform $\mathscr{G}$ into a Weighted Flow Network $\mathscr{G}'$.}
 \label{sssec:flow-network}
 %
 We apply a standard transformation on all vertices of $\mathscr{G}$ except
 $s$ and $t$. Each vertex $v$ is replaced with two new vertices $v'$ and $v''$
 along with an edge of weight zero from $v'$ to $v''$. All incoming edges
 of $v$ are added as incoming edges of $v'$ and all of $v$'s outgoing edges are added
 as outgoing edges of $v''$. All edges of the new graph $\mathscr{G}'$
 have unit capacity.

 \subsubsection{Apply a Min-cost Max-flow Algorithm on $\mathscr{G}'$.}
 \label{sssec:min-cost-max-flow}
We now find the minimum cost maximum flow from $s$ to $t$ in the flow network $\mathscr{G}'$.
Since $s$ has exactly $N$ outgoing edges, the maximum possible flow value cannot be
larger than $N$. If there exists a collision-free set of $N$ rotamers, one for each residue,
then the maximum flow value will be $N$. A rotamer $r_{i,p}$  will be selected provided
there is a flow from $v''_{i}$ to $v'_{i, p}$ in $\mathscr{G}'$.  Because of the costs
assigned to the edges from the residue layer to the rotamer layer, the set of chosen
rotamers will have the minimum $E_{0} + \sum_{i}{E\left( r_{i, p} \right)}$ value among
all possible sets of collision-free rotamers.

We can find a minimum cost maximum flow in $\Oh{m^2 \textrm{polylog}{ m }}$ time, where $m$
is the number of edges in the flow network \cite{T85}. For $\mathscr{G}'$, $m = \Oh{n_{rot}N }$.
Therefore, the algorithm runs in $\Oh{ (n_{rot}N)^2 \textrm{polylog} (n_{rot}N) }$ time on $\mathscr{G}'$.
Since $\mathscr{G}'$ contains only unit capacity edges and all edge costs are nonnegative,
we can modify the Ford-Fulkerson max-flow algorithm \cite{KT05} so that each round results
in an augmenting path with the minimum total edge weight using Dijkstra's single-source shortest
path algorithm \cite{KT05}. Since the maximum flow value is at most $N$, the algorithm will
terminate in no more than $N$ rounds with each round taking $\Oh{m + n \log{ n }}$ time,
where $n$ is the number of vertices in the graph. Since $n = \Oh{n_{rot}N }$, the algorithm will
terminate in $\Oh{ N ( m + n \log{ n } ) } = \Oh{ n_{rot}N^2 \log( n_{rot}N ) }$ time.
This polynomial complexity is much better than the non-polynomial complexity of
TreePack \cite{X05,XB06} .

\subsection{Flow Network for the General SCP Problem}~\\
\label{ssec:approach}

\vspace{-0.3cm}
In Section \ref{ssec:poly-scp} we described a polynomial-time flow-based SCP algorithm
for a transitive pairwise energy function that handles only steric collisions.
In this Section we extend the energy function in two ways: $(i)$ make it non-transitive,
and $(ii)$ make it continuous (i.e. not a binary function describing collision). This general SCP problem also reduces to a  min cost max flow  of the generalized flow network, that we describe below.
A switch gadget from \cite{FHW80} plays a central role in this extension, which we
describe first.

 \begin{figure}[h!]
 \centerline{ { \epsfxsize=2.2in \epsfysize=2.7in \epsffile[30 330 380 760]{../figures1/switch-a.eps}}
              { \epsfxsize=2.2in \epsfysize=2.7in \epsffile[30 330 380 760]{../figures1/switch-b.eps}}
	      { \epsfxsize=2.2in \epsfysize=2.7in \epsffile[30 330 380 760]{../figures1/switch-c.eps}} }
 \caption{A switch gadget \cite{FHW80}. This subgraph is connected to the rest of the graph
 only through the labeled vertices as indicated by the arrows. Figures $(b)$ and $(c)$
 show the only two different ways three vertex disjoint paths pass through the switch
 covering all vertices of the subgraph.}
 \label{fi:switch}
 \end{figure}

\subsubsection{The Switch.}
\label{sssec:switch}
The switch \cite{FHW80} (see Figure \ref{fi:switch}) is a directed subgraph which is
connected to the rest of the graph through eight labeled vertices. Four directed edges enter
the subgraph through $s_{1}$, $s_{2}$, $p_{1}$ and $p_{2}$, and four exit
through $t_{1}$, $t_{2}$, $q_{1}$ and $q_{2}$. The following observation
captures a key property of the switch which we will exploit in our algorithm.

 \begin{observation}[\cite{FHW80}]
 \label{obs:switch}
 Suppose two vertex disjoint paths pass through the switch in Figure \ref{fi:switch}$(a)$
 -- one leaving at $t_{1}$ and the other entering at $s_{2}$. Then the path leaving
 at $t_{1}$ must have entered the switch at $s_{1}$, and the one entering at $s_{2}$
 must leave at $t_{2}$. The only other vertex disjoint path through the switch
 connects either $p_{1}$ to $q_{1}$ or $p_{2}$ to $q_{2}$ depending on the actual
 routing of the first two paths. Moreover, if all three vertex disjoint paths
 are present then all vertices of the switch are traversed.
 \end{observation}

 \noindent
 Figures \ref{fi:switch}$(b)$ and \ref{fi:switch}$(c)$ show the only two possible
 routing of the three vertex disjoint paths through the switch.

 We say that a switch is {\em active} provided the two vertex disjoint paths
 $s_{1}$ to $t_{1}$ and $s_{2}$ to $t_{2}$ pass through the switch. An active
 switch allows only at most one other vertex disjoint path through
 the switch: either $p_{1}$ to $q_{1}$ or $p_{2}$ to $q_{2}$.

 \begin{figure}[t!]
 \centerline{ { \epsfysize=3.5in \epsffile{../figures1/general-scp.eps} }}
 \caption{The figure shows the construction of a collision digraph for general SCP. All paths originate
 at the source vertex $s$ and terminate at the target vertex $t$. For each residue $i$, there is a
 vertex $v_{i}$ in the residue layer, and for each rotamer $r_{i, p}$ there is a vertex $v_{i, p}$ in the
 rotamer layer. For each collision pair $( r_{i, p}, r_{j, q} )$ a switch $x_{k}$
 is placed in the collision layer, and the two paths from $r_{i, p}$ and $r_{j, q}$ to
 target $t$ pass through the switch. Each $v_{i}$ to $v_{i, p}$ edge is given a weight value of
 $E( r_{i, p} )$, while all remaining edges have zero weight. Figure \ref{fi:connect-switch} shows how to connect the
 switches to the rest of the graph.}
 \label{fi:general-scp}
 \end{figure}

\begin{figure}[t!]
 \centerline{ { \epsfxsize=6.5in \epsfysize=4in \epsffile[30 310 760 760]{../figures1/cg-sw.eps}} }
 \caption{We show how to connect switch $x_{1}$ from Figure \ref{fi:general-scp}
 to the rest of the graph. This switch allows at most one of the two paths
 $v_{1, 1}$ -- $t$ and $v_{2, 1}$ -- $t$ pass through it. Path $v_{1, 1}$ -- $t$
 enters the switch at vertex $p_{1}$ and exits at $q_{1}$, while path $v_{2, 1}$ -- $t$
 enters at $p_{2}$ and exits at $q_{2}$. Two edges from the source vertex $s$ connect
 to $s_{1}$ and $s_{2}$, while two other edges connect $t_{1}$ and $t_{2}$ to the
 target vertex $t$.}
 \label{fi:connect-switch}
 \end{figure}

 \begin{figure}[t!]
 \centerline{ { \epsfysize=2in \epsffile{../figures1/compound-switch.eps}} }
 \caption{A compound switch.}
 \label{fi:compound-switch}
 \end{figure}

 \subsubsection{Handling Non-transitive Energy Functions.}
 \label{sssec:non-trans}
 %
 One way of extending the collision graph described in
 Section \ref{sssec:collision} to handle a non-transitive
 pairwise energy function is to use switching gadgets as
 collision layer vertices to handle pairwise collisions
 (see Figure \ref{fi:general-scp}).
 %
 The collision layer consists of $M$ switches $x_{k}$, $k \in [1, M]$,
 and each such switch handles the collision between a pair of rotamers.
 Two edges from the source $s$ connect to vertices $s_{1}$ and
 $s_{2}$ of each switch while two edges from $t_{1}$ and $t_{2}$
 connect to the target $t$ (see Figure \ref{fi:connect-switch}).
 Now consider a switch $x_{k}$ that handles the collision between
 rotamers $r_{i, p}$ and $r_{j, q}$. The path from $v_{i, p}$ to $t$
 (if any) enters $x_{k}$ through vertex $p_{1}$ and exits through
 $q_{1}$, while the $v_{j, q}$ to $t$ path (if any) enters $x_{k}$
 at $p_{2}$ and exits at $q_{2}$. Figure \ref{fi:connect-switch}
 shows  how to connect a switch to the rest of the graph.

 \subsubsection{Handling Continuous Pairwise Energy Functions.}
 \label{sssec:cont-energy}
 %
 To handle continuous pairwise energy functions we can replace each switching
 gadget with a compound switch (see Figure \ref{fi:compound-switch}) composed of
 a weighted and an unweighted graph switches. Only the two outgoing edges from
 the weighted switch have weights $E( r_{i,p}, r_{j,q} )$, and all other
 edges are unweighted. Two paths corresponding to two rotamers (say, $r_{i,p}$ and $r_{j,q}$ )
 enter the compound switch, which may allow 0, 1 or 2 of paths pass through it.
 Since we want the paths to be vertex disjoint and with minimum cost, if only one
 path passes through the compound switch it will use the unweighted. However,
 if both paths pass through the compound switch exactly one of them must take
 the path through the weighted switch and thus incur a cost of $E( r_{i,p}, r_{j,q} )$.

 \bibliographystyle{acm}

\section{Conclusions}
\label{sec:conclusion}
In this paper, we learn the dihedral and/or  rotamer angles of the optimal configuration of a protein structure by using a dynamic and continuous approach. The main continuous reinforcement learning formulation is derived from the stochastic Pontryagin's maximum principle. Our algorithm is based on soft actor-critic and is guaranteed to converge to an optimal solution. Our optimal RL solution for a certain setting of our reward  function and forward stochastic dynamics reduces  to a general Langevin dynamics, which is considered as the suitable optimal dynamics for protein folding and structure prediction problems. We further show a variation of our formulation and its relation with the Nesterov's accelerated gradient method. This suggests that one can use our general framework to derive further efficient reinforcement learning formulation for protein structure prediction and for learning dynamical systems in general.

redpajama_set_name: RedPajamaArXiv


text:
Best Axes For Splitting Wood: Definitive Guide 2023
A nice, cozy night with a good steady fire is all sweet and good until you run out of wood and you have to go buy some again. That's why more and more people are now turning back to the…
A nice, cozy night with a good steady fire is all sweet and good until you run out of wood and you have to go buy some again. That's why...
Power-up your bow and hit the bullseye with these Archery Targets! Or are you just looking for the most ideal archery targets for you? Are you just looking for a fun activity to do while camping? Have you ever tried target archery, then? Since there are so many different archery...
Worry no more - With the RV Backup Cameras, you'll...
These devices provide top-notch protection for your hearing so you...
Say goodbye to wilting plants and a big hello to...
Check in your chickens now with these amazing and 'chic'...
Bug Out Bill is here to give information about how to survive and strive with this craziness world that we live in! You don't want to wait until something that happen but have the knowledge now before it's to late.
Best Bushcraft Knife Under 100: Definitive Guide 2023
In camping, you're going to do a lot of chores. You'll need to find shelter, food and survive the unexpected situations that may arise.
At the same time…
You wouldn't want to be bogged down….
The Best Throwing Knives
These stainless steel knives will level up your knife-throwing talents.
Searching for the next throwing knife to add to your collection?
Or perhaps you're looking for one that will amp up your..
Best Self Defense Knife: Definitive Guide 2023
I'm sure most of us have this memory of being you, going out of the house in the wee hours of the night and carrying a butter knife in the pocket for protection….or maybe just a sense of security.
The Best Handheld Spotlight: Definitive Guide 2023
The sun is the best light source in our universe, but what will you do when it goes down, and darkness spreads on your campground or fishing lake?
Whether you're working at home in the dark…
Best Throwing Axe: Definitive Guide 2023
Whether you're a camper, a lumberjack's apprentice, or a competitive axe thrower – you're going to need a good throwing axe.
What makes an axe tool 'good'?
To measure that…..
The Best Ice Auger: Definitive Guide 2023
This is the Ultimate BugOutBill.com review of the best Ice Augers on the market for 2021.
While learning to ice fish is an acquired taste that not every angler is necessarily going to love the….
bugoutbill.com
50 Gene Jordan Rd
Columbia, NH 03576
info@bugoutbill.com

redpajama_set_name: RedPajamaCommonCrawl


text:
Agenparl is the press agency in Italy. It focuses on politics and it provides daily news coming from the Italian parliament. Articles published by the press agency are published in most Italian newspapers.

External links
 AgenParl

References

Mass media companies of Italy

redpajama_set_name: RedPajamaWikipedia


text:
COVID-19 UPDATES: Non-essential construction further restricted as Ontario declares second provincial emergency to address COVID-19 crisis - January 12, 2021
Strategies & Innovations
Regulation & Guidelines
Pricing & Finance
Ground Water Awareness Week
Features Surface Water Water Issues
B.C.'s Port Mann water supply tunnel now in service
By Ground Water Canada
Vancouver – After almost six years of construction, Metro Vancouver's $240-million Port Mann Water Supply Tunnel is now delivering clean, safe drinking water to residents south of the Fraser River.
The Port Mann connection is one of the key links between Metro Vancouver's watersheds and the communities south of the Fraser, the city said in a news release. Replacing this crossing was the first in a series of major water transmission system upgrades throughout the region.
Located more than 30 metres under the Fraser River bed, the new tunnel more than doubles the capacity of the old water main and provides enhanced earthquake and river erosion resiliency to help ensure the continued delivery of drinking water to Metro Vancouver residents.
"This was one of the most challenging and rewarding projects we've ever built," said Darrell Mussatto, Chair of Metro Vancouver's Utilities Committee. "Our engineering and construction team came up with innovative solutions to tunnel through difficult terrain, at exceptional depths and under extreme groundwater pressures."
The Port Mann Water Supply Tunnel consists of two 60-metre deep shafts – one in Coquitlam and another in Surrey – connected by a one-kilometre-long, 3.5-meter-diametre tunnel housing a 2.1-meter-diametre steel water main. A new valve chamber was built at the top of each shaft to allow the new tunnel to connect with the existing land-based water mains on either side of the river.
Construction started in the spring of 2011, when ground was broken for the south shaft. Tunnelling began in February 2014 and achieved breakthrough in June 2015. Installation of the water main inside the shafts and tunnel was completed in July 2016 and the valve chambers were finished in January 2017.
The tunnel had to be located deep underground to maximize its earthquake resilience, and tunnelling 30 metres below the bed of the Fraser River posed significant challenges. Engineers had to overcome difficult soil conditions and the highest groundwater pressures ever observed in Canadian soft ground tunnelling.
The soil deposits under the Fraser River consist of several different geologic layers. Some places have sand, gravel and clay, while others have dense, glacial soil strewn with immense granite boulders. These soil conditions, combined with the high groundwater pressures, required an advanced, custom-designed tunnel boring machine to complete the tunnel under the river. At one point, engineers had to inject liquid nitrogen from an in-river work platform to freeze and harden the soil at the front of the tunnel boring machine so that it could be repaired.
"This was an extremely complex project that took a lot of skill and creativity," said Metro Vancouver Board Chair Greg Moore. "The insights and experience gained will benefit future tunnelling projects in Metro Vancouver, throughout Canada, and abroad."
The project team consisted of Metro Vancouver and McNally Construction Inc.-Aecon Constructors Joint Venture, with construction management provided by Hatch Infrastructure. Design and engineering services were provided by the Fraser River Tunnel Group, which comprised Ausenco Engineering Canada, McMillen Jacobs Associates, and Golder Associates.
Metro Vancouver's Port Mann Water Supply Tunnel Project received the "2016 Canadian Innovation Project of the Year Award" from the Tunnelling Association of Canada.
Fraser River drinking water tunnel nears completion
Surging river forces evacuation alert in Vancouver
Fuel-tainted ground water, red tape delay Port Stanley, Ont., sewer construction
MLA says B.C.'s plan to leave contaminated soil above Shawnigan Lake water supply unacceptable
January 12, 2021 Non-essential construction further restricted as Ontario declares second provincial emergency to address COVID-19 crisis
January 12, 2021 Study indicates 'Union Safety Effect' on building trades construction sites
January 7, 2021 NGWA applauds extension of geothermal heat pump tax credit in U.S. COVID-19 relief bill
Video: A quick look inside the latest issue of Ground Water Canada with the editor
Drilling down: Industry snapshot
Groundwater associations in Canada
Stay safe, stay in business
Canadian National Ground Water Association: Become a voting member
Princeton Geoscience, Inc.
SJE Rhombus
Detroit Lakes, MN
Carriff Canada
Water treatment plants are failing on reserves across Canada: Globe report
Water well rehab covered in new NGWA book
HPAC Magazine
Ground Water Canada@GroundWaterMag·
It's good to see Ontario promoting the skilled trades – would be ideal to see this happen in every province and territory https://www.groundwatercanada.com/ontario-promotes-skilled-trades/
Response to fire impacts water levels 40 years into future, Washington State University study says
USask has a new bachelor of science in environmental geoscience
Grant Ferguson@geosomething·
Dusting off some notes for upcoming lectures on regional groundwater flow & was reminded of how much I liked this paper on the carbonates in Manitoba by Grasby & Betcher. We don't do enough work like this in Canada these days & there's so much left to map.
https://www.nrcresearchpress.com/doi/abs/10.1139/e02-021#.Xhtmb1NKh-U
Good to see an awareness campaign in full swing: Ontario Federation of Agriculture prioritizes well water testing in 2020 https://www.groundwatercanada.com/ontario-federation-of-agriculture-prioritizes-well-water-testing-in-2020/


text:
Posted 20th May 2019
"It has been a great season and the class of 2019 have certainly created memories – the future is bright for the Blue & Gold"
Following her final game with Team Bath Netball, Saturday's 62-56 victory over Loughborough Lightning that secured third place in the 2019 Vitality Superleague, Head Coach Jess Thirlby has been reflecting on a great match to finish on and an encouraging campaign overall…
"The team showed the passion, grit and will to win on Saturday that has become a welcome trademark of this squad throughout the 2019 season to earn a well-deserved bronze medal. While it may not have been the match we had hoped to be in at the start of the year it was 'our final' and it certainly matched up to the standard of a final.
"Both teams really wanted the win and you could tell that with a first half full of tension. It was a well contested game and despite Loughborough finding their shooter Mary Cholhok with ease at times I felt that Summer Artman really led from the back with some critical turnover in the early stages of the game. It was worrying to lose Ebs so early in the first quarter after rolling her ankle but Leila Thomas entered the game in fine form and combined well with Summer.
"It was the second half where we really stepped up and closed the gap to just one going into the final quarter. We played with more freedom in attack, opening up space more effectively. Combine that with some critical turnover from Serena, Imo and our attacking unit of Sophie and Rach we were able to really build momentum, turning a five-goal deficit at half-time to a six-goal winning margin.
"That is really impressive 11-goal swing and the fourth quarter was a period that we can all be proud of as we witnessed the team really come together and collectively pull through.
So pleased that this bunch got to finish on a high and dug deep against a spirited @LboroLightning side to grab 3rd place for 2019. The grit and passion this team have shown a number of times this season has been impressive and has become a welcome trademark of our performances👊🏻 pic.twitter.com/WFAC1gfX4k
— Jess (@JessThirlby) May 19, 2019
"In true Team Bath style we will reflect and review the season, knowing that critical losses to Surrey Storm early on in the season and Mavericks on the Easter Weekend double header maybe altered our course slightly and prevented us to secure a home semi-final. However, we can and must be very proud of the way in which we have gone about our work this season.
"We have had so many close encounters and managed to find a way to win, which is a huge step forward for us and it came to fruition again in the play-off on Saturday. The margins between winning and losing this season have been smaller than ever and I feel we have risen to that challenge more often than not.
"Not only has the team shown improvement and progression with every game but it has been fantastic to see individuals recognised for their performances across the season. Both Imogen Allison and Sophie Drakeford-Lewis were nominated for the Vitality Superleague Young Player of the Year award, with Sophie taking the accolade. They were both very deserving of their nominations and have had their best seasons so far in the Blue & Gold. I have every confidence that they will, one day, represent the England Roses.
"Serena Guthrie was voted the league's best C in the All-Stars VII and Eboni Usoro-Brown was on the three-strong shortlist for the Vitality Player of the Year award. As co-captains they have inspired their team and are true leaders.
"We were more competitive at the semi-final stages than we have been in recent seasons and bettered both our league and overall final position from fourth to third. The future is very bright for this team and a final is more than achievable next season.
"Thank you to all our amazing fans, the team of staff behind the team and the University of Bath for their continued support. It has been a great season and the class of 2019 have certainly created memories."


Elderly people need specialised rehabilitation in order to maintain as long as possible their independence and mobility. An elderly person who does not move, does not walk or sees no one for too long is a person who stiffens mentally and physically. It is therefore important to help them move as long as possible, for circulatory, musculoskeletal, joint or psychological reasons. The movement is the key to successful and non aching aging. On the contrary to popular belief, the more one moves the less it hurts. Moreover, as therapists, we never let aside the psychological support during the sessions because it is important to stimulate the mind as well as the body in order to avoid the appearance of symptoms such as fear or depression that may limit and slow down the process of rehabilitation.
Some seniors prefer staying at home as long as possible while some other prefer going to nursing and care homes. Whatever your choice is, LHVP physiotherapists will be able to provide you with the most appropriate care as they move anywhere in London to come to you, whether you need help to walk, a mobilisation when you have to stay in bed, a therapy massage or a post operative rehabilitation after a hip or knee replacement for example. As treatments are being held at home, it helps to develop rehabilitation protocols suitable to the place the person lives and also teach seniors efficient and safe transfers and moves.
We also treat women who suffer from incontinence, prolapse, loss of sexual sensation and pelvic pain due to age or any other causes.
Finaly offer soft exercises and fitness classes tailored to keep joints flexible , muscles toned and a good physical and mental shape.
It is common for elderly people to develop difficulties walking. It can appear insidiously with the person walking a bit less every day or after an important traumatism (often a fall or a hospitalisation).
In both cases, the patient develops a Psychomotor Regression Syndrome (PRS). It is a complex term meaning that the systems and mechanisms affected in postural and walking automatisms are impaired. This syndrome is composed of motor symptoms and psychological disorder.
At London Home Visit Physiotherapy, we have specialists with experience in treating patients with Psychomotor Regression Syndrome. We are able to arrange home visits within London or visits in any home care facility.
When necessary we also organise care and administrative duties with family members.
Our first aim is to break this vicious circle by promoting movement and helping the person to go back to normal activity and even develop new hobbies.
The first step of our approach is as always to understand our patients and their situation. To do so, we use a variety of evidence based tests to evaluate the level of impairment of the walk and different posture. Then, we will develop a treatment plan in accordance with the patient expectations.
Work on the change of position to teach the easy and safe way to stand up, sit down, and lie down.


text:
Drone technology continues building on solid foundations in construction
CONSTRUCTION MONITORING
Drone technology continues building on solid foundations in construction.
Managing a construction project is no small task. From tracking site progress and monitoring safety, overseeing subcontractors and, keeping stakeholders informed, there is almost no end to the level of coordination required on any given day.
It's no wonder project managers on job sites of all sizes are turning to drones for support. The construction industry has been one of the earliest and most enthusiastic adopters of drone technology. In the last year alone, the commercial drone industry has grown 240% and much of that growth is driven by the construction industry.
Aerial photos, maps, and 3D models have the power to transform your workflow.
Not only can drones save your construction projects money and resources, but they also give your team a technical set of data for more informed communication and decision making.
Sites can be monitored at any time interval to allow new data on the progress of the site to maintain an efficient workflow and site monitoring.
Alongside surveys, innovation in drone software systems allows accurate contour maps and 3D models to be produced, based on footage and data gathered.
The process of high-resolution aerial imagery 3D modeling comes in
various formats but essentially enables images of the whole project area to be captured and merged into a comprehensive model of the area.
2D images can also be generated based on mapping technology.
Whilst 3D imagery offers full model benefits, 2D images allow accurate measurements and adjustment.
ONE DATA SET – MULTIPLE OUTPUTS
In most cases a single data set can be used in multiple outputs producing an Orth mosaic 2D image to look at the whole site in minute detail allowing you to assess site progress, safety issues and anything else you need to monitor.
As a 2d data set this is taking imagery in NADIR (camera facing straight down), it is very hard to understand levels and relief on the ground.
Using the same data set a DTM (Digital Terrain Model) can be produced at the same time, allowing site managers and planners to understand the elevation of the site, heights, and levels.
CUT AND FILL AND VOLUMETRIC CALCULATIONS INSTANTLY
During the early stages of the project, drones can play an important role in the levelling of the prospective site. due to the Geo tagged images and onboard sensors the drone can take land level calculations it can then through specific software calculate cut and fill amounts and locations. Stockpiles can also be monitored to ensure safety and compliance.
All this can be converted into a site-specific report for all stakeholders.
PROGRESS MONITORING
Many of the latest drone systems incorporate advanced situational and positional awareness for enhanced security and in-the-moment evaluation, response, and planning. Drones make the production of weekly progress maps far quicker, easier, and less costly than traditional methods. They also facilitate greater and easier information exchanges between construction companies and their clients, boosting overall efficiency, transparency, and communication.
SECURITY, MAINTENANCE, SAFETY
Security must be one of the main contributing factors to site safety using drones can allow site managers and stakeholders to quickly assess the installation and maintenance of security measures such as fencing, on-site cameras, equipment security and storage etc.
Using thermal cameras can give added security to help monitor and detect site activity during silent hours.
Due to the height of operations of the drone, a unique data aspect can be obtained to identify further risks such as proximity issues of waste and storage or other materials which could cause fire hazards and other issues.
Where areas are completely inaccessible, drones can be used and where there's uncertainty about safety issues, drone technology can be used to save time as well as minimize risk. For example, drones can analyze roof structures from above, and with the use of a specialist internal drone which can be used in internal structures to gather data.
RISK AND SAFETY MANAGEMENT
Any areas considered too risky for sending in personnel, such as dangerous structures, areas where hazardous materials are leaking or where there's risk of fire from flammable materials, mean using a drone presents a safer option as the drone operator can remain at a safe distance.
Specialist drones can be employed where access is restricted with onboard thermal, RGB cameras, and high-intensity spotlights all housed in a collision-proof cage to avoid damage.
SUPPLIED OR IN-HOUSE DRONE SOLUTION?
We know that one size never fits all, so we tailor our services to your needs. We can provide a full service of the pilot, equipment, data collection and, delivery.
Or if you want to add drones to your workflow, we can help you manage your drone proposition, source pilots and we'll take care of all aspects of the drone management down to licensing and provide a white-labeled case management system for the delivery and monitoring of your data.
For further information, contact info@iprosurv.com
About Iprosurv
Established in 2014, Iprosurv provides companies with the in-house capability of drone and data/media delivery services. Our CAMERA system and optimum drone operator platform provides bespoke services be that an on-demand, fully managed service to independent data/media delivery services.
Iprosurv is a pioneer in the provision of drone technology across a range of industries. Its current network of pilots, covering the entirety of the UK, use a proprietary system to record, store and deliver drone data to clients in a fast and secure way. From building surveys to flood response to assisting emergency services, Iprosurv continues to push the boundaries of how drones can be used in business.
Tagged in claims, construction, Drone Inspections, dronesforgood, evolution of claims, facilities management, healthandsafety, insurance, insurtech, research in insurance, sedgwick, survey drones
Drones Take on High Rise and Tower Block Inspection
In the last five years, drones have been making their presence felt across the economic landscape from agriculture and architecture to environmental monitoring and supporting the emergency services.
They have been adopted so easily in so many markets because they are a fast, cost effective and safe alternative to traditional methods of working. And they have proven themselves to be effortlessly adaptable to any natural or built environment.
Wherever drones go next, and we're sure it will be far and wide, one of the original uses for this technology was in building inspections, a use that is more pertinent than ever in the post-Grenfell landscape.
There is a huge challenge facing the UK Government and property managers up and down the country, to inspect and survey the nation's high-rise residential buildings to ensure they are safe, secure and fit for purpose.
It's a daunting task but one that can be tackled quickly, effectively and at a vastly reduced cost with the support of drone technology.
Let us show you how …
The Grenfell Tower tragedy has exposed the vulnerability of thousands of blocks of flats up and down the country. That vulnerability lies almost entirely within the use of cladding that was applied to improve heating and energy efficiency and to improve the appearance of buildings, many of which had been constructed in the 60s and 70s.
While the use of cladding may have delivered on those needs, the Grenfell fire showed just how dangerous the use of certain types of cladding is. Every single block of flats in the country that has been cladded has to be surveyed and inspected to understand the risk profile of each one.
Which creates a huge challenge for the Government and property managers. An estimated 1,700 high rise blocks of flats have to be surveyed and the cladding removed to make them safe to live in. And they need to do it quickly.
TRADITIONAL METHODS
The Government has pledged billions to facilitate this nationwide structural survey but the Housing, Communities and Local Government Committee has estimated that as large as this fund is, it will only be enough to repair a third of the properties currently considered unsafe.
The reason? A huge chunk of that cost is because the traditional method of inspecting high rise buildings is to erect scaffolding around the structure, a process that takes months to complete and costs on average £250,000 per building.
Technology-Led Methods
But there is another way, a way that is led by, facilitated and completed by drone technology. A method that could reduce the cost of inspection by a factor of 100. Sound too good to be true? It's not, it's reality and we are helping clients across the country bring those theoretical savings to life.
Like anything in life, there is no quickfire solution to the UK's high-rise living problem. But with a little creative thinking and the smart application of technology, a real difference can be made to the way we approach it.
Which is exactly what Iprosurv has been doing with our property management clients.
WHY DRONES
Put simply, drones are a faster, cheaper and safer way to conduct surveys at height but their ability to provide the necessary data digitally, in a secure fashion, means that they can deliver a high-quality result in days, rather than the months the traditional method requires.
So what does a high rise drone inspection involve?
Our pilot will attend the site, secure it and be airborne within an hour
They will conduct a full survey using RGB – Optional Thermal Technology
Depending on the size or complexity of the structure, the flight and gathering of all the necessary data will be completed, on average, in 1 – 6 hours
The data is then shared with all relevant parties via Iprosurv's proprietary, GDPR-compliant data delivery system
Clients receive the data in the form of a detailed, interactive 3D model of the building which can be analysed down to 5mm per pixel.
All of this is delivered for an average cost of a £2,500 compared to the average £250,000 cost for the traditional scaffolding approach
And there is no need to involve or disturb tenants beyond informing them of the flight taking place
INSPECTING YOUR BUILDING
While the drone does the hard work, our clients still have to bring their expertise to bear in analysing the drone data and planning the next steps. And we make that as simple or detailed as necessary with the creation of our interactive, 3D models.
With the 3D model, delivered directly to their desktop, Iprosurv clients can:
Make detailed measurements of the building
Gain a full 360 view of the building
Annotate the model and the individual images of the site to share with others, assign tasks or share insight
Compare condition of the building pre and post-works
Create risk ratings on different aspects of the building
Share the data with other stakeholders with the click of a button, all fully secure and compliant with GDPR regulations
Access our proprietary software for ordering, case tracking and case delivery
Click Image Below to inspect a 3D Interactive Inspection Model
This is a genuine revolution in the inspection of high-rise buildings, one that not only does the job faster, cheaper and more safely than traditional methods, it provides more granular data that can be accessed, manipulated and shared securely in real time.
"The answer to your and our nation's high-rise problem is here, so what are you waiting for? Contact one of our advisors today to find out how you can become part of the drone revolution".
Download Multi-storey Asset Inspection Brochure
Tagged in cladding, construction, dronesforgood, dronesurveys, insurance, insurtech, towerblockinspection
A Daunting Task – High Rise and Tower Block Inspection
https://iprosurv.com/2022/01/18/drone-technology-continues-building-on-solid-foundations-in-construction/Drone technology continues building on solid foundations in construction
https://iprosurv.com/2021/08/28/drones-take-on-high-rise-and-tower-block-inspection/Drones Take on High Rise and Tower Block Inspection
https://iprosurv.com/2021/02/11/a-daunting-task-high-rise-asset-inspection/A Daunting Task – High Rise and Tower Block Inspection


The Redskin Peach has excellent qualities for fresh eating, shipping, or canning. The firm yellow flesh is covered with velvety deep red skin. This fine flavored large peach ripens in early August, and requires 750 chill hours.