text:
\section{Introduction}
Recent developments in new learning methods allow the generation of complex anthropomorphic motions in robotics. However, current systems still struggle with real-world scenarios beyond the narrow conditions of laboratory experiments. Humans, on the other hand, are capable of quickly adapting to uncertain, complex, and changing environments in a sheer endless variety of tasks. One key difference between biological and robotic systems lies in their actuator morphology: robotic drives are mostly designed to yield a linear relation between control signal and output torque. In contrast, muscles have complex nonlinear characteristics.

It has already been demonstrated, that muscular nonlinearities may provide a benefit for stability and robustness, especially under environmental uncertainties or perturbations~\citep{wagner1999stabilizing,eriten2009rigorous,Brandle2019a}. A benefit over linear torque actuator morphology has been observed in computer simulations by exchanging the actuator morphology in otherwise identical anthropomorphic tasks like reaching \citep{Stollenmaier2020} or locomotion \citep{van1993contribution,Haeufle2010a,gerritsen1998intrinsic,john2013stabilisation}. Similarly, reduced demand on the information processing capacity has been shown for muscles when compared to torque actuator morphology~\citep{full1999templates,holmes2006dynamics,blickhan2007intelligence,haeufle2014quantifying,haeufle2020muscles}. This opens the question whether muscular morphology could also be beneficial for robustness and data-efficiency in the process of \textit{learning} movement control.

Recent advances in deep learning facilitated the generation of complex movements like point-reaching ~\cite{fischerReinforcementLearningControl2021a, joosReinforcementLearningMusculoskeletal2020a,myosuite2022, schumacherdeprl2022} and locomotion~\cite{barbera2021ostrichrl, kidzinski_learning_2018-2,kidzinski_artificial_2019-1, Song2021,schumacherdeprl2022} in simulations with muscular actuator morphology.
In the real world, optimization and learning approaches can also find controllers for robotic systems with pneumatic muscles exhibiting somewhat muscle-like actuator morphology \cite{driess2018learning, buchler2019learning}. These examples demonstrate that learning and optimization methods \textit{can} control muscle-driven systems and may enable benefits such as safe learning and robustness \cite{buchler2019learning}. However, investigating advantages of nonlinear muscular actuator morphology over linear torque actuator morphology requires a direct comparison of both, which is---to our knowledge---missing in the literature.

The purpose of this study is to test if learning with muscular actuator morphology is more data-efficient and results in more robust performance as compared to torque actuator morphology. We investigate this in a very broad approach: we employ different learning strategies on multiple anthropomorphic models for multiple variants of reaching and locomotion tasks solved in physics simulators of differing levels of detail. This provides new and comprehensive evidence of the beneficial contribution of muscular morphology to the learning of diverse movements.

\section{Morphological difference between torque and muscle actuators}
\label{sec:muscles}
In contrast to idealized torque actuators, where torque is simply proportional to the control signal $u_{\text{torque}} \in [-1,1]$,
\begin{equation}
	\tau = \tau_{\text{max}}\,u_{\text{torque}}
\end{equation}
muscular force output nonlinearly depends on the muscle control signal $u_{\text{muscle}}$, the muscle length $l_\text{MTU}$ and contraction velocity $\dot{l}_\text{MTU}$. These biologically observed dependencies can be predicted by so-called \emph{Hill-type} muscle models \citep{Siebert2014}.
In a nutshell, the model captures biochemical processes transforming muscle stimulation $u_{\text{muscle}} \in [0,1]$ to the force-generating calcium ion activity $a$. This can be modeled by a first-order differential equation of the form \citep{Rockenfeller2015}
\begin{equation}
	\dot{a} = f_a(u_{\text{muscle}} - a)
\end{equation}
which induces low-pass filter characteristics (Fig.~\ref{fig:muscle_flva_plot}).
The model further captures the nonlinear \textit{force-length} and \textit{force-velocity relations} \citep{Siebert2014}. The \textit{force-length relation} is characterized by a positive slope (increasing force with increasing muscle fiber length) in the typical operating range of biological muscle fibres (Fig.~\ref{fig:muscle_flva_plot}). The \textit{force-velocity} relation is characterized by decreasing force for faster shortening velocities and increasing force if the muscle is externally stretched against its contraction direction (Fig.~\ref{fig:muscle_flva_plot}). A lever arm $\rho(\alpha)$ translates joint angle $\alpha$ into muscle-tendon-unit length $l_{\text{MTU}}$ and muscle force into joint torque
\begin{equation}
	\tau = \sum_{i=1}^{N} \rho_{i}(\alpha) f_{\tau}\left(l_{\text{MTU},i}(\alpha), \dot{l}_{\text{MTU},i}(\dot{\alpha}), a_{i}\right).
\end{equation}
for $N$ muscles which span a joint---typically at least two in an antagonistic arrangement.

In practice, we employ two different muscle models: A detailed one with more physiological details, contained in demoa~\citep{darus-2550_2022}, and a simpler model that efficiently adds muscular properties to existing MuJoCo~\cite{todorovMuJoCoPhysicsEngine2012a} simulations without sacrificing computational speed. See \supp{supp:muscles} for details.
\section{Methods}
\begin{table}
	\centering \small
	\caption{\textbf{Overview of all models and tasks}}
	{\begin{tabular}{llccc} \toprule
			Model & Task & Control & Environment  \\  \midrule
			ArmMuJoCo\xspace & precise reaching & RL & MuJoCo  \\
			ArmMuJoCo\xspace & fast reaching & RL & MuJoCo \\
			ArmDemoa\xspace  & smooth point-reaching & OC, MPC & demoa \\
			ArmDemoa\xspace  & hitting ball with high-velocity & OC, MPC & demoa \\
			Biped\xspace & hopping & RL & MuJoCo \\
			FullBody\xspace  & squatting & OC, MPC & demoa \\
			FullBody\xspace  & high-jumping & OC, MPC & demoa\\ \bottomrule
	\end{tabular}}
	\label{tab:overviewmodelstasks}
\end{table}
\begin{figure}
	\centering
	\begin{subfigure}[b]{0.24\textwidth}
		\centering
		\includegraphics[width=0.55\textwidth]{figures/arm26_torque.png}
		\caption{ArmMuJoCo\xspace}
		\label{subfig:armdemoa}
	\end{subfigure}
	\hfill
		\begin{subfigure}[b]{0.24\textwidth}
		\centering
		\includegraphics[width=0.6\textwidth]{figures/arm26_figure.png}
		\caption{ArmDemoa\xspace}
		\label{subfig:armmujoco}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.24\textwidth}
		\centering
		\includegraphics[width=0.62\textwidth]{figures/biped.png}
		\caption{Biped\xspace}
		\label{subfig:biped}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.24\textwidth}
		\centering
		\includegraphics[width=0.2\textwidth]{figures/allmin_figure.png}
		\caption{FullBody\xspace}
		\label{subfig:allmin}
	\end{subfigure}
	\caption{Models used for the experiments.}
	\label{fig:models}
\end{figure}
\subsection{Learning approaches for movement control}
We test if muscle actuator morphology facilitates learning by applying state-of-the-art learning algorithms covering an extensive range of approaches currently used in robotics. The common thread of the selected algorithms lies in their independence of the actuator morphology: this allows us to easily exchange idealized torque actuator morphology with muscle actuator morphology. We choose optimal control, model-predictive control and reinforcement learning as learning approaches.

\paragraph{Optimal control (OC)}
The control problem with horizon $N$ can be defined as:
\begin{align}
	\min_{\pi_{k}} J = \min_{\pi_k} \sum_{k=0}^{N} l(x(k),u(k),k),\qquad	\text{subject to}\ \  x(k+1) &= f(x(k),u(k),k),& \nonumber\\
	u(k) &= \pi_k(x(0),...,x(k)).& \label{eq:policy}
\end{align}
where $x(k)\in \mathbb{R}^{n_x}$ denotes the current state at time $k$, and $u(k)\in \mathbb{R}^{n_u}$ is the applied input at time $k$. Furthermore, $l$ specifies the cost function, and $f$ denotes the system dynamics. To optimize for the best control policy, we use the covariance matrix adaptation evolution strategy (CMA-ES) \citep{hansen2003reducing} in the optimal control case (open-loop strategy). CMA-ES is a derivative-free algorithm and widely used in machine learning. It combines different learning mechanisms for adapting the parameters of a multivariate normal distribution. Note, that we choose the same optimization parameters for both actuator morphologies to allow for a fair comparison even though the number of decision variables $n_u$ is always larger in the muscle-actuated case due to the antagonistic setup.
\paragraph{Model predictive control (MPC)}
 In MPC, we solve the control problem in a receding-horizon fashion with varying prediction horizons and recursively apply only the first element of the predicted optimal control sequence $u(0)$ (closed-loop strategy). We employ a warm start procedure using the CMA-ES optimizer and afterwards start the MPC routine with the local optimizer BOBYQA \citep{powell2009bobyqa}. The parameters of the optimizers are either varied (see \sec{sec:results}) or given in \supp{supp:implementation}

\paragraph{Reinforcement learning (RL)}
RL allows learning of reusable feedback controllers. Instead of \textit{minimizing} a cost function (see \eqn{eq:policy}), conventionally the discounted expected reward is \textit{maximized}:
\begin{equation}
    \max_{\pi} J =\max_{\pi}\,\mathbb{E}\left[\sum_{k=0}^{N-1}\gamma^{k-1}\,r(k)\right]
\end{equation}
where $r(k)$ is the reward at time $k$, $\pi$ is a control policy and $\gamma\in[0,1]$ is a discount factor such that long-term rewards are weighted less strongly. RL consequently solves a similar problem to MPC, but the resulting controllers are able to act in closed-loop fashion without being given an explicit prediction model. For the point-reaching tasks, we additionally employ goals $g$ characterizing the desired hand position, which then constitutes an additional dependence of the reward function. The aim of the learning process is to learn a controller policy $\pi(u(k)|x(k))$ that takes as input the current sensor values, or state x(k), and outputs a control signal, or action, $u(k)$ such that a task is solved. In practice, we use the RL algorithm MPO~\cite{abdolmaleki2018maximum}, implemented in TonicRL~\cite{pardo2020tonic}.

\subsection{Models}
\paragraph{Arm\xspace{}}
\label{sec:arm26}
The Arm\xspace{} model (\fig{fig:models} a, b) consists of two segments connected with hinge joints (2 joints total) moving against gravity. The ArmMuJoCo\xspace\,\cite{todorovMuJoCoPhysicsEngine2012a,schumacherdeprl2022} model contains four muscles, two for each joint. In the muscle-actuated case in ArmDemoa\xspace~\cite{darus-2871_2022,wochner2020optimality}, six Hill-Type muscles generate forces: two muscles for the shoulder and two for the elbow joint, plus two biarticular muscles acting on both joints. All joints are individually controllable.
\vspace{-10pt}

\paragraph{Biped\xspace}
\label{sec:biped}
We converted the geometrical model of an OpenSim bipedal human without arms~\cite{kidzinski_learning_2018-2} for use in MuJoCo. The model, consisting of 7 controllable joints (lower back, hips, knees, ankles) moves in a 2D plane. Each joint is actuated by two antagonistic muscles or one torque actuator.
\vspace{-10pt}
\paragraph{FullBody\xspace}
\label{sec:allmin}
The FullBody\xspace model \citep{walter2021geometry,darus-2982_2022} consists of two legs and an upper body including arms based on a human skeletal geometry. It consists of 8 controllable joints (ankles, knees, hips, lumbar and cervical spine) in 3D, and 14 movable joints in total including the arms. Each controllable joint was either actuated by two antagonistic muscles (muscle-actuated case) or by one idealized torque actuator (torque-actuated case).
For more details, we refer to \supp{supp:models}.

\subsection{Objectives and rewards}
We choose anthropomorphic movement objectives which are highly relevant for robotic applications. We expect that muscular actuator morphology provides benefits for such tasks. All task formulations allow application in muscle and torque actuator morphologies with an identical reward or objective function. For a precise formulation of the used functions and conditions, see \supp{supp:tasks}.
\label{sec:costfuncs}
\vspace{-9pt}
\paragraph{Smooth point-reaching}
This task encourages \textit{smooth} point-reaching. Therefore, the objective minimizes the L2-error between the desired and current joint angle while penalizing the angle velocity and jerk to ensure a smooth motion. The desired angle is $90^{\circ}$ for both the shoulder and the elbow joint, as this requires a large motion.
\vspace{-12pt}
\paragraph{Precise point-reaching}
Similar to~\cite{haeufle2020muscles}, we incentivize reaching a random hand position in a pre-determined rectangle, while minimizing the distance of the end effector to the goal. We specifically add a reward term that gives a much larger reward for precise motions that reach the center of the target area. The episode does not terminate until a time limit of 1000 steps elapses.
\vspace{-12pt}
\paragraph{Fast point-reaching}
The same objective as for precise point-reaching is used, but the episode terminates if the target is reached, incentivizing reaching speed over precision.
\vspace{-12pt}
\paragraph{High-velocity ball serve}
A ball is dropped in front of the Arm\xspace{} model and the controller learns to hit the ball to achieve maximum ball velocity.
\vspace{-12pt}
\paragraph{Squatting}
This squatting objective is taken from \cite{walter2021geometry} and encourages desired hip, knee, and ankle angles for a squatting position.
 \vspace{-12pt}
 \paragraph{Maximum height jump}
 The objective for the high-jumping task is taken from \citep{pandy1990optimal} and maximizes the position and velocity of the centre of mass of the human body model at the time of lift-off. The model is initialized to start from a squatting position.
 \vspace{-12pt}
\paragraph{Hopping}
We developed an objective based on the z-axis velocity of the center of mass~(COM) of the system that encourages periodic hopping with maximum height. The episode terminates if extreme joint angles are exceeded.
\section{Results}
\label{sec:results}
In the following, we present three major results for the investigated approaches and environments: (\textbf{1}) Muscle-like actuators in general improve data-efficiency compared to torque-actuators. (\textbf{2}) The investigated learning and optimization algorithms exhibit greater robustness to hyperparameter variations when applied to muscle-driven systems. (\textbf{3}) The motions and controllers obtained from the muscular morphology are more robust against force perturbations that were not present during learning. We average results over $5$ and $8$ random seeds for OC/MPC and RL respectively.
\subsection{Data efficiency: Learning with limited resources}
Robotics applications in real-world scenarios often suffer from limited resources, which holds true for training and inference time. Therefore, we investigate the advantages of muscle-like actuator morphology in terms of overall learning efficiency and temporal control resolution.
\vspace{-12pt}
\paragraph{Advantages of muscular morphology} Smooth and precise point-reaching generally require more data with torque-driven systems, as seen in \fig{fig:data_efficiency}. The performance of the muscle actuator, in contrast to torque morphology, varies very little for different settings of the temporal control resolution $c$. Precise reaching with RL also results in stable performance with fewer training iterations, and a very small standard deviation across runs. Similar findings are seen for the squatting and hopping task, where muscle-actuators achieve better data-efficiency and smaller variation across runs and are able to find a good-enough optimum with fewer iterations.
\vspace{-12pt}
\paragraph{Advantages of torque morphology}
In tasks requiring fast and strong motions, without emphasis on stabilization, we find torque actuators to hold certain advantages. In ball hitting and fast reaching, the torque cases show similar or smaller variance, even though both actuators perform well for singular runs. The high-jumping task, where only a strong, swift motion is required to launch the system upwards, is solved much faster in the torque case. We can also observe in the hopping task that, although only after a considerable number of training iterations and exhibiting a large variance, some torque-actuated runs achieve a larger overall return than the best muscle-actuated runs.

\newlength\mylinewidth
\setlength\mylinewidth{0.5pt}

\tikzset{
    ultra thin/.style= {line width=0.25\mylinewidth},
    very thin/.style=  {line width=0.5\mylinewidth},
    thin/.style=       {line width=\mylinewidth},
    semithick/.style=  {line width=1.5\mylinewidth},
    thick/.style=      {line width=2\mylinewidth},
    very thick/.style= {line width=3\mylinewidth},
    ultra thick/.style={line width=4\mylinewidth},
    every picture/.style={semithick}
}

\begin{figure}
	\tikzsetnextfilename{compare_dataefficiency_all}
	\centering
	\tiny
	\pgfplotsset{
		compat=1.6,
		try min ticks = 3,
		legend image code/.code={
			\draw[mark repeat=2,mark phase=2]
			plot coordinates {
				(0cm,0cm)
				(0.15cm,0cm)
				(0.3cm,0cm)
			};%
		}
	}
    \definecolor{mycolor1}{rgb}{0.00000,0.50000,0.75000}%
    \definecolor{mycolor2}{rgb}{0.55000,0.66250,0.94375}%
    \definecolor{mycolor3}{rgb}{0.77500,0.55000,0.55000}%
	\input{figures/compare_dataefficiency_all}
	\caption{\textbf{Cost value or returns for different tasks.} Plotting the mean and standard deviation (shaded area) for 5 (OC/MPC) or 8 (RL) repeated runs for the two actuator morphologies (muscle in red, torque in blue).}
	\label{fig:data_efficiency}
\end{figure}

\subsection{Robustness to hyperparameter variations}
Tuning a growing number of hyperparameters for learning models typically requires considerable time and computational resources. By analysing hyperparameter sensitivity, we test if tuning with torque or muscle actuator morphologies requires less resources.

\Fig{fig:sigmaanalysis} shows the cost curves for smooth point-reaching for the evolutionary optimization algorithm CMA-ES for different values of $\sigma$, which is the principal tuneable parameter for this algorithm. The performance curves vary much stronger for torque actuators for all considered cases. Furthermore, all muscle-actuated cases find a good-enough optimum with fewer iterations and a smaller variance, independent of the hyperparameter $\sigma$ and the control resolution $c$.

The same task was repeated using MPC while varying the main hyperparameter $t_\text{pred}$, which represents the prediction horizon in moving horizon strategies. The performance curves and the final cost vary much stronger for torque actuators (\fig{fig:mpc_pred_horizon}a, note, the cost is plotted logarithmically).

Finally, we performed an extensive hyperparameter optimization for precise point-reaching. For each iteration, $50$ sets of parameters are randomly chosen and the final task performance is evaluated after $2 \times 10^{6}$ learning iterations. The sampling distributions for the parameters are then fit to the best performing runs and $50$ additional sets are evaluated for the next iteration. We optimize the learning rates of MPO, as well as gradient-clipping thresholds, as these have a strong influence on learning speed and stability. Muscle actuators already outperform torque-actuators in the first iteration, with a greater number of well performing parameter sets (\fig{fig:hyperpararl}). Almost no low-performing runs remain for iteration $7$, while a large torque-performance is only achieved by a small subset of parameter settings. See \supp{supp:additional} for more hyperparameter variations.

\begin{figure}
	\tikzsetnextfilename{pointreaching_hyperparameters_sigma}
	\centering
	\tiny
	\pgfplotsset{
		compat=1.6,
		legend image code/.code={
			\draw[mark repeat=2,mark phase=2]
			plot coordinates {
				(0cm,0cm)
				(0.15cm,0cm)
				(0.3cm,0cm)
			};%
		}
	}
	\setlength{\figH}{0.13\textwidth}
	\setlength{\figW}{0.3\textwidth}
	\input{figures/arm26_reaching_hyperparameter_sigmaanalysis_all.tex}
	\caption{\textbf{Muscle morphology is more robust towards hyperparameter variation ($\boldsymbol{\sigma}$) in point reaching.} The cost value of the best observation is shown. The mean and standard deviation (shaded area) are plotted for five repeated runs for the two actuator morphologies (muscle in red, torque in blue) with different control resolutions $c$. }
	\label{fig:sigmaanalysis}
\end{figure}

 \begin{figure}
\centering
{\centering\small\textcolor{ourred}{\rule[2pt]{10pt}{1pt}} muscle \quad \textcolor{ourblue}{\rule[2pt]{10pt}{1pt}} torque}\\
\includegraphics[width=1.0\textwidth]{figures/percentage_goodness_version2_titles.pdf}
	\caption{\textbf{Muscle actuators need less parameter tuning for good performance.} Hyperparameters are optimized for precise point-reaching following an iterative sampling scheme, each run trains for $2\times 10^{6}$ iterations. Fifty sets of parameters are sampled randomly from pre-determined distributions, the final performance is evaluated and used to adapt the sampling distributions for the next iteration. We plot the return distributions over the sampled parameters at different iterations.}
\label{fig:hyperpararl}
\end{figure}

\begin{figure}
	\tikzsetnextfilename{pointreaching_hyperparameter_tpred}
	\centering
	\tiny
	\pgfplotsset{
		compat=1.11,
		legend image code/.code={
			\draw[mark repeat=2,mark phase=2]
			plot coordinates {
				(0cm,0cm)
				(0.15cm,0cm)
				(0.3cm,0cm)
			};%
		}
	}
	\setlength{\figH}{0.13\textwidth}
	\setlength{\figW}{0.4\textwidth}
	\input{figures/arm26_mpcpointreaching_cost_withwithoutpert.tex}
	\caption{\textbf{Muscle morphology is more robust towards variation of hyperparameter $\mathbf{t}_\text{pred}$ and unknown perturbations.}  Plotting the development of cost over time for the two actuator morphologies (muscle in red, torque in blue) while varying the hyperparameter $t_\text{pred}$ denoting the length of the prediction horizon. Left: The unperturbed case. Right: The prediction model is not aware of the added weight to the lower arm.}
	\label{fig:mpc_pred_horizon}
\end{figure}
\subsection{Robustness to perturbations}
In this section, we probe the robustness of the obtained policies against unknown perturbations. In precise point-reaching, we evaluated the RL reaching policies for two modifications that were not present during training: First, the hand-weight of the model is increased by $1.5$ kg (dynamic load), and secondly a free spherical weight is attached to the end effector with a cable (chaotic load). We can see in \fig{fig:rl_robustness_arm} that the muscle-based policy does not suffer significant changes in performance, except for a small circular motion ($3$ cm) around the goal position in the chaotic load case. In contrast, the torque actuator morphology leads to unstable reaching and strong oscillations. Both morphologies seem to handle the dynamic load well. See \supp{supp:additional} for more goals.

For the MPC controller, we evaluated robustness by introducing environment changes that are unknown to the prediction model. One example is the lifting of an object with unknown weight, a typical robotics task.  When adding $1$ kg to the lower arm of the ArmDemoa\xspace model (\fig{fig:mpc_pred_horizon}), the performance in both actuator cases is worse than in the unperturbed case (left); the movement is also slower. However, the variance and absolute value of the final cost in the muscle-actuated case are still much lower compared to the torque-actuated case (plotted logarithmically).

For periodic hopping with the Biped\xspace model, we evaluated trained RL policies with random forces that were drawn from a Gaussian distribution $F\sim\mathcal{N}(\cdot|0,\sigma_{F})$ and applied to the hip, knee, and ankle joints with a probability of $0.05$ at each time step. We see in \fig{fig:rl_robustness_biped} that the torque actuator morphology is stronger affected in relative performance than the muscle morphology. In the robustness investigation with MPC in the FullBody\xspace squatting task, a force is applied to the hip joint after the system has reached its desired position. \Fig{fig:squat_mpc_force} (left) shows that the desired joint angles are much less affected by the perturbation when muscle actuators are controlled. Furthermore, the cost value associated with the movement recovers much slower for torque actuators (\fig{fig:squat_mpc_force} right).

\begin{figure}
	\tikzsetnextfilename{squatting_perturbation_unknownforce}
	\centering
	\tiny
	\pgfplotsset{
		compat=1.11,
		legend image code/.code={
			\draw[mark repeat=2,mark phase=2]
			plot coordinates {
				(0cm,0cm)
				(0.15cm,0cm)
				(0.3cm,0cm)
			};%
		}
	}
	\setlength{\figH}{0.15\textwidth}
	\setlength{\figW}{0.4\textwidth}
	\input{figures/allmin_squatting_perturbation_hiptraj_cost.tex}
	\caption{\textbf{Hip and cost trajectory for squatting with unknown perturbation forces.}
	The muscle morphology is shown in red, the torque morphology in blue while varying the perturbation force $F_\text{pert}$ [N] (applied between $t>=0.45$ s and $t<=0.5$ s).}
	\label{fig:squat_mpc_force}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.49\textwidth]{figures/robustnessarm26_ee.pdf}
\includegraphics[width=0.49\textwidth]{figures/robustnessarm26_joint.pdf}
	\caption{\textbf{Trajectories for dynamic ($\mathbf{1.5}$ kg weight) and chaotic (attached ball) load.} Left: Three random goals are exemplarily shown, the respective goal position is marked as a circle. The unperturbed baseline for each goal is shown with a dashed line. Right: Joint trajectories for the same experiment, the unperturbed baselines are shown with dashed lines. Vertical bars mark episode resets.}
\label{fig:rl_robustness_arm}
\end{figure}
\begin{figure}
\centering
\begin{subfigure}[b]{0.51\textwidth}
\centering
{\hspace{0.75cm}\small\textcolor{red}{\rule[1.5pt]{8pt}{1pt}} muscle \quad \textcolor{ourblue}{\rule[1.5pt]{8pt}{1pt}} torque}\\
\includegraphics[width=1.0\textwidth]{figures/robustness_biped.pdf}
\end{subfigure}
\hspace{1cm}
\begin{subfigure}[b]{0.301\textwidth}
\vfill
\includegraphics[width=1.0\textwidth]{figures/robustness_biped_errorplot.pdf}
\vspace{-0.4cm}
\end{subfigure}
	\caption{\textbf{Muscle actuators lead to more robust hopping.} At each time step, there is a 5 \% chance of a random force being applied to the hip, knee, and ankle joints. Left: COM motion over time. Vertical bars mark episode resets due to falls of the Biped\xspace. Right: Relative performance for different standard deviations of the random force $\sigma_{F}$. Performance is scaled by the unperturbed mean return.}
\label{fig:rl_robustness_biped}
\end{figure}
\section{Discussion}
\label{sec:discussion}
We investigated if muscle-like actuators have beneficial effects for modern learning methods in terms of data-efficiency, hyperparameter sensitivity  and robustness to perturbations. A multitude of variations across physics simulators, learning algorithms and task domains was considered in order to showcase the potential of the considered morphologies independently of the underlying implementation. We showed that muscles yield benefits in tasks requiring stable motion, even when compared to idealized torque actuators, which can be considered an upper performance bound. Indeed, the used torque actuators are able to instantaneously output any desired force at any point of the trajectory, while muscles only slowly change their output due to activation dynamics and can only produce kinematics-dependant force output.

Despite these limitations, the considered learning algorithms learn more efficiently with muscle actuation in all tasks, except for extreme motions where objectives require a strong force application without stability considerations, such as ball-hitting and high-jumping. In bipedal hopping, it was found that muscles result in more efficient learning, even though some torque-runs achieve higher asymptotic performance. Finally, we observe muscle actuation to result in increased robustness to perturbations and hyperparameter variations, which can facilitate learning on real robotic systems that not only present sensor and motor noise, but also prohibit extensive parameter searches.

To harness these biological properties, artificial muscles can be directly built as hardware~\cite{driess2018learning,wolfen2018bioinspired,buchler2019learning,haselcontrol2022} or virtually implemented through software~\cite{impedancelearning}. In particular, the muscle implementation that we apply to the MuJoCo models can efficiently create muscle-like behavior in real-time, which might be used to induce virtual muscle actuation in real systems, provided a sufficient control frequency.
\paragraph{Limitations}
Although we have reported results for a wide variety of algorithms and tasks, we cannot give theoretical statements about the general applicability of our findings. Additionally, some of the tasks we employed were limited in complexity and might also be solvable with classical control algorithms. The MuJoCo muscle model, while computationally efficient, only captures rudimentary properties of biological systems. The demoa implementation, on the other hand, includes visco-elastic, passive tendon characteristics and muscle routing as joint angle-dependent lever arms to account for many physiological details---at substantial additional computational cost. Finally, learning with intermediate control signals given to impedance or position controllers, instead of direct torque commands, might also improve learning performance, while muscle-like properties could have been introduced by learning priors or additional cost terms.
\clearpage

\acknowledgments{We thank Daniel Höglinger for help during the development of the hopping reward function. Furthermore, we like to thank Marc Toussaint, Danny Driess and David Holzm\"{u}ller for initial discussions regarding the topic of learning with muscles. This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016 (SimTech). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS)
for supporting all authors. Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645.
This work was supported by the Cyber Valley Research Fund (CyVy-RF-2020-11 to DH and GM).
We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039B).}

\section{Muscle model}
In this section, the implementations of the muscle models for demoa and MuJoCo are described. The demoa model approximates biological muscles with more physiological detail and accuracy, whereas the simpler MuJoCo model allows the simulation of rudimentary muscular properties at minimal computational cost, rendering it usable for machine learning.
\label{supp:muscles}
\subsection{Muscle model in Mujoco}
Even though the MuJoCo simulator includes the capability of simulating muscles, it requires the explicit definition of tendon insertion points and wrapping surfaces for each model. We, therefore, use our own muscle implementation for the MuJoCo experiments, that does not contain tendons. As a direct consequence, the muscle-fiber length is \textit{uniquely} determined by the joint angle.

In the following, we describe activation dynamics, definitions of muscle-fiber length and velocity, the computation of the resulting torque and the parametrization.

\paragraph{Muscle-tendon-unit}
Each controllable joint of the MuJoCo model is actuated by two monoarticular muscles and we do \textbf{not} compute tendon length.
We assume that:
\begin{equation}
l_{\mathrm{MTU}} = l_{\mathrm{CE}},
\label{eq:length}
\end{equation}
where $l_{\mathrm{MTU}}$ is the length of the entire muscle-tendon-unit and $l_{\mathrm{CE}}$ is the length of the muscle fiber, or contractile element.
While the MuJoCo internal model assumes an \textit{inelastic} tendon, such that the lack of tendon state speeds up the computation, we assume \textit{no} tendon, which allows the application to a large range of systems without the need to hand-define tendon geometry.\\
We define muscle-fiber length and velocity by a linear equation~\cite{Kistemaker2006, Geyer2010, Bayer2017}:
\begin{align}
    l_{\mathrm{CE},i} &= m_{i}\, \phi_{j} + l_{\mathrm{ref},i}\\
    \dot{l}_{\mathrm{CE},i} &= m_{i}\, \dot{\phi_{j}},
\end{align}
where $\phi_{j}$ is the joint angle, $m_{i}$ and $l_{\mathrm{ref},i}$ are computed from user-defined parameters, and $i\in\{1, 2\}$, as we assume two antagonistic muscles per joint. The parameter $m_{i}$ acts as a constant moment arm in our model, see \eqn{eq:torquemujoco}.
\paragraph{Activation dynamics}
The evolution of muscle activity obeys the following first-order ordinary differential equation:
\begin{equation}
    \dot{a}(t) = \frac{1}{\Delta t_{a}} (u(t) - a(t)),
\end{equation}
where $u(t)$ is a control signal.
\paragraph{Muscle force}
Given the previous quantities, the muscle force is computed by:
\begin{equation}
F_{i} = \left[ \mathrm{FL}(l_{\mathrm{CE},i}) \, \mathrm{FV}(v_{\mathrm{scale}}\,\dot{l}_{\mathrm{CE},i}) a_{i} + \mathrm{FP}(l_{\mathrm{CE},i})\right] \, F_{\mathrm{max},}
\label{eq:forcemujoco}
\end{equation}\\
where $v_{\mathrm{scale}}$ is a scaling parameter to adjust in which region of the force-velocity (FV)-curve typical fiber velocities operate.
We can then compute the resulting joint torque:
\begin{equation}
    \tau = - (m_{1}\, F_{1} + m_{2}\, F_{2}).
    \label{eq:torquemujoco}
\end{equation}
The functions FL, FV and FP are given by MuJoCo internal functions that phenomenologically model experimental data and are applied to normalized muscle lengths and velocities, see MuJoCo documentation~\cite{todorovMuJoCoPhysicsEngine2012a} and \fig{fig:musclemodelmujoco}.
\begin{figure}
    \centering
    \includegraphics[width=1.0\textwidth]{figures/muscle_model_mujoco.pdf}
    \caption{Force-length (FL) and force-velocity (FV) relationships and passive force (FP) used in MuJoCo~\cite{todorovMuJoCoPhysicsEngine2012a}. We use the same phenomenological functions in our own MuJoCo muscle model.  While FL and FV get scaled by the current muscle activity $a$, FP does not (see \eqn{eq:forcemujoco}).}
    \label{fig:musclemodelmujoco}
\end{figure}
\paragraph{Parametrization} As the there is a one-to-one mapping of joint angle to muscle lengths in our model, we can determine the required parameters $m_{i}$ and $l_{\mathrm{ref},i}$, if a mapping of $l_{\mathrm{min}}$ to $\phi_{\mathrm{min}}$ and $l_{\mathrm{max}}$ to $\phi_{\mathrm{max}}$ is specified (assuming $l_{\mathrm{max}}$ to be the maximal muscle-fiber length $l_{\mathrm{CE}}$). Inserting them into \eqn{eq:length} and solving the resulting system of equations gives:
\begin{align}
m_{1} &= \frac{l^{\mathrm{max}} - l^{\mathrm{min}}}{\phi^{\mathrm{max}} - \phi^{\mathrm{min}} + \epsilon}\\
l^{\mathrm{ref}}_{1} &= l_{\mathrm{min}} - m_{1}\, \phi_{\mathrm{min}}\\
m_{2} &= \frac{l^{\mathrm{max}} - l^{\mathrm{min}}}{\phi^{\mathrm{min}} - \phi^{\mathrm{max}} + \epsilon} \\
l^{\mathrm{ref}}_{2} &= l_{\mathrm{min}} - m_{2}\,\phi_{\mathrm{max}}
\end{align}
All in all, $\phi^{\mathrm{min}}$, $\phi^{\mathrm{max}}$, $l^{\mathrm{min}}$, $l^{\mathrm{max}}$ and $F_{\mathrm{max}}$ are required to be specified. The constant $\epsilon=0.01$ ensures numerical stability. We use the same parametrization for each MuJoCo task, see  \tab{tab:mujocomuscle}.

The maximum and minimum joint angles were chosen to allow for a large range of motion.  They do not constitute hard limits, but the passive elastic force FP will increase strongly when reaching them. The maximum and minimum fiber lengths are identical to the MuJoCo default values. As we want to study the benefits of muscular properties in learning, we chose the time and velocity scales $\Delta t_{a} = 0.01$ and $v_{\mathrm{scale}}=0.5$ to be large enough to produce noticeable effects, such as low-pass filtering and self-stabilization properties, across all performed tasks. To determine maximum muscle forces, we trained muscle-actuator policies for a chosen maximum force value, after which we adjusted maximum torque-actuator forces to be identical or slightly larger to the maximally observed muscle forces in the final task policies. We repeated this procedure with different force values until good performance could be observed for both morphologies, see \supp{supp:forcevariation} for an evaluation across different force values. All other MuJoCo internal parameters related to muscle modeling are kept to their default values.

In practice, we implement the muscle model in Cython~\cite{behnel2011cython}, interfacing with OpenAI gym~\cite{gym}, which achieves similar execution speed to native MuJoCo.
\begin{table}
\caption{Parameters for the MuJoCo muscle morphology.}
\centering
\begin{subtable}[t]{.49\textwidth}
\caption{Muscle parameters}
\centering
\begin{tabular}{@{}ll@{}}
\textbf{Parameter} & \textbf{Value}  \\
 \toprule
 $l_{\mathrm{max}}$ & 1.05 \\
 $l_{\mathrm{min}}$ & 0.95 \\
 $\phi_{\mathrm{max}}$ & $\pi/2$ [rad]\\
 $\phi_{\mathrm{max}}$ & -$\pi/2$ [rad]\\
 $\Delta t_{a}$ & 0.01 [s]\\
 $v_{\mathrm{scale}}$ & 0.5 \\
\bottomrule
\end{tabular}
\end{subtable}
\begin{subtable}[t]{.49\textwidth}
\caption{Maximum isometric force}
\centering
\begin{tabular}{@{}ll@{}}
\textbf{Task} & \textbf{Value}  \\
 \toprule
 ArmMuJoCo\xspace & 295 [N]\\
 Biped\xspace & 5000 [N]\\
\bottomrule
\end{tabular}
\end{subtable}
\label{tab:mujocomuscle}
\end{table}

\subsection{Muscle model in demoa}
\label{sec:muscledemoa}
The muscle model implemented in demoa \citep{darus-2550_2022} includes additionally visco-elastic, passive tendon characteristics and muscle routing as joint angle-dependent lever arms to account for many physiological details. In the following, we describe the activation and contraction dynamics of the muscle model, as well as the tendon characteristics and the nonlinear lever arms.
\paragraph{Activation dynamics}
The muscles are activated with the learned and optimized control signal $u$, which is nonlinearly transformed into an activation signal. The activity $a$ is following a first-order differential equation of normalized calcium ion concentration $\gamma$ as introduced by \citet{hatze1977myocybernetic} and simplified by Rockenfeller et al. \citep{Rockenfeller2015,Rockenfeller2018}:
\begin{equation}
    \dot{\gamma}(t)=M_{\mathrm{H}}(u(t)-\gamma(t)) \label{eq:actdyn1}
\end{equation}
and a nonlinear mapping onto the muscles activity
\begin{equation}
a(t)=\frac{a_0+\varpi}{1+\varpi}, \label{eq:actdyn2}
\end{equation}
with $\varpi(\gamma(t), l_\text{CE}(t))=(\gamma(t)\cdot\rho(l_\text{CE}))^\nu$ and $\rho(l_\text{CE})=\varpi_\mathrm{opt}\cdot\tfrac{l_\text{CE}}{l_\text{opt}}=\gamma_\mathrm{c}\cdot\rho_0\cdot\tfrac{l_\text{CE}}{l_\mathrm{opt}}$. The parameter values are chosen muscle non-specifically and are given in the description of the models (see \citep{darus-2871_2022,darus-2982_2022}).
\paragraph{Muscle-tendon-unit}
The predicted forces are modeled using Hill-type muscle models \citep{haeufle2014hill} including four spring-damper components (see Fig. \ref{fig:hilltype}): The contractile element (CE) models the active force production of biological muscle fibers, including the nonlinear \textit{force-length} and nonlinear \textit{force-velocity} relation. The parallel elastic element (PEE) models the passive connective tissue in the muscle belly and is arranged in parallel to the CE. The visco-elastic properties of the tendons are modeled using a serial elastic element (SEE) and a serial damping element (SDE).
All in all, the governing model dependencies for all muscles $i=1,...,n$ are:
\begin{eqnarray}
\dot{l}_{\text{CE},i} &=& f_\text{CE} (l_{\text{CE},i}, l_{\text{MTU},i}, \dot{l}_{\text{MTU},i}, a_i) \label{eq:contractdyn}\\
\dot a_i &=& f_a (a_i, u_i, l_{\text{CE},i}) \label{eq:actdyn3}\\
f_{\text{MTU},i} &=& f_{\text{MTU},i}(l_{\text{MTU},i}, \dot{l}_{\text{MTU},i}, l_{\text{CE},i}, a_i) \label{eq:forcemusc}\,,
\end{eqnarray}
where the first differential equation (Eq. \ref{eq:contractdyn}) denotes the contraction dynamics which models the velocity $\dot{l}_\text{CE}$ of the contractile element.
This contraction velocity is dependent on the current CE length $l_\text{CE}$, the length and contraction velocity of the muscle-tendon unit $l_\text{MTU}$ and $\dot{l}_\text{MTU} $ respectively, and the activity $a$. The latter is modeled by the activation dynamics (see Eq. \ref{eq:actdyn1},\ref{eq:actdyn2},\ref{eq:actdyn3}). Finally, a force $f_{\text{MTU},i}$  for each muscle is produced which is translated into joint torques.
\paragraph{Nonlinear lever arms}
To translate the force into joint torques, the muscle path around the joints is routed via deflection ellipses in demoa \citep{hammer2019tailoring}. If the length of the half-axises of all ellipses are set to zero, this approach can be simplified to the more commonly used fixed via-point approach for muscle routing. Based on the resulting moment arms of the muscles, the force $f_\text{MTU}$ is translated to generalized torques acting on the degrees of freedom of the system.
\paragraph{State of the system}
Using a musculoskeletal model with a Hill-type muscle model, as described in this section, increases the number of state variables because two additional differential equations need to be solved for each included muscle. The entire state vector $x$ can therefore be formulated as:
\begin{equation}
    x \in \mathbb{R}^{2n_\text{musc}+ 2n_\theta} = \{\gamma_i, l_{\text{CE},i}, \theta_j, \dot{\theta}_j\}
\end{equation}
where $\theta$ and $\dot{\theta}$ represent the generalized joint angle coordinates and their respective velocities, and $n_\text{musc}$ and $n_\theta$ denote the number of muscles and the number of joints, respectively.
\begin{figure}
    \centering
	\includegraphics[height=2.5cm]{figures/muscle_model.eps}
	\caption{The muscle model in demoa is modeled as lumped Hill-type muscle model (figure adapted from \citet{haeufle2014hill}).  \label{fig:hilltype}}
\end{figure}
\section{Experimental details and hyperparameters}
In this section, we describe algorithm implementation details while also reporting additional settings that were used to obtain previously shown results, as well as used hyperparameters. The section is divided such that experiment details are shown with the control algorithm that was used to generate the results.
\label{supp:implementation}
\subsection{Optimal Control (OC)}
\label{supp:optimal_control}
In the optimal control case, we used the covariance matrix adaptation evolution strategy (CMA-ES) \citep{hansen2003reducing} to find the control policy $u(k)$. As mentioned in the main paper, we chose the same population size and number of generations for both actuator morphologies to allow for a fair comparison, even though the number of decision variables $n_u$ is always larger in the muscle-actuated case. In all cases, if not otherwise mentioned, we use a fixed population size of $36$ and a fixed number of generations of $100$ while varying the control resolution $c$. If the control resolution is refined, this correlates to an increase in the number of decision variables $n_u$, however, we specifically did not change the population size or generation number because we wanted to compare the data-efficiency and learning with limited resources for the chosen actuators.
The temporal control resolution $c$ was typically varied for $c = \{0.05, 0.15, 0.3\}$ s. The upper bound of these control resolutions ($c = 0.3$ s) corresponds to a triphasic control pattern for a typical movement duration of $0.9$ s as it was selected in the smooth point-reaching and squatting task. This selection of $c$ was inspired by biological experiments, where it was shown that triphasic patterns occur in muscle surface electromyograms in typical point-reaching movements (e.g. see \citep{wierzbicka1986role,kistemaker2006equilibrium}). The main hyperparameter of the CMA-ES algorithm $\sigma$ was set to the default value of $0.2$ if not otherwise stated.
\subsection{Model Predictive Control {MPC}}
We employed a warm start procedure using the CMA-ES optimizer and afterwards started the MPC routine with a local optimizer BOBYQA \citep{powell2009bobyqa} (part of the standard python optimization package NLOPT). As temporal control resolution in this closed-loop setting, we chose a very fine resolution of $c = 0.01$ s, similar to the RL setup. This allows counteracting perturbations. The prediction horizon was varied between $t_\text{pred} = \{0.2, 0.3, 0.4, 0.5\}$ s as shown in the result section of the main paper.
\subsection{Reinforcement Learning (RL)}
\label{supp:rl}
We use the RL algorithm MPO~\cite{abdolmaleki2018maximum}, implemented in TonicRL~\cite{pardo2020tonic}. Hyperparameters were optimized with a simplified in-house CEM optimizer. All RL experiments are averaged over $8$ random seeds except for the hyperparameter optimization, which would have been computationally intractable. Each experimental run was computed with 1 NVIDIA A100 GPU and 20 CPUs of varying speed and type. We use a fixed control resolution of $c = 0.01$ s for all RL experiments.
\subsubsection{Experimental details}
We  optimized parameters for the hopping task that were not available at the time of submission. We will update the manuscript at a later date and have included the corrected results in \fig{fig:supphoppingnew}. Note that the findings are similar: muscle actuators lead to faster initial learning and less variance across runs, while asymptotic performance is stronger for torque actuators.
\label{supp:expdetails}
\paragraph{Data-efficiency} For the point-reaching experiments, we used the hyperparameters that were found in the meta-optimization (see \fig{fig:hyperpararl}) for both morphologies. For the hopping task, we used default MPO parameters. The updated learning curves with optimized parameters, as well as additional results on hyperparameters and maximum force settings can be found in \supp{supp:additional}.

\paragraph{Hyperparameter optimization} We optimized the performance of both actuator morphologies in the precise point-reaching task in MuJoCo (see \fig{fig:supphyperpararlarm}). For each iteration, $N_{\mathrm{sets}}$ sets of random parameters are drawn from fixed normal and log-normal distributions. For each of these sets, the task performance is evaluated after $T_{\mathrm{train}}$ environment interactions, where $T_{\mathrm{train}}$ is chosen such that a noticeable increase in performance can be observed with both actuator morphologies. After each iteration, $M_{\mathrm{elite}}$ elite parameter sets are chosen and the mean and standard deviation of each parameter-generating distribution is updated by fitting a (log-)normal distribution to the $M_{\mathrm{elite}}$ elite sets with maximum-likelihood estimation. See \tab{tab:hyperpara} for exact specifications. The meta-optimization for hopping can be found in \fig{fig:supphyperpararlbiped}.

In the experiments, we only used truncated log-normal distributions to generate parameters. The samples were clipped to the bounds given in \tab{tab:hyperpara} and initial mean and standard deviation were chosen to lie inside the bounded interval. More precisely, we defined $\log(\mu) = (a + b)/2$ and $\log(\sigma) = (b - a)/4$, where $a$ and $b$ are the chosen bounds. The chosen parameters were the actor learning rate $\mathrm{lr}_{\mathrm{a}}$, the critic learning rate $\mathrm{lr}_{\mathrm{c}}$, the learning rate of the dual optimizer $\mathrm{lr}_{\mathrm{d}}$, the gradient clipping threshold for the actor $\mathrm{clip}_{\mathrm{a}}$ and the critic $\mathrm{clip}_{\mathrm{a}}$.

\paragraph{Robustness point-reaching} We trained policies with both morphologies in precise point-reaching for $1.5\times 10^{7}$ iterations. The best performing policies were then evaluated for the perturbation experiments. For dynamic load, the mass of the hand is increased by $1.5$ kg to simulate an object. For chaotic load, a ball with radius $0.12$ m and a density of $1000$ kg/m$^{3}$ is attached to a cable of length $0.6$ m, that is connected to the hand. We sample $10$ random goals from the training distribution and visualize three trajectories such that there are no overlapping paths. All $10$ goals are shown in \fig{fig:suppmoregoals}.

\paragraph{Robustness hopping} We trained policies for both morphologies for hopping with the hyperparameters obtained in \fig{fig:supphyperpararlbiped} and for $1.5\times 10^{7}$ iterations. We then record $100$ evaluation episodes where random forces drawn from $F_{i} \sim \mathcal{N}(\cdot|0, \sigma_{F})$ are applied with a probability of $p=0.05$ to the hip, knee and ankle joints and to the pelvis position and rotation. Center of mass trajectories are shown for an interval of $15$ s in the main manuscript, black vertical bars mark episode resets due to extreme angles of the biped, which would cause it to fall to the ground. The performance for each perturbation level is divided by the unperturbed performance for each morphology to yield a relative performance comparison.
\begin{table}
\centering
\caption{Settings for the hyperparameter search.}
\begin{subtable}[t]{.49\textwidth}
\caption{Precise point-reaching}
\centering
\begin{tabular}{@{}lll@{}}
& \textbf{Parameter} & \textbf{Value}  \\
 \toprule
 & $N_{\mathrm{sets}}$ & $50$\\
 & $T_{\mathrm{train}}$ & $2\times 10^{6}$\\
 & $M_{\mathrm{elite}}$ & $10$ \\
\bottomrule
\end{tabular}
\end{subtable}
\begin{subtable}[t]{.49\textwidth}
\caption{Hopping}
\centering
\begin{tabular}{@{}lll@{}}
& \textbf{Parameter} & \textbf{Value}  \\
 \toprule
 & $N_{\mathrm{sets}}$ & $20$\\
 & $T_{\mathrm{train}}$ & $5\times 10^{6}$\\
 & $M_{\mathrm{elite}}$ & $10$ \\
\bottomrule
\end{tabular}
\end{subtable}\\
\begin{subtable}[t]{.49\textwidth}
\caption{Initial distributions}
\centering
\begin{tabular}{@{}lll@{}}
\textbf{Parameter} & \textbf{Distribution} & \textbf{Bounds}  \\
 \toprule
 $\mathrm{lr}_{\mathrm{a}}$ & truncated log-normal & $[2.5\times 10^{-4}, 3\times 10^{-2}]$ \\
 $\mathrm{lr}_{\mathrm{c}}$ & truncated log-normal & $[0.5\times 10^{-2}, 10^{-1}]$ \\
 $\mathrm{lr}_{\mathrm{d}}$ & truncated log-normal & $[0.5\times 10^{-2}, 1]$ \\
 $\mathrm{clip}_{\mathrm{a}}$ & truncated log-normal & $[10^{-7}, 10^{-4}]$ \\
 $\mathrm{clip}_{\mathrm{c}}$ & truncated log-normal & $[10^{-7}, 10^{-4}]$ \\
\bottomrule
\end{tabular}
\end{subtable}

\label{tab:hyperpara}
\end{table}
\subsubsection{Hyperparameters}
The hyperparameters for all RL point-reaching tasks were set to the best performing runs in the shown hyperparameter optimization. They best trained policies from the precise point-reaching experiment were then used for the perturbation experiments. The parameters for hopping were left to default values in TonicRL as our hyperparameter search had not finished in time to train new hopping policies with a sufficient number of random seeds. We were, however, able to train a single muscle- and a single torque-actuated policy with the optimized hyperparameters, which were then used for the hopping perturbation experiments. See \tab{tab:hyperparaused} for the exact values. We will update the manuscript with the updated learning curves based on optimized hyperparameters, see \fig{fig:supphoppingnew}.
\begin{table*}
    \centering
     \caption{RL parameters for MPO in TonicRL for the different tasks. Non-reported values are left to their default setting in TonicRL~\cite{pardo2020tonic}. Common MPO settings are equal for all experiments.}
    \label{tab:hyperparaused}
    \begin{subtable}[t]{.5\textwidth}
        \centering
        \caption{Point-reaching MPO (muscle)}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
        & $\mathrm{lr}_{\mathrm{a}}$ & $3\times 10^{-4}$ \\
        & $\mathrm{lr}_{\mathrm{c}}$ & $10^{-3}$ \\
        & $\mathrm{lr}_{\mathrm{d}}$ & $2\times 10^{-2}$ \\
        & $\mathrm{clip}_{\mathrm{a}}$ & $4 \times 10^{-5}$ \\
        & $\mathrm{clip}_{\mathrm{c}}$ & $3 \times 10^{-5}$ \\
        & batch size & $100$ \\
        & return-normalizer & No \\
        \bottomrule
        \end{tabular}
    \end{subtable}%
    \begin{subtable}[t]{.5\textwidth}
        \centering
        \caption{Point-reaching MPO (torque)}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
              & $\mathrm{lr}_{\mathrm{a}}$ & $10^{-3}$ \\
        & $\mathrm{lr}_{\mathrm{c}}$ & $5\times 10^{-3}$ \\
        & $\mathrm{lr}_{\mathrm{d}}$ & $8.2 \times 10^{-3}$ \\
        & $\mathrm{clip}_{\mathrm{a}}$ & $7 \times 10^{-6}$ \\
        & $\mathrm{clip}_{\mathrm{c}}$ & $10^{-6}$ \\
        & batch size & $100$ \\
        & return-normalizer & No \\
        \bottomrule
        \end{tabular}
    \end{subtable}\\
    \vspace{0.5cm}
 \begin{subtable}[t]{.5\textwidth}
     \centering
        \caption{Hopping MPO (muscle and torque)}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
        & $\mathrm{lr}_{\mathrm{a}}$ & $3\times 10^{-4}$ \\
        & $\mathrm{lr}_{\mathrm{c}}$ & $3\times 10^{-4}$ \\
        & $\mathrm{lr}_{\mathrm{d}}$ & $10^{-2}$ \\
        & $\mathrm{clip}_{\mathrm{a}}$ & None \\
        & $\mathrm{clip}_{\mathrm{c}}$ & None \\
        & batch size & $256$ \\
        & return-normalizer & Yes \\
        \bottomrule
        \end{tabular}
    \end{subtable}\\
    \vspace{0.5cm}
       \begin{subtable}[t]{.5\textwidth}
        \centering
        \caption{Hopping perturbation (muscle)}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
        & $\mathrm{lr}_{\mathrm{a}}$ & $9\times 10^{-4}$ \\
        & $\mathrm{lr}_{\mathrm{c}}$ & $3\times 10^{-3}$ \\
        & $\mathrm{lr}_{\mathrm{d}}$ & $10^{-2}$ \\
        & $\mathrm{clip}_{\mathrm{a}}$ & $10^{-5}$ \\
        & $\mathrm{clip}_{\mathrm{c}}$ & $3\times 10^{-7}$ \\
        & batch size & $256$ \\
        & return-normalizer & Yes \\
        \bottomrule
        \end{tabular}
    \end{subtable}%
    \begin{subtable}[t]{.5\textwidth}
        \centering
        \caption{Hopping perturbation (torque)}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
        & $\mathrm{lr}_{\mathrm{a}}$ & $10^{-3}$ \\
        & $\mathrm{lr}_{\mathrm{c}}$ & $7\times 10^{-4}$ \\
        & $\mathrm{lr}_{\mathrm{d}}$ & $2\times 10^{-2}$ \\
        & $\mathrm{clip}_{\mathrm{a}}$ & $2\times 10^{-5}$ \\
        & $\mathrm{clip}_{\mathrm{c}}$ & $10^{-6}$ \\
        & batch size & $256$ \\
        & return-normalizer & Yes \\
        \bottomrule
        \end{tabular}
    \end{subtable}\\
    \vspace{0.5cm}
    \begin{subtable}[t]{.5\textwidth}
        \centering
        \caption{Common MPO settings}
        \begin{tabular}{@{}lll@{}}
        \toprule
        &\textbf{Parameter} & \textbf{Value} \\
        \midrule
        & buffer size  & $10^{6}$ \\
        & steps before batches & $5\times 10^{4}$ \\
        & steps between batches & $50$ \\
        & number of batches & $50$ \\
        & n-step return & $3$ \\
        & n parallel & $20$ \\
        & n sequential & $10$ \\
        \bottomrule
        \end{tabular}
    \end{subtable}%
\end{table*}
\section{Models}
We give detailed descriptions of the used models in this section. See \tab{tab:mujocomodels} for more information about the MuJoCo models.
\label{supp:models}
\begin{table}
\caption{State information for all MuJoCo environments. The elements actuator lengths and velocities are directly derived from MuJoCo internal attributes \texttt{actuator\_length} and \texttt{actuator\_velocity} and keep the two morphologies as consistent as possible.}
\begin{tabular}{p{0.25\linewidth} p{0.68\linewidth}}
\textbf{model} & \textbf{observations}\\
 \toprule
 ArmMuJoCo\xspace{} (muscle) & joint positions, joint velocities, muscle positions, muscle velocities, muscle forces, muscle activities, goal position, hand position\\
ArmMuJoCo\xspace{} (torque) & joint positions, joint velocities, actuator positions, actuator velocities, actuator forces, goal position, hand position\\
Biped\xspace{} (muscle) & joint positions, joint velocities, muscle lengths, muscle velocities, muscle forces, muscle activities, head position, pelvis position, torso angle, scaled COM-velocity \\
Biped\xspace{} (torque) & joint positions, joint velocities, actuator lengths, actuator velocities, actuator forces, head position, pelvis position, torso angle, scaled COM-velocity\\
\bottomrule
\end{tabular}

\label{tab:mujocomodels}
\end{table}
\subsection{Arm\xspace}
\label{supp:arm26}
The Arm\xspace model consists of two segments connected with hinge joints moving against gravity.
The ArmDemoa\xspace \citep{darus-2871_2022} is freely available using the multi-body software demoa \cite{darus-2550_2022}. In the muscle-actuated case, six muscles were included, modeled as Hill-Type muscles (\ref{sec:muscledemoa}). Here, two monoarticular muscles, each for the shoulder and elbow joint, and two biarticular muscles acting on both joints are included. The segments are modeled as rigid bodies, and the dynamics are solved using the Euler-Lagrange equation. In the torque-actuated case, each joint is driven by one torque actuator. For more details on the demoa model, we refer to the Technical Report \citep{darus-2871_2022}. The variant ArmMuJoCo\xspace was derived from an implementation of Arm26 included in MuJoCo\,\cite{todorovMuJoCoPhysicsEngine2012a}, it was modified to yield a torque-variant similar to~\cite{schumacherdeprl2022}. We additionally created a muscle-variant consisting of 2 muscles per joint. The maximum torques for the torque actuators were matched to the highest achieved torques by the trained muscle policies for both versions independently.
\subsection{Biped\xspace}
\label{supp:biped}
We converted the geometrical model of an OpenSim bipedal human without arms~\cite{kidzinski_learning_2018-2} for use in MuJoCo. The model, consisting of 7 controllable joints (lower back, hip, knee, ankle) moves in a 2D-plane. Each joint is actuated by two antagonistic muscles or one idealized torque actuator. During execution, we only allow control signals for one leg, the actions for the other leg are kept identical to the first one. This incentivizes symmetric hopping motions, even though both legs can still move differently due to differing initial configurations or external forces. The maximum torques for the torque actuators were matched to the highest achieved torques by the trained muscle policies.
\subsection{FullBody\xspace}
\label{supp:allmin}
For the squatting and high-jumping task, we used the FullBody\xspace  (allmin) model \citep{darus-2982_2022} which is freely available using the multi-body software demoa \cite{darus-2550_2022}. It consists of two legs and an upper body with a skeletal geometry similar to humans and moves in 3D. The ankle, knee and hip joints, as well as a lumbar and a cervical spine joint are controllable (8 controllable joints). The model also consists of two arms with their respective joints, however, these joints were not controlled in this study. In total, 14 joints are modeled with 20 degrees of freedom. Each controllable joint was either actuated by two muscles (\ref{sec:muscledemoa}) set up in an agonistic-antagonistic setup (muscle-actuated case) or by one idealized torque actuator (torque-actuated case). The maximum allowed torques were matched to the highest torques that occurred in the optimization in the muscle-actuated case to allow for a fair comparison.  Only monoarticular muscles (spanning one joint) were used. Furthermore, we reduced the number of control inputs $n_u$ for this study by using symmetrical control signals for the left and right legs. Additional to the torques generated by the actuators, also joint limitations are modeled as linear one-sided spring-damper elements. We refer to the Technical Report \citep{darus-2982_2022} for more details.
\section{Tasks}
\label{supp:tasks}
We chose movement objectives which represent both, robotic challenges and naturally observed movements of humans.

\paragraph{Smooth point-reaching (OC/MPC)} This task encourages smooth point-reaching. Therefore, the objective minimizes the L2-error between the desired angle endpoint and the desired joint angle velocity, as well as penalizing the angle jerk to ensure a smooth motion.
The objective for smooth point-reaching is given by:
\begin{equation}\label{eq:smoothpointreaching}
		\varepsilon = \frac{\omega_i}{{S_\mathrm{i}}}(\theta_\mathrm{i}-\theta_\mathrm{i}^\mathrm{des})^2+\frac{\omega_i}{{S_\mathrm{i}}}(\dot{\theta}_\mathrm{i}-\dot{\theta}_\mathrm{i}^\mathrm{des})^2+\dddot{\theta}^2,
\end{equation}
where $\theta_\mathrm{i}$ denotes the joint angle, $\dot{\theta}_\mathrm{i}$ the joint angle velocity and the last term $\dddot{\theta}$ penalizes the angle jerk to ensure a smooth motion. $\omega_i$ and $S_\mathrm{i}$ are weighting and scaling parameters, respectively. Their values (shoulder and elbow) are given in Table \ref{tab:scalingweightingparams}. The scaling parameters were chosen based on measured upper limits for human joint angular velocity \citep{jessop2016maximum} and human joint angle limits (Table 2 in \citep{darus-2982_2022}). The desired angle $\theta_\mathrm{i}^\mathrm{des}$ is set to $90^{\circ}$ for both the shoulder (sh) and the elbow (elb) joint, as this requires a large motion. The movement duration in this task was set to $0.9$ s.
\begin{table}
	\caption{Parameters for cost functions of OC/MPC tasks.}
	\centering
	\begin{subtable}[t]{.49\textwidth}
	\centering
	\caption{Scaling parameters}
	{\begin{tabular}{ll} \toprule
			parameter & value   \\  \midrule
			$S_{\theta,\textrm{sh}}$  & 2.45 [rad] \\
			$S_{\theta,\textrm{elb}}$ & 2.45 [rad] \\
			$S_{\theta,\textrm{hip}}$  & 1.92 [rad] \\
			$S_{\theta,\textrm{knee}}$ & 2.11 [rad] \\
			$S_{\theta,\textrm{ank}}$ & 1.05 [rad] \\
			$S_{\theta,\textrm{ls}}$ & 0.52 [rad] \\
			$S_{\theta,\textrm{cs}}$ & 1.05 [rad] \\
			$S_{\dot{\theta},\textrm{sh}}$  & 18.7 [rad/s]\\
			$S_{\dot{\theta},\textrm{elb}}$ & 27.9 [rad/s] \\
			$S_{\dot{\theta},\textrm{hip}}$  & 14.1 [rad/s]\\
			$S_{\dot{\theta},\textrm{knee}}$ & 28.4 [rad/s] \\
			$S_{\dot{\theta},\textrm{ank}}$ & 12.6 [rad/s] \\
			$S_{\dot{\theta},\textrm{ls}}$ & 5.2 [rad/s] \\
			$S_{\dot{\theta},\textrm{cs}}$ & 10.4 [rad/s] \\ \bottomrule
	\end{tabular}}
	\end{subtable}
	\begin{subtable}[t]{.49\textwidth}
	\centering
	\caption{Weighting parameters}
	{\begin{tabular}{ll} \toprule
			parameter & value   \\  \midrule
			$\omega_{\theta,\textrm{sh}}$ & 2  \\
			$\omega_{\theta,\textrm{elb}}$ & 2  \\
			$\omega_{\theta,\textrm{hip}}$ & 2  \\
			$\omega_{\theta,\textrm{knee}}$ & 2  \\
			$\omega_{\theta,\textrm{ank}}$ & 2  \\
			$\omega_{\theta,\textrm{ls}}$ & 2  \\
			$\omega_{\theta,\textrm{cs}}$ & 2  \\
			$\omega_{\dot{\theta},\textrm{sh}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{elb}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{hip}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{knee}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{ank}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{ls}}$ & 1 \\
			$\omega_{\dot{\theta},\textrm{cs}}$ & 1  \\ \bottomrule
	\end{tabular}}
	\end{subtable}
	\label{tab:scalingweightingparams}
\end{table}
\paragraph{Precise point-reaching (RL)}
We employ a similar reward function to~\cite{haeufle2020muscles}:
\begin{equation}
    r = - \lambda_{1} (d - \log(d + \epsilon^{2})) - \frac{\lambda_{2}}{N}\sum a_{i}^{2} - 2,
\end{equation}
where $d$ is the Euclidean distance between end effector and target position, $\epsilon=10^{-4}$ prevents numerical instabilities, $\lambda_{1}=0.1$ and $\lambda_{2}=10^{-4}$. A smaller distance $d$ increases the overall reward, but in contrast to the usual Euclidean distance, the $\log$-term increases rewards for very small distances even further, incentivizing precision.
The episode does not terminate until a time limit of 1000 steps elapses.

\paragraph{Fast point-reaching (RL)}
This task is identical to the previous one, but, in addition to the time limit, the episode also terminates if the distance between end effector and target position is below $5$ cm, which incentivizes reaching speed over precision.
\paragraph{Hitting a ball with a high velocity (OC/MPC)}
A ball with a mass of $250$ g is dropped in front of the arm model and the controller learns to hit the ball with a high velocity by optimizing the following objective:
\begin{equation}\label{eq:ballhitting}
		\varepsilon = -\max{\dot{z}_\text{ball}},
\end{equation}
where $\dot{z}_\text{ball}$ denotes the ball-velocity in z-direction (direction of gravity).
\paragraph{Squatting (OC/MPC)}
The objective for squatting is given by:
\begin{equation}\label{eq:squatting}
	\varepsilon = \frac{\omega_i}{{S_\mathrm{i}}}(\theta_\mathrm{i}-\theta_\mathrm{i}^\mathrm{des})^2+\frac{\omega_i}{{S_\mathrm{i}}}(\dot{\theta}_\mathrm{i}-\dot{\theta}_\mathrm{i}^\mathrm{des})^2,
\end{equation}
where $\theta_\mathrm{i}$ denotes the joint angle, $\dot{\theta}_\mathrm{i}$ the joint angle velocity. $\omega_i$ and $S_\mathrm{i}$ are weighting and scaling parameters, respectively. Their values are given in Table \ref{tab:scalingweightingparams}. The scaling parameters were chosen based on measured upper limits for human joint angular velocity \citep{jessop2016maximum} and human joint angle limits (Table 2 in \citep{darus-2982_2022}). The movement duration in this task was set to $0.9$ s. This squatting objective is taken from \cite{walter2021geometry}, where the desired hip $\theta_\text{hp}^\mathrm{des}$, knee $\theta_\text{kn}^\mathrm{des}$ and ankle $\theta_\text{an}^\mathrm{des}$ joint angle are defined to be:
\begin{align*}
	\theta_\text{an}^\mathrm{des} &= -20^{\circ}, \\
	\theta_\text{kn}^\mathrm{des} &= \sin^{-1}(-\frac{L_s}{L_t} \cdot \sin(\theta_\text{an}^\mathrm{des}))-\theta_\text{an}^\mathrm{des}-\theta_{\text{an,}0}, \\
	\theta_\text{hp}^\mathrm{des} &= -\theta_\text{kn}^\mathrm{des}-\theta_\text{an}^\mathrm{des}. \\
\end{align*}
 \paragraph{High-Jumping (OC/MPC)}
 The objective for the high-jumping is taken from \citep{pandy1990optimal} and maximizes the position and velocity of the centre of mass of the human body model at the time of lift-off $t_l$. Additionally, we slightly expanded this objective to account for the three-dimensionality of our jumping model by penalizing deviations of the centre of mass in the $x$ and $y$-direction:
 \begin{equation}\label{eq:high-jumping}
	\varepsilon = z_\text{com}(t_l) + \frac{\dot{z}_\text{com}^2(t_l)}{2g} - |(x_\text{com}(t_l)-0)| - |(y_\text{com}(t_l)-0)|.
\end{equation}
Note, that $z_\text{com}$ denotes the centre of mass position (CoM) in z-direction (direction of gravity). The model is initialized to start from a squatting position in this task.

\paragraph{Hopping (RL)}
We developed a reward function that is able to induce hopping in different leg-driven systems and can be applied independently of the actuator morphology. We did not obtain good results with height-based rewards or the gym hopper~\cite{gym} reward function.
The reward for hopping is given by:
\begin{equation}
r = \exp(\mathrm{max}\{0, \hat{v}_{z}^{\mathrm{COM}}\}) - 1,
\end{equation}
where $v_{z}^{\mathrm{COM}}$ is the z-velocity of the center of mass. The transformation $\hat{v} = \mathrm{min}\{10, 100\,v\}$ adjusts the sensitivity of the reward function while also preventing numerical overflows of the exponential function. Crucially, large positive velocities are weighted much more strongly than small or negative velocities, driving the system to maximum height periodic hopping. The second term prevents positive rewards for velocities close to zero, as $\exp(0) = 1$.
We additionally use regularizing cost terms:
\begin{equation}
    r_{\mathrm{reg}} = r_{\mathrm{alive}} - \lambda_{1}\,r_{\mathrm{action}} - \lambda_{2}\, r_{\mathrm{joint}},
\end{equation}
where $\lambda_{1} = 10^{-4}$, $\lambda_{2}= 10^{-3}$, $r_{\mathrm{alive}}$ is $1$ if the episode does not terminate and $0$ otherwise, $r_{\mathrm{action}}=\sum a_{i}^{2}/N$ punishes large actions and $r_{\mathrm{joint}}$ punishes joint angles close to the limits of the system. Specifically:
\begin{equation}
    r_{\mathrm{joint}} = \begin{cases}
        -1, & \text{if } |q_{\mathrm{max},i} - q_{i}| < 0.1\\
        -1, & \text{if }|q_{\mathrm{min},i} - q_{i}| < 0.1\\
         0, & \text{otherwise}.
    \end{cases}
\end{equation}
Finally, we terminate the episode after the lapse of a time limit of 1000 iterations, or if different parts of the model are very close to the ground, as this indicates a fall. The termination conditions are:
\begin{align*}
    h_{\mathrm{skull}} &< 0.3 ~\mathrm{[m]}\\
    h_{\mathrm{pelvis}} &< 0.2~\mathrm{[m]}\\
    h_{\mathrm{tibia_l}} &< 0.3~\mathrm{[m]}\\
    h_{\mathrm{tibia_r}} &< 0.3~\mathrm{[m]}\\
    \theta_{\mathrm{torso}} &> 1.22~\mathrm{[rad]}\\
    \theta_{\mathrm{torso}} &< -0.88~\mathrm{[rad]},
\end{align*}
where $h$ is the height of the respective body part and $\theta_{\mathrm{torso}}$ marks the torso angle deviation from the upright position.
 \section{Additional experiments (RL)}
 \label{supp:additional}
\subsection{Corrected hopping experiments}
We present corrected results in \fig{fig:supphoppingnew} for the hopping experiments in the main manuscript.
\begin{figure}
    \centering
    {\hspace{1.2cm}\small\textcolor{red}{\rule[2pt]{10pt}{1pt}} muscle \quad \textcolor{ourblue}{\rule[2pt]{10pt}{1pt}} torque}\\
    \includegraphics[width=0.45\textwidth]{figures/bipeds_hopping_new.pdf}
    \caption{The hyperparameter optimization for hopping was not available at the time of submission. The original and the corrected data show the same trend: Muscle actuators lead to faster initial learning and less variance, while torque actuators present larger asymptotic performance.}
    \label{fig:supphoppingnew}
\end{figure}
\subsection{Maximum force variation}
\label{supp:forcevariation}
Although the maximum force of the actuators is not freely adjustable in real systems, it is trivial to do so in simulation and has a strong influence on performance. We, therefore, used the biped parameters resulting from our hyperparameter optimization and recorded learning curves for the hopping task for both actuator morphologies for different maximum actuator forces. For each setting, we set the maximum isometric force for all muscle actuators to a certain value, trained the systems to convergence, and then recorded the torque values occurring at each controllable joint during execution of the hopping behavior. We then trained torque-actuator policies while setting $\tau_{\mathrm{max}}$ to the previously observed maximum values for \textit{each} individual joint. The results are shown in \fig{fig:forcevariation_hopping}. Even though singular torque-driven runs are able to outperform all muscle-driven runs at the end of training, this not only takes a considerable number of learning iterations, but also comes at the cost of strong learning instabilities. Looking at the learned behaviors, the torque-driven policies tend to jump very high, but violate the allowed torso-angles at the peak due to their unstable explorative policies. No periodic hopping could be observed. The muscle-driven policies, on the other hand, achieve periodic hopping, even though the apex hopping height is smaller.
 \begin{figure}
\centering
{\hspace{1.5cm}\small\textcolor{red}{\rule[2pt]{10pt}{1pt}} muscle \quad \textcolor{ourblue}{\rule[2pt]{10pt}{1pt}} torque}\\
\includegraphics[width=1.0\textwidth]{figures/bipeds_hopping_force_variation.pdf}
	\caption{\textbf{Hopping performance for different actuator strengths.} Hyperparameters are optimized for hopping with a maximum muscle strengh of $F_{\mathrm{max}}=5000$ N as used in the previous hopping experiment. The maximum isometric muscle force is set to different values and the policies are trained for the task. Afterwards, the maximum used torques for the learned behaviors are recorded for each joint and set to identical values for the torque actuator. Muscle-actuators lead to more consistent performance and yield periodic hopping. Torque-actuators yield unstable policies that manage to jump very high once, but terminate the episode due to falls.}
\label{fig:forcevariation_hopping}
\end{figure}
\subsection{Additional goals for point-reaching with perturbations}
\label{supp:moregoals}
We show ten random arm goals for precise point-reaching with perturbations that were not present during training in \fig{fig:suppmoregoals}.
\begin{figure}
\centering
\includegraphics[width=1.0\textwidth]{figures/robustnessarm26_ee_all.pdf}
	\caption{\textbf{Trajectories for dynamic ($\mathbf{1.5}$ kg weight) and chaotic (attached ball) load.} Left: The torque actuator handles the dynamic load case slightly better than the muscle actuator for all goals, especially compared to the goals in grey, brown and purple. Right: The muscle-actuator performs very well for all chaotic load goals, except for a small deviation from the end-point. The torque actuator exhibits strong instabilities. The respective goal positions are marked as circles, the unperturbed baseline for each goal is shown with a dashed line, the perturbed trajectories with slightly transparent solid lines.}
\label{fig:suppmoregoals}
\end{figure}
\subsection{Additional hyperparameter variations}
We show the relative performance of all runs of the hyperparameter searches in polar coordinates for precise point-reaching and hopping for both actuator morphologies (\fig{fig:supphyperpararlarm} and \fig{fig:supphyperpararlbiped}). The angles mark the specified hyperparameter (see \supp{supp:expdetails} for definitions), while the radius marks the chosen value in $\log_{10}$-coordinates. The top row marks performance with muscle-actuators, the middle row with torque-actuators and the bottom row shows histograms of returns for both morphologies at different iterations. For point-reaching, muscle morphology leads to a return distribution that is centered around the top-performing parameter sets, with almost no badly performing sets left at iteration 7. In contrast, for torque-morphology a large number of runs is still distributed at low return values. For hopping, a much harder task, muscle-morphology quickly leads to a large number of runs at the top-performance level, while some badly performing parameter sets remain even at iteration 5. For torque-morphology, a large peak can be observed for returns close to 0, as most sampled parameter sets do not achieve any kind of hopping. Only at iteration 7, a few singular well-performing runs appear, that strongly outperform even the best muscle-driven run. This was to be expected, as any muscle-actuator behavior can in principle be replicated by torque actuators, given that the policy is able to learn it. Muscle actuators, on the other hand, are restricted to trajectory-dependent output.
\begin{figure}
\centering
\begin{subfigure}{1.0\textwidth}
\includegraphics[width=1.0\textwidth]{figures/radarplot_torque.pdf}
\end{subfigure}\\
\begin{subfigure}{1.0\textwidth}
\hspace{0.23cm}
\includegraphics[width=1.0\textwidth]{figures/percentage_goodness_version2.pdf}
\end{subfigure}
	\caption{\textbf{Hyperparameter variation for precise point-reaching.} Hyperparameters are optimized following an iterative sampling scheme and individual runs train for $2\times 10^{6}$ iterations. Fifty sets of parameters are sampled randomly from pre-determined distributions, the final performance is evaluated and used to adapt the sampling distributions for the next iteration. We record 7 iterations which equals 350 runs in total. We optimize 5 parameters related to MPO. The angle of the radarplot marks the parameter, the radius marks the value (in $\log_{10}$-coordinates). Top: Radarplot of parameters for the muscle in precise point-reaching at iteration 1, 2 and 7. The color marks the achieved performance of the parameter sample relative to the best achieved performance over \textbf{all} sampled parameters. Middle: precise point-reaching torque. Bottom: Histogram of returns for all sets of parameters at iterations 1, 2 and 7.}
	\label{fig:supphyperpararlarm}
\end{figure}
\begin{figure}
\centering
\begin{subfigure}{1.0\textwidth}
\includegraphics[width=1.0\textwidth]{figures/radarplot_torque_bipeds.pdf}
\end{subfigure}\\
\begin{subfigure}{1.0\textwidth}
\hspace{0.24cm}
\includegraphics[width=1.0\textwidth]{figures/percentage_goodness_bipeds.pdf}
\end{subfigure}
	\caption{\textbf{Hyperparameter variation for hopping.} Hyperparameters are optimized following an iterative sampling scheme and individual runs train for $5\times 10^{6}$ iterations. Twenty sets of parameters are sampled randomly from pre-determined distributions, the final performance is evaluated and used to adapt the sampling distributions for the next iteration. We record $5$ iterations which equals 100 runs in total. We optimize 5 parameters related to MPO. The angle of the radarplot marks the parameter, the radius marks the value (in $\log_{10}$-coordinates). Top: Radarplot of parameters for the muscle in the hopping task at iteration 1, 2 and 5. The color marks the achieved performance of the parameter sample relative to the best achieved performance over \textbf{all} sampled parameters. Middle: hopping torque. Bottom: Histogram of returns for all sets of parameters at iterations 1, 2 and 5.}
	\label{fig:supphyperpararlbiped}
\end{figure}

redpajama_set_name: RedPajamaArXiv


SIA Awards is the only advertising awards to specifically recognize the advertising achievements in the service industry. Many other advertising awards are dominated by consumer goods, packaged goods and other tangible products. The service industry is adding a significant contribution to marketing and advertising.
SIAAwards recognizes the creativity and communication accomplishments of the service industry, recognizing the best service industry advertisement, print communications, websites, TV Radio and electronic communications, and social media.


"Estimation of Subpixel Land Surface Temperature Using an Endmember Ind" by Guijun Yang, Ruiliang Pu et al.
Land surface temperature (LST) is a key parameter in numerous environmental studies. Surface heterogeneity induces uncertainty in estimating subpixel temperature. To take an advantage of simultaneous, multi-resolution observations at coincident nadirs by the Advanced Spaceborne Thermal Emission Reflection Radiometer (ASTER) and the MODerate-resolution Imaging Spectroradiometer (MODIS), LST products from the two sensors were examined for a portion of suburb area in Beijing, China. We selected Soil-Adjusted Vegetation Index (SAVI), Normalized Multi-band Drought Index (NMDI), Normalized Difference Built-up Index (NDBI) and Normalized Difference Water Index (NDWI) as representative remote sensing indices for four land cover types (vegetation, bare soil, impervious and water area), respectively. By using support vector machines, the overall classification accuracy of the four land cover types with inputs of the four remote sensing indices, extracted from ASTER visible near infrared (VNIR) bands and shortwave infrared (SWIR) bands, reached 97.66%, and Kappa coefficient was 0.9632. In order to lower the subpixel temperature estimation error caused by re-sampling of remote sensing data, a disaggregation method for subpixel temperature using the remote sensing endmember index based technique (DisEMI) was established in this study. Firstly, the area ratios and statistical information of endmember remote sensing indices were calculated from ASTER VNIR/SWIR data at 990 m and 90 m resolutions, respectively. Secondly, the relationship between the 990 m resolution MODIS LST and the corresponding input parameters (area ratios and endmember indices at the 990 m resolution) was trained by a genetic algorithm and self-organizing feature map artificial neural network (GA-SOFM-ANN). Finally, the trained models were employed to estimate the 90 m resolution subpixel temperature with inputs of area ratios and endmember indices at the 90 m resolution. ASTER LST product was used for verifying the estimated subpixel temperature, and the verified results indicate that the estimated temperature distribution was basically consistent with that of ASTER LST product. A better agreement was found between temperatures derived by our proposed method (DisEMI) and the ASTER 90 m data (R2 = 0.709 and RMSE = 2.702 K).
Yang, Guijun; Pu, Ruiliang; Zhao, Chunjiang; Huang, Wenjiang; and Wang, Jihua, "Estimation of Subpixel Land Surface Temperature Using an Endmember Index Based Technique: A Case Examination on ASTER and MODIS Temperature Products Over a Heterogeneous Area" (2011). School of Geosciences Faculty and Staff Publications. 364.


Was Clifford here last night?
I'd like a Bloody Damone.
It looks like Naren has fallen back to sleep.
Dwayne was lying flat on his back on a hospital bed.
Why haven't you told Roland when and where the meeting will be?
Huey drove straight to school.
I don't believe everything Morgan tells me.
Ninja said you had a heart attack last year.
Konstantinos has been there three times.
Jarvis, are you in here?
Jack has got to be at least thirty years old.
Jacques called out to John and asked him to turn himself in.
I don't think Tran will ever forgive me.
Jimmy realized that he might be in danger.
Christophe was a trusted friend.
I know you don't want to disappoint Rudolf.
Bernard called Sharan a liar.
Clare was trying to appear calm.
Dannie has a degree in biology.
I think you should find out when Meehan plans to leave.
Just because Ravindranath said he didn't have a key doesn't mean it's true.
What makes Bernard so special?
Elaine is a chemistry professor.
Nick is in a meeting, but Stefan's in her office.
Gregg set up a company recently.
Are you sure Debbie can do it?
Major is wearing low-rise jeans.


The British tabloid The Sun has released a story that says that Finnish singing sensation Saara Aalto has been offered the role of the "fifth Spice Girl" in the reunion tour of the superstar British pop group The Spice Girls.
"With party pooper, Victoria Beckham refusing to be part of the Spice Girls reunion tour this summer, massive stars from Katy Perry to Adele have been touted to join the band on stage in place of Posh. But it turns out Mel C has already offered the gig — for at least part of the summer — to Saara Aalto," the tabloid writes.
In a video linked to the piece, Aalto asks Mel C if she could join the group as the sixth Spice Girl, to which 'Sporty Spice' replied that "you could be the fifth Spice Girl if you want."
The Sun reporter says Saara considers herself "a Spice Girls superfan" and is very excited about the prospects of joining at least some of their concerts.
"They want to have a fifth member for the tour. I was like, 'Is this a real question?' It's insane. Posh was my favorite Spice Girl but she didn't have that many lines to sing," Aalto told the tabloid.
Four of the original five Spice Girls are planning a reunion tour this summer that would include six stadium performances. Saara Aalto would appear in at least part of them, The Sun reports.
Aalto became a major star in the UK after coming in second in Britain's X-Factor talent show. After this she represented Finland at the 2018 Eurovision Song Contest.


When did you start fishing? What is your angling earliest memory?
I started fishing local farm ponds and neighborhood lakes when I was about five. I fished with my older brothers. My earliest fishing memory is one of catching bluegill on wax worms under a small cork bobber. It was on Mr. Beringer's property. I thought it was the greatest thing in the world. To this day, I think there is something satisfying about watching a bobber dance and then dive under.
I'm 39 years old and I'm married to Kimberly. My wife and I have two boys; Braden, age 5 and Cole, age 3. I currently live in Cincinnati. When I'm not working as a supervisor for Cincinnati Bell I make jigs and spinnerbaits, and I hand-pour special soft-plastic baits designed with regional conditions and species in mind. Of course I also tie flies and build rods. I fish from my Poke Boat, which I've had for a dozen years.
You're known as, shall we say, a "committed" angler. When did the bug really bite you?
I became an avid angler as I entered seventh grade. By high school I was fully engaged. I used to Caddy at Western Hills Country club in the summer, and as I earned some money I upgraded my equipment, and started collecting lures for all types of fishing. I began to get species-specific as my knowledge increased. I compiled tackle and rods for bass, panfish, Ohio river fishing, and rough fishing. I bought my first fly rod during my freshmen year of high school-- it was a Martin 6wt that I purchased from Swallen's Department Store. I still have that rod and use it today for Float and Fly fishing! I used that rod to pester Bream in ponds and rock bass in some local creeks and I think that is when I really developed the passion.
How often are you on the water? What fish do you chase the most?
With work and family obligations I don't get out as much as I used to. I once fished thirteen states in one year, but now I get out around 75 days a year. I am lucky that my wife is from the Virginia and her family lives smack-dab in an area that has fine fisheries for everything from trout to smallmouth to musky. I incorporate fishing with weekend mini-vacations.
It's hard for me to pick a favorite fish. I really like so many species and techniques. Locally I fish smallmouth the most, but the Ohio river fishery is growing on me! White Bass and bad-arse Hybrids RULE!! I recently caught my first Mad River trout. It's not a stretch to say I'll be doing more of that. I dream about pig-beast smallies from the New in VA, but I also love to take my two weight fly rod to harass the local panfish populations. I consider myself to be a versatile angler and am well-versed in most fresh water applications.
How did Buckeye Baits get started?
Buckeye Baits started out of necessity. I was young and short of funds. I needed bass baits for fishing some creeks. The local stores had limited selections and I couldn't afford to do a ton of mail orders, so I started making my own. My first Buzzbait that I made was when I was 15. I ordered the parts from Jann's Netcraft out of Toledo, Ohio. My home-made lures caught fish, and I started getting requests from my brothers and friends to make some from them. Word of mouth spread and I now have baits in the tackle boxes of folks all over the country. It's a labor of love.
As with most of us, you are as quick to grab a spinning or casting rod as you are a fly rod when the conditions demand. Do you have a favorite technique?
My favorite technique is top-water. Whether it's the sound of a gurgling Buzzbait on a stream, or tossing a size 20 Para-Adams to a current seam on a Mountain Trout stream; I love the visual take! I consider myself to be a jig expert as well.
Buckeye Baits is known for spinner baits, buzz baits, jigs and soft plastics geared towards the stream fisherman. Does your intimate knowledge of hardware design influence how you fly fish and which flies you use?
I do take components of spin lure design and incorporate them into my pattern selection. For example, I tie a fly rod jig-n-pig that suggests a bottom critter much better than a wooly bugger or Clouser. Those are both top-shelf patterns, but they are not as strong in the pre-spawn when the big females start to stage and a bigger, slower, deeper presentation is required. My familiarity with the rivers and creeks helps me select the best and most efficient pattern for any given day. Another example is when I'm targeting panfish. I love to take an ultralight spinning rod and toss tiny tube jigs. I tie up an imitation that matches that local bait and it is a real fly rod killer.
Both Tom Nixon and Jack Ellis espouse the use of soft plastics with a fly rod for a true "finesse" presentation. Have you fished your soft plastics using a fly rod? Is this something you're interested in?
I am starting to experiment more with soft plastics on the long rod, but I'm still a novice. I think it's a super idea and worthy of experimentation. To me, I can't see why it's such a "problem" for purists. If the object is to enjoy your time on the water and catch fish, it's just the logical progression of the sport. I'll be trying it out on panfish and bass trips this spring. Pouring my own plastic allows me the unique opportunity to try some unconventional baits. I'm looking forward to pushing the envelope.
What emerging trends do you see occurring in the bass fishing community. What's the hot technique?
I think more future specialization will be seen in the fly fishing community. This trend is evidenced by the explosion in species specific fly lines. There's a true niche market out there into which companies can expand. Pushing this envelope will continue as more fly anglers pursue carp and catfish on the fly. And the steelhead crowd writes new chapters on the sport every season!
I think rod manufactures realize that the "cookie cutter approach" of their core selection can be effectively augmented with more specialized rods. For example; on the spinning rod side there is an explosion in float&fly rods. The same thing is evident with the G. Loomis Bronzeback series. The companies realize that there are new markets out there. There are new revenue streams and new businesses being associated with this metamorphosis of fishing technology. This technology will lead the charge into the future starting with new rods, reels, lines, and tippet materials.
Another important trend is outreach to non-conventional fly fishing population segments. We are seeing more women, more diversity, more new cultures assimilating the ethos and the lifestyle of outdoor sports. As the sport of angling ages, new folks will be introduced to the "quiet sport." And that's a good thing! Of course, the downside is that may start a trend where there will be more pay-to-play water. We're seeing that all over the country, and that alarms me a bit.
The hot bait in fly fishing right now the gummy minnow. It's popping fish everywhere! Kreel Tackle's Spoon and Baitfish are also making names for themselves. Soft hackles are making a comeback. The hot technique most applicable to fly rodders is float&fly. It's "en fuego" all over.
What fish does Matt Frondorf chase in his dreams?
Click on the Images Below for a Full Size Picture!
" Low and slow" is the way to go with this bait. Lift, drop, tick....... FISH ON!
A closer view of a bass fisherman's bread and butter bait.
What fish can resist such a face?
Ring the dinner bell - this madtom/sculpin/waterdog imitator is a big fish favorite!
Keep the heart medication close at hand, fish will absolutely detonate on this one!
A "must-have" for creek smallmouth fishermen.
Hand-poured custom colors are a Buckeye Baits specialty!
Contact Matt for recommendations about what's hot for the species you're after!


text:
What's Behind The Latest Move To Kill The Plaza Project
Is this the beginning of the end for Yakima's Downtown Plaza? I hope not. We deserve this great economic development tool and community "living room". I'm not giving up but there is now a pretty big hill to climb. Let's review recent history that lead to today's radio program.
A year ago plaza proponents were told that if they raised 9-million dollars without state and federal grants that the project would proceed. The community responded and by year's end the giant and unprecedented fundraising task was accomplished. Full speed ahead, right? Wrong.
Back on March 6th, assistant mayor Dulce Gutierrez made a motion to kill the project. So much for that promise to move ahead huh? Why the drastic change madam assistant mayor? What motivated such a community betrayal? What made you decide to go back on your word? Oh, we'll get to that.
At that time council members Jason White and Holly Cousens voted to join Gutierrez to end the 5+ year old project with one million, two-hundred thousand City dollars already invested. Council members Kathy Coffey, Carmen Mendez and Brad Hill voted to honor their obligation and continue the project. Swing vote is Council member Dr. Kay Funk. She said she had some concerns but that the project was too far along and it wouldn't be fair to terminate it. Motion to kill the plaza fails 4 to 3. Full speed ahead, right? Wrong.
On June 19th councilman Jason White offered a motion to put the future of the plaza up to a public vote -in yet another effort to kill it. Cousens and Gutierrez dutifully got in line with White in favor of a vote and Coffey, Mendez and Hill lined up again against. Swing vote again is Dr. Funk. What would she do? Would she hold true to her word - that the project was too far along to stop--that fairness and respect mandated it be allowed to move ahead? OR would she change her mind and join the council plaza haters with a "put it on the ballot" vote? Full speed ahead, right? Wrong.
Tonight the city council considers a measure to put the plaza up to a public vote. 3 of the 4 council members who voted to put the plaza on the ballot hope the vote kills the project. Jason White, Holly Cousens and Dulce Gutierrez say they have their reasons and all are tied to lack of money but are they really? We'll use their own words to explore that in the audio that follows. See if you think their reasoning hold up.
The 4th vote to put it to a vote belongs to Dr. Kay Funk. Three months ago she voted to let the project live with the prospect that is could be successful and "wonderful". Now, three months farther down the road from that, she voted to join those who want to kill it with a public vote. What changed? What's going on? What's the truth?
Hear for yourself. The comments are from the March 6th meeting to kill the project as well as the June 19th vote where Ms. Funk changes her mind. I would love to hear your thoughts.
But I would rather you show up at council tonight and demand to know why a mutually agreed to, priority city initiative got sold out in this fashion? Demand to know how council can be believed on anything moving forward. And ask council if a lack of money is really the issue, because if it is, why spend 50-thousand on a public vote when a new Funk motion does it for free? Why search for 35-thousand dollars to fund an Eastside pool project design that will ultimately a cost millions to build and that will promptly fail to pay for itself -as all pools do- triggering additional tens of thousands of dollars in maintenance and support? Consider that a pool costs 3-5 million or more and serves perhaps a few thousand people for a 3 month period. While the Plaza costs 1.8 million and serves everyone in Central Washington year round, has the potential to pay for itself like other plaza projects do and is designed to act a lure for up to 70 -million more in economic development. IF money is really the issue, how does this make sense?
If you missed KIT's radio show this morning, then you really should listen to this. And then do the informed and responsible thing.
Filed Under: city council, plaza
Categories: Featured, News, Politics, Yakima News

redpajama_set_name: RedPajamaCommonCrawl


i am exhibiting at the World Trade Center, Cuffe Parade tomorrow: 7th of November and day after: 8th of November 2012!!!! the exhibition time is from 11 am to 7 pm. please come, bring friends and family, shop lots and make it a success...i will not be able to do this without you!!
p.s. yes i screwed up the spellings of bedazzled above....ahh you didn't notice....but its just one of those words that begs to be misspelled!!


detached house, 2 people, 1 room/s, 1 bathroom/s, approx. 30 m², 1 floor/s. Please note: Non-smoking house. Furnishing: friendly, bright, parquet floor. Equipment: satellite TV; internet access Wi-Fi (included). Room layout: Living room/bed room: sofa, double bed. Dining area: in the living room. Kitchenette: in the living room. Kitchen/ cooking equipment: 2 electric plates, refrigerator, coffee machine, water boiler. Sanitary 1: shower, WC.


In light of Papa John's Pizza CEO John Schnatter's intention to cut his employees hours so that he doesn't have to provide them with health care (at a cost of less than 15 cents a pizza), conservatives are calling Friday, "National Papa John's Appreciation Day."
It's also being called, "National Make Uninsured Minimum Wage Employees Work Twice As Hard While You Protest Their Right to Health Care and Hope They Don't Spit In Your Food Day."
Some people are calling it, "Help Papa John Schnatter Recoup the Millions He Wasted on Romney Campaign Contributions Day."
It's kind of like election day all over again, except this time people are voting with their wallets.
You can take a stand for free enterprise by giving your business that day to a local restauranteur, who would no doubt put the profits back into the community as opposed to stashing them in the Cayman Islands.
Maybe I'll go to Square Pizza on Friday ... they've got awesome old-school video games.
I appreciate Papa John because he makes me feel better about myself.
Well never did like Pappa John pizza and now have a very good reason to never try it again.
But afraid we are going to see a lot of this crap. Begining to think we are going to see the so call full time employee become a minority. Yup sad chain of events that is being sold to use as making things better is in fact just going to make them worse.