\section{Introduction}
With the growth of the Internet of Things (IoT), billions of devices are connected through wireless networks, which puts great challenges in satisfying the strict quality of service (QoS). Among the strict QoS, high energy efficiency and ultra-low latency are the important goals in the present 5G and future 6G [1, 2]. In particular, to achieve high energy efficiency and ultra-low latency simultaneously, the cross-layer design is a significant way. To this end, in our work, we focus on minimizing average power for data transmission with the constraint of average delay through the cross-layer design.

In existing works [3-9], the cross-layer design has been widely studied and implemented as a significant way to satisfy the strict QoS through scheduling and resource allocation. More specifically, works [3-6] focused on the optimal scheduling of data transmission under the cross-layer framework. In particular, the Collins and Cruz proposed the pioneering work [3] on cross-layer design to minimize average power for data transmission over a two-state fading channel. In [4], through cross-layer design, Berry and Gallager studied the optimal delay-power tradeoff in the regime of asymptotically large delay. After this, Berry also studied the optimal delay-power tradeoff in the regime of asymptotically small delay in [5]. Authors in [6] proposed a lazy scheduling algorithm to achieve energy-efficient transmission. There are also some works focused on the resource allocation and management with cross-layer design [7-9]. In [7], the power and bandwidth allocation policy were studied in the scenario of Ultra-Reliable and Low-Latency (URLLC) to minimize the power for data transmission under the QoS constraint. The optimal energy allocation was studied in [8] to maximize the throughput with the consideration of energy harvesting. In [9], the energy allocation and management were studied in satellites communications through a dynamic programming approach.

In our previous work [10-12], we studied the optimal delay-power tradeoff and its corresponding scheduling policy through the cross-layer design. More particularly, we first proposed the probabilistic scheduling to minimize average delay under the constraint of average power in a simple scenario with the two-state block-fading channel and fixed modulation [10]. Based on the probabilistic scheduling proposed in [10], in the follow-up works, we considered the optimal delay-power tradeoff in more generalization and complex scenarios. We considered  buffer aware scheduling with adaptive transmission in [11] and multi-state block-fading channel in [12]. However, in the practical scenario, the channel energy gain fluctuates in a continuous range. If we model the state of channel as multiple discrete states, the practical channel state information will not be accurately described.

To make full use of channel state information, in this paper, we study the optimal cross-layer scheduling based on the continuous channel states. Fortunately, the optimal scheduling can be converted to a variational problem. Based on this, we construct a solution within the feasible region of the variational problem. The scheduling policy corresponding to the constructed solution is shown to be the deterministic policy. Meanwhile, the average power corresponding to the constructed solution is shown to approximate the achievable minimum power with an arbitrarily small error. Therefore, based on the constructed solution, we finally show the optimality of deterministic cross-layer scheduling policy. This conclusion greatly compresses the search space for finding the optimal policy.

\section{System Model}

Consider a communication link that a transmitter sends data packets to its receiver over a block fading channel as shown in Fig. 1. The data transmission is scheduled according to the information of the discrete data queue state and continuous channel state under the cross-layer design. The aim of scheduling is to minimize average power for data transmission with the constraint of average delay.

In the network layer, data packets arrive at the transmitter randomly at the beginning of each time slot. Let $a[n]\in \mathcal{A}$ denote the number of data packets arriving in time slot $n$, where $\mathcal{A}=\{0, 1, 2, ..., A\}$. In particular, the parameter $A$ is the maximum number of data packet that can be arrived in one time slot. To capture the practical scenario, we allow that $a[n]$  could follow any discrete probability distribution. Therefore, the probability distribution of $a[n]$ can be given by
 \begin{equation}
\Pr\{a[n]=k\}=\alpha_{k},
 \end{equation}
 where $\alpha_{k}\geq0$ and $\sum_{k=0}^{A}\alpha_{k}=1$. Thus, the average arrival rate of data packets $\overline{a}$ can be obtained as
 \begin{equation}
\overline{a}=\sum_{k=0}^{A}k\alpha_{k}.
 \end{equation}

 In the data link layer, the arrived data packets will be backlogged in the data queue if they are not sent out immediately. We let $Q$ denote the buffer size of data queue. It means that if the number of packets in the data queue exceeds $Q$, the extra packets will be lost. Let $q[n]\in \mathcal{Q}$ denote the data queue length at the beginning of time slot $n$, where $\mathcal{Q}$ is the discrete space that can be expressed as $\{ 1, 2, ..., Q\}$. We implement $q[n]$ to indicate the state of data queue for scheduling of data transmission. The update of $q[n]$ is given by
  \begin{equation}
   q[n+1]=\min\{\max\{q[n]-s[n], 0\}+a[n+1],Q\},
 \end{equation}
 where $s[n]\in\mathcal{S}$ is the number of packets transmitted in time slot $n$, and $\mathcal{S}=\{1, 2, ..., S_{max}\}$. The $S_{max}$ indicates the maximum number of data packets that can be transmitted in one time slot corresponding to the transmitter's capacity for data transmission. We assume that $S_{max}\geq A$ to avoid packet loss caused by overflow of data queue.

In the physical layer, the data packets are transmitted over the block fading channel. We assume that channel energy gain stays invariant during each time slot, and follows an independent and identically distributed ($i.i.d.$) fading process across the time slots. In practice, the channel energy gain is a value that fluctuates in a continuous interval. Let $h[n]$ denote the channel energy gain in time slot $n$. To capture the practical scenario without loss of generality, we assume that $h[n]\in \mathcal{H}$ obeys a probability density function $f_{H}(h)$, where $\mathcal{H}=(h_{min}, h_{max}]$, and $f_{H}(h)$ is a continuous bounded function on $\mathcal{H}$. Therefore, the probability distribution of $h[n]$ can be given by
 \begin{equation}
\Pr\{h[n]\leq x\}=\int_{h_{min}}^{x} f_{H}(h)  dh.
 \end{equation}
  \begin{figure}
   \centering
   \includegraphics[width=0.5\textwidth]{system1.eps}

   \caption{System Model.}
   \label{fig.pathdemo4}
   \end{figure}
The energy consumption for data transmission in time slot $n$ is determined by the data transmission rate $s[n]$ and channel energy gain $h[n]$. Let $\xi(s)h^{-1}$ denote the energy consumption in one time slot when transmission rate is $s$ and the channel energy gain is $h$. The $\xi(s)$ is assumed to be a strictly monotonic increasing function, since more data transmission requires more energy. For example, $\xi(s)=2^{s}-1$ is widely used, which can be derived based on channel capacity.

Our aim is to minimize the average power for data transmission while satisfying that the average delay is below a certain threshold $D_{th}$.

\section{The Deterministic Policy is Optimal}
In this section, we first present the probabilistic scheduling that can cover all feasible stationary policies for data transmission. Then the minimization of average power for data transmission can be formulated as a variational problem. Based on this variational problem, we show the optimality of deterministic policy through a constructive proof.

The probabilistic scheduling can be expressed by a set of scheduling parameters $\{f(s|q,h)\}$. The parameter $f(s|q,h)$ denotes the probability of transmitting $s$ data packets when the data queue length is $q$ and channel energy gain is $h$. Let $g(q, s, h)$ represent the steady state probability density of the system being in a state that the data queue length is $ q $, the transmission rate is $ s$, and the channel energy gain is $h$.

Based on the definition of $g(q, s, h)$, the scheduling parameters $\{f(s|q,h)\}$ can be represented by $g (q, s, h)$ as follows,
 \begin{equation}
 f(s|q,h)=\frac{g (q, s, h)}{\sum_{x=0}^{S_{max}}g (q, x, h)}.
 \end{equation}
 Furthermore, the average delay $D_{ave}$ and average power $P_{ave}$ can also be represented by $g (q, s, h)$. The average delay is associated with average data queue length. The average data queue length can be obtained as
 \begin{equation}
\mathbb{E}\{q[n]\}=\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}q g(q, s, h)dh.
\end{equation}
According to Little's Law, the expression of $D_{ave}$ can be obtained as
 \begin{equation}
D_{ave}=\frac{1}{\overline{a}}\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}qg(q, s, h)dh.
\end{equation}
The average power for data transmission is associated with transmission rate and channel energy gain. Thus, the expression of $P_{ave}$ can be obtained as
 \begin{equation}
 P_{ave}=\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\xi(s)h^{-1}g(q, s, h)dh.
\end{equation}

In Eqs. (5), (7), and (8), the probabilistic scheduling parameters $\{f(s|q,h)\}$, the average delay $D_{ave}$, and the average power $P_{ave}$ are all represented by steady state probability density function $g (q, s, h)$, respectively. This allows us to use $g (q, s, h)$ as the optimization variable to minimize the average power as follows,
\begin{subequations} \label{10151408}
\renewcommand{\theequation}{\theparentequation.\alph{equation}}
\begin{align}
\min_{g(q,s,h)}\ & ~  \int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\xi(s)h^{-1}g(q, s, h)dh\\
 s.t.\ \ &~ \frac{1}{\overline{a}}\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}qg(q, s, h)dh\leq D_{th} \label{b} \\
& ~\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}g(q, s, h)=f_{H}(h)\ \ \ \forall h\in \mathcal{H}\label{c}\\
& ~\sum_{k=0}^{A}\sum_{s=0}^{S_{max}}\alpha_{k}\int_{h_{min}}^{h_{max}}g(q+s-k, s, h)dh=\notag\\ & ~ \sum_{s=0}^{S_{max}}\int_{h_{min}}^{h_{max}}g(q, s, h)dh\ \ \ \forall q\in \mathcal{Q}\label{c}\\
& ~ g(q,s,h)\geq 0 \ \ \forall q\in \mathcal{Q}, s\in \mathcal{S}, h\in \mathcal{H}\\
& ~ g(q,s,h)=0 \ \ \ \forall q\!-\!s<0\notag \\ & ~  \ \text{or}\ q\!-\!s>Q\!-\!A \ \text{or}\ q \notin \mathcal{Q}\ \text{or}\ s \notin \mathcal{S}.
\end{align
\end{subequations}

In problem (9), the objective function (9.a) is the average power, as shown in Eq. (8). The first constraint (9.b) is the constraint of average delay, as shown in Eq. (7). The second constraint (9.c) is the constraint that channel energy gain obeys the probability density function $f_{H}(h)$. The constraint (9.d) represents the state balance equation of the data queue state. The left side of the equation represents the summation of probability that the data queue state are transferred to the queue state $q$ at the next time slot, while the right side of the equation is the steady-state probability that the system is in the queue state $q$. Constraint (9.e) is the non-negative condition of probability. Finally, constraint (9.f) avoids the overflow or underflow of the data queue. However, problem (9) is a variational problem, since its optimization variable $g (q, s, h)$ is a probability density function.

Though, it is not a trivial work to solve problem (9), we may show a light on the structural properties of its optimal solution. Let $g ^{*}(q, s, h)$ denote the optimal solution of the problem (9). Based on $g ^{*}(q, s, h)$, we define a set of functions $\{U_{q}(x)\}$ as follows,
 \begin{equation}
U_{q}(x)=\int_{h_{min}}^{x}\sum_{s=0}^{S_{max}}g ^{*}(q, s, h)dh,
 \end{equation}
where $q\in\mathcal{Q}$. Then, we construct a solution $y_{M}(q, s, h)$ as follows
 \begin{equation}
y_{M}(q, s, h)=\sum_{k=1}^{M}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)\mathbb{I}\{h\in(h_{k,s,q}^{min}, h_{k,s,q}^{max}]\},
 \end{equation}
where $M\in\mathbb{N}$, $q\in\mathcal{Q}$, $h\in \mathcal{H}$. The definition of $h_{k,s,q}^{max}$ in Eq. (11) is given by
 \begin{equation}
 \begin{aligned}
 & ~h_{k,s,q}^{max}=U_{q}^{-1}(\int_{h_{min}}^{h_{k}^{min}}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)dh \\ & ~+\int_{h_{k}^{min}}^{h_{k}^{max}}\sum_{x=0}^{s}g ^{*}(q, x, h)dh),
  \end{aligned}
 \end{equation}
 where $U_{q}^{-1}(.)$ is the inverse of $U_{q}(.)$, and $h_{k}^{min}=h_{min}+\frac{(k-1)(h_{max}-h_{min})}{M}$, $h_{k}^{max}=h_{min}+\frac{k(h_{max}-h_{min})}{M}$. We also define $h_{k,-1,q}^{max}=h_{k}^{min}$. Then, the definition of $h_{k,s,q}^{min}$ in Eq. (11) is given by
 \begin{equation}
h_{k,s,q}^{min}=h_{k,s-1,q}^{max}, \ \ \ \forall \ \ 0\leq s \leq S_{max}.
 \end{equation}

 For the constructed solution $y_{M}(q, s, h)$ in Eq. (11), we first show that $y_{M}(q, s, h)$ is in the feasible region of problem (9). On this basis, we show that the scheduling policy corresponding to $y_{M}(q, s, h)$ is the deterministic policy. Finally, we show that the average power corresponding to $y_{M}(q, s, h)$ can approximate that of the optimal solution $g ^{*}(q, s, h)$ with an arbitrarily small error for the sufficiently large $M$. To sum up, the optimality of the deterministic scheduling policy can be shown.
   \newtheorem{lem}{\bf Lemma}
   \begin{lem}\label{lem1}
   The constructed solution $y_{M}(q, s, h)$ in Eq. (11) satisfies the constraint (9.c), i.e.,
     \begin{equation}
    \sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}y_{M}(q, s, h)=f_{H}(h)\ \ \ \forall h\in \mathcal{H}.
     \end{equation}
     \end{lem}
 \begin{proof}
 Based on Eq. (12), we obtain the $h_{k,S_{max},q}^{max}$ given by
  \begin{equation}
 \begin{aligned}
  h_{k,S_{max},q}^{max}&=U_{q}^{-1}(\int_{h_{min}}^{h_{k}^{min}}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)dh \\ & ~+\int_{h_{k}^{min}}^{h_{k}^{max}}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)dh)\\
              &=U_{q}^{-1}(\int_{h_{min}}^{h_{k}^{max}}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)dh)\\
              &=h_{k}^{max}.
   \end{aligned}
   \end{equation}

   Therefore, we have
    \begin{equation}
 \begin{aligned}
  \bigcup_{k=1}^{M}\bigcup_{s=0}^{S_{max}} (h_{k,s,q}^{min}, h_{k,s,q}^{max}]&=\bigcup_{k=1}^{M} (h_{k,0,q}^{min}, h_{k,S_{max},q}^{max}]\\
              &=\bigcup_{k=1}^{M} (h_{k}^{min}, h_{k}^{max}]\\
              &=(h_{min}, h_{max}].
   \end{aligned}
   \end{equation}
 According to Eqs. (11) and (16), we can simply $\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}y_{M}(q, s, h)$ as follows,
      \begin{equation}
 \begin{aligned}
 \sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}y_{M}(q, s, h)&=\sum_{q=0}^{Q}\sum_{x=0}^{S_{max}}\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\\ & ~g ^{*}(q, x, h)\mathbb{I}\{h\in(h_{k,s,q}^{min}, h_{k,s,q}^{max}]\}\\
              &=\!\!\sum_{q=0}^{Q}\!\!\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)\mathbb{I}\{h\!\!\in\!\!(h_{min}, h_{max}]\}\\
              &=\sum_{q=0}^{Q}\!\!\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)\\
              &=f_{H}(h).
   \end{aligned}
   \end{equation}

 \end{proof}

  \begin{lem}\label{lem2}
   The constructed solution $y_{M}(q, s, h)$ in Eq. (11) satisfies the constraint (9.d), i.e.,
  \begin{equation}
 \begin{aligned}
& ~\sum_{k=0}^{A}\sum_{s=0}^{S_{max}}\alpha_{k}\int_{h_{min}}^{h_{max}}y_{M}(q+s-k, s, h)dh=\\ & ~ \sum_{s=0}^{S_{max}}\int_{h_{min}}^{h_{max}}y_{M}(q, s, h)dh\ \ \ \forall q\in \mathcal{Q}.
   \end{aligned}
   \end{equation}
     \end{lem}
 \begin{proof}
 We first show that for any $q\in \mathcal{Q}$, $s\in \mathcal{S}$, we have
 \begin{subequations} \label{10151408}
\renewcommand{\theequation}{\theparentequation.\alph{equation}}
 \begin{align}
& ~\int_{h_{min}}^{h_{max}}y_{M}(q, s, h)dh\\
 & ~ =\int_{h_{min}}^{h_{max}}\!\!\sum_{k=1}^{M}\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)\mathbb{I}\{h\!\!\in\!\!(h_{k,s,q}^{min}, h_{k,s,q}^{max}]\}dh\\
  & ~ =\sum_{k=1}^{M}\int_{h_{k,s,q}^{min}}^{h_{k,s,q}^{max}}\sum_{x=0}^{S_{max}}g ^{*}(q, x, h)dh\\
    & ~ =\sum_{k=1}^{M}(U_{q}(h_{k,s,q}^{max})-U_{q}(h_{k,s,q}^{min}))\\
      & ~ =\sum_{k=1}^{M}(U_{q}(h_{k,s,q}^{max})-U_{q}(h_{k,s-1,q}^{max}))\\
      & ~ =\sum_{k=1}^{M}((\!\!\int_{h_{min}}^{h_{k}^{min}}\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)dh \!\!+\!\!\!\int_{h_{k}^{min}}^{h_{k}^{max}}\!\!\sum_{x=0}^{s}\!\!g ^{*}(q, x, h)dh)\notag\\
      & ~-\!\!(\!\!\int_{h_{min}}^{h_{k}^{min}}\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)dh \!\!+\!\!\!\int_{h_{k}^{min}}^{h_{k}^{max}}\sum_{x=0}^{s-1}\!\!g ^{*}(q, x, h)dh))\\
          & ~ =\sum_{k=1}^{M}\int_{h_{k}^{min}}^{h_{k}^{max}}g ^{*}(q, s, h)dh\\
          & ~ =\int_{h_{min}}^{h_{max}}g ^{*}(q, s, h)dh.
   \end{align}
 \end{subequations}

We obtain (19.b) through substituting $y_{M}(q, s, h)$ in (19.a) which is based on Eq. (11). By changing the integration interval, we can eliminate the indicative function in (19.b), as shown in (19.c). The definition of $U_{q}(x)$ in Eq. (10) helps us convert (19.d) to (19.c). We obtain (19.e) through substituting $h_{k,s,q}^{min}$ in (19.a) by $h_{k,s-1,q}^{max}$ which is based on Eq. (13). Based on the definition of $h_{k,s,q}^{max}$ in Eq. (12), we obtain (19.f) from (19.e). Finally, (19.h) is derived.

The optimal solution $g ^{*}(q, s, h)$ satisfies the constraint (9.d). Therefore, based on Eq. (19), we have
  \begin{equation}
 \begin{aligned}
& ~\sum_{k=0}^{A}\sum_{s=0}^{S_{max}}\alpha_{k}\int_{h_{min}}^{h_{max}}y_{M}(q+s-k, s, h)dh\\& ~ = \sum_{k=0}^{A}\sum_{s=0}^{S_{max}}\alpha_{k}\int_{h_{min}}^{h_{max}}g ^{*}(q+s-k, s, h)dh\\& ~
=\sum_{s=0}^{S_{max}}\int_{h_{min}}^{h_{max}}g ^{*}(q, s, h)dh\\& ~
=\sum_{s=0}^{S_{max}}\int_{h_{min}}^{h_{max}}y_{M}(q, s, h)dh,
   \end{aligned}
   \end{equation}
where $ q\in \mathcal{Q}$.

  \end{proof}

  \begin{lem}\label{lem2}
   The constructed solution $y_{M}(q, s, h)$ in Eq. (11) satisfies the constraint (9.b), i.e.,
  \begin{equation}
 \frac{1}{\overline{a}}\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}qy_{M}(q, s, h)dh\leq D_{th}.
   \end{equation}
     \end{lem}
\begin{proof}
   Based on Eq. (19), we have
  \begin{equation}
 \begin{aligned}
& ~\frac{1}{\overline{a}}\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}qy_{M}(q, s, h)dh\\ & ~ =\frac{1}{\overline{a}}\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}qg^{*}(q, s, h)dh\\ & ~
\leq D_{th}.
   \end{aligned}
   \end{equation}
\end{proof}
  \begin{lem}\label{lem2}
   The constructed solution $y_{M}(q, s, h)$ in Eq. (11) satisfies the constraint (9.e), i.e.,
  \begin{equation}
y_{M}(q,s,h)\geq 0 \ \ \forall q\in \mathcal{Q}, s\in \mathcal{S}, h\in \mathcal{H}.
   \end{equation}
     \end{lem}
\begin{proof}
Since $g^{*}(q, s, h)$ satisfies the constraint (9.e) in problem (9), based on Eq. (11) $y_{M}(q, s, h)$ also satisfies the constraint (9.e).
\end{proof}

  \begin{lem}\label{lem2}
   The constructed solution $y_{M}(q, s, h)$ satisfies the constraint (9.f) in problem (9), i.e.,
  \begin{equation}
  y_{M}(q,s,h)=0 \ \ \ \forall q\!-\!s<0 \ \text{or}\ q\!-\!s>Q\!-\!A \ \text{or}\ q \notin \mathcal{Q}\ \text{or}\ s \notin \mathcal{S}.
   \end{equation}
     \end{lem}
\begin{proof}
Since $g^{*}(q, s, h)$ satisfies the constraint (9.f), if $q-s<0\ \text{or}\ q-s>Q-A\ \text{or}\ q \notin \mathcal{Q}\ \text{or}\ s \notin \mathcal{S}$, then we have
  \begin{equation}
  g^{*}(q, s, h)=0,
   \end{equation}
 When $g^{*}(q, s, h)=0$, we have
   \begin{equation}
 \begin{aligned}
h_{k,s,q}^{max}-h_{k,s-1,q}^{max}& ~ =\int_{h_{k}^{min}}^{h_{k}^{max}}g ^{*}(q, s, h)dh\\ & ~
=0.
   \end{aligned}
   \end{equation}
 Based on Eqs. (13) and (26), we have
 \begin{equation}
h_{k,s,q}^{max}=h_{k,s,q}^{min}.
 \end{equation}
 Finally, based on Eq. (11), using the relationship $h_{k,s,q}^{max}=h_{k,s,q}^{min}$ gives $y_{M}(q, s, h)=0$.
\end{proof}

   \newtheorem{thm}{\bf Theorem}
   \begin{thm}\label{thm1}
   The constructed solution $y_{M}(q, s, h)$ is in the feasible region of problem (9).
\end{thm}
\begin{proof}
Based on Lemma 1-5, Theorem 1 is proved.
\end{proof}
 \begin{lem}\label{lem2}
   For any $q\in\mathcal{Q}$, $h\in\mathcal{H}$, there is at most one $s\in\mathcal{S}$ satisfies that

    \begin{equation}
    y_{M}(q, s, h)>0.
    \end{equation}

  \end{lem}
 \begin{proof}
 Based on Eqs. (13) (15), for any $k_{1}, k_{2}\in [1:M]$, $s_{1}, s_{2}\in \mathcal{S}$ and $q\in\mathcal{Q}$, we have $(h_{k_{1},s_{1},q}^{min}, h_{k_{1},s_{1},q}^{max}]\in (h_{k_{1}}^{min}, h_{k_{1}}^{max}]$ and $(h_{k_{2},s_{2},q}^{min}, h_{k_{2},s_{2},q}^{max}]\in (h_{k_{2}}^{min}, h_{k_{2}}^{max}]$.

 If $k_{1}\neq k_{2}$, then based on the definition of $h_{k}^{min}$ and $h_{k}^{max}$ we have
     \begin{equation}
  (h_{k_{1},s_{1},q}^{min}, h_{k_{1},s_{1},q}^{max}]\bigcap (h_{k_{2},s_{2},q}^{min}, h_{k_{2},s_{2},q}^{max}]=\emptyset.
   \end{equation}
  If $k_{1}= k_{2}$ and $s_{1}\neq s_{2}$, then based on Eqs. (12) (13), we still  have
     \begin{equation}
  (h_{k_{1},s_{1},q}^{min}, h_{k_{1},s_{1},q}^{max}]\bigcap (h_{k_{2},s_{2},q}^{min}, h_{k_{2},s_{2},q}^{max}]=\emptyset.
   \end{equation}
   Furthermore, in Eq. (16), we have obtained that
    \begin{equation}
  \bigcup_{k=1}^{M}\bigcup_{s=0}^{S_{max}} (h_{k,s,q}^{min}, h_{k,s,q}^{max}]=(h_{min}, h_{max}].
   \end{equation}
  Therefore, based on Eqs. (29) (30) and (31), for any $h\in(h_{min}, h_{max}]$ and $q\in\mathcal{Q}$, there is only one $k^{*}$ and $s^{*}$ satisfies that
   \begin{equation}
  \mathbb{I}\{h\in(h_{k^{*},s^{*},q}^{min}, h_{k^{*},s^{*},q}^{max}]\}=1.
   \end{equation}
   Finally, based on Eqs. (11) and (32), Lemma 6 is proved.
\end{proof}
Let $\{f_{M}(s|q,h)\}$ denote the scheduling policy corresponding to constructed $y_{M}(q, s, h)$.
\begin{thm}\label{thm1}
    The scheduling policy $\{f_{M}(s|q,h)\}$ is the deterministic policy.
\end{thm}
\begin{proof}
Based on Eq. (5), the scheduling parameters $\{f_{M}(s|q,h)\}$ can be obtained as
 \begin{equation}
 f_{M}(s|q,h)=\frac{y_{M}(q, s, h)}{\sum_{x=0}^{S_{max}}y_{M}(q, x, h)}.
 \end{equation}
 If the denominator in Eq. (33) is $0$, the queue state $q$ is the transient state. If the denominator in Eq. (33) is not $0$. Then, based on Lemma 6, the $f_{M}(s|q,h)$ in Eq. (33) is $0$ or $1$, which means that the scheduling policy corresponding to $y_{M}(q, s, h)$ is the deterministic policy.
\end{proof}
Let $P_{M}$ and $P^{*}$ denote the average power corresponding to the constructed solution $y_{M}(q, s, h)$ and the optimal solution $g^{*}(q, s, h)$, respectively. Next, we explore the relationship between the $P_{M}$ and $P^{*}$.

\begin{lem}\label{lem2}
   The relationship between $P_{M}$ and $P^{*}$ satisfies that
    \begin{equation}
    P^{*}\leq P_{M}\leq (1+\frac{h_{max}-h_{min}}{M h_{min}})P^{*}.
    \end{equation}
  \end{lem}
 \begin{proof}
 Since $g^{*}(q, s, h)$ is the optimal solution of problem (9), then $P^{*}\leq P_{M}$ is established.
  \begin{subequations} \label{10151408}
\renewcommand{\theequation}{\theparentequation.\alph{equation}}
 \begin{align}
P_{M}& ~= \int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\xi(s)h^{-1}y_{M}(q, s, h)dh\\
 & ~ =\int_{h_{min}}^{h_{max}}\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\xi(s)h^{-1}\sum_{k=1}^{M}\sum_{x=0}^{S_{max}}\notag\\  & ~ g ^{*}(q, x, h)\mathbb{I}\{h\in(h_{k,s,q}^{min}, h_{k,s,q}^{max}]\}dh\\
 & ~ =\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\int_{h_{k,s,q}^{min}}^{h_{k,s,q}^{max}}\xi(s)h^{-1}  \!\!\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)dh\\
& ~ \leq\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\frac{\xi(s)}{h_{k}^{min}}\int_{h_{k,s,q}^{min}}^{h_{k,s,q}^{max}}  \!\!\sum_{x=0}^{S_{max}}\!\!g ^{*}(q, x, h)dh\\
 & ~  =\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\frac{\xi(s)}{h_{k}^{min}}(U_{q}(h_{k,s,q}^{max})-U_{q}(h_{k,s,q}^{min}))\\
& ~  =\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\frac{\xi(s)}{h_{k}^{min}}(U_{q}(h_{k,s,q}^{max})-U_{q}(h_{k,s-1,q}^{max}))\\
      & ~ =\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\frac{\xi(s)}{h_{k}^{min}}\int_{h_{k}^{min}}^{h_{k}^{max}}g ^{*}(q, s, h)dh\\
       & ~ \leq\!\!\sum_{s=0}^{S_{max}}\sum_{k=1}^{M}\sum_{q=0}^{Q}\frac{\xi(s)}{h_{k}^{min}}\int_{h_{k}^{min}}^{h_{k}^{max}}\frac{h_{k}^{max}}{h}g ^{*}(q, s, h)dh\\
          & ~ =\!\!\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\sum_{k=1}^{M}\frac{h_{k}^{max}}{h_{k}^{min}}\int_{h_{k}^{min}}^{h_{k}^{max}}\!\!\!\xi(s)h^{-1}g ^{*}(q, s, h)dh\\
   & ~ \leq\!\!\sum_{s=0}^{S_{max}}\sum_{q=0}^{Q}\sum_{k=1}^{M}\frac{h_{1}^{max}}{h_{1}^{min}}\int_{h_{k}^{min}}^{h_{k}^{max}}\!\!\!\xi(s)h^{-1}g ^{*}(q, s, h)dh\\
      &~= (1+\frac{h_{max}-h_{min}}{M h_{min}})P^{*}.
   \end{align}
 \end{subequations}

The indicative function in (35.b) is eliminated by changing the integration interval of $h$, which is shown in (35.c). Since $h_{k}^{min}\leq h_{k,s,q}^{min}$, formula (35.c) can be enlarged to (35.d) by replacing $h$ with $h_{k}^{min}$. The definition of $U_{q}(x)$ in Eq. (10) helps us convert formula (35.d) to (35.e). Based on Eq. (13), the $h_{k,s,q}^{min}$ in (35.e) can be replaced by  $h_{k,s-1,q}^{max}$ as shown in (35.f). Fortunately, based on the definition of $h_{k,s,q}^{max}$ in Eq. (12), the $U_{q}(h_{k,s,q}^{max})-U_{q}(h_{k,s-1,q}^{max})$ in (35.f) can be simplified, which is shown in (35.g). Since $\frac{h_{k}^{max}}{h_{k}^{min}}\leq \frac{h_{1}^{max}}{h_{1}^{min}}$ for any $k\in[1:M]$, formula (35.i) can be enlarged to (35.j). Finally, based on Eq. (8), we can obtain (35.k).
  \end{proof}
 \begin{thm}\label{thm1}
    For any $\epsilon>0$, there exists an $m=\frac{h_{max}-h_{min}}{\epsilon h_{min}}$. For all $ M\geq m$, we have
        \begin{equation}
    1\leq \frac{P_{M}}{P^{*}}\leq 1+\epsilon.
    \end{equation}
\end{thm}
 \begin{proof}
 The proof of Theorem 3 is based on Lemma 7. Since  $P_{M}\geq P^{*}$, we can obtain $\frac{P_{M}}{P^{*}}\geq 1$. Based on Eq. (34), we have $\frac{P_{M}}{P^{*}}\leq (1+\frac{h_{max}-h_{min}}{M h_{min}})$.
 Since $M\geq \frac{h_{max}-h_{min}}{\epsilon h_{min}}$, we have $\frac{P_{M}}{P^{*}}\leq 1+\epsilon$.

  \end{proof}
\section{ Numerical Results }
In this section, we adopt Constrained Markov Decision Process (CMDP) to demonstrate the optimality of deterministic policies. More specifically, the continuous state-space of channel $\mathcal{H}$ can be equally divided into multiple subspaces. By judging which subspace the current channel state belongs to, we convert the continuous channel state into the discrete channel state for utilization. Based on this, the optimal scheduling can be approximately converted to CMDP. In the simulations, we set $A=2$, $S_{max}=2$, $\alpha_{0}=0.4$, $\alpha_{1}=0.3$, $\alpha_{2}=0.3$, $Q=10$, $\xi(s)=2^{s}-1$, $\mathcal{H}=(0.5, 10]$, and $ f_{H} (h) $ is a uniform distribution on $\mathcal{H}$.

 In Fig. 2, delay-power tradeoff curves are obtained by CMDP with $\mathcal{H}$ being equally divided into 2, 4, 8, 16 subspaces respectively. As we can see, when $\mathcal{H}$ is more densely divided, the delay-power tradeoff curve becomes better and converges asymptotically. The reason is that when the state-space of channel $\mathcal{H}$ is divided sufficiently densely, the discrete channel state information can be approximately equal to continuous channel state information. Therefore, the curve converges to the optimal delay-power tradeoff curve.

 More importantly, the delay-power tradeoff curve achieved by the scheduling based on discrete system state is piecewise linear. The policy corresponding to the vertex on this piecewise linear curve is the deterministic policy which is proofed in [11]. As we can see, when the state space of channel $\mathcal{H}$ is divided into 2 channel states, the distance between two adjacent vertices is shown to be 0.4944. When $\mathcal{H}$ is divided into 16 channel states, the distance decreases to 0.0753. It indicates that if $\mathcal{H}$ is divided more densely, the distance between two adjacent vertices will decrease. Therefore, when $\mathcal{H}$ is divided sufficiently densely, for any point on the delay-power tradeoff curve, we can always find a vertex on the curve to approximate it with sufficient small error, which exactly demonstrates the optimality of deterministic policies.
\section{Conclusion}
In this paper, we studied the optimal cross-layer scheduling based on the continuous channel state. More specifically, a variational problem was formulated to minimize average power for data transmission with the constraint of average delay. Based on this, we constructed a solution within the feasible region of the variational problem. The scheduling policy corresponding to the constructed solution was shown to be deterministic. Meanwhile, we show that the average power of the constructed solution can approximate that of the optimal solution with an arbitrarily small error. Therefore, the optimal scheduling policy could be found in the deterministic policies.

  \begin{figure}
   \centering
   \includegraphics[width=0.5\textwidth]{sim2.eps}

   \caption{The delay-power tradeoff curves with discrete channel states.}
   \label{fig.pathdemo4}
   \end{figure}
\small


\section{Introduction}\label{sec:intro}

Manipulating deformable objects is a long-standing challenge in robotics with a wide range of real-world applications. In contrast to rigid object manipulation, deformable object manipulation presents additional challenges due to more complex configuration spaces, dynamics, and sensing.

While several prior works have made progress largely centered around tasks with 1D (e.g.,  rope~\cite{nair_rope_2017}) or 2D (e.g.,  fabric~\cite{seita_fabrics_2020}) deformable structures, little prior work has addressed generalizable vision-based methods for manipulating 3D deformable structures such as the task of: ``insert objects into a bag, and then carry the bag away''.  These types of tasks are especially challenging with diverse goal conditioning: the goal states of deformable objects are not easily specified, for example, by compact pose representations.

In this work, we propose a new suite of benchmark tasks, called \textit{DeformableRavens}, to test manipulation of cables, fabrics, and bags spanning 1D, 2D, and 3D deformables. For several tasks in the benchmark, we propose to tackle them using novel goal-conditioned variants of Transporter Network~\cite{zeng_transporters_2020} architectures.
Our experiments also significantly extend results of using Transporter Networks for deformable manipulation tasks --- while~\cite{zeng_transporters_2020} demonstrated one 1D deformable task, we show results on 12 tasks, including those with fabrics and bags.

\begin{figure}[t]
\center
\includegraphics[width=0.49\textwidth]{teaser_v02.png}
\caption{
Example of a trained Transporter Network policy in action on the \emph{bag-items-1} task (see Table~\ref{tab:tasks}). The setup involves a simulated UR5 robot arm, a green target zone, a drawstring-style bag, and a red cube. The starting configuration is shown in the top left frame. The robot, with suction cup, must grasp and sufficiently open the bag to insert a cube, then (bottom row) picks and pulls upwards, enclosing the cube. The robot concludes by bringing the bag with the cube in the target zone. We overlay arrows to indicate the movement of the robot's arm just before a given frame.
}
\vspace*{-15pt}
\label{fig:teaser}
\end{figure}

The main contributions of this paper are: (i) an open-source simulated benchmark, \textit{DeformableRavens}, with 12 tasks manipulating 1D, 2D, and 3D deformable objects to help accelerate research progress in robotic manipulation of deformables, (ii) end-to-end goal-conditioned Transporter Network architectures that learn vision-based multi-step manipulation of deformable 1D, 2D, and 3D structures, and (iii) experiments demonstrating that the proposed vision-based architectures are competitive with or superior to baselines that consume ground truth simulated pose and vertex information. We also discuss the shortcomings of the system, which point to interesting areas for future research. The project website contains supplementary material, including the appendix, code, data, and videos.

\begin{figure*}[t]
\center
\includegraphics[width=1.00\textwidth]{tasks_v06-min.png}
\caption{
The 12 tasks in the proposed \emph{DeformableRavens} benchmark (see Table~\ref{tab:tasks}) with suction cup gripper and deformable objects. \textbf{Top row}: (a) cable-ring, (b) cable-ring-notarget, (c) cable-shape, (d) cable-shape-notarget, (e) cable-line-notarget, (f) fabric-cover. \textbf{Bottom row}: (g) fabric-flat, (h) fabric-flat-notarget, (i) bag-alone-open, (j) bag-items-1, (k) bag-items-2, (l) bag-color-goal. Of the tasks shown, 4 are ``goal-conditioned,'' i.e.,  cable-shape-notarget, cable-line-notarget, fabric-flat-notarget, bag-color-goal. These use a separate goal image $\mathbf{o}_g$ for each episode to specify a success configuration. The other 8 tasks do not use goal images. Examples of successful configurations for each task are shown with overlaid (cropped) images to the top left.
}
\vspace*{-15pt}
\label{fig:tasks}
\end{figure*}

\section{Related Work}\label{sec:rw}

\subsection{Deformable Object Manipulation}

In this work, deformables~\cite{manip_deformable_survey_2018} refers to 1D structures such as ropes and cables (we use the terms interchangeably), 2D structures such as fabrics, and 3D structures such as bags.

Rope has been extensively explored in early robotics literature, such as for knot-tying~\cite{case_study_knots_1991,knot_planning_2003}. In recent years, learning-based approaches for manipulation have grown in popularity to improve the generalization of knot-tying and also to manipulate rope towards general target configurations~\cite{schulman_isrr_2013,nair_rope_2017,ZSVI_2018,priya_2020,state_estimation_LDO,wang_visual_planning_2019}. The approach we propose is focused primarily in manipulating cables to target locations, which may be specified by a target zone or a target image, and does not require modeling rope physics.

Much research on robotic fabric manipulation~\cite{grasp_centered_survey_2019,manip_deformable_survey_2018,corona_2018} uses bilateral robots and gravity to expose corners for ease of manipulation, which enables iteratively re-grasping the fabric's lowest hanging point~\cite{osawa_2007,kita_2009_iros,kita_2009_icra}. Subsequent work~\cite{maitin2010cloth,cusumano2011bringing,unfolding_rf_2014}, generalized to a wider variety of initial configurations of new fabrics. Other approaches focus on fabric smoothing.
For example, Sun~et~al.~\cite{heuristic_wrinkles_2014,cloth_icra_2015} attempt to detect and then pull perpendicular to the largest wrinkle. Another task of interest is wrapping rigid objects with fabric, which bridges the gap between manipulating 2D and 3D structures. Hayashi~et~al.~\cite{planning_wrapping_fabric_2017,fabric_wrapping_2014} present an approach for wrapping fabric around a cylinder using a bimanual robot.

Work on 3D deformable structures such as bags has been limited due to the complexities of manipulating bags. Early work focused on mechanical design of robots suitable for grasping~\cite{grasping_sacks_2005} or unloading~\cite{unloading_sacks_2008} sacks. Some work assumes that a sturdy, brown bag is already open for item insertion, as with a grocery checkout robot~\cite{checkout_robot_2011}, or uses reinforcement learning for zipping bags~\cite{contour_ziplock_2018} in constrained setups.

These works generally focus on optimizing a specific problem in deformable object manipulation, often with task-specific algorithms or manipulators. In contrast, we propose a simulation setup with a single robot arm that applies to a wide range of tasks with 1D, 2D, and 3D deformables. In independent and concurrent work, Lin~et~al.~\cite{corl2020softgym}  propose \emph{SoftGym}, a set of environments with cables, fabrics, and liquids simulated using NVIDIA FleX, and use it to benchmark standard reinforcement learning algorithms.

\subsection{Data-Driven Robot Manipulation}

Robot manipulation using learned, data-driven techniques has become a highly effective paradigm for robot manipulation, as exemplified in the pioneering work of Levine~et~al.~\cite{levine_finn_2016}, Mahler~et~al.~\cite{mahler2019learning}, Florence~et~al.~\cite{dense_object_nets_2018}, Kalashnikov~et~al.~\cite{QT-Opt}, and others. Tools used often involve either Imitation Learning (IL)~\cite{argall2009survey} or Reinforcement Learning (RL)~\cite{Sutton_2018}, and in recent years, such techniques have been applied for manipulation of deformables.

For example, IL is used with human~\cite{seita-bedmaking} and scripted~\cite{seita_fabrics_2020} demonstrators for fabric smoothing, while RL is applied for smoothing and folding in~\cite{sim2real_deform_2018,rishabh_2020,lerrel_2020}. Using model-based RL,~\cite{visual_foresight_2018} were able to train a video prediction model to enable robots to fold pants and fabric, and a similar approach was investigated further in~\cite{fabric_vsf_2020} for model-based fabric manipulation. Some approaches combine IL and RL~\cite{balaguer2011combining}, and others use techniques such as latent space planning~\cite{latent_space_roadmap_2020,yan_fabrics_latent_2020} or using dense object descriptors~\cite{descriptors_fabrics_2020}. In this work, we use IL.

As described in Section~\ref{sec:PS}, the deep architecture we propose for image-based manipulation uses Fully Convolutional Neural Networks (FCNs)~\cite{fcn_2015} to produce per-pixel scores in an image, where each pixel corresponds to an action. This technique~\cite{thesis_andyzeng} has shown promising results for robotic warehouse order picking~\cite{zeng_icra_2018}, pushing and grasping~\cite{zeng_iros_2018}, tossing objects~\cite{zeng_tossing_2019}, kit assembly~\cite{form2fit_2020}, and mobile manipulation~\cite{spatial_action_maps_2020}. We show that similar approaches may be effective for picking and placing of deformables. In independent and concurrent work, Lee~et~al.~\cite{folding_fabric_fcn_2020} show how an approach can be used for fabric folding.

\section{Background}\label{sec:PS}

We first describe the problem formulation, followed by background on Transporter Networks~\cite{zeng_transporters_2020}. Section~\ref{sec:goal-conditioned} then describes a novel goal-conditioned framework.

\subsection{Problem Formulation}\label{ssec:problem-formulation}

We formulate the problem of rearranging deformable objects as learning a policy $\pi$ that sequences pick and place actions $\mathbf{a}_t \in \mathcal{A}$ with a robot from visual observations $\mathbf{o}_t \in \mathcal{O}$:
\begin{equation}\label{eq:action}
\pi(\mathbf{o}_t)\rightarrow{\mathbf{a}_t}=(\mathcal{T}_\textrm{pick},\mathcal{T}_\textrm{place})\in{\mathcal{A}}
\end{equation}
where $\mathcal{T}_\textrm{pick}$ is the pose of the end effector when grasping part of an object, and $\mathcal{T}_\textrm{place}$ is the pose of the end effector when releasing the grasp.
Since deformable objects may require more than a single pick and place action to reach a desired configuration, it is important that the policy $\pi$ learns from closed-loop visual feedback. Some rearrangement tasks~\cite{rearrangement_2020} may also be specified by a target goal observation $\mathbf{o}_g$, in which case we refer to the policy as goal-conditioned:
\begin{equation}\label{eq:action_goal}
\pi(\mathbf{o}_t, \mathbf{o}_g)\rightarrow{\mathbf{a}_t}=(\mathcal{T}_\textrm{pick},\mathcal{T}_\textrm{place})\in{\mathcal{A}}.
\end{equation}
In this work, we consider tabletop manipulation tasks where both poses $\mathcal{T}_\textrm{pick}$ and $\mathcal{T}_\textrm{place}$ are defined in SE(2). Picking and placing positions are sampled from a fixed-height planar surface, while rotations are defined around the z-axis (aligned with gravity direction). The $\mathcal{T}_\textrm{pick}$ and $\mathcal{T}_\textrm{place}$ both parameterize a motion primitive \cite{frazzoli2005maneuver} that controls the end effector to approach $\mathcal{T}_\textrm{pick}$ until contact is detected,
activates the end effector to execute a grasp, moves upward to a fixed z-height, approaches $\mathcal{T}_\textrm{place}$, and lowers the end effector until contact is detected, then releases the grasp.
While this discrete-time planar action parameterization has its limitations, we find that it remains sufficiently expressive for a number of tabletop tasks involving manipulation of bags, in which gravity helps to induce a natural adhesive force that brings objects back towards the tabletop plane.

To train the policy, we assume access to a small dataset of $N$ stochastic expert demonstrations $\mathcal{D} = \{\xi_i\}_{i=1}^N$, where each \emph{episode} $\xi_i$ of length $T_i$ (i.e.,  the number of actions) consists of a sequence of observations and actions:
\begin{equation}\label{eq:tau}
\xi_i = (\mathbf{o}_1, \mathbf{a}_1, \mathbf{o}_2, \ldots,
\mathbf{o}_{T_i}, \mathbf{a}_{T_i}, \mathbf{o}_{T_i + 1})
\end{equation}
used to supervise $\pi$, which may be goal-conditioned.

\begin{figure*}[t]
\center
\includegraphics[width=0.95\textwidth]{transporter-goal_v04.png}
\caption{
The proposed \emph{Transporter-Goal-Split}, applied to an example on \emph{bag-color-goal}, where given the current image $\mathbf{o}_t$ and goal $\mathbf{o}_g$, the objective is to insert the block in the correct, open bag; the goal $\mathbf{o}_g$ specifies that the yellow bag (with the block) is the target, not the blue bag. Transporter Networks use three Fully Convolutional Networks (FCNs), $f_\textrm{pick}$ for the attention module, and $\Phi_\textrm{query}$ and $\Phi_\textrm{key}$ for the transport module. \emph{Transporter-Goal-Split} considers a separate goal image $\mathbf{o}_g$ and passes that through a new FCN, $\Phi_\textrm{goal}$. These are then combined with other deep features (labeled as ``logits'') through element-wise multiplication. Finally, the standard cross-convolution operator is applied. \emph{Transporter-Goal-Stack} dispenses with $\Phi_\textrm{goal}$ and instead stacks $\mathbf{o}_t$ and $\mathbf{o}_g$ as one image, and does not use $\Phi_\textrm{goal}$). All tested Transporter architectures produce $\mathcal{Q}_{\rm pick}$ and $\mathcal{Q}_{\rm place}$, which are each $320{\times}160$ dimensional heat maps colored so that darker pixels are low values and lighter pixels are high. The largest value in a heat map represents the \emph{pixel} to pick or place.
}
\vspace*{-10pt}
\label{fig:transporter-goal}
\end{figure*}

\subsection{Background: Transporter Networks}\label{ssec:vanilla-transporter}

\emph{Transporter Networks}~\cite{zeng_transporters_2020} is a model architecture for manipulation that learns to rearrange objects by (i) attending to a local region of interest, then (ii) predicting its target spatial displacement by cross-correlating its dense deep visual features over the scene. It can parameterize actions for pick and place, and has shown to work well for variety of tabletop manipulation tasks including stacking, sweeping, and kit assembly~\cite{form2fit_2020}.

Transporter Networks consist of 3 Fully Convolutional Networks (FCNs). The first FCN $f_\textrm{pick}$ takes as input the visual observation $\mathbf{o}_t$, and outputs a dense per-pixel prediction of action-values $\mathcal{Q}_\textrm{pick}$ that correlate with picking success: $\mathcal{T}_\textrm{pick} = \argmax_{(u, v)} \ \mathcal{Q}_\textrm{pick}((u,v)|\mathbf{o}_t)$ where each pixel $(u, v)$ corresponds to a picking action at that location via camera-to-robot calibration. The second FCN $\Phi_\textrm{key}$ also takes as input $\mathbf{o}_t$, while the third FCN $\Phi_\textrm{query}$ takes as input a partial crop $\mathbf{o}_t[\mathcal{T}_\textrm{pick}]$ from $\mathbf{o}_t$ centered on $\mathcal{T}_\textrm{pick}$. Both the second and third FCNs output dense feature embeddings, which are then cross-correlated with each other to output a dense per-pixel prediction of action-values $\mathcal{Q}_\textrm{place}$ that correlate with placing success: $\mathcal{T}_\textrm{place} = \argmax_{\{\Delta\tau_i\}} \ \mathcal{Q}_\textrm{place}(\Delta\tau_i|\mathbf{o}_t,\mathcal{T}_\textrm{pick})$ where
\begin{equation}
\mathcal{Q}_\textrm{place}(\Delta\tau|\mathbf{o}_t,\mathcal{T}_\textrm{pick}) = \Phi_\textrm{query}(\mathbf{o}_t[\mathcal{T}_\textrm{pick}])*\Phi_\textrm{key}(\mathbf{o}_t)[\Delta\tau]
\end{equation}
and $\Delta\tau$ covers the space of all possible placing poses.

A key aspect of Transporter Networks is that the visual observations $\mathbf{o}_t$ must be spatially consistent, so that the 3D structure of the data is preserved under different visuo-spatial transforms $\Delta\tau$. In practice, this property also improves training data augmentation (since rotations and translations of the orthographic image appear as different configurations of objects in the scene). To leverage this, observations $\mathbf{o}_t$ are top-down orthographic images of the tabletop scene, where each pixel represents a vertical column of 3D space.

\section{Goal-Conditioned Transporter Networks}\label{sec:goal-conditioned}

In some tasks, it is more natural to specify a success criteria by passing in a goal observation $\mathbf{o}_g$ showing objects in a desired configuration. In this case, we assume $\mathbf{o}_g$ is fixed and available as an extra input with the current observation $\mathbf{o}_t$, and that $\mathbf{o}_t$ and $\mathbf{o}_g$ have the same pixel resolution.

\subsection{Goal-Conditioned Transporter Networks}\label{ssec:goal-conditioned}

We propose two goal-conditioned architectures based on Transporter Networks. The first, \emph{Transporter-Goal-Stack}, stacks the current $\mathbf{o}_t$ and goal $\mathbf{o}_g$ images channel-wise, then passes it as input through a standard Transporter Network.
The second, \emph{Transporter-Goal-Split}, separates processing of the goal image $\mathbf{o}_g$ through a fourth FCN $\Phi_\textrm{goal}$ to output dense features that are combined with features from the query $\Phi_\textrm{query}$ and key $\Phi_\textrm{key}$ networks using the Hadamard product:
\begin{equation}
\psi_\textrm{query}(\mathbf{o}_t) = \Phi_\textrm{query}(\mathbf{o}_t)\odot\Phi_\textrm{goal}(\mathbf{o}_t)
\end{equation}
\begin{equation}
\psi_\textrm{key}(\mathbf{o}_t) = \Phi_\textrm{key}(\mathbf{o}_t)\odot\Phi_\textrm{goal}(\mathbf{o}_t)
\end{equation}
\begin{equation}
\mathcal{Q}_\textrm{place}(\Delta\tau|\mathbf{o}_t,\mathcal{T}_\textrm{pick}) = \psi_\textrm{query}(\mathbf{o}_t)[\mathcal{T}_\textrm{pick}]*\psi_\textrm{key}(\mathbf{o}_t)[\Delta\tau]
\end{equation} We hypothesize that this separation is beneficial for learning, since convolutional kernels may otherwise struggle to disambiguate which channels correspond to $\mathbf{o}_t$ or $\mathbf{o}_g$.
Figure~\ref{fig:transporter-goal} shows the architecture of \emph{Transporter-Goal-Split}.
For consistency, all Transporter modules $\Phi_\textrm{query}$, $\Phi_\textrm{key}$, and $\Phi_\textrm{goal}$ employ a 43-layer, 9.9M parameter FCN~\cite{fcn_2015} with residual connections~\cite{resnets_2016}, given that similar architectures have been used for FCN-based picking and placing~\cite{zeng_icra_2018,zeng_transporters_2020}.

A key aspect of the goal-conditioned model is that the spatial structure of the goal images are preserved in the architecture. This prior encourages the deep networks to learn features that infer pick and place actions anchored on visual correspondences between objects in the current and goal images. This is useful for manipulating deformable objects, where changes between current and desired configurations are better captured with dense correspondences \cite{descriptors_fabrics_2020,dense_object_nets_2018}.

\subsection{Training Details for Goal Conditioned Models}

To train both goal-conditioned architectures, we use an approach based on Hindsight Experience Replay~\cite{HER_2017}. For each task (Table~\ref{tab:tasks}), we assume a dataset of $N$ demonstrations $\mathcal{D} = \{\xi_i\}_{i=1}^N$, where each episode $\xi_i$ (see Eq.~\ref{eq:tau}) is of length $T_i$ with final observation $\mathbf{o}_g = \mathbf{o}_{T_i + 1}$.
In standard Transporter Networks, a single sample $(\mathbf{o}_k, \mathbf{a}_k)$ containing the observation and the resulting action at time $k$, is drawn at random from $\mathcal{D}$. For the goal-conditioned Transporter Networks, we use the same procedure to get $(\mathbf{o}_k, \mathbf{a}_k)$, then additionally use the corresponding observation $\mathbf{o}_g$ after the last action from the demonstration episode containing $\mathbf{o}_k$, to get training sample $(\mathbf{o}_k, \mathbf{a}_k, \mathbf{o}_g)$.

As described in Section~\ref{ssec:vanilla-transporter}, because observations $\mathbf{o}_t$ are top-down orthographic images and are spatially consistent with rotations and translations, this enables data augmentation by randomizing a rotation and translation for each training image. To handle the goal-conditioned case, both $\mathbf{o}_t$ and $\mathbf{o}_g$ are augmented using the same random rotation and translation, to ensure consistency.

\section{Simulator and Tasks}\label{sec:simulator-tasks}

\begin{table}
  \setlength\tabcolsep{3.0pt}

  \centering
  \caption{
  \textbf{DeformableRavens}. Tasks involve rearranging deformable objects (e.g.,  cables, fabrics, and bags). Each comes with a scripted expert demonstrator that succeeds with high probability, except for the four bag tasks which are challenging; for these, we filter to use only successful episodes in training. Some require \emph{precise placing} to trigger a success. Tasks with a \emph{visible zone} will have a green target zone on the workspace to indicate where items should be placed (e.g., a square target zone that a fabric must cover); other \emph{goal-conditioned} tasks use a separate goal image to specify the success criteria for object rearrangement. See Figure~\ref{fig:tasks} for visualizations.
  }
  \footnotesize
  \begin{tabular}{@{}lrccc}
  \toprule
  Task                  & demos$\;$  & precise & visible & goal  \\
  (Max. Episode Length) & stats(\%)  & placing & zone    & cond. \\
  \midrule
  (a) cable-ring$^\mathsection$ (20)          &  99.1 & \xmark & \cmark & \xmark \\
  (b) cable-ring-notarget$^\mathsection$ (20) &  99.3 & \xmark & \xmark & \xmark \\
  (c) cable-shape$^*$ (20)                    &  98.8 & \cmark & \cmark & \xmark \\
  (d) cable-shape-notarget$^*$ (20)           &  99.1 & \cmark & \xmark & \cmark \\
  (e) cable-line-notarget$^*$ (20)            & 100.0 & \cmark & \xmark & \cmark \\
  \midrule
  (f) fabric-cover (2)                        &  97.0 & \xmark & \xmark & \xmark \\
  (g) fabric-flat$^{\dagger}$ (10)            &  98.3 & \cmark & \cmark & \xmark \\
  (h) fabric-flat-notarget$^{\dagger}$ (10)   &  97.4 & \cmark & \xmark & \cmark \\
  \midrule
  (i) bag-alone-open$^{\mathsection}$ (8)     &  60.2 & \xmark & \xmark & \xmark \\
  (j) bag-items-1 (8)                         &  41.7 & \xmark & \cmark & \xmark \\
  (k) bag-items-2 (9)                         &  32.5 & \xmark & \cmark & \xmark \\
  (l) bag-color-goal (8)                      &  89.1 & \xmark & \xmark & \cmark \\
  \bottomrule
  \end{tabular}
  \\$^\mathsection$evaluated based on maximizing the convex hull area of a ring.
  \\$^*$evaluated by the percentage of a cable within a target zone.
  \\$^\dagger$evaluated using fabric coverage, as in Seita~et~al.~\cite{seita_fabrics_2020,seita-bedmaking}.
  \vspace{-2.0em}
  \label{tab:tasks}
\end{table}

We evaluate the system on \emph{DeformableRavens}, a novel suite of simulated manipulation tasks involving cables, fabrics, and bags, using PyBullet~\cite{coumans2019} with an OpenAI gym~\cite{gym} interface. See Table~\ref{tab:tasks} and Figure~\ref{fig:tasks} for overviews.

\subsection{Deformable Objects (Soft Bodies) in PyBullet}\label{ssec:simulator}

Data-driven methods in robot learning often require substantial amounts of data, which can be expensive to obtain in the real-world~\cite{QT-Opt}.
While simulators have been used in robotics to help alleviate this difficulty, such as for locomotion~\cite{sim-to-real-locomotion} and rigid object manipulation~\cite{openai-dactyl}, many simulated benchmarks for manipulation focus on rigid objects, partially because of difficulties in simulation of deformables~\cite{manipulation_cloth_liu_2016}.

Motivated by these challenges, we provide support for deformable objects (called ``soft bodies'') in PyBullet~\cite{coumans2019}, a widely-used publicly available simulator for robotics research. While prior work with soft bodies in PyBullet~\cite{deep_dressing_2018,assistive_gym_2020,sim2real_deform_2018} use position-based dynamics solvers, we use new soft body physics simulation based on the Finite Element Method~\cite{bathe2006finite} with mass-springs and self-collisions among vertices~\cite{cloth-cloth-collisions}. Contact and friction constraints between soft bodies and multi bodies are solved in a unified constraint solver at the velocity level. Implicit damping of the velocity uses a Krylov style method and soft body contact and friction is based on Conjugate Gradient for symmetric positive definite systems and Conjugate Residual for indefinite systems~\cite{Verschoor2019}.

In simulation, cables consist of a sequence of rigid bodies (``beads''). Fabrics and bags are soft bodies, where each consists of a set of vertices. For fabrics and bags, we create a plane and a sphere object, respectively, using Blender~\cite{blender}. (For bags, we remove vertices to form an opening.) We then import the object mesh files into PyBullet. To improve physics stability of drawstring-style bagging behavior in PyBullet simulation, we additionally attach a ring of rigid beads at the bag's opening, using the same beads that form cables. See Figure~\ref{fig:pybullet-sim} for an overview.

\begin{figure}[t]
\center
\includegraphics[width=0.47\textwidth]{bags_v03-min.png}
\caption{
\textbf{Top row}: examples of physical bags. The bags we use follow a design similar to the sack (top left) and drawstring (top middle). Bags with handles (e.g.,  top right) or with more stiffness will be addressed in future work. \textbf{Bottom row}: to make bags, we create a sphere in Blender and remove vertices above a certain height (bottom left). We import the mesh into PyBullet and add a series of beads at the bag opening. For bag tasks, we initialize the bag by randomly sampling a starting pose (bottom middle), applying a small force, and allowing the bag to crumple (bottom right).
}
\vspace*{-15pt}
\label{fig:pybullet-sim}
\end{figure}

\subsection{Benchmark for Manipulating Deformable Objects}\label{ssec:benchmark}

We design 12 tasks with deformables, listed in Table~\ref{tab:tasks}. For consistency, each uses a standardized setup: a UR5 arm, a $0.5{\times}1$m tabletop workspace, and 3 calibrated RGB-D cameras diagonally overlooking the workspace, producing top-down observations $\mathbf{o}_t \in \mathbb{R}^{320\times 160\times 6}$ of pixel resolution $320{\times}160$, where each pixel represents a $3.125{\times}3.125$mm vertical column of 3D space. Observations contain 3 RGB channels and 3 channel-wise depth values. The goal is to train a policy that executes a sequence of pick and place actions in the workspace to achieve an objective, learned from demonstrations with behavior cloning~\cite{Pomerleau_behavior_cloning}. Some tasks (e.g.,  \emph{cable-shape-notarget} and \emph{fabric-flat-notarget}) are specified by goal images of a target scene configuration, while others are described solely by the distribution of demonstrations (e.g.,  \emph{cable-ring-notarget} and \emph{bag-alone-open}).

The grasping motion primitive (introduced in Section~\ref{sec:PS}) is similar to the implementations in~\cite{fabric_vsf_2020,seita_fabrics_2020} and approximates a pinch-grasp on a deformable by indexing the nearest vertex that comes in contact with the end effector tip, and locking in its degrees of freedom to match that of the end effector using fixed constraints. Certain tasks are divided into different stages, such as \emph{bag-items-1} and \emph{bag-items-2} which involve opening a bag, inserting items inside, and lifting a bag. Depending on the task stage and the gripped item, actions lift the gripped object to a different hard-coded height. For example, pulling the bag upwards requires a higher vertical displacement than opening the bag.

\section{Experiments}\label{sec:experiments}

We use scripted, stochastic demonstrator policies to get 1000 demonstrations (i.e.,  episodes) per task, and train policies using 1, 10, 100, or all 1000 demonstrations. We provide a detailed overview of the data generation process in the Appendix. We test the following models:

\noindent\textbf{Transporter.} This model is directly from~\cite{zeng_transporters_2020}.

\noindent\textbf{Transporter-Goal-Split.} The Transport model that includes a separate goal module $\Phi_\textrm{goal}$ to process $\mathbf{o}_g$.

\noindent\textbf{Transporter-Goal-Stack.} A model that stacks $\mathbf{o}_t$ and $\mathbf{o}_g$ channel-wise to form a 12-channel image, then passes it as input to standard Transporter Networks.

\noindent\textbf{GT-State MLP.} A model that uses ground truth pose information as observations $\mathbf{o}_t$, and does not use images. It processes its input with a multi-layer perception (MLP) and directly regresses $\mathcal{T}_{\rm pick}$ and $\mathcal{T}_{\rm place}$.

\noindent\textbf{GT-State MLP 2-Step.} A baseline similar to the prior ground truth baseline, except it regresses in a two-step fashion, first regressing $\mathcal{T}_{\rm pick}$, and then concatenates this result again to the ground truth information to regress $\mathcal{T}_{\rm place}$.

We test \emph{Transporter} on non-goal conditioned tasks, whereas for those that use goals $\mathbf{o}_g$, we test with \emph{Transporter-Goal-Split} and \emph{Transporter-Goal-Stack}. Ground truth baselines are tested in both settings, where in the goal-conditioned case, we concatenate the ground truth pose information in both the current and goal configurations to form a single input vector. Following~\cite{zeng_transporters_2020}, to handle multi-modality of picking and placing distributions, the models output a mixture density~\cite{mixture_density_networks} represented by a 26-D multivariate Gaussian. During training, all models use the same data augmentation procedures for a fair comparison.

\noindent\textbf{Evaluation metrics.}
Episodes for \emph{cable-shape}, \emph{cable-shape-notarget}, and \emph{cable-line-notarget}, are successful if all cable beads have reached designated target poses.
For \emph{cable-ring}, \emph{cable-ring-notarget}, and \emph{bag-alone-open}, success is triggered when the convex hull area of the ring of beads (forming the cable or the bag opening) exceeds a threshold.
Episodes for \emph{fabric-cover} are successful if the fabric covers the cube.
\emph{Fabric-flat} and \emph{fabric-flat-notarget} are evaluated based on coverage~\cite{seita_fabrics_2020}.
\emph{Bag-items-1} and \emph{bag-items-2} consider a success when all blocks have been transported to the target zone with the bag.
Finally, \emph{bag-color-goal} evaluates success if the item has been inserted into the correct colored bag.

\begin{table*}[t]
  \setlength\tabcolsep{5.0pt}
  \caption{
  \textbf{Results.} Task success rate (mean \% over 60 test-time episodes in simulation of the best saved snapshot) vs. \# of demonstration episodes (1, 10, 100, or 1000) used in training. For the first eight tasks listed, we benchmark with Transporter Networks~\cite{zeng_transporters_2020} (``Transporter'') and two baselines that use ground-truth pose information instead of images as input. The last row of four tasks tests goal-conditioned policies, and we test with proposed goal-conditioned Transporter Network architectures (see Section~\ref{ssec:goal-conditioned}), along with similar ground-truth baselines. We do not test the last four tasks with ``Transporter'' since the architecture does not support including an extra goal image as input. Section~\ref{sec:experiments} and the Appendix contain more details.
  }
  \centering
  \footnotesize

  \begin{tabular}{@{}lrrrrrrrrrrrrrrrr@{}}
  \toprule
  & \multicolumn{4}{c}{cable-ring} & \multicolumn{4}{c}{cable-ring-notarget} & \multicolumn{4}{c}{cable-shape} & \multicolumn{4}{c}{fabric-cover}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 \\
 \midrule
 GT-State MLP            & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.7 & 3.3 & 5.0 & 0.4 & 0.8 & 1.0 & 0.5 & 3.3 & 25.0 & 18.3 & 21.7  \\
 GT-State MLP 2-Step     & 0.0 & 1.7 & 1.7 & 0.0 & 1.7 & 0.0 & 0.0 & 1.7 & 0.7 & 0.6 & 0.9 & 0.5 & 3.3 & 16.7 & 6.7 & 3.3  \\
 Transporter             & 16.7 & 50.0 & 55.0 & 68.3 & 15.0 & 68.3 & 73.3 & 70.0 & 75.6 & 80.6 & 90.1 & 86.5 & 85.0 & 100.0 & 100.0 & 100.0  \\
 \midrule
 & \multicolumn{4}{c}{fabric-flat} & \multicolumn{4}{c}{bag-alone-open} & \multicolumn{4}{c}{bag-items-1} & \multicolumn{4}{c}{bag-items-2}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 \\
 \midrule
 GT-State MLP            & 26.0 & 45.6 & 65.6 & 71.3 & 15.0 & 16.7 & 35.0 & 43.3 & 1.7 & 20.0 & 30.0 & 31.7 & 0.0 & 0.0 & 6.7 & 8.3  \\
 GT-State MLP 2-Step     & 21.8 & 30.9 & 45.5 & 41.7 & 11.7 & 15.0 & 18.3 & 26.7 & 0.0 & 8.3 & 28.3 & 31.7 & 0.0 & 1.7 & 6.7 & 11.7  \\
 Transporter             & 42.1 & 86.5 & 89.5 & 88.8 & 18.3 & 50.0 & 61.7 & 63.3 & 25.0 & 36.7 & 48.3 & 51.7 & 5.0 & 30.0 & 41.7 & 46.7  \\
 \midrule
 & \multicolumn{4}{c}{cable-line-notarget} & \multicolumn{4}{c}{cable-shape-notarget} & \multicolumn{4}{c}{fabric-flat-notarget} & \multicolumn{4}{c}{bag-color-goal}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000  \\
 \midrule
 GT-State MLP            & 11.1 & 44.5 & 72.7 & 77.4 & 11.1 & 42.7 & 66.0 & 65.4 & 14.8 & 49.8 & 62.1 & 63.2 & 0.8 & 0.8 & 10.0 & 14.9  \\
 GT-State MLP 2-Step     & 8.5 & 39.4 & 58.8 & 65.4 & 9.4 & 44.9 & 54.9 & 56.4 & 23.0 & 51.1 & 59.6 & 61.9 & 4.9 & 5.0 & 5.0 & 15.0  \\
 Transporter-Goal-Stack  & 63.5 & 82.8 & 53.0 & 54.4 & 54.4 & 47.5 & 45.1 & 45.6 & 16.5 & 26.3 & 25.3 & 20.1 & 12.4 & 21.6 & 65.4* & 70.3*  \\
 Transporter-Goal-Split  & 74.9 & 95.6 & 53.9 & 99.2 & 48.4 & 75.1 & 64.9 & 76.4 & 27.6 & 35.6 & 30.1 & 77.0 & 10.0 & 63.1 & 40.1* & 49.8*  \\
  \bottomrule \vspace{-0.8em} \\
  *trained with 40K iterations.
  \end{tabular}
  \label{tab:results}
  \vspace{-2.0em}
\end{table*}

\section{Results}\label{sec:results}

We summarize results in Table~\ref{tab:results}. For each model type and demo count $N \in \{1, 10, 100, 1000\}$, we train 3 models randomly initialized with different TensorFlow~\cite{tensorflow2015-whitepaper} seeds, except for \emph{bag-color-goal}, where we report results from 1 model since the task is computationally expensive. Models are trained for 20K iterations, with a batch size of 1 for the three Transporter models and 128 for the two ground truth models (to strengthen the baseline). We save 10 snapshots throughout training at equally spaced intervals, and for each, we roll out 20 evaluation episodes. This gives 60 metrics (Section~\ref{sec:experiments}) for each snapshot, due to training 3 models. For each of the 10 snapshots and their associated metric, we average the 60 metrics and report the maximum over the 10 in Table~\ref{tab:results}. The Appendix has more extensive analysis.

\subsection{Non-Goal Conditioned Tasks}

In these tasks, \emph{Transporter} generally achieves orders of magnitude better sample efficiency than ground truth models. It performs reliably on \emph{fabric-cover} with 100\% success rates with just 10 demos (compared to 25\% or less for all ground truth models), and does well on \emph{cable-shape} ($\ge$~90.1\%) and \emph{fabric-flat} ($\ge$~86.5\%) given enough demos. On \emph{bag-alone-open}, \emph{Transporter} attains performance of 61.7\% and 63.3\% with 100 and 1000 demos, which is comparable to the scripted demonstrator performance of 60.2\% (1000 successes out of 1661) before data filtering. Similarly, on \emph{bag-items-1} and \emph{bag-items-2}, the best raw performance numbers (with 1000 demos) are 51.7\% and 46.7\%, which exceed demonstrator performance of 41.7\% and 32.5\% (see Table~\ref{tab:tasks}).

Qualitatively, the learned policies may generalize to new starting configurations from the same data distribution. Figure~\ref{fig:teaser} shows an example successful rollout on \emph{bag-items-1}, where the robot opens the bag, inserts the cube, picks and pulls the bag (which encloses the cube in it), then deposits the bag to the target zone. The project website contains videos.

\subsection{Goal Conditioned Tasks}

For \emph{cable-line-notarget} and \emph{cable-shape-notarget} with 1 and 10 demos, \emph{Transporter-Goal-Stack} and \emph{Transporter-Goal-Split} outperform ground-truth baselines, but the performance gap narrows with larger datasets, perhaps because ground-truth models should eventually do well given sufficient data. Furthermore, we observe from training curves (see the supplementary material) that at around 16-20K training iterations, both goal-conditioned Transporter models with larger datasets (100 and 1000 demos) start to get significantly higher test-time performance across the goal-conditioned tasks (especially \emph{bag-color-goal}). From inspecting test roll outs, we hypothesize that these models need to train longer to disambiguate $\mathbf{o}_t$ and $\mathbf{o}_g$ for the picking network $f_{\rm pick}$.

\begin{figure}[t]
\vspace{-0.5em}
\center
\includegraphics[width=0.46\textwidth]{failures_v02.png}
\caption{
Failure cases we observe from trained Transporter policies on bag tasks. Left: in all bag tasks, a failure case may result from covering up the bag opening; these are hard for subsequent actions to recover from. Right: even if items are inserted into the bag, they may not be fully enclosed and can fall out when the bag is lifted. We overlay frames with orange arrows to indicate the direction of motion of the robot's end effector.
}
\vspace*{-15pt}
\label{fig:failure}
\end{figure}

\section{Limitations and Future Work}
\label{sec:limitations}

In this work, we present a new suite of tasks that evaluate end-to-end vision-based manipulation of deformable objects, spanning cables, fabrics, and bags. In addition, we propose goal-conditioned model architectures with Transporter Networks for rearranging objects to match goal configurations.

Manipulating deformable objects is challenging, and while the proposed method yields promising results, its limitations point to interesting directions for future work.
For example, Figure~\ref{fig:failure} shows several failure modes in the bag tasks.
The bag opening step is challenging because a single counterproductive action that visually occludes the bag opening makes it difficult to recover.
In addition, for \emph{bag-items-1} and \emph{bag-items-2}, items may fail to remain enclosed in the bag when lifted vertically.
Future work may investigate higher rates of control that learn recovery policies to react in real-time, or stateful systems that can track objects under visual occlusions, or bimanual manipulation policies, which may help prevent items from falling out of bags. We also observe that the performance of policies on goal-conditioned tasks are slightly below those that explicitly rely on visual anchors in the workspace (e.g.,  \emph{fabric-flat} vs. \emph{fabric-flat-notarget}), suggesting a performance gap. Finally, given the COVID-19 pandemic which limited access to robots, as conditions improve, we will conduct physical experiments with bags.

\section*{Acknowledgments}

\small Daniel Seita is supported by the Graduate Fellowships for STEM Diversity. We thank Xuchen Han for assistance with deformables in PyBullet, and Julian Ibarz for helpful feedback on writing.

\bibliographystyle{IEEEtranS}