text:
# 6.7 Integer exponents and scientific notation

 Page 1 / 10
By the end of this section, you will be able to:
• Use the definition of a negative exponent
• Simplify expressions with integer exponents
• Convert from decimal notation to scientific notation
• Convert scientific notation to decimal form
• Multiply and divide using scientific notation

Before you get started, take this readiness quiz.

1. What is the place value of the $6$ in the number $64,891$ ?
If you missed this problem, review [link] .
2. Name the decimal: $0.0012.$
If you missed this problem, review [link] .
3. Subtract: $5-\left(-3\right).$
If you missed this problem, review [link] .

## Use the definition of a negative exponent

We saw that the Quotient Property for Exponents introduced earlier in this chapter, has two forms depending on whether the exponent is larger in the numerator or the denominator.

## Quotient property for exponents

If $a$ is a real number, $a\ne 0$ , and $m\phantom{\rule{0.2em}{0ex}}\text{and}\phantom{\rule{0.2em}{0ex}}n$ are whole numbers, then

$\begin{array}{c}\frac{{a}^{m}}{{a}^{n}}={a}^{m-n},m>n\phantom{\rule{1em}{0ex}}\text{and}\phantom{\rule{1em}{0ex}}\frac{{a}^{m}}{{a}^{n}}=\frac{1}{{a}^{n-m}},n>m\hfill \end{array}$

What if we just subtract exponents regardless of which is larger?

Let’s consider $\frac{{x}^{2}}{{x}^{5}}$ .

We subtract the exponent in the denominator from the exponent in the numerator.

$\begin{array}{c}\hfill \frac{{x}^{2}}{{x}^{5}}\hfill \\ \hfill {x}^{2-5}\hfill \\ \hfill {x}^{-3}\hfill \end{array}$

We can also simplify $\frac{{x}^{2}}{{x}^{5}}$ by dividing out common factors:

This implies that ${x}^{-3}=\frac{1}{{x}^{3}}$ and it leads us to the definition of a negative exponent .

## Negative exponent

If $n$ is an integer and $a\ne 0$ , then ${a}^{\text{−}n}=\frac{1}{{a}^{n}}$ .

The negative exponent    tells us we can re-write the expression by taking the reciprocal of the base and then changing the sign of the exponent.

Any expression that has negative exponents is not considered to be in simplest form. We will use the definition of a negative exponent    and other properties of exponents to write the expression with only positive exponents.

For example, if after simplifying an expression we end up with the expression ${x}^{-3}$ , we will take one more step and write $\frac{1}{{x}^{3}}$ . The answer is considered to be in simplest form when it has only positive exponents.

Simplify: ${4}^{-2}$ ${10}^{-3}.$

## Solution

1. $\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}{4}^{-2}\hfill \\ \text{Use the definition of a negative exponent,}\phantom{\rule{0.2em}{0ex}}{a}^{\text{−}n}=\frac{1}{{a}^{n}}.\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{{4}^{2}}\hfill \\ \text{Simplify.}\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{16}\hfill \end{array}$

2. $\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}{10}^{-3}\hfill \\ \text{Use the definition of a negative exponent,}\phantom{\rule{0.2em}{0ex}}{a}^{\text{−}n}=\frac{1}{{a}^{n}}.\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{{10}^{3}}\hfill \\ \text{Simplify.}\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{1000}\hfill \end{array}$

Simplify: ${2}^{-3}$ ${10}^{-7}.$

$\frac{1}{8}$ $\frac{1}{{10}^{7}}$

Simplify: ${3}^{-2}$ ${10}^{-4}.$

$\frac{1}{9}$ $\frac{1}{10,000}$

In [link] we raised an integer to a negative exponent. What happens when we raise a fraction to a negative exponent? We’ll start by looking at what happens to a fraction whose numerator is one and whose denominator is an integer raised to a negative exponent.

$\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}\frac{1}{{a}^{\text{−}n}}\hfill \\ \text{Use the definition of a negative exponent,}\phantom{\rule{0.2em}{0ex}}{a}^{\text{−}n}=\frac{1}{{a}^{n}}.\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{\frac{1}{{a}^{n}}}\hfill \\ \text{Simplify the complex fraction.}\hfill & & & \phantom{\rule{4em}{0ex}}1·\frac{{a}^{n}}{1}\hfill \\ \text{Multiply.}\hfill & & & \phantom{\rule{4em}{0ex}}{a}^{n}\hfill \end{array}$

This leads to the Property of Negative Exponents.

## Property of negative exponents

If $n$ is an integer and $a\ne 0$ , then $\frac{1}{{a}^{\text{−}n}}={a}^{n}$ .

Simplify: $\frac{1}{{y}^{-4}}$ $\frac{1}{{3}^{-2}}.$

## Solution

1. $\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}\frac{1}{{y}^{-4}}\hfill \\ \text{Use the property of a negative exponent,}\phantom{\rule{0.2em}{0ex}}\frac{1}{{a}^{\text{−}n}}={a}^{n}.\hfill & & & \phantom{\rule{4em}{0ex}}{y}^{4}\hfill \end{array}$

2. $\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}\frac{1}{{3}^{-2}}\hfill \\ \text{Use the property of a negative exponent,}\phantom{\rule{0.2em}{0ex}}\frac{1}{{a}^{\text{−}n}}={a}^{n}.\hfill & & & \phantom{\rule{4em}{0ex}}{3}^{2}\hfill \\ \text{Simplify.}\hfill & & & \phantom{\rule{4em}{0ex}}9\hfill \end{array}$

Simplify: $\frac{1}{{p}^{-8}}$ $\frac{1}{{4}^{-3}}.$

${p}^{8}$ $64$

Simplify: $\frac{1}{{q}^{-7}}$ $\frac{1}{{2}^{-4}}.$

${q}^{7}$ $16$

Suppose now we have a fraction raised to a negative exponent. Let’s use our definition of negative exponents to lead us to a new property.

$\begin{array}{cccc}& & & \phantom{\rule{4em}{0ex}}{\left(\frac{3}{4}\right)}^{-2}\hfill \\ \text{Use the definition of a negative exponent,}\phantom{\rule{0.2em}{0ex}}{a}^{\text{−}n}=\frac{1}{{a}^{n}}.\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{{\left(\frac{3}{4}\right)}^{2}}\hfill \\ \text{Simplify the denominator.}\hfill & & & \phantom{\rule{4em}{0ex}}\frac{1}{\frac{9}{16}}\hfill \\ \text{Simplify the complex fraction.}\hfill & & & \phantom{\rule{4em}{0ex}}\frac{16}{9}\hfill \\ \text{But we know that}\phantom{\rule{0.2em}{0ex}}\frac{16}{9}\phantom{\rule{0.2em}{0ex}}\text{is}\phantom{\rule{0.2em}{0ex}}{\left(\frac{4}{3}\right)}^{2}.\hfill & & & \\ \text{This tells us that:}\hfill & & & \phantom{\rule{4em}{0ex}}{\left(\frac{3}{4}\right)}^{-2}={\left(\frac{4}{3}\right)}^{2}\hfill \end{array}$

Bruce drives his car for his job. The equation R=0.575m+42 models the relation between the amount in dollars, R, that he is reimbursed and the number of miles, m, he drives in one day. Find the amount Bruce is reimbursed on a day when he drives 220 miles.
LeBron needs 150 milliliters of a 30% solution of sulfuric acid for a lab experiment but only has access to a 25% and a 50% solution. How much of the 25% and how much of the 50% solution should he mix to make the 30% solution?
5%
Michael
hey everyone how to do algebra
Felecia answer 1.5 hours before he reaches her
I would like to solve the problem -6/2x
12x
Andrew
how
Christian
Does the x represent a number or does it need to be graphed ?
latonya
-3/x
Venugopal
-3x is correct
Atul
Arnold invested $64,000, some at 5.5% interest and the rest at 9%. How much did he invest at each rate if he received$4,500 in interest in one year?
Tickets for the community fair cost $12 for adults and$5 for children. On the first day of the fair, 312 tickets were sold for a total of $2204. How many adult tickets and how many child tickets were sold? Alpha Reply 220 gayla Three-fourths of the people at a concert are children. If there are 87 children, what is the total number of people at the concert? Tsimmuaj Reply Erica earned a total of$50,450 last year from her two jobs. The amount she earned from her job at the store was $1,250 more than four times the amount she earned from her job at the college. How much did she earn from her job at the college? Tsimmuaj Erica earned a total of$50,450 last year from her two jobs. The amount she earned from her job at the store was $1,250 more than four times the amount she earned from her job at the college. How much did she earn from her job at the college? Tsimmuaj ? Is there anything wrong with this passage I found the total sum for 2 jobs, but found why elaborate on extra If I total one week from the store *4 would = the month than the total is = x than x can't calculate 10 month of a year candido what would be wong candido 87 divided by 3 then multiply that by 4. 116 people total. Melissa the actual number that has 3 out of 4 of a whole pie candido was having a hard time finding Teddy use Matrices for the 2nd question Daniel One number is 11 less than the other number. If their sum is increased by 8, the result is 71. Find the numbers. Tsimmuaj Reply 26 + 37 = 63 + 8 = 71 gayla 26+37=63+8=71 ziad 11+52=63+8=71 Thisha how do we know the answer is correct? Thisha 23 is 11 less than 37. 23+37=63. 63+8=71. that is what the question asked for. gayla 23 +11 = 37. 23+37=63 63+8=71 Gayla by following the question. one number is 11 less than the other number 26+11=37 so 26+37=63+8=71 Gayla your answer did not fit the guidelines of the question 11 is 41 less than 52. gayla 71-8-11 =52 is this correct? Ruel let the number is 'x' and the other number is "x-11". if their sum is increased means: x+(x-11)+8 result will be 71. so x+(x-11)+8=71 2x-11+8=71 2x-3=71 2x=71+3 2x=74 1/2(2x=74)1/2 x=37 final answer tesfu just new Muwanga Amara currently sells televisions for company A at a salary of$17,000 plus a $100 commission for each television she sells. Company B offers her a position with a salary of$29,000 plus a $20 commission for each television she sells. How televisions would Amara need to sell for the options to be equal? Tsimmuaj Reply yes math Kenneth company A 13 company b 5. A 17,000+13×100=29,100 B 29,000+5×20=29,100 gayla need help with math to do tsi test Toocute me too Christian have you tried the TSI practice test ***tsipracticetest.com gayla DaMarcus and Fabian live 23 miles apart and play soccer at a park between their homes. DaMarcus rode his bike for 34 of an hour and Fabian rode his bike for 12 of an hour to get to the park. Fabian’s speed was 6 miles per hour faster than DaMarcus’s speed. Find the speed of both soccer players. gustavo Reply ? Ann DaMarcus: 16 mi/hr Fabian: 22 mi/hr Sherman Joy is preparing 20 liters of a 25% saline solution. She has only a 40% solution and a 10% solution in her lab. How many liters of the 40% solution and how many liters of the 10% solution should she mix to make the 25% solution? Wenda Reply 15 and 5 32 is 40% , & 8 is 10 % , & any 4 letters is 5%. Karen It felt that something is missing on the question like: 40% of what solution? 10% of what solution? Jhea its confusing Sparcast 3% & 2% to complete the 25% Sparcast because she already has 20 liters. Sparcast ok I was a little confused I agree 15% & 5% Sparcast 8,2 Karen Jim and Debbie earned$7200. Debbie earned \$1600 more than Jim earned. How much did they earned
5600
Gloria
1600
Gloria
Bebbie: 4,400 Jim: 2,800
Jhea
A river cruise boat sailed 80 miles down the Mississippi River for 4 hours. It took 5 hours to return. Find the rate of the cruise boat in still water and the rate of the current.
A veterinarian is enclosing a rectangular outdoor running area against his building for the dogs he cares for. He needs to maximize the area using 100 feet of fencing. The quadratic equation A=x(100−2x) gives the area, A , of the dog run for the length, x , of the building that will border the dog run. Find the length of the building that should border the dog run to give the maximum area, and then find the maximum area of the dog run.
ggfcc
Mike


text:
mdp

# MDPs Using Bayesian Control Rule/Thompson Sampling

This is the model-free reinforcement learning algorithm that we originally used as an example to showcase the Bayesian control rule, inspired by the “Bayesian Q-Learning” paper by Dearden et al.1) Please cite as: Ortega, P.A. and Braun D.A. “A minimum relative entropy principle for learning and acting”. Journal of Artificial Intelligence Research 38, pp. 475-511, 2010.

Before you move on, please make sure you understand the example with the bandits.

This time we want to do something more fun: controlling a Markov Decision Process (MDP) using Thompson sampling. In particular, our goal is to:

1. learn to control an MDP with unknown transition probabilities;
2. and optimize the average reward, rather than the the discounted cumulative reward.

If we use Thompson sampling, then the straightforward way to do this consists in:

1. placing a probabilistic model over the transition probabilities and reward distributions;
2. iterating by sampling an MDP from this model, solving it using your favorite method (e.g. policy iteration), and use the policy for one or more iterations;
3. and finally, collecting the environmental responses and conditioning your probabilistic model.

This works, but we will end up solving a much harder problem than necessary: system identification plus optimal control, or model-based reinforcement learning. It's important to realize that all we really need to know about the environment is how to control it optimally, and this information is nicely encapsulated in the Q-function [following Watkins 1989]. So let's just place a probabilistic model over the Q-function instead. And, to do some different, let's have a look at how to cope with undiscounted MDPs.

### What we will achieve here

We will derive a Bayes-causal algorithm for off-policy, model-free reinforcement learning. At any time, the agent holds beliefs about the Q-table (and the average reward) which it updates using its experience. In order to act, the agent instantiates a random Q-table that it takes as the true one, and thus issues actions that are optimal w.r.t. these simulated beliefs. Q-tables are sampled using MCMC: more precisely, it uses a Gibbs sampler that samples each Q-value from a Normal distribution.

## Undiscounted MDPs

Let's introduce some notation. An MDP is a tuple $(\mathcal{X}, \mathcal{A}, T, r)$, where

• $\mathcal{X}$ is the state space;
• $\mathcal{A}$ is the action space;
• $T_a(x;x') = P(x'|a,x)$ is the probability that an action $a \in \mathcal{A}$ taken in state $x \in \mathcal{X}$ will lead to state $x' \in \mathcal{X}$;
• and $r(x,a) \in \mathcal{R} := \mathbb{R}$ is the immediate reward obtained in state $x \in \mathcal{X}$ under action $a \in \mathcal{A}$.

The agent must play actions to maximize the average reward: $\lim_{t\rightarrow \infty} \frac{1}{T} \sum_{t=1}^T r_t.$ These MDPs are (rather oddly, IMHO) called undiscounted MDPs, and they are useful to build controllers that have to operate indefinitely.

## Optimal policies

For every given MDP there's always an optimal stationary policy, hence we can center our attention on them and ignore the stochastic and nonstationary ones. Thus, we only consider policies $\pi$ that are mappings $\pi: \mathcal{X} \rightarrow \mathcal{A}$.

### Ergodicity

Now we need to make a technical assumption. Imagine we pick a policy $\pi$ and run it on a given MDP for a very long time until we feel that the average reward has converged. Let's say we repeat this procedure several times with different initial states. If the MDP has different “absorbing regions” that the agent could enter but never leave again, then it is possible that the average reward converges to different values depending on the initial state. We now rule out this possibility: we assume that, for every policy, the average reward converges to the same value no matter the initial state (although of course different policies can yield different values). Such an MDP is said to be ergodic (a Greek-ish term dreamt up by Boltzmann).

### Characterizing Optimal Policies

Before we move on to place a probabilistic model over optimal policies, we need to come up with a way of characterizing them. Fortunately, Bellman optimality equations come to the rescue. For undiscounted MDPs, the optimality equations are: \begin{align*} Q(x,a) + \rho &= r(x,a) + \sum_{x' \in \mathcal{X}} P(x'|x,a) \max_{a'} Q(x',a') \\ &= r(x,a) + \mathbb{E}\Bigl[\max_{a'} Q(X',a') \Bigl|x,a \Bigr] \end{align*} where $Q(x,a), Q(x',a')$ are Q-values and $\rho$ is the average reward of the MDP under the optimal policy. It's important to realize here that the set of optimality equations form a system of nonlinear equations that are strongly coupled. If we knew the transition probabilities $P(x'|x,a)$ and rewards $r(x,a)$ then the problem would simply consist in finding a Q-table $\mathbf{Q}$ with entries $Q(x,a)$ and an optimal average reward $\rho$ that solve the system of equations.

The previous optimality equations look very similar to the discounted case, safe for the average reward $\rho$. Therefore, given the Q-table, the optimal action $a^\ast(x)$ in state $x$ is given by $a^\ast(x) = \arg \max_a Q(x,a).$

## Causal-probabilistic model

In order to apply Thompson sampling (using the Bayesian control rule), we now need to describe the stochastic processes that would result when the agent and the environment interact with each other under the assumption that both the Q-table $\mathbf{Q}$ and the average reward $\rho$ are known. To shorten our notation, we collect these parameters into a single parameter structure $\theta = (\rho,\mathbf{Q})$.

In the case of model-free reinforcement learning we consider experience tuples of the form $\langle x, a, r, x' \rangle,$ containing the current state $x$, the current action $a$, the current reward $r$ and the next state $x'$. In particular, since we are not interested in the transition model, we will take

1. the states $x$ and $x'$ as given,
2. the action $a$ as drawn from a policy $P(a|\theta,x)$;
3. and the reward $r$ as drawn from $P(r|\theta,x,a,x')$.

Our next goal is to write down these distributions.

### Actions

This distribution is the policy used by the agent to generate the actions, and from our characterization of the optimal policy from above, we know that the optimal action is the one that maximizes the Q-value of the current state: $P(a|\theta,x) = \begin{cases} 1 & \text{if } a = \arg\max_{a'} Q(x,a') \\ 0 & \text{otherwise.} \end{cases}$

### Rewards

To model the distribution over the rewards, we take the Bellman optimality equation and add and subtract $\max_{a'} Q(x',a')$ to get the equation \begin{align*} r(x,a) = &Q(x,a) + \rho - \max_{a'} Q(x',a') \\ &+ \max_{a'} Q(x',a') - \mathbb{E}\Bigl[ \max_{a'} Q(X',a')\Bigl|x,a \Bigr]. \end{align*} Here, we identity two parts. The first is the mean instantaneous reward defined as $\bar{r}(x,a,x') := Q(x,a) + \rho - \max_{a'} Q(x',a').$ The second is a noise term $\nu := \max_{a'} Q(x',a') - \mathbb{E}\Bigl[ \max_{a'} Q(X',a') \Bigl| x,a \Bigr]$ representing the deviation of the Q-value in the experienced transition from the expected Q-value. This peculiar interpretation might look a bit odd at first, but remember that here we are ignoring the transition model. Hence, we need to purge every mention of transition probabilities from our representation, and in particular also the expectations. Instead, we focus on representing every quantity solely in terms of the average reward $\rho$ and the Q-table $\mathbf{Q}$. Thus, in this restricted representation, the reward becomes a random variable $r(x,a) = \bar{r}(x,a,x') + \nu.$ To simplify things, we will assume that the noise $\nu$ is normal $\mathcal{N}(0,1/p)$ with a known precision $p$. That's a fairly reasonable assumption. The good news is that we can now write down a likelihood model for the immediate reward $r$ using the $Q$-values and the average reward, i.e. \begin{align*} P(r|\theta,x,a,x') &= \mathcal{N}\bigl(\bar{r}(x,a,x'), 1/p \bigr) \\ &= \sqrt{\frac{p}{2\pi}} \exp\Bigl\{ -\frac{p}{2} (r- \bar{r}(x,a,x'))^2 \Bigr\}. \end{align*}

## Learning to act optimally

The previous model assumes that the Q-table and the average reward are known and encapsulated in the big parameter structure $\theta = (\rho,\mathbf{Q})$. Here we are interested in learning it, so we place a prior over the parameter space $\Theta$ to represent our uncertainty. Fortunately, doing so is fairly straightforward.

### Bayesian Model for Rewards

First, we focus on the reward $r$ for a given transition $(x,a,x')$. The likelihood $P(r|\theta,x,a,x') = P(r|\bar{r}(x,a,x')) = \mathcal{N}(r; \bar{r}(x,a,x'), 1/p)$ is a Normal distribution with unknown mean and known precision, and so we can place another Normal $P\bigl(\bar{r}(x,a,x')|\mu(x,a,x'), \lambda(x,a,x')\bigr) = \mathcal{N}\Bigl(\bar{r}(x,a,x'); \mu(x,a,x'), 1/\lambda(x,a,x')\Bigr)$ to model the uncertainty over $\bar{r}(x,a,x')$, because it is conjugate to the likelihood. In this prior, the hyperparameters $\mu(x,a,x')$ and $\lambda(x,a,x')$ represent the mean and precision of the Normal distribution.

When we experience the tuple $(x,a,r,x')$, the new posterior distribution becomes $P\bigl(\bar{r}(x,a,x')|r, \lambda(x,a,x'), \mu(x,a,x')\bigr) = P\bigl(\bar{r}(x,a,x')|\mu'(x,a,x'),\lambda'(x,a,x')\bigr)$ where $\mu'(x,a,x')$ and $\lambda(x,a,x')$ are the new hyperparameters calculated as $\mu'(x,a,x') = \frac{\lambda(x,a,x') \mu(x,a,x') + p r}{ \lambda(x,a,x') + p } \quad\text{and}\quad \lambda'(x,a,x') = \lambda(x,a,x') + p$ that is, the standard update for a Gaussian belief model.

To maintain a model over all the rewards in the MDP, we need to keep track of $\mu(x,a,x')$ and $\lambda(x,a,x')$ for all possible state transitions. In a brute-force implementation, this bookkeeping is done with two 3-dimensional matrices, one for the means $\mu$ and the other for precisions $\lambda$.

### Gibbs Sampler for Q-values and Average Reward

So far, we have only modelled the rewards but not the Q-values, which are really what we're interested in. Fortunately, we can use the previous Bayesian model over instantaneous rewards to derive an approximate Gibbs sampler for the parameter structure $\theta = (\rho,\mathbf{Q})$. For this, we recall that mean instantaneous reward $\bar{r}(x,a,x')$ is equal to $Q(x,a) - \rho + \max_{a'} Q(x',a')$, and hence posterior over the paramater structure is just equal to the product \begin{align*} P(\theta|\mu,\lambda) &= \prod_{x,a,x'} P(\bar{r}(x,a,x')|\mu,\lambda) \\ &\propto \prod_{x,a,x'} \exp\Bigl\{ -\frac{1}{2} \lambda(x,a,x') \bigl(\bar{r}(x,a,x')-\mu(x,a,x')\bigr)^2\Bigr\}\\ &\propto \exp\Bigl\{ -\frac{1}{2} \sum_{x,a,x'} \lambda(x,a,x') \bigl(\bar{r}(x,a,x')-\mu(x,a,x')\bigr)^2\Bigr\}. \end{align*} By replacing each mean instantaneous reward with its representation in terms of $\rho$ and $\mathbf{Q}$, we can calculate various conditional distributions through rearranging the terms and completing the square.

#### Average Reward

With the previous replacement and rearranging the terms, we get that the conditional distribution over the average reward is given by $P(\rho|\mathbf{Q},\mu,\lambda) = \mathcal{N}(\rho; \bar{\rho}, 1/S)$ where the mean and precision are calculated as \begin{align*} \bar{\rho} &:= \frac{1}{S} \sum_{x,a,x'} \lambda(x,a,x)\Bigl(\mu(x,a,x') - Q(x,a) + \max_{a'}Q(x',a')\Bigr) \\ S &:= \sum_{x,a,x'} \lambda(x,a,x'). \end{align*}

The remarkable thing is that the average reward can be just sampled from a Normal distribution with parameters that are simple functions of the hyperparameters $\mu$ and $\lambda$.

#### Q-Values

The conditional distribution over the Q-values is bit uglier though because each $Q(x,a)$ enters the posterior distribution both linearly and non-linearly through $\bar{r}(x,a,x')$, in the form of $Q(x,a)$ and $\max_{a'} Q(x,a')$ terms. However, if we treat $\max_{a'}Q(x,a')$ as a constant for the Gibbs step, then the conditional distribution can be approximated by a Normal distribution: $P\bigl(Q(x,a)|\rho,\mathbf{Q}^{-(x,a)},\lambda,\mu \bigr) \approx \mathcal{N}\Bigr( Q(x,a); \bar{Q}(x,a), 1/S(x,a) \Bigr)$ where $\mathbf{Q}^{-(x,a)}$ denotes the Q-table without the entry $Q(x,a)$. Here, the mean and precision are given by \begin{align*} \bar{Q}(x,a) &:= \frac{1}{\lambda(x,a)} \sum_{x'} \lambda(x,a,x)\Bigl(\mu(x,a,x') - \rho + \max_{a'}Q(x',a')\Bigr) \\ S(x,a) &:= \sum_{x'} \lambda(x,a,x'). \end{align*} We have found this approximation to work surprisingly well in our simulations.

## Putting everything together

We now have all the ingredients we need for our reinforcement learning algorithm:

1. a policy $P(a|\theta,x)$;
2. a reward model $P(r|\theta,x,a,x')$;
3. and a prior distribution over the parameter structure $P(\theta)$.

This probabilistic model can be represented with a causal DAG, where two subsequent states $x_t$ and $x_{t+1}$ are treated as mute conditionals of the joint probability distribution:

To use it, we proceed as follows. In every step $t=1,2,\ldots$, we carry out the following three steps:

1. Current state: We condition the model on the current state $x_t$.
2. Current action: Now we draw an action $a_t$ directly from the model. Since the parameter $\theta$ is unknown, the distribution over the action is $P(a_t|x_t,q_{1:t-1}) = \int P(a_t|\theta,x_t) P(\theta|q_{1:t-1}) \, d\theta$ where we use the shorthand $q_\tau = (x_\tau,a_\tau,r_\tau)$ to represent the past experience. Algorithmically, this can be done in two steps: first by sampling a parameter structure $\theta_t \sim P(\theta|q_{1:t-1})$ and then by sampling the action $a_t \sim P(a_t|\theta_t)$, where we have used $\theta_t$ as a plugin parameter. However, since we are using a value generated by the model itself, we have to the random variable by performing the causal intervention before conditioning.
3. Current reward and next state: We condition the model on the current reward $r_t$ and the next state $x_{t+1}$. This will update the posterior distribution over the parameter structure $\theta$.

Afer performing these three steps, the unrolled model will look as shown below:

The pseudocode below explains how to do all of this.

### Bayesian Control Rule for Undiscounted MDPs

1. Initialize the entries of $\lambda$ and $\mu$ to zero or any other desired value.
2. Set the initial state to $x \leftarrow x_0$.
3. For every time step $t = 1, 2, 3, \ldots$ do:
1. Sample a new average reward and Q-table using Gibbs sampling:
1. $\rho \sim \mathcal{N}(\bar{\rho}, 1/S)$.
2. For each $(x,a)$, do $Q(x,a) \sim \mathcal{N}(\bar{Q}(x,a),1/S(x,a))$.
2. Interact with the world:
1. Pick the next action $a \leftarrow \max_{a'} Q(x,a')$ and issue it.
2. Get the next observation $o = (r,x')$ from the environment.
3. Update the hyperparameters:
1. $\mu(x,a,x') \leftarrow \frac{ \lambda(x,a,x') \mu(x,a,x') + pr }{ \lambda(x,a,x') + p }$
2. $\lambda(x,a,x') \leftarrow \lambda(x,a,x') + p$
4. Update the current state:
1. Set $x \leftarrow x'$.
1) Dearden, R., Friedman, N., and Russell, S.. “Bayesian Q-learning.” AAAI/IAAI. 1998.

format: plain


text:
## Thinking Mathematically (6th Edition)

$39$ square feet.
The area of a rectangle can be calculated as: $T=a\times b$ The area of this figure can be calculated as: Area of the big rectangle (with sides of $2+5$ and $9+3$) minus the area of the small rectangle (with sides of 5 and 9): $T=(2+5)\times(9+3)-(5\times9)=7\times12-45=84-45=39$ square feet.


text:
# CSI Feedback with Autoencoders

This example shows how to use an autoencoder neural network to compress downlink channel state information (CSI) over a clustered delay line (CDL) channel. CSI feedback is in the form of a raw channel estimate array.

### Introduction

In conventional 5G radio networks, CSI parameters are quantities related to the state of a channel that are extracted from the channel estimate array. The CSI feedback includes several parameters, such as the Channel Quality Indication (CQI), the precoding matrix indices (PMI) with different codebook sets, and the rank indicator (RI). The UE uses the CSI reference signal (CSI-RS) to measure and compute the CSI parameters. The user equipment (UE) reports CSI parameters to the access network node (gNB) as feedback. Upon receiving the CSI parameters, the gNB schedules downlink data transmissions with attributes such as modulation scheme, code rate, number of transmission layers, and MIMO precoding. This figure shows an overview of a CSI-RS transmission, CSI feedback, and the transmission of downlink data that is scheduled based on the CSI parameters.

The UE processes the channel estimate to reduce the amount of CSI feedback data. As an alternative approach, the UE compresses and feeds back the channel estimate array. After receipt, the gNB decompresses and processes the channel estimate to determine downlink data link parameters. The compression and decompression can be achieved using an autoencoder neural network [1, 2]. This approach eliminates the use of existing quantized codebook and can improve overall system performance.

This example uses a 5G downlink channel with these system parameters.

txAntennaSize = [2 2 2 1 1]; % rows, columns, polarizations, panels rxAntennaSize = [2 1 1 1 1]; % rows, columns, polarizations, panels rmsDelaySpread = 300e-9; % s maxDoppler = 5; % Hz nSizeGrid = 52; % Number resource blocks (RB) % 12 subcarriers per RB subcarrierSpacing = 15; % 15, 30, 60, 120 kHz numTrainingChEst = 15000; % Carrier definition carrier = nrCarrierConfig; carrier.NSizeGrid = nSizeGrid; carrier.SubcarrierSpacing = subcarrierSpacing
carrier = nrCarrierConfig with properties: NCellID: 1 SubcarrierSpacing: 15 CyclicPrefix: 'normal' NSizeGrid: 52 NStartGrid: 0 NSlot: 0 NFrame: 0 Read-only properties: SymbolsPerSlot: 14 SlotsPerSubframe: 1 SlotsPerFrame: 10
autoEncOpt.NumSubcarriers = carrier.NSizeGrid*12; autoEncOpt.NumSymbols = carrier.SymbolsPerSlot; autoEncOpt.NumTxAntennas = prod(txAntennaSize); autoEncOpt.NumRxAntennas = prod(rxAntennaSize);

### Generate and Preprocess Data

The first step of designing an AI-based system is to prepare training and testing data. For this example, generate simulated channel estimates and preprocess the data. Use 5G Toolbox™ functions to configure a CDL-C channel.

waveInfo = nrOFDMInfo(carrier); samplesPerSlot = ... sum(waveInfo.SymbolLengths(1:waveInfo.SymbolsPerSlot)); channel = nrCDLChannel; channel.DelayProfile = 'CDL-C'; channel.DelaySpread = rmsDelaySpread; % s channel.MaximumDopplerShift = maxDoppler; % Hz channel.RandomStream = "Global stream"; channel.TransmitAntennaArray.Size = txAntennaSize; channel.ReceiveAntennaArray.Size = rxAntennaSize; channel.ChannelFiltering = false; % No filtering for % perfect estimate channel.NumTimeSamples = samplesPerSlot; % 1 slot worth of samples channel.SampleRate = waveInfo.SampleRate;

#### Simulate Channel

Run the channel and get the perfect channel estimate, Hest.

[pathGains,sampleTimes] = channel(); pathFilters = getPathFilters(channel); offset = nrPerfectTimingEstimate(pathGains,pathFilters); Hest = nrPerfectChannelEstimate(carrier,pathGains,pathFilters, ... offset,sampleTimes);

The channel estimate matrix is an $\left[{\mathit{N}}_{\mathrm{subcarriers}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{symbols}\text{\hspace{0.17em}}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{rx}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{tx}}\right]$ array for each slot.

[nSub,nSym,nRx,nTx] = size(Hest)
nSub = 624
nSym = 14
nRx = 2
nTx = 8

Plot the channel response. The upper left plot shows the channel frequency response as a function of time (symbols) for receive antenna 1 and transmit antenna 1. The lower left plot shows the channel frequency response as a function of transmit antennas for symbol 1 and receive antenna 1. The upper right plot shows the channel frequency response for all receive antennas for symbol 1 and transmit antenna 1. The lower right plot shows the change in channel magnitude response as a function of transmit antennas for all receive antennas for subcarrier 400 and symbol 1.

plotChannelResponse(Hest)

#### Preprocess Channel Estimate

Preprocess the channel estimate to reduce the size and convert it to a real-valued array. This figure shows the channel estimate reduction preprocess.

Assume that the channel coherence time is much larger than the slot time. Average the channel estimate over a slot and obtain a $\left[{\mathit{N}}_{\mathrm{subcarriers}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}1\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{rx}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{tx}}\right]$array.

Hmean = mean(Hest,2);

To enable operation on subcarriers and Tx antennas, move the Tx and Rx antenna dimensions to the second and third dimensions, respectively.

Hmean = permute(Hmean,[1 4 3 2]);

To obtain the delay-angle representation of the channel, apply a 2-D discrete Fourier transform (DFT) over subcarriers and Tx antennas for each Rx antenna and slot. To demonstrate the workflow and reduce runtime, this subsection processes Rx channel 1 only.

Hdft2 = fft2(Hmean(:,:,1));

Since the multipath delay in the channel is limited, truncate the delay dimension to remove values that do not carry information. The sampling period on the delay dimension is ${T}_{delay}=1/\left({N}_{subcarriers}*{F}_{ss}\right)$, where ${F}_{ss}$ is subcarrier spacing. The expected RMS delay spread in delay samples is ${\tau }_{RMS}/{T}_{delay}$, where ${\tau }_{RMS}$ is the RMS delay spread of the channel in seconds.

Tdelay = 1/(autoEncOpt.NumSubcarriers*carrier.SubcarrierSpacing*1e3); rmsTauSamples = channel.DelaySpread / Tdelay; maxTruncationFactor = floor(autoEncOpt.NumSubcarriers / rmsTauSamples);

Truncate the channel estimate to an even number of samples that is 10 times the expected RMS delay spread. Increasing the truncationFactor value can decrease the performance loss due to preprocessing. But, doing so increases the neural network complexity, number of required training data points, and training time. A neural network with more learnable parameters might not converge to a better solution.

truncationFactor = 10; maxDelay = round((channel.DelaySpread/Tdelay)*truncationFactor/2)*2
maxDelay = 28
autoEncOpt.MaxDelay = maxDelay;

Calculate the truncation indices and truncate the channel estimate.

midPoint = floor(nSub/2); lowerEdge = midPoint - (nSub-maxDelay)/2 + 1; upperEdge = midPoint + (nSub-maxDelay)/2; Htemp = Hdft2([1:lowerEdge-1 upperEdge+1:end],:);

To get back to the subcarriers-Tx antennas domain, apply a 2-D inverse discrete Fourier transform (IDFT) to the truncated array [2]. This process effectively decimates the channel estimate in the subcarrier axis.

Htrunc = ifft2(Htemp);

Separate the real and imaginary parts of the channel estimate to obtain a $\left[{\mathit{N}}_{\mathrm{delay}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{tx}}\text{\hspace{0.17em}}2\right]$ array.

HtruncReal = zeros(maxDelay,nTx,2); HtruncReal(:,:,1) = real(Htrunc); HtruncReal(:,:,2) = imag(Htrunc); %#ok<NASGU>

Plot the channel estimate signal through the preprocessing steps. Images are scaled to help visualization.

plotPreprocessingSteps(Hmean(:,:,1),Hdft2,Htemp,Htrunc,nSub,nTx, ... maxDelay)

#### Prepare Data in Bulk

The helperCSINetTrainingData helper function generates numTrainingChEst of preprocessed $\left[{\mathit{N}}_{\mathrm{delay}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{tx}}\text{\hspace{0.17em}}2\right]$ channel estimates by using the process described in this section. The function saves each $\left[{\mathit{N}}_{\mathrm{delay}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}{\mathit{N}}_{\mathrm{tx}}\text{\hspace{0.17em}}2\right]$ channel estimate as an individual file in the dataDir with the prefix of trainingDataFilePrefix. If Parallel Computing Toolbox™ is available, helperCSINetTrainingData function uses parfor to parallelize data generation. Data generation takes less than three minutes on a PC with Intel® Xeon® W-2133 CPU @ 3.60GHz and running in parallel on six workers.

dataDir = fullfile(exRoot(),"Data"); trainingDataFilePrefix = "nr_channel_est"; if validateTrainingFiles(dataDir,trainingDataFilePrefix, ... numTrainingChEst,autoEncOpt,channel,carrier) == false disp("Starting training data generation") tic autoEncOpt.Normalization = false; % Do not normalize data yet helperCSINetTrainingData(dataDir,trainingDataFilePrefix, ... numTrainingChEst,carrier,channel,autoEncOpt); t = seconds(toc); t.Format = "hh:mm:ss"; disp(string(t) + " - Finished training data generation") end
Starting training data generation
6 workers running 00:00:14 - 8% Completed 00:00:26 - 16% Completed 00:00:38 - 24% Completed 00:00:50 - 32% Completed 00:01:03 - 40% Completed 00:01:15 - 48% Completed 00:01:29 - 56% Completed 00:01:41 - 64% Completed 00:01:54 - 72% Completed 00:02:06 - 80% Completed 00:02:18 - 88% Completed 00:02:30 - 96% Completed
00:02:37 - Finished training data generation

Create a signalDatastore object to access the data. The signal datastore uses individual files for each data point.

sds = signalDatastore( ... fullfile(dataDir,"processed",trainingDataFilePrefix+"_*"));

Load data into memory, calculate the mean value and standard deviation, and then use the mean and standard deviation values to normalize the data.

HtruncRealCell = readall(sds); HtruncReal = cat(4,HtruncRealCell{:}); meanVal = mean(HtruncReal,'all')
meanVal = single -0.0236
stdVal = std(HtruncReal,[],'all')
stdVal = single 16.0657

Separate the data into training, validation, and test sets. Also, normalize the data to achieve zero mean and a target standard deviation of 0.0212, which restricts most of the data to the range of [-0.5 0.5].

N = size(HtruncReal, 4); numTrain = floor(N*10/15)
numTrain = 10000
numVal = floor(N*3/15)
numVal = 3000
numTest = floor(N*2/15)
numTest = 2000
targetStd = 0.0212; HTReal = (HtruncReal(:,:,:,1:numTrain)-meanVal) ... /stdVal*targetStd+0.5; HVReal = (HtruncReal(:,:,:,numTrain+(1:numVal))-meanVal) ... /stdVal*targetStd+0.5; HTestReal = (HtruncReal(:,:,:,numTrain+numVal+(1:numTest))-meanVal) ... /stdVal*targetStd+0.5; autoEncOpt.MeanVal = meanVal; autoEncOpt.StdValue = stdVal; autoEncOpt.TargetSTDValue = targetStd; %#ok<STRNU>

### Define and Train Neural Network Model

The second step of designing an AI-based system is to define and train the neural network model.

#### Define Neural Network

This example uses a modified version of the autoencoder neural network proposed in [1].

inputSize = [maxDelay nTx 2]; % Third dimension is real and imaginary parts nLinear = prod(inputSize); nEncoded = 64; autoencoderLGraph = layerGraph([ ... % Encoder imageInputLayer(inputSize,"Name","Htrunc", ... "Normalization","none","Name","Enc_Input") convolution2dLayer([3 3],2,"Padding","same","Name","Enc_Conv") batchNormalizationLayer("Epsilon",0.001,"MeanDecay",0.99, ... "VarianceDecay",0.99,"Name","Enc_BN") leakyReluLayer(0.3,"Name","Enc_leakyRelu") flattenLayer("Name","Enc_flatten") fullyConnectedLayer(nEncoded,"Name","Enc_FC") sigmoidLayer("Name","Enc_Sigmoid") % Decoder fullyConnectedLayer(nLinear,"Name","Dec_FC") functionLayer(@(x)dlarray(reshape(x,maxDelay,nTx,2,[]),'SSCB'), ... "Formattable",true,"Acceleratable",true,"Name","Dec_Reshape") ]); autoencoderLGraph = ... helperCSINetAddResidualLayers(autoencoderLGraph, "Dec_Reshape"); autoencoderLGraph = addLayers(autoencoderLGraph, ... [convolution2dLayer([3 3],2,"Padding","same","Name","Dec_Conv") ... sigmoidLayer("Name","Dec_Sigmoid") ... regressionLayer("Name","Dec_Output")]); autoencoderLGraph = ... connectLayers(autoencoderLGraph,"leakyRelu_2_3","Dec_Conv"); figure plot(autoencoderLGraph) title('CSI Compression Autoencoder')

#### Train Neural Network

Set the training options for the autoencoder neural network and train the network using the trainNetwork (Deep Learning Toolbox) function. Training takes less than 15 minutes on an AMD EPYC 7262 3.2 GHz 8C/16T with 8 NVIDIA RTX A5000 GPUs with ExecutionEnvironment set to 'multi-gpu'. Set trainNow to false to load the pretrained network.

trainNow = false; miniBatchSize = 1000; options = trainingOptions("adam", ... InitialLearnRate=0.0074, ... LearnRateSchedule="piecewise", ... LearnRateDropPeriod=112, ... LearnRateDropFactor=0.6085, ... Epsilon=1e-7, ... MaxEpochs=1000, ... MiniBatchSize=miniBatchSize, ... Shuffle="every-epoch", ... ValidationData={HVReal,HVReal}, ... ValidationFrequency=20, ... Verbose=false, ... ValidationPatience=20, ... OutputNetwork="best-validation-loss", ... ExecutionEnvironment="auto", ... Plots='training-progress') %#ok<NASGU>
options = TrainingOptionsADAM with properties: GradientDecayFactor: 0.9000 SquaredGradientDecayFactor: 0.9990 Epsilon: 1.0000e-07 InitialLearnRate: 0.0074 LearnRateSchedule: 'piecewise' LearnRateDropFactor: 0.6085 LearnRateDropPeriod: 112 L2Regularization: 1.0000e-04 GradientThresholdMethod: 'l2norm' GradientThreshold: Inf MaxEpochs: 1000 MiniBatchSize: 1000 Verbose: 0 VerboseFrequency: 50 ValidationData: {[28×8×2×3000 single] [28×8×2×3000 single]} ValidationFrequency: 20 ValidationPatience: 20 Shuffle: 'every-epoch' CheckpointPath: '' CheckpointFrequency: 1 CheckpointFrequencyUnit: 'epoch' ExecutionEnvironment: 'auto' WorkerLoad: [] OutputFcn: [] Plots: 'training-progress' SequenceLength: 'longest' SequencePaddingValue: 0 SequencePaddingDirection: 'right' DispatchInBackground: 0 ResetInputNormalization: 1 BatchNormalizationStatistics: 'population' OutputNetwork: 'best-validation-loss'
if trainNow [net,trainInfo] = ... trainNetwork(HTReal,HTReal,autoencoderLGraph,options); %#ok<UNRCH> save("csiTrainedNetwork_" ... + string(datetime("now","Format","dd_MM_HH_mm")), ... 'net','trainInfo','options','autoEncOpt') else helperCSINetDownloadData() load("csiTrainedNetwork",'net','trainInfo','options','autoEncOpt') end
Starting download of data files from: https://www.mathworks.com/supportfiles/spc/CSI/TrainedCSIFeedbackAutoencoder_v22b1.tar Download complete. Extracting files. Extract complete.

### Test Trained Network

Use the predict (Deep Learning Toolbox) function to process the test data.

HTestRealHat = predict(net,HTestReal);

Calculate the correlation and normalized mean squared error (NMSE) between the input and output of the autoencoder network. The correlation is defined as

$\rho =\mathbb{E}\left\{\frac{1}{N}\sum _{n=1}^{N}\frac{|{\underset{}{\overset{ˆ}{h}}}_{n}^{H}{h}_{n}|}{‖{\underset{}{\overset{ˆ}{h}}}_{n}{‖}_{2}‖{h}_{n}{‖}_{2}}\right\}$

where ${h}_{n}$ is the channel estimate at the input of the autoencoder and ${\underset{}{\overset{ˆ}{h}}}_{n}$ is the channel estimate at the output of the autoencoder. NMSE is defined as

$NMSE=\mathbb{E}\left\{\frac{{‖H-\underset{}{\overset{ˆ}{H}}‖}_{2}^{2}}{{‖H‖}_{2}^{2}}\right\}$

where $H$ is the channel estimate at the input of the autoencoder and $\underset{}{\overset{ˆ}{H}}$ is the channel estimate at the output of the autoencoder.

rho = zeros(numTest,1); nmse = zeros(numTest,1); for n=1:numTest in = HTestReal(:,:,1,n) + 1i*(HTestReal(:,:,2,n)); out = HTestRealHat(:,:,1,n) + 1i*(HTestRealHat(:,:,2,n)); % Calculate correlation n1 = sqrt(sum(conj(in).*in,'all')); n2 = sqrt(sum(conj(out).*out,'all')); aa = abs(sum(conj(in).*out,'all')); rho(n) = aa / (n1*n2); % Calculate NMSE mse = mean(abs(in-out).^2,'all'); nmse(n) = 10*log10(mse / mean(abs(in).^2,'all')); end figure tiledlayout(2,1) nexttile histogram(rho,"Normalization","probability") grid on title(sprintf("Autoencoder Correlation (Mean \\rho = %1.5f)", ... mean(rho))) xlabel("\rho"); ylabel("PDF") nexttile histogram(nmse,"Normalization","probability") grid on title(sprintf("Autoencoder NMSE (Mean NMSE = %1.2f dB)",mean(nmse))) xlabel("NMSE (dB)"); ylabel("PDF")

### End-to-End CSI Feedback System

This figure shows the end-to-end processing of channel estimates for CSI feedback. The UE uses the CSI-RS signal to estimate the channel response for one slot, ${H}_{est}$. The preprocessed channel estimate, ${H}_{tr}$, is encoded by using the encoder portion of the autoencoder to produce a 1-by-${N}_{enc}$ compressed array. The compressed array is decompressed by the decoder portion of the autoencoder to obtain $\underset{}{\overset{ˆ}{{H}_{tr}}}$. Postprocessing $\underset{}{\overset{ˆ}{{H}_{tr}}}$ produces $\underset{}{\overset{ˆ}{{H}_{est}}}$.

To obtain the encoded array, split the autoencoder into two parts: the encoder network and the decoder network.

[encNet,decNet] = helperCSINetSplitEncoderDecoder(net,"Enc_Sigmoid"); plotNetwork(net,encNet,decNet)

Generate channel estimates.

nSlots = 100; Hest = helperCSINetChannelEstimate(nSlots,carrier,channel);

Encode and decode the channel estimates with Normalization set to true.

autoEncOpt.Normalization = true; codeword = helperCSINetEncode(encNet, Hest, autoEncOpt); Hhat = helperCSINetDecode(decNet, codeword, autoEncOpt);

Calculate the correlation and NMSE for the end-to-end CSI feedback system.

H = squeeze(mean(Hest,2)); rhoE2E = zeros(nRx,nSlots); nmseE2E = zeros(nRx,nSlots); for rx=1:nRx for n=1:nSlots out = Hhat(:,rx,:,n); in = H(:,rx,:,n); rhoE2E(rx,n) = helperCSINetCorrelation(in,out); nmseE2E(rx,n) = helperNMSE(in,out); end end figure tiledlayout(2,1) nexttile histogram(rhoE2E,"Normalization","probability") grid on title(sprintf("End-to-End Correlation (Mean \\rho = %1.5f)", ... mean(rhoE2E,'all'))) xlabel("\rho"); ylabel("PDF") nexttile histogram(nmseE2E,"Normalization","probability") grid on title(sprintf("End-to-End NMSE (Mean NMSE = %1.2f dB)", ... mean(nmseE2E,'all'))) xlabel("NMSE (dB)"); ylabel("PDF")

### Effect of Quantized Codewords

Practical systems require quantizing the encoded codeword by using a small number of bits. Simulate the effect of quantization across the range of [2, 10] bits. The results show that 6-bits is enough to closely approximate the single-precision performance.

maxVal = 1; minVal = -1; idxBits = 1; nBitsVec = 2:10; rhoQ = zeros(nRx,nSlots,length(nBitsVec)); nmseQ = zeros(nRx,nSlots,length(nBitsVec)); for numBits = nBitsVec disp("Running for " + numBits + " bit quantization") % Quantize between 0:2^n-1 to get bits qCodeword = uencode(double(codeword*2-1), numBits); % Get back the floating point, quantized numbers codewordRx = (single(udecode(qCodeword,numBits))+1)/2; Hhat = helperCSINetDecode(decNet, codewordRx, autoEncOpt); H = squeeze(mean(Hest,2)); for rx=1:nRx for n=1:nSlots out = Hhat(:,rx,:,n); in = H(:,rx,:,n); rhoQ(rx,n,idxBits) = helperCSINetCorrelation(in,out); nmseQ(rx,n,idxBits) = helperNMSE(in,out); end end idxBits = idxBits + 1; end
Running for 2 bit quantization Running for 3 bit quantization Running for 4 bit quantization Running for 5 bit quantization Running for 6 bit quantization Running for 7 bit quantization Running for 8 bit quantization Running for 9 bit quantization Running for 10 bit quantization
figure tiledlayout(2,1) nexttile plot(nBitsVec,squeeze(mean(rhoQ,[1 2])),'*-') title("Correlation (Codeword-" + size(codeword,3) + ")") xlabel("Number of Quantization Bits"); ylabel("\rho") grid on nexttile plot(nBitsVec,squeeze(mean(nmseQ,[1 2])),'*-') title("NMSE (Codeword-" + size(codeword,3) + ")") xlabel("Number of Quantization Bits"); ylabel("NMSE (dB)") grid on

### Further Exploration

The autoencoder is able to compress a [624 8] single-precision complex channel estimate array into a [64 1] single-precision array with a mean correlation factor of 0.99 and an NMSE of –16 dB. Using 6-bit quantization requires only 384 bits of CSI feedback data, which equates to a compression ratio of approximately 800:1.

display("Compression ratio is " + (624*8*32*2)/(64*6) + ":" + 1)
 "Compression ratio is 832:1"

Investigate the effect of truncationFactor on the system performance. Vary the 5G system parameters, channel parameters, and number of encoded symbols and then find the optimum values for the defined channel.

The NR PDSCH Throughput Using Channel State Information Feedback (5G Toolbox) example shows how to use channel state information (CSI) feedback to adjust the physical downlink shared channel (PDSCH) parameters and measure throughput. Replace the CSI feedback algorithm with the CSI compression autoencoder and compare performance.

#### Helper Functions

Explore the helper functions to see the detailed implementation of the system.

#### Training Data Generation

helperCSINetChannelEstimate

helperCSINetTrainingData

#### Network Definition and Manipulation

helperCSINetLayerGraph

helperCSINetSplitEncoderDecoder

#### CSI Processing

helperCSINetPreprocessChannelEstimate

helperCSINetPostprocessChannelEstimate

helperCSINetEncode

helperCSINetDecode

#### Performance Measurement

helperCSINetCorrelation

helperNMSE

### Appendix: Optimize Hyperparameters with Experiment Manager

Use the Experiment Manager app to find the optimal parameters. CSITrainingProject.mlproj is a preconfigured project. Extract the project.

if ~exist("CSITrainingProject","dir") projRoot = helperCSINetExtractProject(); else projRoot = fullfile(exRoot(),"CSITrainingProject"); end

To open the project, start the Experiment Manager app and open the following file.

disp(fullfile(".","CSITrainingProject","CSITrainingProject.prj"))
.\CSITrainingProject\CSITrainingProject.prj

The Optimize Hyperparameters experiment uses Bayesian optimization with hyperparameter search ranges specified as in the following figure. The experiment setup function is CSIAutoEncNN_setup. The custom metric function is NMSE.

The optimal parameters are 0.0074 for initial learning rate, 112 iterations for the learning rate drop period, and 0.6085 for learning rate drop factor. After finding the optimal hyperparameters, train the network with same parameters multiple times to find the best trained network. Increase the maximum iterations by a factor of two.

The sixth trial produced the best NMSE. This example uses this trained network as the saved network.

#### Configuring Batch Mode

When execution Mode is set to Batch Sequential or Batch Simultaneous, training data must be accessible to the workers in a location defined by the dataDir variable in the Prepare Data in Bulk section. Set dataDir to a network location that is accessible by the workers. For more information, see Offload Experiments as Batch Jobs to Cluster (Deep Learning Toolbox).

### Local Functions

function plotChannelResponse(Hest) %plotChannelResponse Plot channel response figure tiledlayout(2,2) nexttile waterfall(abs(Hest(:,:,1,1))') xlabel("Subcarriers"); ylabel("Symbols"); zlabel("Channel Magnitude") view(15,30) colormap("cool") title("Rx=1, Tx=1") nexttile plot(squeeze(abs(Hest(:,1,:,1)))) grid on xlabel("Subcarriers"); ylabel("Channel Magnitude") legend("Rx 1", "Rx 2") title("Symbol=1, Tx=1") nexttile waterfall(squeeze(abs(Hest(:,1,1,:)))') view(-45,75) grid on xlabel("Subcarriers"); ylabel("Tx"); zlabel("Channel Magnitude") title("Symbol=1, Rx=1") nexttile plot(squeeze(abs(Hest(400,1,:,:)))') grid on xlabel("Tx"); ylabel("Channel Magnitude") legend("Rx 1", "Rx 2") title("Subcarrier=400, Symbol=1") end function valid = validateTrainingFiles(dataDir,filePrefix,expN, ... opt,channel,carrier) %validateTrainingFiles Validate training data files % V = validateTrainingFiles(DIR,PRE,N,OPT,CH,CR) checks the DIR directory % for training data files with a prefix of PRE. It checks if there are % N*OPT.NumRxAntennas files, channel configuration is same as CH, and % carrier configuration is same as CR. valid = true; files = dir(fullfile(dataDir,filePrefix+"*")); if isempty(files) valid = false; return end if exist(fullfile(dataDir,"info.mat"),"file") infoStr = load(fullfile(dataDir,"info.mat")); if ~isequal(get(infoStr.channel),get(channel)) ... || ~isequal(infoStr.carrier,carrier) valid = false; end else valid = false; end if valid valid = (expN <= (length(files)*opt.NumRxAntennas)); % Check size of Hest in the files load(fullfile(files(1).folder,files(1).name),'H') if ~isequal(size(H),[opt.NumSubcarriers opt.NumSymbols ... opt.NumRxAntennas opt.NumTxAntennas]) valid = false; end end if ~valid || expN < (length(files)*opt.NumRxAntennas) disp("Removing invalid data directory: " + files(1).folder) rmdir(files(1).folder,'s') end end function plotNetwork(net,encNet,decNet) %plotNetwork Plot autoencoder network % plotNetwork(NET,ENC,DEC) plots the full autoencoder network together % with encoder and decoder networks. fig = figure; t1 = tiledlayout(1,2,'TileSpacing','Compact'); t2 = tiledlayout(t1,1,1,'TileSpacing','Tight'); t3 = tiledlayout(t1,2,1,'TileSpacing','Tight'); t3.Layout.Tile = 2; nexttile(t2) plot(net) title("Autoencoder") nexttile(t3) plot(encNet) title("Encoder") nexttile(t3) plot(decNet) title("Decoder") pos = fig.Position; pos(3) = pos(3) + 200; pos(4) = pos(4) + 300; pos(2) = pos(2) - 300; fig.Position = pos; end function plotPreprocessingSteps(Hmean,Hdft2,Htemp,Htrunc, ... nSub,nTx,maxDelay) %plotPreprocessingSteps Plot preprocessing workflow hfig = figure; hfig.Position(3) = hfig.Position(3)*2; subplot(2,5,[1 6]) himg = imagesc(abs(Hmean)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.XTick=''; himg.Parent.YTick=''; xlabel(sprintf('Tx\nAntennas\n(%d)',nTx)); ylabel(sprintf('Subcarriers\n(%d)',nSub')); title("Measured") subplot(2,5,[2 7]) himg = image(abs(Hdft2)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.XTick=''; himg.Parent.YTick=''; title("2-D DFT") xlabel(sprintf('Tx\nAngle\n(%d)',nTx)); ylabel(sprintf('Delay Samples\n(%d)',nSub')); subplot(2,5,[3 8]) himg = image(abs(Htemp)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.Position(4) = himg.Parent.Position(4)*10*maxDelay/nSub; himg.Parent.Position(2) = (1 - himg.Parent.Position(4)) / 2; himg.Parent.XTick=''; himg.Parent.YTick=''; xlabel(sprintf('Tx\nAngle\n(%d)',nTx)); ylabel(sprintf('Delay Samples\n(%d)',maxDelay')); title("Truncated") subplot(2,5,[4 9]) himg = imagesc(abs(Htrunc)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.Position(4) = himg.Parent.Position(4)*10*maxDelay/nSub; himg.Parent.Position(2) = (1 - himg.Parent.Position(4)) / 2; himg.Parent.XTick=''; himg.Parent.YTick=''; xlabel(sprintf('Tx\nAntennas\n(%d)',nTx)); ylabel(sprintf('Subcarriers\n(%d)',maxDelay')); title("2-D IDFT") subplot(2,5,5) himg = imagesc(real(Htrunc)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.Position(4) = himg.Parent.Position(4)*10*maxDelay/nSub; himg.Parent.Position(2) = himg.Parent.Position(2) + 0.18; himg.Parent.XTick=''; himg.Parent.YTick=''; xlabel(sprintf('Tx\nAntennas\n(%d)',nTx)); ylabel(sprintf('Subcarriers\n(%d)',maxDelay')); title("Real") subplot(2,5,10) himg = imagesc(imag(Htrunc)); himg.Parent.YDir = "normal"; himg.Parent.Position(3) = 0.05; himg.Parent.Position(4) = himg.Parent.Position(4)*10*maxDelay/nSub; himg.Parent.Position(2) = himg.Parent.Position(2) + 0.18; himg.Parent.XTick=''; himg.Parent.YTick=''; xlabel(sprintf('Tx\nAntennas\n(%d)',nTx)); ylabel(sprintf('Subcarriers\n(%d)',maxDelay')); title("Imaginary") end function rootDir = exRoot() %exRoot Example root directory rootDir = fileparts(which("helperCSINetLayerGraph")); end

### References

[1] Wen, Chao-Kai, Wan-Ting Shih, and Shi Jin. “Deep Learning for Massive MIMO CSI Feedback.” IEEE Wireless Communications Letters 7, no. 5 (October 2018): 748–51. https://doi.org/10.1109/LWC.2018.2818160.

[2] Zimaglia, Elisa, Daniel G. Riviello, Roberto Garello, and Roberto Fantini. “A Novel Deep Learning Approach to CSI Feedback Reporting for NR 5G Cellular Systems.” In 2020 IEEE Microwave Theory and Techniques in Wireless Communications (MTTW), 47–52. Riga, Latvia: IEEE, 2020. https://doi.org/10.1109/MTTW51045.2020.9245055.


text:
When I use tikz, it always creates a new page. Anybody could help?

I was trying to insert a tex file containing the code for a tikz graph into a main tex file. However, I got a problem. The graph always stands alone in an independent page. So any words before or after the inserting command will be a page ahead or a page afterward. The tex file was automatically generated by R package "tikzDevice"

My Latex code is very simple.

\title{homework}
\author{me}
\date{}
\begin{document}
\maketitle
\setmainfont{Microsoft Sans Serif}
\centering
\include{Plot1}
crazy
\include{Plot1}
\end{document}

My tikz tex code for the graph is as below:

\resizebox{\columnwidth}{!}{
\begin{tikzpicture}[x=1pt,y=1pt]
...
\end{tikzpicture}
}

• Please don't use \resizebox for elements that contain text, like the graph you show us in your image. See tex.stackexchange.com/questions/425453/… for other possibilities instead of \resizebox – samcarter Dec 16 '18 at 12:59
• Thank you samcarter! I would check it out! – Hanqing Wu Dec 16 '18 at 13:17

I just found the solution. I should use \input instead of \include!


text:
# Booting from grub

Published:

## The Problem

I have a dual system window10/Ubuntu16.04 installed in my laptop. Today I can not access window files from Ubuntu and tried one command line from youtube which seems to mess things up :< The system did not boot like before but entering into the grub prompt instead. It seems that system does not know where/what to boot now and may need a manual configuration.

## The solution

I spent some time googling and found these two posts quite useful for me. Here and Here

First, I changed the booting priority in bios so that linux will boot first. I have an Asus laptop. To enter into BIOS, hold F2 when starting your machine.

### Find the root system

When in grub prompt, type ls to check files in the system. You will see things like (hd0),(hd0,gpt1), … ls each one to find where the root system is (you can omit the gpt label). For me, it is (hd0,gpt9). ls (hd0,9)/ outputs things like:

lost+found/ bin/ ... vmlinuz ... initrd.img ...

If you have many linuxes installed, double check if it is the right system by command cat (hd0,9)/etc/issue.

### Boot the system

Now, it is time let GRUB know the system you want to boot by typing commend:

grub> set root (hd0,9)
grub> Linux /vmlinuz root=/dev/sta5
grub> initrd /initrd.img
grub> boot

• If you do not know where your root device is, you can use UUID for specifying the path. type cat /etc/fstab and check the output. The UUID for the root device is located in the entry where \ is in the <mount on> column. Then you can replace root=/dev/sta5 by root=UUID=[UUID you found].

• If you want to specify the kernel version of your linux, you can check available ones by typing ls (hd0,9)/boot/. To suppress the output, you can do set pager=1. Now you can boot the system you want by

grub> set root (hd0,9)
grub> Linux /boot/vmlinuz-[version] root=/dev/sta5
grub> initrd /boot/initrd.img-[version]
grub> boot

The version has to be the same in the second and thrid command line or you will get a kernel panic.

Hurrah, the system now boots correctly. To fix it permantly, after you boot succesfully. Type these in the shell:

$update-grub$ grub-install /dev/sda

Hopefully, I did not mess up with linux again…

Tags:


text:
Browse Questions

# If $3 \cos x \neq 2 \sin x,$ then the general solution of $\sin ^2 x -\cos 2x=2 -\sin 2x$ is $x=$

$\begin {array} {1 1} (1)\;n \pi +(-1)^n \frac{\pi}{2}, n \in Z & \quad (2)\;\frac{n\pi}{2}, n \in Z \\ (3)\;(4 n \pm 1) \frac{\pi}{2}, n \in Z & \quad (4)\;(2n-1) \pi, n \in Z \end {array}$

$(3)\;(4 n \pm 1) \frac{\pi}{2}, n \in Z$


text:
# Knowing the particular solution $y_1 = \frac{\sin(x)}{x}$ solve: $y'' + \frac{2}{x}y' + y = 0$

Knowing the particular solution $$y_1 = \frac{\sin(x)}{x}$$ solve: $$y'' + \frac{2}{x}y' + y = 0$$

I set the substitution $$y = vy_1 = v\frac{\sin(x)}{x}$$. I get the following:

$$y' = v' \frac{\sin(x)}{x} +v\frac{x\cos(x) - \sin(x)}{x^2}$$ $$y'' = v''\frac{\sin(x)}{x} + 2v'\frac{\cos(x)}{x}-2v'\frac{\sin(x)}{x^2}-2v\frac{\cos(x)}{x^2}-v\frac{\sin(x)}{x}-v\frac{2\sin(x)}{x^3}$$

Which results in:

$$v'' + v'(2\cot(x)) = \frac{4}{x^2}$$

Now, this linear differential is not very clean, and I am not sure if there is an error somewhere, or if this approach is invalid.

$$y'' + \frac{2}{x}y' + y = 0$$ The DE is simply: $$(xy)''+xy=0$$ $$\implies r^2+1=0 \implies r=\pm i$$ $$xy=c_1\cos x +c_2\sin x$$ $$y(x)=\dfrac 1 x (c_1\cos x +c_2\sin x)$$
You made a little mistake you should get: $$(xy)''+xy=0$$ $$(v\sin x)''+v \sin x=0$$ $$v'' \sin x +2v' \cos x=0$$ $$v''+2 v'\cot x=0$$


text:
Calculus 2 : Definite Integrals

Example Questions

Example Question #101 : Definite Integrals

Explanation:

First, integrate the expression. Remember to raise the exponent by 1 and then put that result on the denominator:

.

Then, evaluate at 2 and then 0.

Subtract the two:

Example Question #101 : Definite Integrals

What is the area underneath the curve from x=4 to x=6 and bounded by the axis?

Explanation:

First set up the integral expression:

.

Then, integrate each term separately. Remember to raise the exponent by 1 and then put that result on the denominator as well:

.

Then evaluate at 6 and then 4. Subtract the results.

.

.

Example Question #103 : Definite Integrals

Explanation:

First, chop up the fraction into two separate terms:

.

Then, integrate those to get

.

Then evaluate at 2 and then 1. Subtract the results:

.

Example Question #104 : Definite Integrals

Calculate the value of the definite integral

Explanation:

The antiderivative of is .

Using the corollary to the Fundamental Theorem of Calculus we have

Example Question #102 : Definite Integrals

Calculate the value of the definite integral

Explanation:

To solve this definite integral, we use u-substitution.

and

We then solve the integral

Example Question #103 : Definite Integrals

Evaluate.

Explanation:

In this case, .

The antiderivative is  .

Using the Fundamental Theorem of Calculus, evaluate the integral using the antiderivative:

Example Question #104 : Definite Integrals

Evaluate.

Explanation:

In this case, .

The antiderivative is  .

Using the Fundamental Theorem of Calculus, evaluate the integral using the antiderivative:

Example Question #105 : Definite Integrals

Evaluate.

Explanation:

In this case, .

The antiderivative is  .

Using the Fundamental Theorem of Calculus, evaluate the integral using the antiderivative:

Example Question #106 : Definite Integrals

Evaluate.

Explanation:

In this case, .

The antiderivative is  .

Using the Fundamental Theorem of Calculus, evaluate the integral using the antiderivative:

Evaluate.


text:
Memory Sharing - Algorithmica

# Memory Sharing

Starting at some level of the hierarchy, the cache becomes shared between different cores. This reduces the total die area and lets you add more cores on a single chip but also poses some “noisy neighbor” problems as it limits the effective cache size and bandwidth available to a single execution thread.

On most CPUs, only the last layer of cache is shared, and not always in a uniform manner. On my machine, there are 8 physical cores, and the size of the L3 cache is 8M, but it is split into two halves: two groups of 4 cores have access to their own 4M region of the L3 cache, and not all of it.

There are even more complex topologies, where accessing certain regions of memory takes non-constant time, different for each core (which is sometimes unintended). Such architectural feature is called non-uniform memory access (NUMA), and it is the case for multi-socket systems that have several separate CPU chips installed.

On Linux, the topology of the memory system can be retrieved with lstopo:

This has some important implications for parallel algorithms: the performance of multi-threaded memory accesses depends on which cores are running which execution threads. To demonstrate this, we will run the bandwidth benchmarks in parallel.

### #CPU Affinity

Instead of modifying the source code to run on multiple threads, we can simply run multiple identical processes with GNU parallel. To control which cores are executing them, we set their processor affinity with taskset. This combined command runs 4 processes that can run on the first 4 cores of the CPU:

parallel taskset -c 0,1,2,3 ./run ::: {0..3}

Here is what we get when we change the number of processes running simultaneously:

You can now see that the performance decreases with more processes when the array exceeds the L2 cache (which is private to each core), as the cores start competing for the shared L3 cached and the RAM.

We specifically set all processes to run on the first 4 cores because they have a unified L3 cache. If some of the processes were to be scheduled on the other half of the cores, there would be less contention for the L3 cache. The operating system doesn’t monitor such activities — what a process does is its own private business — so by default, it assigns threads to cores arbitrarily during execution, without caring about cache affinity and only taking into account the core load.

Let’s run another benchmark, but now with pinning the processes to different 4-core groups that don’t share L3 cache:

parallel taskset -c 0,1 ./run ::: {0..1}  # L3 cache sharing
parallel taskset -c 0,4 ./run ::: {0..1}  # no L3 cache sharing

It performs better — as if there were twice as much L3 cache and RAM bandwidth available:

These issues are especially tricky when benchmarking and are a huge source of noise when timing parallel applications.

### #Saturating Bandwidth

When looking at the RAM section of the first graph, it may seem that with more cores, the per-process throughput goes ½, ⅓, ¼, and so on, and the total bandwidth remains constant. But this isn’t quite true: the contention hurts, but a single CPU core usually can’t saturate all of the RAM bandwidth.

If we plot it more carefully, we see that the total bandwidth actually increases with the number of cores — although not proportionally, and eventually approaches its theoretical maximum of ~42.4 GB/s:

Note that we still specify processor affinity: the $k$-threaded run uses the first $k$ cores. This is why we have such a huge performance increase when switching from 4 cores to 5: you can have more RAM bandwidth if the requests go through separate L3 caches.

In general, to achieve maximum bandwidth, you should always split the threads of an application symmetrically.


text:
The equation of a circle whose radius is 5 and is tangent to both x -axis and y -axis and lies in first quadrant.

Precalculus: Mathematics for Calcu...

6th Edition
Stewart + 5 others
Publisher: Cengage Learning
ISBN: 9780840068071

Precalculus: Mathematics for Calcu...

6th Edition
Stewart + 5 others
Publisher: Cengage Learning
ISBN: 9780840068071

Solutions

Chapter 1.8, Problem 100E
To determine

To calculate: The equation of a circle whose radius is 5 and is tangent to both x -axis and y -axis and lies in first quadrant.

Expert Solution

The equation of the circle is (x5)2+(y5)2=25 .

Explanation of Solution

Given information:

The radius of the circle is 5 and is tangent to both x -axis and y -axis and lies in first quadrant.

Formula used:

The standard form of the equation of the circle is (xh)2+(yk)2=r2 , where (h,k) denote the center of the circle and r denote the radius.

Distance d(A,B) between two points A=(x1,y1) and B=(x2,y2) in the Cartesian plane is denoted by d(A,B)=(x2x1)2+(y2y1)2 .

Calculation:

Consider the provided conditions that radius of the circle is 5 and is tangent to both x -axis and y -axis and lies in first quadrant.

Since the circle is tangent to x -axis so it touches the x -axis at a single point and y -coordinate is 0.

As the radius of circle is 5 units so point on x -axis through which the circle passes is (5,0) .

Since the circle is tangent to y -axis so it touches the y -axis at a single point and x -coordinate is 0.

As the radius of circle is 5 units so point on y -axis through which the circle passes is (0,5) .

So, the center of the circle is (5,5) .

Recall that the standard form of the equation of the circle is (xh)2+(yk)2=r2 , where (h,k) denote the center of the circle and r denote the radius.

Compare, (h,k) and (5,5) also radius r is 5.

Here, h=5,k=5 and r=5 .

Substitute the values in standard equation of circle,

(x5)2+(y5)2=52(x5)2+(y5)2=25

Thus, the equation of circle is (x5)2+(y5)2=25 .

Have a homework question?

Subscribe to bartleby learn! Ask subject matter experts 30 homework questions each month. Plus, you’ll have access to millions of step-by-step textbook answers!