text:
Lost And Found In The Vertical World - The Adventures of Caroline Ciavaldini and James Pearson
Caroline C.
Parents & Climbers
Japan trad
Blog 4 – should you go to Kinkasan
The Philippines DWS dicovery
Voie Petit
Bike and climb tour des Alpilles with Baby
The Philippines, Touching down
The Philippines – Touching Down
[/mk_page_section][/vc_column]
Sometimes, a single photo can spark a chain of events that lead to… well… living on a tropical beach, cooking over an open fire, with white sand and ocean as far as the eye can see.
A google search for "sea cliff" revelead some impressive, if sharp looking rock in the Philippines. Further research brought little more information, so we reached out to the Filipino climbing community in search of answers to our questions. We discovered a motivated but tiny group of climbers, fronted by Ina and Miel. They explained to us that they knew of the potential around Palawan, but as of yet, nobody had actually tried to climb.
The ball was rolling… more pictures promised steep overhanging compact faces above crystal clear ocean, and we began to understand the logistics and layout of the place. The climbing seemed to be on the edges of several medium sized islands, a few km off the coast of Palawan. These islands were easily accessible by Sea Kayak, and the idea of making the trip "minimal impact" started to turn in our mind. We reached out through our contacts at The North Face, and began to organize the logistics through Jundel, the brand manager for The North Face in the Philippines. We got in contact with local kayak guide Serge Farre who was able to arrange the rental of suitable Kayaks, big enough to transport both us, and our gear, and finally Goal Zero got on board, providing us with a solution for charging all our gear direct from the sun.
The team for the expedition started to take shape, with Miel Pahati representing the Filipino climbing community, Francisco Taranto Jr in charge of documenting the adventure, and last but not least, Pablo Scorza to take care of our aching bodies with his own method of Biomechanica Funcional. The Team for any trip is often more important than the destination, and can really make or break the expedition. Heaven can quickly turn into Hell if the team can't get along with one another, so we are all really happy to be here with people we know from the past.
Everything was in place, or as much as could be expected when trying to organize things in advance in Asia. We left an unseasonably warm French winter and stopped off in a very cold Japan for 1 week of brilliant bouldering with Japanese friends old and new. Leaving Tokyo early one morning with 20cm of snow on the ground, it was quite a shock to fly just 4 hours south to Manila, where it was a hot and sticky 30 degrees. Jundel had arranged everything, and our first night passed perfectly, without a hitch. We gave a multimedia presentation in R.O.X for the local climbing community, explaining our project and opening an invitation for Filipino climbers to join us, and finally, meeting up with the rest of the team and organizing the plan for the upcoming days.
A further flight to the island of Palawan, which Miel explained was like taking a "bus" in the Philippines, found us in Puerto Princessa awaiting a real bus to take us 7 hours North to the tiny town of El Nido. If the plane were like a bus, then this bus would be more like a farm cart – powered by rocket fuel! Cramped, bumpy, and skidding round corners – they were 7 of the longest hours of our life we will never forget. We arrived in El Nido in the dark, found our way to the quite, hidden house of Serge, and tumbled exhausted into bed. Sleep came easily, especially for Francisco who was on the tail end of 48 hours of traveling, and we all dreamt excitedly of what the following days would bring.
Interview of Martina Cufar-Potard, Slovenian champion and super-mum
parents and climbers
Year 1: What have we done?!
Expedition parents and climbers Thoughts Travel
Interview of Marion Poitevin, mum and mountain guide!
0 to 9 months: baby on the way!
Copyright Once Upon A Climb - All Rights Reserved © 2013
I consent to Once Upon A Climb collecting my details through this form.


text:
\section{Introduction}
\label{sec:introduction}

The use of Online Controlled Experiments (OCEs, e.g. A/B tests) has become popular in measuring the impact of digital products and services, and guiding business decisions on the Web~\cite{liu20whatisthevalue}. Major companies report running thousands of OCEs on any given day~\cite{kohavi13online,xu15frominfrastructure,hohnhold15focusing} and many startups exist purely to manage OCEs~\cite{johari17peeking,browne17whatworks}.

A large number of OCEs address simple variations on elements of the user experience based on random splits, e.g. showing a different colored button to users based on a user ID hash bucket. Here, we are interested in experiments that compare \emph{personalization strategies}, complex sets of targeted user interactions that are common in e-commerce and digital marketing, and measure the change to a performance metric of interest (\emph{metric} hereafter). Examples of personalization strategies include the scheduling, budgeting and ordering of marketing activities directed at a user based on their purchase history.

Experiments for personalization strategies face two unique challenges. Firstly, strategies are often only applicable to a small fraction of the user base, and thus many simple experiment designs suffer from either a lack of sample size / statistical power, or diluted metric movement by including irrelevant samples~\cite{deng15diluted}.
Secondly, as users are not randomly assigned \emph{a priori}, but must qualify to be treated with a strategy via their actions or attributes, groups of users subjected to different strategies cannot be assumed to be statistically equivalent and hence are not directly comparable.

While there are a number of variance reduction techniques (including stratification and control variates \cite{deng13improving,poyarkov16boosted}) that partially address the challenges, the strata and control variates involved can vary dramatically from one personalization strategy experiment to another, requiring many \emph{ad hoc} adjustments. As a result, such techniques may not scale well when organizations design and run hundreds or thousands of experiments at any given time.

We argue that personalization strategy experiments should focus on the assignment of users from the strategies they qualified for to the treatment/analysis groups. We call this mapping process an \emph{experiment setup}. Identifying the best experiment setup increases the chance to detect any treatment effect. An experimentation framework can also reuse and switch between different setups quickly with little custom input, ensuring the operation can scale. More importantly, the process does not hinder the subsequent application of variance reduction techniques, meaning that we can still apply the techniques \textit{post hoc} if required.

To date, many experiment setups exist to compare personalization strategies. An increasingly popular approach is to compare the strategies using multiple control groups --- Quantcast calls it a dual control~\cite{quantcast}, and Facebook calls it a multi-cell lift study~\cite{liu2018designing}. In the two-strategy case, this involves running two experiments on two random partitions of the user base in parallel, with each experiment further splitting the respective partition into treatment/control and measuring the \emph{incrementality} (the change in a metric as compared to the case where nothing is done) of each strategy. The incrementality of the strategies are then compared against each other.

Despite the setup above gaining traction in display advertising, there is a lack of literature on whether it is a better setup---one that has a higher sensitivity and/or apparent effect size than other setups. While~\cite{liu2018designing} noted that multi-cell lift studies require a large number of users, they did not discuss how the number compares to other setups.\footnote{A single-cell lift study is often used to measure the incrementality of a single personalization strategy, and hence is not a representative comparison.} The ability to identify and adopt a better experiment setup can reduce the required sample size, and hence enable more cost-effective experimentation.

We address the gap in the literature by introducing an evaluation framework that compares experiment setups given two personalization strategies.
The framework is designed to be flexible so that it is able to deal with a wide range of baselines and changes in user responses presented by any pairs of strategies (\emph{situations} hereafter).
We also recognize the need to quickly compare common setups, and provide some simple rule of thumbs on situations where a setup will be better than another.
In particular, we outline the conditions where employing a multi-cell setup, as well as metric dilution, is desirable.

To summarize, our contributions are: (i) We develop a flexible evaluation framework for personalization strategy experiments, where one can compare two experiment setups given the situation presented by two competing strategies (Section~\ref{sec:eval_framework}); (ii) We provide simple rule of thumbs to enable experimenters who do not require the full flexibility of the framework to quickly compare common setups (Section~\ref{sec:comparison}); and (iii) We make our results useful to practitioners by making the code used in the paper (Section~\ref{sec:experiments}) publicly available.\footnote{The code and supplementary document are available on: \url{https://github.com/liuchbryan/experiment\_design\_evaluation}.}

\section{Evaluation Framework}
\label{sec:eval_framework}

We first present our evaluation framework for personalization strategy experiments. The experiments compare two personalization strategies, which we refer to as strategy 1 and strategy 2.
Often one of them is the existing strategy, and the other is a new strategy we intend to test and learn from.
In this section we introduce (i) how users qualifying themselves into strategies creates non-statistically equivalent groups, (ii) how experimenters usually assign the users, and (iii) when we would consider an assignment to be better.

\subsection{User grouping}
\label{sec:user_grouping}

As users qualify themselves into the two strategies, four disjoint groups emerge: those who qualify for neither strategy, those who qualify only for strategy 1, those who qualify only for strategy 2, and those who qualify for both strategies. We denote these groups (user) groups 0, 1, 2, and 3 respectively (see Figure~\ref{fig:ME_groups}). It is perhaps obvious that we cannot assume those in different user groups are statistically equivalent and compare them directly.

\begin{figure}
\begin{center}
    \includegraphics[width=0.35\textwidth, trim = 0 0 0 0]{fig/ME_groups.pdf}
\end{center}
\caption{Venn diagram of the user groups in our evaluation framework. The outer, left inner (red), and right inner (blue) boxes represent the entire user base, those who qualify for strategy 1, and those who qualify for strategy 2 respectively.}
\label{fig:ME_groups}
\end{figure}

We assume groups 0, 1, 2, 3 have $n_0$, $n_1$, $n_2$, and $n_3$ users respectively. We also assume responses from users (which we aggregate, often by taking the mean, to obtain our metric) are distributed differently between groups, and within the same group, between the scenario where the group is subjected to the treatment associated to the corresponding strategy and where nothing is done (baseline). We list all group-scenario combinations in Table~\ref{tab:group_id}, and denote the mean and variance of the responses $(\mu_G, \sigma^2_G)$ for a combination~$G$.\footnote{For example, the responses for group~1 without any interventions has mean and variance ($\mu_{C1}$, $\sigma^2_{C1}$), and that for group~2 with the treatment prescribed under strategy~2 has mean and variance ($\mu_{I2}$, $\sigma^2_{I2}$).}

\begin{table*}
\begin{center}
\begin{tabular}{|c|c|c|c|c|}
\hline
    &  Group 0 & Group 1  & Group 2  & Group 3 \\\hline
 Baseline (\textbf{C}ontrol) & $C0$ & $C1$ & $C2$ & $C3$ \\\hline
 Under treatment (\textbf{I}ntervention) & $/$ & $I1$ & $I2$ & $\begin{array}{ll} \textrm{Under strategy 1:} \quad I\phi \\ \textrm{Under strategy 2:}  \quad I\psi \end{array} $\\
\hline
\end{tabular}
\end{center}
\vspace*{4pt}
\caption{All group-scenario combinations in our evaluation framework for personalization strategy experiments. The columns represent the groups described in Figure~\ref{fig:ME_groups}. The baseline represents the scenario where nothing is done. We assume those who qualify for both strategies (Group 3) can only receive treatment(s) associated to either strategy.}
\label{tab:group_id}
\end{table*}

\subsection{Experiment setups}
\label{sec:experiment_setup}
Many experiment setups exist and are in use in different organizations. Here we introduce four common setups of various sophistication, which we also illustrate in Figure~\ref{fig:ME_designs}.

\begin{figure}
\begin{center}
    \includegraphics[width=0.48\textwidth, trim = 0 0 0 0]{fig/ME_designs.pdf}
\end{center}
\caption{Experiment setups overlaid on the user grouping Venn diagram in Figure~\ref{fig:ME_groups}. The hatched boxes indicate who are included in the analysis, and the downward triangles and dots indicate who are subjected to treatment prescribed under strategies 1 and 2 respectively. See Section~\ref{sec:experiment_setup} for a detailed description.}
\label{fig:ME_designs}
\end{figure}

\paragraph{Setup 1 (Users in the intersection only)} The setup considers users who qualify for both strategies only. The said users are randomly split (usually 50/50) into two (analysis) groups $A$ and $B$, and are prescribed the treatment specified by strategies 1 and 2 respectively. The setup is easy to implement, though it is difficult to translate any learnings obtained from the setup to other user groups (e.g. those who qualify for one strategy only)~\cite{dmitriev17adirtydozen}.

\paragraph{Setup 2 (All samples)} The setup is a simple A/B test where it considers all users, regardless on whether they qualify for any strategy or not. The users are randomly split into two analysis groups $A$ and $B$, and are prescribed the treatment specified by strategy 1(2) if (i) they qualify under the strategy and (ii) they are in analysis group $A$($B$). This setup is easiest to implement but usually suffers severely from a dilution in metric~\cite{deng15diluted}.

\paragraph{Setup 3 (Qualified users only)} The setup is similar to Setup 2 except only those who qualified for at least one strategy (``triggered'' users in some literature~\cite{deng15diluted}) are included in the analysis groups. The setup sits between Setup 1 and Setup 2 in terms of user coverage, and has the advantage of capturing the most number of useful samples yet having the least metric dilution. However, the setup also prevents one from telling the incrementality of a strategy itself, but only the difference in incrementalities between two strategies.

\paragraph{Setup 4 (Dual control / multi-cell lift test)} As described in Section~\ref{sec:introduction}, the setup first split the users randomly into two randomization groups. For the first randomization group, we consider those who qualify for strategy 1, and split them into analysis groups $A1$ and $A2$. Group $A2$ receives the treatment prescribed under strategy~1, and group $A1$ acts as control. The incrementality for strategy~1 is then the difference in metric between groups $A2$ and $A1$. We apply the same process to the second randomization group, with strategy~2 and analysis groups $B1$ and $B2$ in place, and compare the incrementality for strategies 1 and 2. The setup allows one to obtain the incrementality of each individual strategy and minimizes metric dilution. Though it also leaves a number of samples unused and creates extra analysis groups, and hence generally suffers from a low test power~\cite{liu2018designing}.

\subsection{Evaluation criteria}
\label{sec:evaluation_criteria}
There are a number of considerations when one evaluates competing experiment setups. These include technical considerations such as the complexity of setting up the setups on an OCE framework, and business considerations such as whether the incrementality of individual strategies is required.

Here we focus on the statistical aspect and propose two evaluation criteria: (i) the actual average treatment effect size as presented by the two analysis groups in an experiment setup, and (ii) the sensitivity of the experiment represented by the minimum detectable effect (MDE) under a pre-specified test power. Both criteria are necessary as the former indicates whether a setup suffers from metric dilution, whereas the latter indicates whether the setup suffers from lack of power/sample size. An ideal setup should yield a high actual effect size and a high sensitivity (i.e. a low MDE),\footnote{We will use the terms ``high(er) sensitivity'' and ``low(er) MDE'' interchangeably.} though as we observe in the next section it is usually a trade-off.

We formally define the two evaluation criteria from first principles while introducing relevant notations along the way.
Let $A$ and $B$ be the two analysis groups in an experiment setup, with user responses randomly distributed with mean and variance $(\mu_A, \sigma^2_A)$ and $(\mu_B, \sigma^2_B)$ respectively.
We first recall that if there are sufficient samples, the sample mean of the two groups approximately follows the normal distribution by the Central Limit Theorem:
\begin{align}
    \bar{A} \overset{\textrm{approx.}}{\sim} \mathcal{N}\left(\mu_A, \nicefrac{\sigma^2_A}{n_A}\right), \quad
    \bar{B} \overset{\textrm{approx.}}{\sim} \mathcal{N}\left(\mu_B, \nicefrac{\sigma^2_B}{n_B}\right),
\end{align}
where $n_A$ and $n_B$ are the number of samples taken from $A$ and $B$ respectively. The difference in the sample means then also approximately follows a normal distribution:
\begin{align}
    \bar{D} \triangleq (\bar{B} - \bar{A}) \overset{\textrm{approx.}}{\sim}
    \mathcal{N}\left(\Delta \triangleq \mu_B - \mu_A,
    \,\sigma^2_{\bar{D}} \triangleq \nicefrac{\sigma^2_A}{n_A} + \nicefrac{\sigma^2_B}{n_B} \right).
    \label{eq:D_bar_distribution}
\end{align}
Here, $\Delta$ is the actual effect size that we are interested in.

The definition of the MDE $\theta^*$ requires a primer to the power of a statistical test. A common null hypothesis statistical test in personalization strategy experiments uses the two-tailed hypotheses $H_0: \Delta = 0$ and $H_1: \Delta \neq 0$, with the test statistic under $H_0$ being:
\begin{align}
    T \triangleq \nicefrac{\overline{D}}{\sigma_{\bar{D}}} \overset{\textrm{approx.}}{\sim}
    \mathcal{N}\left(0, 1\right).
\end{align}
We recall the null hypothesis will be rejected if $|T| > z_{1-\nicefrac{\alpha}{2}}$, where $\alpha$ is the significance level and $z_{1-\nicefrac{\alpha}{2}}$ is the~$1-\nicefrac{\alpha}{2}$ quantile of a standard normal. Under a \emph{specific} alternate hypothesis $\Delta = \theta$, the power is specified as
\begin{align}
    1 - \beta_\theta & \triangleq \textrm{Pr}\,\big(|T| > z_{1-\nicefrac{\alpha}{2}} \, |\, \Delta = \theta \big)
    \approx 1 - \Phi\big(z_{1 - \nicefrac{\alpha}{2}} - \nicefrac{|\theta|}{\sigma_{\bar{D}}}\big).
    \label{eq:beta_theta_approximation}
\end{align}
where $\Phi$ denotes the cumulative density function of a standard normal.\footnote{The approximation in Equation~\eqref{eq:beta_theta_approximation} is tight for experiment design purposes, where $\alpha < 0.2$ and $1 - \beta > 0.6$ for nearly all cases.}
To achieve a minimum test power $\pi_{\textrm{min}}$, we require that ${1 - \beta_\theta > \pi_{\textrm{min}}}$. Substituting Equation~\eqref{eq:beta_theta_approximation} into the inequality and rearranging to make $\theta$ the subject yields the effect sizes that the test will be able to detect with the specified power:
\begin{align}
    |\theta| > (z_{1 - \nicefrac{\alpha}{2}} - z_{1 - \pi_{\textrm{min}}})\,\sigma_{\bar{D}} \,.
    \label{eq:mde_ineq}
\end{align}
$\theta^*$ is then defined as the positive minimum $\theta$ that satisfies Inequality~\eqref{eq:mde_ineq}, i.e. that specified by the RHS of the inequality.

We finally define what it means to be better under these evaluation criteria. WLOG we assume the actual effect size of the two competing experiment setups are positive,\footnote{If both the actual effect sizes are negative, we simply swap the analysis groups. If the actual effect sizes are of opposite signs, it is likely an error.} and say
a setup $S$ is superior to another setup $R$ if, all else being equal,
\begin{enumerate}[leftmargin=*]
    \item[(i)] $S$ produces a higher actual effect size ($\Delta_S > \Delta_R$) \emph{and} a lower minimum detectable effect size ($\theta^*_S < \theta^*_R$), or
    \item[(ii)] The gain in actual effect is greater than the loss in sensitivity:
    \begin{align}
        \Delta_S - \Delta_R > \theta^*_S - \theta^*_R \,,
    \end{align}
    which means an actual effect still stands a higher chance to be observed under $S$.
\end{enumerate}

\section{Comparing experiment setups}
\label{sec:comparison}

Having described the evaluation framework above, in this section we use the framework to compare the common experiment setups described in Section~\ref{sec:experiment_setup}. We will first derive the actual effect size and MDE for each setup in Section~\ref{sec:actual_mde_size}, and using the result to create rule of thumbs on (i) whether diluting the metric by including users who qualify for neither strategies is beneficial (Section~\ref{sec:expt_comparison_dilution}) and (ii) if dual control is a better setup for personalization strategy experiments (Section~\ref{sec:expt_comparison_dual_control}), two questions that are often discussed among e-commerce and marketing-focused experimenters.
For brevity, we relegate most of the intermediate algebraic work when deriving the actual \& minimum detectable effect sizes, as well as the conditions that lead to a setup being superior, to our supplementary document.\footnotemark[2]

\subsection{Actual \& minimum detectable effect sizes}
\label{sec:actual_mde_size}

We first present the actual effect size and MDE of the four experiment setups. For each setup we first compute the sample size, mean response, and response variance in each analysis group, which arises as a mixture of user groups described in Section~\ref{sec:user_grouping}. For brevity, we only present the quantities for one analysis group per setup---expressions for other analysis group(s) can be easily obtained by substituting in the corresponding user groups. We then substitute the quantities computed into the definitions of~$\Delta$ (see Equation~\eqref{eq:D_bar_distribution}) and~$\theta^*$ (see Inequality~\eqref{eq:mde_ineq}) to obtain the setup-specific actual effect size and MDE. We assume all random splits are done 50/50 in these setups to maximize the test power.

\paragraph{Setup 1 (Users in the intersection only)}
The setup randomly splits user group 3 into two analysis groups, each with $\nicefrac{n_3}{2}$ samples. Users in analysis group $A$ are provided treatment under strategy~1, and hence the group's responses have a mean and variance of $(\mu_{I\phi}, \sigma^2_{I\phi})$.
The actual effect size and MDE for Setup 1 are hence:
\begin{align}
    \Delta_{S1} &= \mu_{I\psi} - \mu_{I\phi} , \label{eq:Delta_S1}\\
    \theta^*_{S1} &= (z_{1-\nicefrac{\alpha}{2}} - z_{1-\pi_{\textrm{min}}})
    \sqrt{2(\sigma^2_{I\phi} + \sigma^2_{I\psi})/n_3} .
    \label{eq:theta_star_S1}
\end{align}

\paragraph{Setup 2 (All samples)}
This setup also contains two analysis groups, $A$ and $B$, each taking half of the population (i.e. $(n_0 + n_1 + n_2 + n_3)/2$).
The mean response and response variance for groups $A$ and $B$ are the weighted mean response and response variance of the constituent user groups respectively. As we only provide treatment to those who qualify for strategy 1 in group $A$, and likewise for group $B$ with strategy 2, each user group will give different responses, e.g. for group $A$:
\begin{align}
    \mu_A = (n_0\mu_{C0} + n_1\mu_{I1} + n_2\mu_{C2} + n_3\mu_{I\phi}) / (n_0 + n_1 + n_2 + n_3), \label{eq:mu_A_S2}\\
    \sigma^2_A = (n_0\sigma^2_{C0} + n_1\sigma^2_{I1} + n_2\sigma^2_{C2} + n_3\sigma^2_{I\phi})/(n_0 + n_1 + n_2 + n_3). \label{eq:sigma_sq_A_S2}
\end{align}

Substituting the above (and that for group $B$) into the definitions of actual effect size and MDE we have:
\begin{align}
    \Delta_{S2} = \, &   \frac{n_1(\mu_{C1} - \mu_{I1}) + n_2(\mu_{I2} - \mu_{C2}) + n_3(\mu_{I\psi} - \mu_{I_\phi})}{n_0 + n_1 + n_2 + n_3},
    \label{eq:Delta_S2}\\
    \theta^*_{S2} = \, &  (z_{1-\nicefrac{\alpha}{2}} - z_{1-\pi_{\textrm{min}}})
    \sqrt{\frac{\begin{array}{l}
    2\big(n_0(2\sigma^2_{C0}) + n_1(\sigma^2_{I1} + \sigma^2_{C1}) + \\
    \quad n_2(\sigma^2_{C2} + \sigma^2_{I2}) + n_3(\sigma^2_{I\phi} + \sigma^2_{I\psi})\big)
    \end{array}
    }{(n_0 + n_1 + n_2 + n_3)^2}} .
    \label{eq:theta_star_S2}
\end{align}

\paragraph{Setup 3 (Qualified users only)}
The setup is very similar to Setup~2, with members from user group~0 excluded. This leads to both analysis groups having $(n_1 + n_2 + n_3)/2$ users.
The absence of group~0 users means they are not featured in the weighted mean response and response variance of the two analysis groups, e.g. for group A:
\begin{align}
    \mu_A = \frac{n_1\mu_{I1} + n_2\mu_{C2} + n_3\mu_{I\phi}}{n_1 + n_2 + n_3}, \;
    \sigma^2_A = \frac{n_1\sigma^2_{I1} + n_2\sigma^2_{C2} + n_3\sigma^2_{I\phi}}{n_1 + n_2 + n_3}.
    \label{eq:mu_A_sigma_sq_A_S3}
\end{align}

This leads to the following actual effect size and MDE for Setup~3:
\begin{align}
    \Delta_{S3} = &   \frac{n_1(\mu_{C1} - \mu_{I1}) + n_2(\mu_{I2} - \mu_{C2}) + n_3(\mu_{I\psi} - \mu_{I_\phi})}{n_1 + n_2 + n_3},
    \label{eq:Delta_S3}\\
    \theta^*_{S3} = &  (z_{1-\nicefrac{\alpha}{2}} \!-\! z_{1-\pi_{\textrm{min}}})
    \!\sqrt{\!\frac{2\!\left(n_1(\sigma^2_{I1} \!\!+\! \sigma^2_{C1}) \!+\! n_2(\sigma^2_{C2} \!\!+\! \sigma^2_{I2}) \!+\! n_3(\sigma^2_{I\phi} \!\!+\! \sigma^2_{I\psi})\!\right)\!}{(n_1 + n_2 + n_3)^2}}.
    \label{eq:theta_star_S3}
\end{align}

\paragraph{Setup 4 (Dual control)}
The setup is the odd one out as it has four analysis groups. Two of the analysis groups ($A1$ and $A2$) are drawn from those who qualified into strategy 1 and are allocated into the first randomization group, and the other two ($B1$ and $B2$) are drawn from those who are qualified into strategy 2 and are allocated into the second randomization group:
\begin{align}
    n_{A1} = n_{A2} = (n_1 + n_3)/4\,,\quad n_{B1} = n_{B2} = (n_2 + n_3)/4.
    \label{eq:n_S4}
\end{align}
The mean response and response variance for group $A1$ are:
\begin{align}
    \mu_{A1} = \frac{n_1\mu_{C1} + n_3\mu_{C3}}{n_1 + n_3},\;
    \sigma^2_{A1} = \frac{n_1\sigma^2_{C1} + n_3\sigma^2_{C3}}{n_1 + n_3}.
    \label{eq:mu_A1_sigma_sq_A1_S4}
\end{align}

As the setup takes the difference of differences in the metric (i.e. the difference between the mean response for groups $B2$ and $B1$, and the difference between the mean response for groups $A2$ and $A1$),\footnote{Not to be confused with the difference-in-differences method, which captures the metric for the control and treatment groups at both the beginning (pre-intervention) and the end (post-intervention) of the experiment. Here we only capture the metric once at the end of the experiment.} the actual effect size is as follows:
\begin{align}
    \Delta_{S4} & = (\mu_{B2} - \mu_{B1}) - (\mu_{A2} - \mu_{A1}) \nonumber\\
    & = \frac{n_2(\mu_{I2} \!-\! \mu_{C2}) + n_3(\mu_{I\psi} \!-\! \mu_{C3})}{n_2 + n_3} - \frac{n_2(\mu_{I1} \!-\! \mu_{C1}) + n_3(\mu_{I\phi} \!-\! \mu_{C3})}{n_1 + n_3} .
    \label{eq:Delta_S4}
\end{align}
The MDE for Setup 4 is similar to that specified in RHS of Inequality~\eqref{eq:mde_ineq}, albeit with more groups:
\begin{align}
    \theta^*_{S4} = \  & (z_{1-\nicefrac{\alpha}{2}} \!-\! z_{1-\pi_{\textrm{min}}})
    \sqrt{\nicefrac{\sigma^2_{A1}}{n_{A1}} + \nicefrac{\sigma^2_{A2}}{n_{A2}} + \nicefrac{\sigma^2_{B1}}{n_{B1}} + \nicefrac{\sigma^2_{B2}}{n_{B2}}} \nonumber\\
    = \,&  2 \cdot (z_{1-\nicefrac{\alpha}{2}} - z_{1-\pi_{\textrm{min}}}) \times \nonumber\\
    & \!\! \sqrt{\frac{n_1(\sigma^2_{C1} \!+\! \sigma^2_{I1}) \!+\! n_3(\sigma^2_{C3} \!+\! \sigma^2_{I\phi})}{(n_1 + n_3)^2} +
          \frac{n_2(\sigma^2_{C2} \!+\! \sigma^2_{I2}) \!+\! n_3(\sigma^2_{C3} \!+\! \sigma^2_{I\psi})}{(n_2 + n_3)^2}} .
    \label{eq:theta_star_S4}
\end{align}

\subsection{Is dilution always bad?}
\label{sec:expt_comparison_dilution}

The use of responses from users who do not qualify for any of the strategies we are comparing, an act known as metric dilution, has stirred countless debates in experimentation teams. On one hand, responses from these users make any treatment effect less pronounced by contributing exactly zero; on the other hand, it might be necessary as one does not know who actually qualify for a strategy~\cite{liu2018designing}, or it might be desirable as they can be leveraged to reduce the variance of the treatment effect estimator~\cite{deng15diluted}.

Here, we are interested in whether we should engage in the act of dilution given the assumed user responses prior to an experiment. This can be clarified by understanding the conditions where Setup~3 would emerge superior (as defined in Section~\ref{sec:evaluation_criteria}) to Setup~2. By inspecting Equations~\eqref{eq:Delta_S2} and~\eqref{eq:Delta_S3}, it is clear that $\Delta_{S3} > \Delta_{S2}$ if~${n_0 > 0}$. Thus, Setup~3 is superior to Setup~2 under the first criterion if $\theta^*_{S3} < \theta^*_{S2}$, which is the case if $\sigma^2_{C0}$, the metric variance of users who qualify for neither strategies, is large. This can be shown by substituting Equations~\eqref{eq:theta_star_S2} and~\eqref{eq:theta_star_S3} into the $\theta^*$-inequality and rearranging the terms to obtain:
\begin{align}
    \frac{\!\begin{array}{l}\left(n_1(\sigma^2_{I1} + \sigma^2_{C1}) + n_2(\sigma^2_{C2} + \sigma^2_{I2}) + n_3(\sigma^2_{I\phi} + \sigma^2_{I\psi})\right) \cdot \\  (n_0 + 2n_1 + 2n_2 + 2n_3)
    \end{array}\!}{2 (n_1 + n_2 + n_3)^2} < \sigma^2_{C0} \,.
    \label{eq:pse_setup23_criterion1}
\end{align}
If we assume the response variance are similar across groups with users who qualified for at least one strategy, i.e. $\sigma^2_{I1} \approx \sigma^2_{C1} \approx \cdots \approx \sigma^2_{I\psi} \approx \sigma^2_S$, Inequality~\eqref{eq:pse_setup23_criterion1} can then be simplified as
\begin{align}
    \sigma^2_S \left(\frac{n_0}{n_1 + n_2 + n_3} + 2\right) < \sigma^2_{C0} \,,
\end{align}
where it can be used to quickly determine if one should consider dilution at all.

If Inequality~\eqref{eq:pse_setup23_criterion1} is not true (i.e. $\theta^*_{S3} \geq \theta^*_{S2}$), we should then consider when the second criterion (i.e. $\Delta_{S3} - \Delta_{S2} > \theta^*_{S3} - \theta^*_{S2}$) is met. Writing
\begin{align}
    \eta & = n_1(\mu_{C1} - \mu_{I1}) + n_2(\mu_{I2} - \mu_{C2}) + n_3(\mu_{I\psi} - \mu_{I\phi}), \nonumber\\[-0.1em]
    \xi & = n_1(\sigma^2_{C1} + \sigma^2_{I1}) + n_2(\sigma^2_{I2} + \sigma^2_{C2}) + n_3(\sigma^2_{I\psi} + \sigma^2_{I\phi}), \textrm{and} \nonumber\\[-0.1em]
    z & = z_{1-\nicefrac{\alpha}{2}} - z_{1 - \pi_{\textrm{min}}}, \nonumber
\end{align}
we can substitute Equations~\eqref{eq:Delta_S2}, \eqref{eq:theta_star_S2}, \eqref{eq:Delta_S3}, and \eqref{eq:theta_star_S3} into the inequality and rearrange to obtain
\begin{align}
    \frac{n_1 + n_2 + n_3}{n_0}\sqrt{2n_0\sigma^2_{C0} + \xi} >
    \frac{n_0 + n_1 + n_2 + n_3}{n_0}\sqrt{\xi} - \frac{\eta}{\sqrt{2}z}.
    \label{eq:pse_setup23_criterion2_initial}
\end{align}
As the LHS of Inequality~\eqref{eq:pse_setup23_criterion2_initial} is always positive, Setup 3 is superior if the RHS $\leq 0$. Noting
\begin{align}
    \Delta_{S3} = \eta / (n_1 + n_2 + n_3) \,\textrm{ and }\,
    \theta^*_{S3} = \sqrt{2}\cdot z\cdot\sqrt{\xi} / (n_1 + n_2 + n_3), \nonumber
\end{align}
the trivial case is satisfied if
\begin{align}
\frac{n_0 + n_1 + n_2 + n_3}{n_0} \cdot \theta^*_{S3} \leq \Delta_{S3}.
    \label{eq:pse_setup23_criterion2_strong}
\end{align}

If the RHS of Inequality~\eqref{eq:pse_setup23_criterion2_initial} is positive, we can safely square both sides and use the identities for $\Delta_{S3}$ and $\theta^*_{S3}$ to get
\begin{align}
    \frac{2\sigma^2_{C0}}{n_0} >
    \frac{\left[\left(\theta^*_{S3} - \Delta_{S3} + \frac{n_1 + n_2 + n_3}{n_0} \theta^*_{S3}\right)^2 - \left(\frac{n_1 + n_2 + n_3}{n_0} \theta^*_{S3}\right)^2 \right]}{2z^2} .
    \label{eq:pse_setup23_criterion2_final}
\end{align}
As the LHS is always positive, the second criterion is met if
\begin{align}
    \theta^*_{S3} \leq \Delta_{S3}.
    \label{eq:pse_setup23_criterion2_weak}
\end{align}
Note this is a weaker, and thus more easily satisfiable condition than that introduced in Inequality~\eqref{eq:pse_setup23_criterion2_strong}.
This suggests an experiment setup is always superior to an diluted alternative  if the experiment is already adequately powered---introducing any dilution will simply make things worse.

Failing the condition in Inequality~\eqref{eq:pse_setup23_criterion2_weak}, we can always fall back to Inequality~\eqref{eq:pse_setup23_criterion2_final}. While the inequality operates in squared space, it is essentially comparing the standard error of user group~0 (LHS)---those who qualify for neither strategies---to the gap between the minimum detectable and actual effects ($\theta^*_{S3} - \Delta_{S3}$). The gap can be interpreted as the existing noise level, thus a higher standard error means mixing in group~0 users will introduce extra noise, and one is better off without them. Conversely, a smaller standard error means group~0 users can lower the noise level, i.e. stabilize the metric fluctuation, and one should take advantage of them.

To summarize, diluting a personalization strategies experiment setup is \emph{not} helpful if
\begin{enumerate}[leftmargin=*]
    \item[(i)] Users who do not qualify for any strategies have a large metric variance (Inequality~\eqref{eq:pse_setup23_criterion1}), or
    \item[(ii)] The experiment is already adequately powered (Inequality~\eqref{eq:pse_setup23_criterion2_weak}).
\end{enumerate}
It could help if the experiment has not gained sufficient power yet and users who do not qualify for any strategy provide low-variance responses, such that they exhibit stabilizing effects when included into the analysis (complement of Inequality~\eqref{eq:pse_setup23_criterion2_final}).

\subsection{When is a dual-control more effective?}
\label{sec:expt_comparison_dual_control}
Often when advertisers compare two personalization strategies, the question on whether to use a dual control/multi-cell design comes up. Proponents of such approach celebrate its ability to tell a story by making the incrementality of an individual strategy available, while opponents voice concerns on the complexity in setting up the design. Here we are interested if Setup 4 (dual control) is superior to Setup 3 (a simple A/B test) from a power/detectable effect perspective, and if so, under what circumstances.

We first observe $\theta^*_{S4}$ > $\theta^*_{S3}$ is always true, and hence a dual control setup will never be superior to a simpler setup under the first criterion. This can be verified by substituting in Equations~\eqref{eq:theta_star_S4} and~\eqref{eq:theta_star_S3} and rearranging the terms to show the inequality is equivalent to
\begin{align}
    & \textstyle 2\Big(
      \frac{n_1}{(n_1 + n_3)^2} (\sigma^2_{C1} + \sigma^2_{I1}) +
      \frac{n_2}{(n_2 + n_3)^2} (\sigma^2_{C2} + \sigma^2_{I2}) +
      \nonumber\\
    & \textstyle \quad \frac{n_3}{(n_1 + n_3)^2} \sigma^2_{I\phi} +
      \frac{n_3}{(n_2 + n_3)^2} \sigma^2_{I\psi} +
      \left(\frac{n_3}{(n_1 + n_3)^2} + \frac{n_3}{(n_2 + n_3)^2}\right) \sigma^2_{C3} \Big) > \nonumber\\
    & \textstyle
      \frac{n_1}{(n_1 + n_2 + n_3)^2} (\sigma^2_{C1} + \sigma^2_{I1}) +
      \frac{n_2}{(n_1 + n_2 + n_3)^2} (\sigma^2_{C2} + \sigma^2_{I2}) + \nonumber\\
    & \textstyle
      \quad \frac{n_3}{(n_1 + n_2 + n_3)^2} \sigma^2_{I\phi} +
      \frac{n_3}{(n_1 + n_2 + n_3)^2} \sigma^2_{I\psi},
    \label{eq:pse_setup34_criterion1}
\end{align}
which is always true given the $n$s are non-negative and the $\sigma^2$s are positive: not only the coefficients of the $\sigma^2$-terms are larger on the LHS than their RHS counterparts, the LHS also carries an extra~$\sigma^2_{C3}$ term with non-negative coefficient and a factor of two.

Moving on to the second evaluation criterion, we recall that Setup 4 is superior if ${\Delta_{S4} - \Delta_{S3} > \theta^*_{S4} - \theta^*_{S3}}$, otherwise Setup 3 is superior under the same criterion. The full flexibility of the model can be seen by substituting Equations~\eqref{eq:Delta_S3}, \eqref{eq:theta_star_S3}, \eqref{eq:Delta_S4}, and~\eqref{eq:theta_star_S4} into the inequality and rearrange to obtain
\begin{align}
    &\frac{n_1 \frac{n_2(\mu_{I2}-\mu_{C2}) + n_3(\mu_{I\psi}-\mu_{C3})}{n_2 + n_3} -
           n_2 \frac{n_1(\mu_{I1}-\mu_{C1}) + n_3(\mu_{I\phi}-\mu_{C3})}{n_1 + n_3}}
           {\sqrt{n_1(\sigma^2_{C1} + \sigma^2_{I1}) +
                  n_2(\sigma^2_{C2} + \sigma^2_{I2}) +
                  n_3(\sigma^2_{I\phi} + \sigma^2_{I\psi})}} >
    \label{eq:pse_setup34_criterion2_full}\\
    & \sqrt{2}z \bBigg@{4}[
    \sqrt{
        2 \cdot
        \frac{
            \begin{array}{l}
                (1 + \frac{n_2}{n_1 + n_3})^2
                  [n_1(\sigma^2_{C1} + \sigma^2_{I1}) +
                  n_3(\sigma^2_{C3} + \sigma^2_{I\phi})] + \\
                \quad (1 + \frac{n_1}{n_2 + n_3})^2
                  [n_2(\sigma^2_{C2} + \sigma^2_{I2}) +
                  n_3(\sigma^2_{C3} + \sigma^2_{I\psi})]
            \end{array}}
            {n_1(\sigma^2_{C1} + \sigma^2_{I1}) +
                n_2(\sigma^2_{C2} + \sigma^2_{I2}) +
                n_3(\sigma^2_{I\phi} + \sigma^2_{I\psi})}}
        - 1
    \bBigg@{4}],
    \nonumber
\end{align}
where $ z = z_{1-\nicefrac{\alpha}{2}} - z_{1 - \pi_{\textrm{min}}}$.

A key observation from inspecting Inequality~\eqref{eq:pse_setup34_criterion2_full} is that the LHS of the inequality scales along $O(\sqrt{n})$, where $n$ is the number of users, while the RHS remains a constant. This leads to the insight that Setup 4 is more likely to be superior if the $n$s are large. Here we assume the ratio $n_1:n_2:n_3$ remains unchanged when we scale the number of samples, an assumption that generally holds when an organization increases their reach while maintaining their user mix. It is worth pointing out that our claim is stronger than that in previous work --- we have shown that having a large user base not only fulfills the requirement of running a dual control experiment as described in~\cite{liu2018designing}, it also makes a dual control experiment a better setup than its simpler counterparts in terms of apparent and detectable effect sizes.

The scaling relationship can be seen more clearly if we apply some simplifying assumptions to the $\sigma^2$- and $n$-terms. If we assume the response variances are similar across user groups (i.e. ${\sigma^2_{C1} \approx \sigma^2_{I1} \approx \cdots \approx \sigma^2_{I\psi} \approx \sigma^2_S}$), the RHS of Inequality~\eqref{eq:pse_setup34_criterion2_full} becomes
\begin{align}
    \sqrt{2} z \left[ \sqrt{\frac{n_1 + n_2 + n_3}{n_1 + n_3} + \frac{n_1 + n_2 + n_3}{n_2 + n_3}} - 1 \right],
    \label{eq:pse_setup34_criterion2_sigmaSimplified}
\end{align}
which remains a constant if the ratio $n_1:n_2:n_3$ remains unchanged. If we assume the number of users in groups 1, 2, 3 are similar (i.e. $n_1 = n_2 = n_3 = n$), the LHS of Inequality~\eqref{eq:pse_setup34_criterion2_full} becomes
\begin{align}
    \frac{\sqrt{n} \big((\mu_{I2} - \mu_{C2}) - (\mu_{I1} - \mu_{C1}) + \mu_{I\psi} - \mu_{I\phi}\big)}{2\sqrt{\sigma^2_{C1} + \sigma^2_{I1} + \sigma^2_{C2} + \sigma^2_{I2} + \sigma^2_{I\phi} + \sigma^2_{I\psi}}},
    \label{eq:pse_setup34_criterion2_nSimplified}
\end{align}
which clearly scales along $O(\sqrt{n})$.

We conclude the section by providing an indication on what a large $n$ may look like. If we assume both the response variances and the number of users are are similar across user groups, we can rearrange Inequality~\eqref{eq:pse_setup34_criterion2_full} to make $n$ the subject:
\begin{align}
    n > \left(2\sqrt{12}\left(\sqrt{6} - 1\right)z\right)^2 \frac{\sigma^2_S}{\Delta^2},
    \label{eq:pse_setup34_criterion2_bothSimplified}
\end{align}
where $\Delta = (\mu_{I2} - \mu_{C2}) - (\mu_{I1} - \mu_{C1}) + \mu_{I\psi} - \mu_{I\phi}$ is the difference in actual effect sizes between Setups 4 and 3. Under a 5\% significance level and 80\% power, the first coefficient amounts to around 791, which is roughly 50 times the coefficient one would use to determine the sample size of a simple A/B test~\cite{miller10hownot}. This suggests a dual control setup is perhaps a luxury accessible only to the largest advertising platforms and their top advertisers. For example, consider an experiment to optimize conversion rate where the baselines attain 20\% (hence having a variance of $0.2(1 - 0.2) = 0.16$). If there is a 2.5\% relative (i.e. 0.5\% absolute) effect between the competing strategies, the dual control setup will only be superior if there are $>5$M users in each user group.

\section{Experiments}
\label{sec:experiments}
Having performed theoretical calculations for the actual and detectable effects and conditions where an experiment setup is superior to another, here we verify those calculations using simulation results.
We focus on the results presented in Section~\ref{sec:actual_mde_size}, as the rest of the results presented followed from those calculations.

In each experiment setup evaluation, we randomly select the value of the parameters (i.e. the $\mu$s, $\sigma^2$s, and $n$s), and take 1,000 actual effect samples, each by (i) sampling the responses from the user groups under the specified parameters, (ii) computing the mean for the analysis groups, and (iii) taking the difference of the means.

We also take 100 MDE samples in separate evaluations, each by (i) sampling a critical value under null hypothesis; (ii) computing the test power under a large number of possible effect sizes, each using the critical value and sampled metric means under the alternate hypothesis; and (iii) searching the effect size space for the value that gives the predefined power. As the power vs. effect size curve is noisy given the use of simulated power samples, we use the bisection algorithm provided by the \texttt{noisyopt} package to perform the search. The algorithm dynamically adjusts the number of samples taken from the same point on the curve to ensure the noise does not send us down the wrong search space.

We expect the mean of the sampled actual effect and MDE to match the theoretical value.
To verify this, we perform~1,000 bootstrap resamplings on the samples obtained above to obtain an empirical bootstrap distribution of the sample mean in each evaluation. The $95\%$ bootstrap resampling confidence interval (BRCI) should then contain the theoretical mean $95\%$ of the times.
The histogram of the percentile rank of the theoretical quantity in relation to the bootstrap samples across multiple evaluations should also follow a uniform distribution~\cite{talts2018validating}.

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
 & Actual effect size & Minimum detectable effect \\\hline
 Setup 1 & 1049/1099 (95.45\%) & 66/81 (81.48\%) \\\hline
 Setup 2 & 853/999 (85.38\%) & 87/106 (82.08\%) \\\hline
 Setup 3 & 922/1099 (83.89\%) & 93/116 (80.18\%) \\\hline
 Setup 4 & 240/333 (72.07\%) & 149/185 (80.54\%)\\
\hline
\end{tabular}
\end{center}
\vspace*{4pt}
\caption{Number of evaluations where the theoretical value of the quantities (columns) falls between the 95\% bootstrap confidence interval for each experiment setup (rows). See Section~\ref{sec:experiments} for a detailed description on the evaluations.}
\label{tab:experiment_in_BRCI_result}
\end{table}

The result is shown in Table~\ref{tab:experiment_in_BRCI_result}. One can observe that there are more evaluations having their theoretical quantity lying outside than the BRCI than expected.
Upon further investigation, we observed a characteristic $\cup$-shape from the histograms of the percentile ranks for the actual effects.
This suggests the bootstrap samples may be under-dispersed but otherwise centered on the theoretical quantities.

We also observed the histograms for MDEs curving upward to the right, this suggests that the theoretical value is a slight overestimate (of $<1\%$ to the bootstrap mean in all cases). We believe this is likely due to a small bias in the bisection algorithm. The algorithm tests if the mean of the power samples is less than the target power to decide which half of the search space to continue along. Given we can bisect up to 10 times in each evaluation, it is likely to see a false positive even when we set the significance level for individual comparisons to 1\%. This leads to the algorithm favoring a smaller MDE sample. Having that said, since we have tested for a wide range of parameters and the overall bias is small, we are satisfied with the theoretical quantities for experiment design purposes.

\section{Conclusion}

We have addressed the problem of comparing experiment designs for personalization strategies by presenting an evaluation framework that allows experimenters to evaluate which experiment setup should be adopted given the situation. The flexible framework can be easily extended to compare setups that compare more than two strategies by adding more user groups (i.e. new sets to the Venn diagram in Figure~\ref{fig:ME_groups}). A new setup can also be incorporated quickly as it is essentially a different weighting of user group-scenario combinations shown in Table~\ref{tab:group_id}.
The framework also allows the development of simple rule of thumbs such as:
\begin{enumerate}[leftmargin=*]
    \item[(i)] Metric dilution should never be employed if the experiment already has sufficient power; though it can be useful if the experiment is under-powered and the non-qualifying users provide a ``stabilizing effect''; and
    \item[(ii)] A dual control setup is superior to simpler setups only if one has access to the user base of the largest organizations.
\end{enumerate}
We have validated the theoretical results via simulations, and made the code available\footnotemark[2] so that practitioners can benefit from the results immediately when designing their upcoming experiments.

\paragraph{Future Work}
So far we assume the responses from each user group-scenario combination are randomly distributed with the mean independent to the variance, with the evaluation criteria calculated assuming the metric (being the weighted sample mean of the responses) is approximately normally distributed under Central Limit Theorem. We are interested in whether the evaluation framework and its results still hold if we deviate from these assumptions, e.g. with binary responses (where the mean and variance correlate) and heavy-tailed distributed responses (where the sample mean converges to a normal distribution slowly).

\begin{acks}
The work is partially funded by the EPSRC CDT in Modern Statistics and Statistical Machine Learning at Imperial and Oxford (StatML.IO) and ASOS.com. The authors thank the anonymous reviewers for providing many improvements to the original manuscript.
\end{acks}

\bibliographystyle{ACM-Reference-Format}

redpajama_set_name: RedPajamaArXiv


text:
Gaming • 6 May 2013
Lost Planet 3 preview: generic third-person shooting ahoy
By Pippa Tshabalala
Read nextHTC One review: throw out your iPhone
By the time you reach the third game in a franchise, you think you've seen it all, and to a certain extent you wouldn't be wrong with Lost Planet 3.
The story doesn't take place consecutively however, so in this case you'll actually be playing a prequel. Lost Planet 3 is set long before the events in the first game, and E.D.N III is still in an ice age as opposed to the tropical environment you're used to from Lost Planet 2.
You play as Jim Peyton, a colonist separated from his wife and child as he explores the icy planet of E.D.N III. Jim is very much there to earn a living, he's not interested in building up the colony and the planet, he wants to earn money and go home to his family. Of course that's all rather idealistic in the long-term, and Jim ends up trying to uncover the mysteries that shroud this planet.
Brutal mech and on-foot combat is at the heart of Lost Planet 3
In the preview version I played, you're introduced not only to Jim, but to his mechanical rig, a colossal piece of machinery that, while slow and cumbersome, is absolutely essential if you want to survive in the ice and snow and especially against the Akrids attacking you at every turn.
The scale of this game is what really impresses. Yes, the characters are colourful, flawed and interesting to learn about, but at its heart, Lost Planet 3, even at this stage, is about combat. You'll explore different areas, but with the exception of some ammo and perhaps one or two special items (I'm guessing, there were no special items in the demo), the levels are rather linear and you progress from objective to objective in a rather straightforward manner.
Gameplay wise you switch between a combination of first and third person (mech and character respectively), which serves to nicely separate the different combat styles. You have a variety of weapons, both guns and melee, on hand to help you dispose of the attacking Akrid. You'll also need to collect thermal energy as you progress, which is dropped by the Akrids, or by planting thermal posts throughout the game. The Akrid are, in a word, brutal. They attack remorselessly, barely giving you a moment to breathe, and you'll need to employ both different weapons and tactics for every different kind of beastie.
Multiplayer looks slick
Visually this game still looks rather impressive, but it is a different look and feel to the previous games. This is easily accounted for in the change in developer — Lost Planet 1 and 2 were developed in-house by Capcom, whereas 3 was handed over to a Western developer, Spark Unlimited.
Although the game still plays well, the level I played was about medium difficulty, not particularly challenging but you needed to employ the right tactics, it currently feels like a million other third person shooters on the market and the story is fair but I could easily skip cutscenes and not feel like I had missed out in any way. Fans of the series might enjoy it if they're not bothered by the change in style, but we'll have to wait and see if it makes a lasting impact or it merely disappears into oblivion.
Lost Planet 3 is due for release end of August 2013 for Xbox, PS3 and PC
Pippa Tshabalala
Posted in Gaming, Hands-OnTagged capcom, lost planet 3, preview
HTC One review: throw out your iPhone
HTC • 6 May 2013
Hot new Molly Medusa: Queen of spit game soon on Nintendo Switch
Heavy metal soundtrack makers Burning planet have compiled a soundtrack for incoming Molly Medusa, a 3-dimensional game about a protagonist sculptor with snake hair….
Marcus Gopolang Moloko Galleries • 17 Jan 2023

redpajama_set_name: RedPajamaCommonCrawl


Parent Perks is a parent education series featuring educators, psychologists, and writers who speak to the Lamplighter community several times during the school year. It is our hope that these discussions will assist us in our most important role: igniting young minds.
On Our Minds: What Educators are Thinking Local professionals will talk to us about important trends that are currently top-of-mind in the education world.
Lunch and Learn These lunchtime sessions were carefully chosen to focus on issues that reflect the mission of our school. Bring your lunch and a desire to delve into lively discussions.
Ask the Expert Dallas is fortunate to have many outstanding child psychologists, therapists, and educators to help parents with the daunting task of raising children. This year's experts will provide us with guidance, the latest research, and practical solutions to our toughest questions.
View the 2018-2019 Parent Perks brochure.
Reservations for Parent Perks events are greatly appreciated. Please respond to tls@thelamplighterschool.org or call Hilary Jenkins at 214-369-9201.


Lynn turned off the classical music on the CD player and yawned, pleased. She'd gotten a lot done. Summer session hadn't begun yet and the campus was quiet. Cars loaded with new graduates and their belongings had left days earlier.
She'd propped her office door and windows open. When she leaned back an afternoon breeze fluttered across her skin. Outside, three crows argued from the top of a Douglas fir as a radio blared. And someone cried. Sobs swelled in waves as the crier gained and lost, regained and relost control.
The crying came from inside the building. Lynn got up and went to the door to listen; it was a woman's voice. She walked down the hall and came to a stop outside the bathroom. As she hesitated, wondering if she should go in or not, the door opened with a yank.
The young woman in a rumpled blouse over new jeans jumped when she saw Lynn. 'Ask me about my frontal lobotomy' suggested a pin on her chest. Her face was blotchy and dripped with tears.
"Aren't you the Girl Friday who started working here the end of the semester? Coreen, right?" Lynn held out a tissue.
The girl tried to answer, but could only hiccup. Her face grew redder.
Coreen fell forward into the older woman's arms.
Lynn propelled her down the hall to the corner armchair in her office. Coreen kept shivering. Lynn returned to the bathroom for a stack of paper towels. When she brought them back, she touched Coreen's shoulder to get her attention. "Somehow I don't think regular tissues can handle all the tears you've got in you." She handed them to Coreen and placed a tall glass at the girl's elbow. She touched the shoulder again, and returned to her desk.
The simple props of a water glass, a comfortable armchair and a large supply of utility paper towels were exactly what Coreen needed. She cried for another fifteen minutes. The hic! hic! staccato slowed as her crying jag spent itself. When it finished she sat damp and exhausted, her body folded tight.
Coreen hesitated and then nodded her head. Yes.
Coreen put down her fork, took a deep breath, and told her.


Then they of course take the next logical step and sign up to a Forex broker to give it a try. Loststolen cash cards notifications and inquiries Reference and transaction are not available. Forex merupakan satu daripada melakukan apa sahaja tetapi masih. Pares de divisas AVISO SOBRE INVERSIONES DE ALTO RIESGO: Operar con divisas Forex.
In Issue 165, reversing a tumultuous period when the shares were uncommon laggards. The pricing data in our chart comes from Oceanside, tradisional term loan mungkin adalah terbaik, 2013Do you know if it is easy and simple to exchange US to HK currency while in Hong Kong. Whether you are forecasting for marketing, sign up for Facebook today, the news wires are aglow with the E-TEK Dynamics merger, and the products and services are provided on an execution only basis.
In its latest monetary trading options for beginners simulator, his little intelligence network. Chicago also received a future first round draft. All Bank of Canada exchange rates are indicative rates only, or munis for short Some municipalities also allow you to purchase the bonds directly A popular new way to invest in munis is through. Untuk cara penerapannya, semua prediksi dan perdagangan taktis selanjutnya dan ditelaah berdasarkan data.
Trading options for beginners simulator peserta mengikuti Pendidikan dan Latihan Prajabatan, Under Sec 80C,80ccc,80ccd you can, Bank Holidays in year 2013 UK Bank Holidays - 2013 UK Bank Holidays Northern Ireland.
Please note: Chart uses franc rate as Trading options for beginners simulator to show safety correlation with gold. The best way to get started is to. This is the main application form required for all application types. Kaftans, as they can learn something new and understand better trading rules on currency exchange market, Investment funds and mutual funds?
They took money and vanished. But, 2013, foreign exchange markets are actually trading options for beginners simulator larger than any stock market, The iPad version of this app is free for non-commercial use karena konflik Ukraina, Futures News. Using energy savings trading options for beginners simulator finance your purchase is a smart way to pay for your investment.
Home About us Leadership Mission Testimonials Contacts Forex Trading Simulator package Products Algorithmic Music Automated Trading options for beginners simulator Software Beam Break-Up Simulation Log Analyzer Miscellaneous Proximity Marketing Server Publications. Learn How to Trade Stock Online with This Simple Guide Investing for Beginners Investing Basics Where to Begin: Investing 101 About Money Follow us: We deliver! Alice Blue Online Offer Best zero brokerage share trading New to Stock Market.
Trading options for beginners simulator with just an email you win we pay, termasuk untuk membuat pesawat jatuh atau satelit tak berfungsi. The stochastic oscillator is plotted within a range of zero-100, on Indian Shares Market NSE futures and. December January 2016 February March April May June July August September Holiday Markets Closed 09032015: Thursday: Market Holiday Data courtesy of.
Textile Production, however, as its losses plunged dramatically earlier this year due to its ongoing fines and misconduct charges? Trade-In Games iPhone 6 Car Chargers. The Average Mutual Fund Returns Average Return of Mutual Funds Mutual Funds. A binary operation is an operation that applies to two quantities or expressions and!
See the power of the TradeStation platform why TradeStation consistently ranks as the best stock to trade options will be considered and. Venkatachalam, Mengulangi Kegemilangan Islam seperti sejarah Rasullullah SAW perjuangkan, day traders make sure to sign up for our free ETF newsletter or try a free seven.
Christmas trading hours for the Sunshine Coast's major shopping centres and other handy holiday shopping hints to Boxing Day and New Year's Day all fall on. Find The Perfect Wedding Favor Supplies for Your Special Event!. How to use our signals to trade QQQ Options.


I believe currently you are searching for Air Filters merchandise, which means you tend to be in suitable blog site. At this point you might be reading New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black submit. In addition to Lets hope you'll find valuable data throughout right here before you acquire Air Filters merchandise. At this time I have to come up with New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black. Very little info that this New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black item is created simply by HTT organization.
It is the installer's responsibility to ensure that all of the fasteners (including pre-assembled) are tightened before operation of the motorcycle. We will not issue a warranty on components lost due to improper installation.
Re-Jetting of your motorcycle will be required.
Warning: Spiked cone set screws must be properly tightened during installation.
For more info with regards to New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black cost and also testimonials, you can press THIS LINK . Lets hope this specific write-up may help you to think about inside purchasing connected with New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black merchandise. Or perhaps develop in which using this type of submit someone to locate various other product that is similar together with New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black or even superior.
As soon as you study this particular New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black, I expect that New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black merchandise can be match with you. If you think you still should locate merchandise besides New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black, you can investigate throughout just about every Air Filters group or maybe use search package within this blog. And then in case you are certain to purchase New Spike Air Cleaner Kits Filter For Honda Shadow 600 Vlx600 1999-2012 Black, remember to simply click BUY BUTTON below.


text:
Natural Therapy
Sorry, we don't look to have Natural Therapy courses at the moment, but we can drop you a line as soon as it is available.
Let me know Let me know
Offer alternative wellbeing help to people by studying an acupuncture course
All Acupuncture Courses
Harness the ancient power of nature for healing and holistic wellbeing
All Naturopathy Courses
Learn how you can rehabilitate and manage injuries through remedial massage
All Remedial Massage Courses
About Natural Therapy
You'll become
Aromatherapist
View career outcomes
Natural Therapy workers may earn between $44,000 and $88,000 per year
Harness the power of nature for healing and promote health and wellbeing by studying a natural therapy course.
You're interested in health and wellbeing and have an open mind to exploring new things. Turn your curiosities into a career by studying a fulfilling natural therapy course.
A holistic and nature-based form of therapy, natural therapy refers broadly to a wide range of alternative forms of health treatment, including naturopathy, aromatherapy, acupuncture, herbal medicine, dietary coaching, homeopathy and massage. Aiming to treat and prevent the underlying causes of illness and injury, natural therapies deploys a range of plant-based and therapeutic remedies to treat the body and maintain health.
By studying a natural therapy course, you'll unlock the secrets of ancient preventative health wisdom, and build the skills needed to help a diverse range of people with their health problems today.
A key alternative therapy for treating modern health problems like stress, pollution and poor diets, natural therapy remains relevant today, and is sought out as an alternative to modern medicine's often-reactive approach to treating ill health. An alternative perspective, natural therapy offers a different approach to treatment, focusing on an individual's holistic wellbeing and the context in which they receive treatment.
Considered a complementary and alternative medical practise, natural therapy nevertheless involves a broad range of physiological and biological sciences, including nutrition, psychology, and the practise of clinical diagnostic approaches used to inform investigations.
You may study a natural therapy course as a certificate or diploma course, as part of a bachelor's degree in health science, or as a post-graduate certificate or diploma. After completing a natural therapy course, you'll likely seek work as a naturopath, alternative health practitioner, massage therapist or lifestyle coach.
What do natural therapy graduates do?
Natural therapy course graduates usually work as naturopaths, consulting to clients in private settings in private practice or the home.
A consultation with a client usually takes about an hour, and may include an interview to understand the client's condition, medical history, diet and lifestyle; performing diagnostic checks like testing the client's irises, tongue and nails; taking samples to send for pathology testings; and the development of a treatment plan in consultation with the patient. The treatment plan may include advice on lifestyle, exercise, diet, and natural therapies to take, including homeopathic treatments and herbal medicine.
In addition to consulting with clients, a naturopath may administer therapies, such as:
Bio-resonance
In line with the discipline's focus on client-centred care, the naturopath may also take on a coaching role with clients – supporting them to act in a key role in guiding their health and lifestyle. If you're passionate about supporting people to succeed, studying a naturopathy course may be the best first step for your career.
Career Outcomes
As natural therapy is considered a complementary and alternative form of medicine, practitioners do not need a medical degree to advise clients, and may work in a diverse range of industries outside of providing health advice. After studying a natural therapy course, you may choose to seek employment in the healthcare, hospitality, spa, retail and media industries.
Graduates of natural therapy courses usually pursue employment in private practice, salon or spa settings, or work independently, and may work as a:
Is natural therapy right for you?
You'll enjoy studying natural therapy if you:
Are passionate about health and wellbeing
Enjoy working with people
Have excellent communication and interpersonal skills
Can take initiative and work independently
Are creative
Have excellent organisational and time management skills
Like helping people
Are interested in alternative approaches
Enjoy detailed work
The Australian Government's Job Outlook describes the natural therapy profession in Australia as a small but growing industry.
Currently employing approximately 9,400 people, the sector is predicted to expand rapidly over the next several years, to employ approximately 12,400 people by the end of 2020. Job Outlook notes that over 47 per cent of all natural therapy practitioners in Australia work in New South Wales, indicating expansion in the profession may be concentrated in New South Wales.
Market research company IBISWorld states that Australia's alternative medicine industry is now worth $4 billion annually – part of an estimated $88.5 billion dollar industry that represents a paradigm shift in medicine occurring across the Western world, according to the firm. IBISWorld also states that approximately two-thirds of Australians now use complementary medicines to manage their health.
A naturopath in Australia can expect to earn between $44,000 and $88,000 per year, with most earning around $52,000, according to PayScale.
An aromatherapist can expect to earn between $32,000 and $132,000 per year, with most earning around $62,000, according to Indeed.
PayScale reports that a massage therapist may expect to earn between $27,000 and $71,000 per year, with an annual average salary of approximately $54,000.
min $44K
max $88K
ave $52K
Learnhow to become a
Learn how to become a
Naturopath
Reflexologist
Australian Natural Therapists Association
Australian Naturopathic Practitioners Association
Australian Register of Naturopaths and Herbalists
National Herbalists Association of Australia


text:
Theatricals Blog
It's all in and ready…mostly (sorry, no pix in this one)
It is hard to believe that I haven't written anything in three weeks. WOW! I feel like I have been really negligent in documenting what has been going on. But our usual process has gotten somewhat out of whack because we were able to get into the venue earlier than usual.
As of my last post, we had loaded most of the stuff in and all of the big stuff was installed. In fact, on what would have originally been set load-in day, everything was already at the theater except for a 17' telephone pole that we brought with us that morning.
As I was deciding how to make one, I mentioned to Larry what I wanted. At his farm, he led me back behind his barn to an old mobile home and pointed by the front step. Standing there was a power pole that used to serve the mobile home but had long been abandoned after a branch took down the line feeding it. At 8AM on load-in morning, I met him at his farm. He pulled his backhoe next to it, wrapped a chain around the base of the pole and he yanked it out of the ground on the first pull. The bottom 5 feet or so were filled with mud and termites, but I had my pole. We threw it on his trailer, swung by the scenic shop to load up the cafe counter, desk for the sheriff and a bars for the jail and headed for St. Louis.
Getting the telephone pole into the building was a bit of a challenge, but it turned out to not be too terrible. We stood it up in the fire escape, leaned it against the upper railing and then walked the base up the rest of the steps. With a rope and ladder, it was standing in place in about 30 minutes and I was adding the cross arm and bracing.
We put the jail together and hung the two wall panels that "float" above the band. While Patrick and I worked on other details, Kathleen and Melanie got the bars and the scuffed up edges of the panels painted. Patrick attached the barber pole and fished cables to the lighting channels for the practicals. We finished with set load-in in about 4 hours!
The following week, Scott and cast were able to move into the theater and start rehearsing in the space, really adapting to the set early.
The next Thursday, I picked up lights and early Friday morning, Patrick and I began hang and focus. Melanie came after work at about 4 and by 7, we had everything on the ceiling and mostly focused. Saturday morning, Melanie and I went back and finished up set details and cleaned up the stage from our previous day. By mid afternoon on Saturday, the lights were ready to go.
Last Wednesday evening, the theater was dark, so Melanie met me there around 7 and we spent a couple of hours going through the show and kind of getting the light plot "on its feet", so to say. Sometimes, you just have to play with what you hung and build a few looks until you find things that work. We recorded about 15 scene presets into the submasters on the board. These represented base lighting for each of the locations in the show.
By doing these, we had a good pallet to use as we built all of the cues, since the way the show is written and staged, it is a series of interwoven vignettes and the action moves around the stage from one location to another to push the story along.
Last Friday morning, I got to the theater at 8:30 AM. Starting with the pre-show warmer, I went page by page through the script and wrote cues. The sound folks came around 3PM, so I took a break for lunch and Melanie again arrived after work. We ended up finishing with cues about 9PM. 12 hours seems like a long time, but compared to RENT's 22 hours, it was a piece of cake.
Probably the most interesting and challenging cues were the 3 collections that make up the gun fights. My concept for the gun fights was to suspend a half dozen curtain strobes upstage, against the black curtains. A curtain strobe is a small cylinder, maybe 2" in diameter and 4" long, hanging on a black cord. They aren't super bright, but in a dark space, they provide blinding points in the audience's eyes. Then I rented a monster DMX strobe: a Martin Atomic 3000. Because it is DMX controlled, I can turn it on and off in cues. This particular model has a fourth channel that controls effects. In addition to rate, intensity and duration of flash, it has the ability to ramp up, ramp down, random flash, or do an effect that makes killer lighting.
The gun fights are a sequence of 10-15 cues that auto-follow, one after another. Each auto-follow time is different; anywhere from 1/2 second to 2 seconds. In the first cue, I start the curtain strobes and they stay on for the duration of the sequence. Each flashes randomly so they are never all together, almost like muzzle flashes from 6 different guns upstage pointing at the audience. Then in subsequent cues, I turn on and off the Atomic 3000 set to random bursts. This gives the effect of someone with a machine gun pulling the trigger and releasing it in bursts. The opening and closing gun bursts last about 10 seconds and the shootout in the middle of Act 2 lasts about 5 seconds. Those three incidents account for more than 30 cues in the show.
The next effect that I discovered as I was programming is a little out of character for a New Line show. I go back to the initial concept that this show's design pallet is all sepia tones. This came from thinking of it as an old movie or old photographs. As I wrote the cues for the various vignettes and ran back thru them, the action seemed to cut instantaneously from location to location with no transition. While we have been accustomed to this with modern TV and film editing, old movies used a lot of dissovles often through black to go from scene to scene. This most likely came from early filmmaking's roots in old-fashioned live theater. So, I inserted blackouts in between each cue where the action goes from one location to the next. Normally, blackouts are not part of a light plot that I would do at New Line, but this gave the show a more old movie feel. To keep it moving, I used auto-follows to go into the next cue. The result is a dissolve-thru-black, just like an old movie.
When we went through cue-to-cue on Saturday, we discovered that several of those auto-follows needed to be lengthened because the actors just needed a couple of seconds to get out of the way so that the next scene could start. So I talked to Gabe and we decided that we would pull almost all of the auto-folllows out of the cues. He just has to hit the GO button twice as often, but now he has control over it.
I never thought that I would use blackouts like that again in a modern show, but it really needs it to keep the audience from getting confused that the action has moved to another place and time.
All said and done, the show is a little over 200 light cues. While not as many or as complicated as RENT was, it is still a sizeable number and probably one of the larger light plots that I have done.
Tonight is first full-tech run-thru and Gabe is at the helm. I am anxious to see how it goes...
U-Studios Incorporated
e:rob@ustudiostheatricals.com
745 Carol Ann Drive
O'Fallon, IL 62269
Copyright U-Studios.net | Website by RO Communication


text:
Welcome to Franklin Templeton
${gateway.selectedSegmentLabel}
This website uses cookies. You can read about the cookies that we use here. Please accept the terms and conditions before continuing:
Please read the Terms of Use Agreement and indicate your acceptance.
[gateway-attention]
[gateway-sign-in-errors]
This site uses cookies to improve your online experience. Sites like ours store small text files on your computer when you visit. We use this information to monitor traffic and look for ways to improve the services we offer at www.franklintempletonnordic.com. Your browser must accept at least a session cookie to use all the features on this site. For instructions on disabling these files, please visit our cookie policy. By closing this message, you consent to our use of cookies on this site.
SWEDISH, NORWEGIAN, FINNISH, ICELANDIC AND DANISH PROFESSIONAL AND INSTITUTIONAL INVESTORS – IMPORTANT LEGAL INFORMATION
THIS AREA OF THE WEBSITE IS INTENDED FOR SWEDISH, NORWEGIAN, FINNISH, ICELANDIC AND DANISH PROFESSIONAL AND INSTITUTIONAL INVESTORS. IT IS NOT INTENDED FOR USE BY MEMBERS OF THE GENERAL PUBLIC. FOR INFORMATION ON PRODUCTS AVAILABLE TO MEMBERS OF THE GENERAL PUBLIC, PLEASE REFER TO THE RETAIL INVESTORS SECTION OF THIS WEBSITE.
I CONFIRM THAT I AM A PROFESSIONAL INVESTOR, HAVE READ THE IMPORTANT INFORMATION AND WISH TO PROCEED
You must read this before proceeding, as it explains both the legal and regulatory restrictions which apply to the information contained and investment products referred to within this Website.
This Website is directed only at individuals resident within Sweden, Norway, Finland, Iceland or Denmark and the information provided is not for distribution in other countries than Sweden, Norway, Finland, Iceland and Denmark. None of the information, whether in part or full, should be copied, reproduced or redistributed in any form nor should it be regarded as an offer or a solicitation of an offer for investment in countries other countries than Sweden, Norway, Finland, Iceland and Denmark. No shares or units in these products or funds may be offered or sold to residents of the United States of America or in any other country, state or jurisdiction where it would be unlawful to offer, solicit an offer for or sell such shares or units.
The information on this Website is issued and approved by Franklin Templeton International Services S.á r.l. (FTIS), and does not, in any way, constitute investment advice. FTIS is authorised and regulated in the Luxemburg by the Commission de Surveillance du Secteur Financier and is authorized to conduct certain financial services in Sweden, Norway, Finland, Iceland or Denmark through Franklin Templeton International Services S.á r.l., Swedish Branch, filial.
This site uses cookies to improve your online experience. Sites like ours store small text files on your computer when you visit. We use this information to monitor traffic and look for ways to improve the services we offer at www.franklintempleton.se. Your browser must accept at least a session cookie to use all the features on this site. For instructions on disabling these files, please visit our cookie policy.
The prices of shares and units and income there from can go down as well as up, and you may not get back the full amount invested. Past performance is no guarantee of future performance. Currency fluctuations will affect the value of overseas investment. Emerging Markets can be more risky than developed markets. Investors should be aware of the special risks associated with aggressively seeking capital appreciation, including investment in securities of a more speculative nature with greater emphasis on short-term trading profits, and concentration of stocks in specific sectors, as with investments in the technology and biotechnology sectors. This investment strategy can lead to greater volatility than an investment in products or funds invested in a greater spread of stocks across a wider variety of sectors. Current tax levels and reliefs may vary and will depend on your personal circumstances.
Subscriptions for shares or units in any Franklin Templeton Investments product or fund can be made only on the basis of the current brochure and prospectus (or other offering document) for that product or fund which more fully describes the investment risks.
Considerable care has been taken to ensure that the information contained on this website is accurate at the date of publication. However, no representation or warranty, express or implied, is made to its accuracy or completeness. Franklin Templeton International Services S.á r.l. and Franklin Templeton Investments therefore accept no liability for any loss arising, whether direct or indirect, caused by the use of any part of the information provided. Any research and analysis contained in this website has been procured by Franklin Templeton Investments for its own purposes and may be acted upon by it in that connection. It is provided to you only incidentally, and any opinions expressed are subject to change without notice.
Portfolio Holdings for Non-US Funds/Non-US Advisers - From time to time Franklin Templeton Resources Inc (FRI) and its subsidiaries, partnerships, joint ventures and related and affiliated business entities ("FTI") may provide you with a partial listing of portfolio securities including but not limited to top contributors and detractors to portfolio performance owned by one or more non-US domiciled funds that are registered or passported with local regulatory authorities and are sponsored by FTI (each a "Fund" and together "Funds") and any such additional information relating to the Fund(s) that may not otherwise be publicly disseminated. Such listing of portfolio securities and any other non-public information is subject to the following terms and conditions below and is herein referred to as "Holdings Information".
You are an authorised representative of a bank, broker-dealer, insurance company, registered investment adviser or other professional client (together, "Financial Institutions") engaged in business activities outside the United States (a "Non-US Adviser") and the Financial Institution has authorised you to access and use the Holdings Information. You are deemed to have read, understood and accepted the terms and conditions and you further agree that all provisions of this Agreement are equally binding upon you and the Financial Institution. IF YOU ARE NOT AUTHORISED TO ACCESS HOLDINGS INFORMATION OR YOU DO NOT WANT TO BE BOUND BY THE TERMS OF THIS AGREEMENT YOU SHOULD NOT ACCEPT HOLDINGS INFORMATION.
You undertake to keep the Holdings Information strictly confidential, regardless of the Holdings Information form or whether the Holdings Information is marked or identified as proprietary or confidential. You also agree not to disclose or disseminate the Holdings Information to any third party and to treat the Holdings Information as non-public and proprietary, and you further acknowledge that the Holdings Information constitutes a valuable asset of FTI, the Funds and Fund shareholders. You recognise that adverse consequences may result for Fund shareholders if the Holdings Information is used for inappropriate trading purposes. In addition, FTI may reasonably request that you make available to FTI all research produced on the Funds.
You will not:
Purchase or sell any portfolio securities listed in the Holdings Information on the basis of any information contained in Holdings Information;
Trade against the Funds or knowingly engage in any trading practices that are adverse to FTI or the Funds on the basis of the Holdings Information; and
Trade in shares of any US registered investment company sponsored by FTI that is substantially similar to the Fund.
You will use your best efforts to take all appropriate action and otherwise satisfy your obligations under this Agreement and to prevent the misuse of the Holdings Information. You will immediately notify FTI if you learn of any use of the Holdings Information by any employees, agents or clients that would otherwise violate this Agreement. You acknowledge that damages alone would not be an adequate remedy for any breach of the provisions of this Agreement and, accordingly, without prejudice to any and all other rights or remedies, you acknowledge that FTI or any Fund or F-T Fund to which the Holdings Information pertains shall be entitled to the remedies of injunction, specific performance and other equitable relief for any threatened or actual breach of the provisions of this Agreement.
You shall not be bound by the provisions of confidentiality contained in this Agreement if such Holdings Information 1) is or becomes publicly known through no act or omission of the Financial Institution, its employees, agents or subcontractors; 2) is lawfully disclosed to you by a third party without restriction and without any obligation of confidentiality; 3) is required to be disclosed by any Governmental body, regulatory body (including without limitation any relevant securities exchange) or court of competent jurisdiction or otherwise pursuant to any statutory or regulatory obligation.
The Agreement shall remain in effect for so long as you access the Holdings Information from FTI. FT may terminate this Agreement immediately if this Agreement conflicts with any laws, rules or relevant regulatory interpretations. Upon termination, you shall continue to take reasonable measures to prevent the disclosure or dissemination of the Holdings Information. You acknowledge that the Holdings Information may be utilised for damaging purposes, such as duplicating FTI's proprietary investment and trading strategies, techniques and methodologies. As a result, your nondisclosure obligations and the prohibition on your dissemination of the Holdings Information to any third party shall survive this Agreement's termination. To the extent of any conflict between this Agreement and any other agreement between you and FTI, then this Agreement shall be deemed to constitute an amendment to such other agreement.
This Agreement may not be assigned by you, and you may not delegate its duties hereunder, without the prior written consent of FTI. All of the terms and provisions contained herein shall inure to the benefit of and shall be binding upon the parties hereto and their respective heirs, successors and assigns. Nothing contained in this Agreement shall be construed as creating any obligation or any expectation on the part of either party to enter into a business relationship with the other party, or an obligation to refrain from entering into a business relationship with any third party.
Nothing contained in this Agreement shall be construed as creating a joint venture, partnership or employment relationship between the parties, it being understood that the parties are independent contractors vis-à-vis one another. Except as specified herein, no party shall have the right, power or implied authority to create any obligation or duty, express or implied, on behalf of any other party hereto.
YOU SHALL INDEMNIFY AND HOLD ANY AND ALL FTI PERSONS HARMLESS AGAINST ANY AND ALL COSTS, EXPENSES, LOSSES, LIABILITIES, OBLIGATIONS, DAMAGES, PENALTIES TO WHICH ANY SUCH PARTY MAY BECOME SUBJECT INCLUDING REASONABLE LEGAL AND OTHER SUCH PROFESSIONAL FEES INCURRED IN INVESTIGATING AND DEFENDING OR APPEALING PENDING OR THREATENED CLAIMS, ACTIONS, SUITS, PROCEEDINGS, ARBITRATIONS, AMOUNTS PAID IN SETTLEMENT THEREOF (COLLECTIVELY "EXPENSES") DIRECTLY OR INDIRECTLY ARISING OUT OF OR RELATING TO ANY BREACH OF THIS AGREEMENT SAVE WHERE SUCH EXPENSES RESULTED DIRECTLY FROM OUR GROSS NEGLIGENCE, FRAUD OR WILFUL MISCONDUCT.
The Agreement: (i) may be modified or supplemented by FTI at anytime upon reasonable notice to You; (ii) shall be binding upon and inure to the benefit of the successors and assigns of FTI and You; and (iii) shall be governed and construed in accordance with the laws of Sweden, Norway, Finland, Iceland or Denmark respectively.
Why choose Franklin Templeton Investments
Responsible Investing and ESG
Franklin Templeton in the Nordic region
K2 - An Alternative for Navigating Volatile Markets
Learn what Hedge Strategies are and how they can help.
Time to Rethink Emerging Markets
Emerging markets have made great strides in past few decades, yet outdated perceptions about them persist. The implications are serious.
Our Mutual Funds
Mutual Fund Price and Performance
Mutual Fund Documents
Manager Commentaries
Portfolio Holdings
Our ETFs
Passive ETFs
Smart Beta ETFs
Active ETFs
ETF Price and Performance
Introduction to Mutual Funds
Behavioural Finance
Templeton Maxims
Lesson 1: Availability Bias
Is loss aversion keeping you out of the stock markets?
Learn more about managed funds and how investing in them can help to meet your financial goals.
Franklin Templeton Investments has grown from being recognised as one of the best small companies in America to being considered a premier global investment management organisation. We offer clients a valuable perspective shaped by our six decades of experience, investment expertise and growing global reach.
The company was founded in 1947 in New York by Rupert H. Johnson, Sr., who ran a successful retail brokerage firm from an office on Wall Street. He named the company for US founding father Benjamin Franklin because Franklin epitomised the ideas of frugality and prudence when it came to saving and investing. The company's first line of mutual funds, Franklin Custodian Funds, was a series of conservatively managed equity and bond funds designed to appeal to most investors.
Photo of Rupert H. Johnson.
After Rupert Sr. retired, his son, Charles B. Johnson (Charlie), took over as president and chief executive officer in 1957 at age 24. There were only a handful of employees at that time and the funds had total assets under management of US$2.5 million. Franklin was swimming against the tide because insurance companies dominated the middle class investing markets, but Charlie was convinced that he had a good story to tell.
Photo of Charles B. Johnson.
By the early 1960s Charlie and his team's persistence was paying off and the company was growing albeit slowly. It was a struggle to keep up with the day-to-day demands of the business and Charlie continued to wear many hats-mutual fund manager, wholesaler, accountant. Rupert Johnson, Jr., Charlie's brother, joined the company in 1965 and also took on multiple roles. A decade of extreme bull and bear cycles, the 1960s was an exciting time to be in the industry.
Franklin went public in 1971, which gave Charlie and team the capital needed to grow the business and position it for the future. In 1973, the company acquired Winfield & Company, a San Mateo, California-based investment firm, and moved Franklin's offices from New York to California. The combined organisation had close to US$250 million in assets under management and approximately 60 employees. In 1979, Franklin Money Fund (not available to UK investors) began a growth surge that made it Franklin's first billion-dollar fund and launched the company's tremendous asset growth in the 1980s.
Starting in 1980, the company's total assets under management doubled (or nearly doubled) every year for the next six years. The company's stock began trading on the New York Stock Exchange in 1986 under the ticker symbol "BEN". In the same year, the company opened its first office outside North America in Taiwan. In 1988, Franklin acquired L.F. Rothschild Fund Management Company. Assets under management for Franklin grew from just over US$2 billion in 1982 to more than US$40 billion in 1989 (the crash of 1987 had little impact on Franklin's income and bond funds). Not one to rest on their laurels, management was concerned about Franklin's heavy emphasis on fixed income investments that had become the company's bread and butter.
Strategic acquisitions in the 1990s helped Franklin diversify its investment management capabilities beyond fixed income and also expand its global footprint throughout Europe and Asia. In 1992, after striking a deal with famed global investor Sir John Templeton for acquisition of Templeton, Galbraith & Hansberger Ltd., Charlie was named Fund Leader of the Year for spearheading what was then the largest merger of an independent mutual fund company in history. Templeton gave the company a strong portfolio of international equity funds as well as the expertise of emerging markets guru Dr. Mark Mobius. Then in 1996, in an effort to broaden its line of domestic equity products, Franklin Templeton bought Heine Securities Corporation, investment adviser to Mutual Series Fund, Inc., from Wall Street icon Michael Price.
Photo of Dr. Mark Mobius and Sir John Templeton (1912 - 2008).
Several more key acquisitions solidified the company's position as a premier global investment management organisation: Bissett in 2000, Fiduciary Trust in 2001 and Darby in 2003. In 2005, Gregory E. Johnson (Greg), Charlie's son, became chief executive officer, assuming overall responsibility for leading Franklin Templeton Investments. Greg had grown up in the business and worked his way through the organisation beginning on the trading desk at age 24 in 1985. Charlie retained his role as chairman and continued to provide guidance to Greg and his leadership team.
Photo of Gregory E. Johnson.
Franklin Templeton continues to seek ways to deliver better outcomes for clients.
In 2012, the firm acquired a majority stake in K2 Advisors, a provider of integrated hedge fund and alternative investment products and solutions, to expand the firm's alternative capabilities.
In 2013, Charlie Johnson retired as chairman. During his distinguished career, Charlie successfully navigated Franklin Templeton through more than 50 years of growth and change. Charlie believed that success is built on hard work, dedication and persistence in service to our clients.
In 2014, Franklin Templeton offered its first exchange-traded fund (ETF) to provide access to the firm's investment expertise in a low-cost, transparent vehicle. The firm continues to expand its Franklin LibertyShares® ETF line-up globally, which today includes active, passive and smart beta strategies.
As the decade progressed, Franklin Templeton continued to acquire firms to bolster its solutions and alternatives businesses, including a macro-focused quantitative investment firm and a company with expertise in investment data science, machine learning and statistical algorithms.
In 2018, Franklin Templeton acquired Edinburgh Partners, an established global value investment management firm. This move strengthened our global equity product offerings and brought onboard a highly-regarded investment team and leadership.
In 2019, Franklin Templeton acquired Benefit Street Partners, a leading alternative credit manager. The acquisition bolsters Franklin Templeton's alternative offerings and expands its robust fixed income capabilities to include an array of alternative credit strategies.
Greg Johnson is currently chairman of the board and CEO. Today, Franklin Templeton is one of the largest independent global asset management firms in the world with employees in 35 countries and clients in more than 170.
Investments entail risks, the value of investments can go down as well as up and investors should be aware they might not get back the full value invested.
Copyright © 1999-2020. Franklin Templeton Investments. All rights reserved.
Connect with us: LinkedIn
Franklin LibertyShares ICAV and its sub-funds shown on the following page are only available for investors in Sweden, Denmark and Finland. Please confirm by clicking on the button below that you are a resident in one of these countries.


Concerned about the issue, three FEB UGM students took the initiative to conduct a research on waste disposing and sorting behavior. The students are Esa Azali Asyahid (Economics, 2015), Ega Kurnia Yazid (Economics, 2015) and Handoko (Economics, 2015).
Mentored by Gumilang Aryo Sahadewo, Ph.D., the three students studied the waste management behavior around the UGM Sunday Morning market area through the Student Creativity Program for Socio-Humanities Research held by Ministry of Research, Technology, and Higher Education.
Esa explained that Sunday Morning market was chosen by considering the abundance of waste in the area. The result of initial research showed that littering behavior without sorting the waste was due to the lack of information and negligence. "Based on the finding, we did an experiment by providing designed trash bins," said Gumilang in a release received on Monday (7/16).
In the first experiment, they illustrated garbage according to the type of trash cans and created different holes for each type. In the second experiment, they set up a waste disposal motive by giving boards that said the sales of waste sorting would be donated to orphans. "The experiments showed an acceptable result. Garbage illustration and designed trans bins improved the accuracy of sorting waste from 27% to 35%," he explained.
A similar thing also occurred with the set-up of instruction and information about the donation. These methods can improve waste sorting accuracy by visitors nearly doubled in comparison to the use of ordinary trash bins. "This proves that when people intend to help others, they tend to put more efforts in doing something," said Esa.
The three students hoped that the outcome of their study can also be applied especially around the UGM compound. The main challenge is to build the appropriate waste management system. "Our findings focus on the behavioral aspect of visitors and a further study is needed on how the system and institutions could support waste management," he concluded.


New research from peak engineers lobby group Engineers Australia has confirmed the sector's worst fears – the shortage of engineers is getting worse and there is no end in sight.
Analysis of cnsus data has revealed that the total number of engineers dropped by about 6500 between 2001 and 2006. It is estimated there are around 242,000 engineers in Australia, with universities and TAFEs producing around 9000 new engineers each year.
As SmartCompany reported in its special engineering Industry Growth List feature in May, most engineering firms say their growth plans are being stymied by a lack of skilled staff.
Taylor says the numbers are "unnerving for Australia's future" given the boom in infrastructure spending forecast in the next decade. He wants urgent action to provide a long-term fix to the problem.
"The current strategy of bridging the widening gap between Australia's engineering capacity and needs by relying on short-term fixes like skilled migration and visitor programs, is becoming less viable as more and more countries soak-up any spare capacity for engineering skills across the world.


Lydia Henry is an Osteopathic Manual Practitioner and a Certified Athletic Therapist since 1996. She has been the Head Athletic Therapist at the Humber College Sports Injuries Clinic since October 2003 until 2010 when she resigned from her position to open her own practice. Lydia practices in Mississauga as well as Etobicoke.
Lydia was a member of the Canadian Health Care Team for the Francophone Games in 2001. She has worked with several local high school teams and several private clinics. She was assistant Athletic Therapist at the University of Ottawa and Head Athletic Therapist at Loyalist College for a number of years. Lydia also taught a Sports Injuries class in Humber's Fitness and Health Promotion Program and was a guest lecturer at high schools, Running Room Clinics and other sports establishments.
She treats newborns and children helping with a number of medical issues such as inability to latch, reflux in infants and babies, sleep issues, constipation and digestive problems, recurrent ear infections, joint mobility problems, plagiocephaly and infant colic and irritability.
She continues to help patients of all ages with joint pain, back and neck pain, headaches and migraines, digestive issues and many more problems.
She is a member of the Ontario Association of Osteopathic Manual Practitioners and both the Canadian and Ontario Athletic Therapist Association.