There's a landfill's worth of nutrition and health nonsense out there. With news headlines contradicting each other daily and social media stars disagreeing on whether you should eat nothing but cabbage or anything but cabbage, how are you supposed to know what to believe?
The answer: a licensed nutritionist.
1. A juice cleanse is a good idea.
To be fair, juice cleanses sound good: weight loss, decreased toxins, increased energy, and even a lower risk of heart disease. But, the research doesn't support those outcomes1, and neither will our nutritionists. Juice tends to be high in sugar, low in protein, and missing all the fiber that can support weight loss and prevent heart disease.
Check out methods for a natural detox here.
2. You should eat just 500 to 1,000 calories per day for weight loss.
Will you lose weight if you only eat 500 to 1,000 calories a day? Probably – because you're starving your body. Is it sustainable? No. Most people can lose weight at a more reasonable calorie level. When you think about what you can maintain long term, 500 calories a day doesn't fit into this picture.
Eating fewer than 1,200 calories per day would make it very difficult to meet your nutritional needs and support sustainable weight loss.
3. You need to work out 7 days a week.
Working out every day of the week can cause higher levels of inflammation and make it difficult for your body to recover from a workout fully2, 3. Six days a week – at most – is what our nutritionists will recommend. A day of rest is good for the body. On rest days, a light walk or yin (restorative) yoga class could be a good option.
This blog on chronic inflammation discusses overworking your body.
4. You only need 4 hours of sleep a night.
Getting enough sleep is one of the most important ways you can stay healthy. Research shows that seven to nine hours is the most appropriate for most people4 and that this amount will support heart health5, brain health6, stress management7, and even BMI8.
5. Don't do what your doctor says.
Your doctor is usually right. In the few instances when your doctor may have said something that doesn't match up, our nutritionists can help guide you to have a constructive conversation with your doctor to clarify their intentions.
6. Cholesterol in eggs and shrimp will raise your cholesterol.
This is a perfect example of how staying on top of the current research or having a coach who knows the current research is vital. Bring back the eggs and shrimp! Research shows dietary cholesterol, such as that found in these eggs and shrimp, is hardly absorbed by the body9. Instead, saturated fat from foods like red meat, bacon, cheese, full-fat dairy, and many take-out foods contributes the most to your cholesterol levels. In fact, eggs may even be good for your heart! Want to know more about cholesterol? Check it out here.
7. You should never eat French fries.
Are French fries every day going to part of a healthy diet? No. Every week? Probably not. But, never? That's not really a healthy diet either. All foods in moderation can be part of a healthy diet. Saying you can never have a specific food can cause a scarcity mindset that itself be unhealthy. Savor your French fries, when you do choose to have them, eating them slowly, mindfully, and without any distractions. You may be surprised how few you eat when you focus on the act of eating them!
8. You need to sweat to have a good workout.
Curious what's needed for heart health, exercise-wise? This blog has got you covered.
9. Guacamole is not healthy.
Actually, for many of us, it is! Guacamole has lots of heart-healthy fats and is high in fiber. Choosing guacamole without any creamy additives – no sour cream, crema, or mayonnaise – will ensure it stays in the healthy category.
10. We have the perfect diet for your blood type.
There isn't any research to show your blood type indicates how you should eat. What the blood-type diet does get right is your diet should be personalized to you. Each individual needs to figure out which foods help them feel the best and should be part of their healthy diet and which are best to limit or choose less frequently.
11. It's better to eat nothing than to eat something unhealthy.
Our nutritionists will never say this because it simply isn't true. If you're hungry or needing food, eating something unhealthy will be better than eating nothing. Food is fuel, and if you don't fuel your body, you're likely to become so hungry you can't make supportive food choices (like a grain bowl or salad), which can lead to overeating or making choices that are high in calories and potentially high in fat.
Think of the last time you were so hungry you ate a cheeseburger, French fries, and a shake instead of the salad you were hoping for. Fuel your body regularly – especially when you're hungry. This can also help improve your self-control around foods.
12. Exercising 60 minutes a day will negate the doughnut you ate.
13. White foods are unhealthy.
White foods have gotten a bad rap because many are simple carbohydrates and provide minimal amounts of nutritional value per serving. White bread, white pasta, white rice, and potatoes are all foods that have fallen into this "unhealthy" category.
However, there are lots of white foods that are nutritious (and delicious)! Cauliflower, cannellini beans, some mushrooms, turnips, and – believe it or not – potatoes are all healthier white foods. As long as you aren't covering your potatoes in butter or gravy and are choosing to eat the skin, you'll be choosing a naturally low-fat complex carbohydrate with protein and fiber.
14. Coffee is bad for you.
It depends. Despite the panic when California put coffee on its list of carcinogens (due to a possibly cancer-causing byproduct of the roasting process), countless studies indicate coffee is good for you and may be associated with a reduced risk of several types of cancer, heart disease, and type 2 diabetes10. Read more here.
However, coffee may be bad for you if you have the genetic variants that mean regular consumption is likely to increase your blood pressure or disrupt your sleep. And, if you take your coffee with cream and sugar, you're adding extra calories and saturated fats to your day. Choosing a plant-based creamer, such as almond or oat milk, and reducing the sugar makes it more likely your daily cup or two is contributing to your health.
15. If you eat out more than 3 times a week, you're unhealthy.
It's true that it can be more difficult to stay healthy if you eat out a lot because it's difficult to know how much salt, fat, and calories are in the meals you're ordering. But, with careful ordering and restaurant selection, it can be done!
Foods that are lower in saturated fats, high in fiber, and contain a lean protein are going to be your best choice. Maybe a grilled chicken salad or bean burrito bowl (without cream or sour cream and topped with guac), for example. If you're eating out more than three times a week, consider what you're ordering and choose veggies every chance you get. Check out these tips on what to look for when eating out.
16. Carbs are the devil.
No research supports that carbs are the personification of evil. "Carbs" can encompass everything from beans and fruit to whole grains and pasta. They got their villainous reputation because "carbs" also means things like sugary cereals, pastries, and candy. Sugar, which is a carb, has been pointed to as a significant contributor toward the rise of type 2 diabetes worldwide11.
Portion size is important when you're choosing carbs. Having too large a portion gives your body a big load of fuel to take care of. Additionally, the quality of your carbs will make a difference. Choose complex carbohydrates, such as beans, whole grains, fruit, and some vegetables.
17. A glass of wine will help you sleep.
A glass of wine can relax the body, so it may help you fall asleep initially. However, alcohol can block REM sleep, cause the brain to create brain waves that usually only occur when you're awake, and cause you to wake up too early12. All of this works against you to create a less restful and shorter sleep cycle.
18. Your weight or number on the scale shows how healthy you are.
While the physical component of health may often be tied to a number, especially when health care is concerned, food choices, exercise, self-care, and mental wellbeing play a larger part in how healthy you are.
19. If you aren't willing to stop going out with your friends, you'll never lose weight.
Weight loss is challenging and often requires changes in both your behavior and your environment. This doesn't mean you need to stop going out with your friends – social wellbeing is a huge part of health (see above). However, what you choose to do with your friends could be impacting your weight. If you're constantly going out to bars, having too many drinks, and choosing foods that are deep fried or covered in cheese, it can make it challenging to meet your weight goals.
Working with a nutrition coach who can help you figure out the behavioral and environmental changes that fit you and your lifestyle best is one way to have the best of both worlds.
20. You must buy these supplements to be healthy.
You can be completely healthy without buying supplements. The most a supplement will ever do is possibly enhance your health (and, in some cases, can cause harm or side effects). It can't make up for unhealthy eating, not exercising, or not getting enough sleep.
That being said, are some supplements better than others? Definitely. If you aren't getting certain nutrients, a supplement can help enhance your diet. But, taking a supplement when you're already hitting it out of the park with your diet and exercise won't push you into superman status. You're already doing that yourself every time you choose to hit the gym or pick nourishing, nutrient-rich foods. Go you!
Kreider, R. B., Fry, A.C., & O'Toole, M.L. (1998). Overtraining in Sport. Champaign, IL: Human Kinetics.
National Sleep Foundation's sleep time duration recommendations: methodology and results Hirshkowitz, Max et al. Sleep Health: Journal of the National Sleep Foundation, Volume 1, Issue 1 , 40 – 43.
Xiao Q, Keadle SK, Hollenbeck AR, Matthews CE. (2014). "Sleep duration and total and cause-specific mortality in a large US cohort: interrelationships with physical activity, sedentary behavior, and body mass index." American Journal of Epidemiology, 2014 Nov 15;180(10):997-1006. doi: 10.1093/aje/kwu222.
Lutsey PL, Misialek JR, Mosley TH, Gottesman RF, Punjabi NM, Shahar E, MacLehose R, Ogilvie RP, Knopman D, Alans A. (2018). "Sleep characteristics and risk of dementia and Alzheimer's disease: The Atherosclerosis Risk in Communities Study." Alzheimer's & Dementia, 18 Feb;14(2):157-166. doi: 10.1016/j.jalz.2017.06.2269.
Sacks, F. M. et al. (2014) "Effects of high vs low glycemic index of dietary carbohydrate on cardiovascular disease risk factors and insulin sensitivity: the OmniCarb randomized clinical trial." JAMA 312, 2531–2541.


text:
Revealed: Average salary of Everton FC players and how they compare to other … – Liverpool Echo
Everton FC players earn an average of £1.63m per year or £31,447 per week according to a new report undertaken by website Sporting Intelligence.
The Blues remain in the top 100 and are the 90th biggest wage payers in sport, two places below Premier League strugglers Newcastle.
Aston Villa (80th) and Tottenham (30th) also sit above Everton in the list, paying their players £34,815 and £54,077 respectively.
Rivals Liverpool pay their players more than double that what Everton pay theirs, with players earning per week £67,486 on average.
Manchester United in sixth (£89,988 per week/£4.68m per year), Chelsea in eighth (£83,713 per week/£4.35m per year) and Arsenal in 10th (£77,963 per week/£4.05m per year) are on average the top four payers in the Premier League
Sunderland (94th) and Swansea (95th) also make the top 100 with their average wages being £31,078 and £31,015.
In his introduction to the report, Nick Harris from Sporting Intelligence said: "The GSSS looks solely at earnings for playing sport, not for endorsements or other extra-curricular activities. The 2015 report considers 333 teams in 17 leagues across 13 countries in seven sports: football, baseball, basketball, gridiron, cricket, ice hockey and Australian Rules football.
"The NBA is the highest-paying league as a whole, with 448 players at 30 teams in the 2014-15 season earning an average of £2.67m per year each, or $4.58m each on average.
"The Premier League is the best paying football league in the world, with average annual pay at £2.23m per player."
For up-to-the-minute transfer rumours and football stories subscribe to the RSS feed.
For the rest of today we have a one-off special offer on new Everton photos (autographed).
Posted by admin at 4:06 pm
Everton FC transfer rumours: Blues in pole position for Jonny Evans, Oviedo … – Liverpool Echo Live: Everton FC end of season awards – Liverpool Echo

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: How can I assign session id of php in a jQuery file? I have a main file
index.php

in which I include four other files like
header_get_started.php,
content_main.php,
right_sec_home.php,
footer.php.

Here is my code
"index.php"

<?php include_once('header_getstarted.php'); ?>
<?php include_once('content_main.php'); ?>
<?php include_once('right_sect_home.php'); ?>
<?php include_once('footer.php'); ?>

"header_getstarted.php"

       <?php
         $sqlCmd = "some query";
         echo combo('cmb_command','custom-class1 custom-class2','cmd_name','cmd_id','0',$sqlCmd,'sendpostmtu()' $width="style='width:250px;cursor:pointer;'")
       ?>

       <input type="submit" name="" class="sub_bg" value="" onclick="get_request_by_command($('#cmb_command').val());">

In header_get_started.php
When select any command from select box, I want to assign it's id to $_SESSION['id'].
Onclick of selection box I have an ajax request which can refresh the main content (content_main.php). Also I want that selected id in another page in right_sec_home.php to refresh it's content.
How can I assign the php session id in JS file for 2nd page?
My JS file is
function  get_request_by_command (commandId) {
          var cmdTitle=$('#cmb_command :selected').text();

          if(cmdTitle=="") {
              cmdTitle='ABC';
          }

          $("#main_wrap").hide();

          if(cmdTitle=='--Select--'){
              $("#cmdTitle").html("");
          }else{
              $("#cmdTitle").html(cmdTitle);
          }

          $("#cmbs_cmd").val(commandId);

          document.getElementById('request_list').innerHTML='';

          var strURL="request_by_command_tbl.php?id="+commandId;
          var req = getXMLHTTP();

          if (req) {
            req.onreadystatechange = function() {
              if (req.readyState == 4) {
                // only if "OK"
                if (req.status == 200) {
                    var ret = req.responseText;
                    $("#requestTitle").html(""+cmdTitle+");
                    $('#request_list').html('');
                    $('#request_list').html(ret);
                    $('#main').show();
                    $("#searchDiv").show();
                } else {
                    alert("There was a problem while using XMLHTTP:\n" + req.statusText);
                }
             }
           }
           req.open("GET", strURL, true);
           req.send(null);
         }
}

With using this jQuery function, I have created an ajax request, and send the id to the "request_by_command_tbl.php" file.
In "request_by_command_tbl.php" file i assigned,
$_SESSION['id'] = $_REQUEST['id'];

Also I want this $_SESSION['id'] in right_sec_home.php at same instant.
So is their any way to assign php $_SESSION['id'] in the jQuery script file before sending ajax request.
My other files
"content_main.php"

Right section home file in which i need session id
    "right_sec_home.php"

<?php

function getRequestByMonth($month, $year, $id){

        $que ="SELECT distinct(MS.date) FROM commands AS MC , ranks AS MR ,steady_tours AS MST, preferred_tours AS MPT, registration AS MMS where date_format(rm_date, '%c-%Y') = '".$month."-".$year."'
    AND    MMS.cmd_id ='".$_SESSION['id']."'
        order by MMS.date";
        $res = mysql_query($que);
    while($rows[]=mysql_fetch_array($res)){}
        foreach ($rows as $res):
        if($res['date'] == "") continue;
        $dt = date("Y-n-j", strtotime($res['date']));
        $return[getDateB($dt)] = $res['date'];
        endforeach;
        return $return;
}

I hope that this is clear enough.
Any ideas?
Please help.

A: there is no way for you to access the session information with jquery ..
explanation
sessions are files stored on the server --> where is java script is only a client side language ..
there is always a work around .. but i guess you should explain more about what exactly you want to achieve

A: <?php $sessionId = session_id(); ?>

<input id="session_id" name="session_id" type="hidden" value="<?php echo $sessionId; ?>" />

Get the value of the hidden field using jquery.

A: You can use some hidden fields or script variables inside the php file.
example :

varible id can be accessible using the javascript or jquery

redpajama_set_name: RedPajamaStackExchange


THERE'S good news and bad news for Tyrone on the injury front with Kieran McGeary set to be available for their first round Qualifier with Meath in Navan next weekend, but Declan McClure is the latest doubt.
Midfielder McClure suffered a shoulder injury while playing for Clonoe against Errigal Ciaran on Monday evening and was forced out of the action midway through the second half of the Division One league game.
On a brighter note from the same game, McClure's Clonoe team-mate Connor McAliskey, Tyrone's best player as they bowed out of the provincial title race, maintained his impressive form by hitting six points in a 2-12 to 1-10 defeat.
McGeary, who captained the Red Hands to All-Ireland U21 glory in 2015, didn't finish training last Thursday night after sustaining a knock to the knee which was thought to be serious.
He had an MRI scan over the weekend and didn't play for his club Pomeroy on Sunday.
The results on Monday, though, have given him the all-clear and he was due to resume training last night.
He featured in all seven games of Tyrone's Allianz Football League Division One campaign and came on in the 56th minute for the injured Lee Brennan in their Ulster SFC quarter-final defeat by Monaghan.
With Brennan and Mark Bradley both ruled out through injury and Peter Harte suspended, McGeary could well get the nod for a starting berth against the Royals.
Brennan, who was the NFL Division One top scorer this season, is out for eight weeks with a hamstring injury and will miss the entire Qualifier series, while a damaged ankle will keep Bradley on the sidelines for five weeks.
Colm Cavanagh and Tiernan McCann have not yet regained full fitness after returning from quad muscle and kneecap injuries respectively, and both were unable to complete against Monaghan.
It is understood, though, that both Cavanagh and McCann played full games for their clubs at the weekend and came through unscathed.
Harte will play no part at Pairc Tailteann because of suspension. He received a straight red card in the dying moments of the Healy Park clash.
There is no throw-in time yet confirmed for Tyrone's trip to Meath, but it will take place on June 9.


text:
\section{Experimental Setup and Evaluations}\label{sec:experiments}
This section presents the results of evaluating the proposed reduced empirical distribution-based comparison operator as well as the techniques from~\cite{KMGT15}, \cite{KBT18}, and~\cite{MAG12} in the context of multi-objective optimization with uncertain objectives.
All comparison operators and the proposed UDTLZ benchmark have been integrated into the open-source Opt4J optimization framework~\cite{opt4jpaper} and can be paired with different optimization techniques.
The following experiments use the well-known \emph{Nondominated Sorting Genetic Algorithm II (NSGA-II)}~\cite{DAPM00} to optimize instances of UDTLZ as well as UZDT~\cite{KBT18} benchmarks.
NSGA-II is configured with a population size of $\lambda=25$ and performs optimization runs for 400 generations in each case.
The UZDT benchmark extends the well-known ZDT benchmark suite~\cite{ZDT00} and incorporates various uncertainties in the specification of all six bi-objective ZDT problems.
Also, the proposed UDTLZ problems are configured with $n=7$ decision variables and $m=3$ objective functions.

A sample size $N = 100$ is used for all instances of the UZDT and UDTLZ problems and for all comparison operators.
This requires 100 iterative evaluations of each objective function of each candidate solution, and allows a good balance between the comparison accuracy and execution time.
The proposed comparison operator uses a maximum of 20 steps in the approximated \acp{CDF}, while the histogram-based approach partitions each uncertainty distribution into intervals with a width of $\omega = 0.01$.
Both of these comparison operators are evaluated for a comparison threshold $\gamma = 0.7$ as suggested in~\cite{KBT18}.
Moreover, the three-stage approach is configured to use mean value in the average criterion and the 95\% quantile intervals in the spread criterion, as suggested in~\cite{KMGT15}.
Also, the threshold values $0.1$ and $0.3$ are used for these criteria, respectively.

\subsection{Convergence and Diversity}
Figure~\ref{fig:pareto_one} depicts the Pareto front approximations for the UZDT1 and UZDT3 test problems~\cite{KBT18} obtained by NSGA-II incorporating the proposed $C_{\it reduce}$, the histogram-based $\AHistogram{0.01}$~\cite{KBT18}, the three-stage $C_{\it ThreeStage}$~\cite{KMGT15} and the mean-based $C_{\it Mean}$~\cite{MAG12} comparison operators.
\begin{figure*}
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTone{reduced}{8}{$C_{\it reduce}$}
    \end{subfigure}\hspace*{-.0\textwidth}
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTone{hist}{1}{$\AHistogram{0.01}$}
    \end{subfigure}\\[-1mm]
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTone{threestage}{7}{$C_{\it ThreeStage}$}
    \end{subfigure}\hspace*{-.0\textwidth}
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTone{naive}{1}{$C_{\it Mean}$}
    \end{subfigure}\\[-1mm]
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTthree{reduced}{8}{$C_{\it reduce}$}
    \end{subfigure}\hspace*{-.0\textwidth}
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTthree{hist}{6}{$\AHistogram{0.01}$}
    \end{subfigure}\\[-1mm]
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTthree{threestage}{4}{$C_{\it ThreeStage}$}
    \end{subfigure}\hspace*{-.0\textwidth}
    \begin{subfigure}[]{0.52\textwidth}
    \frontsUZDTthree{naive}{6}{$C_{\it Mean}$}
    \end{subfigure}
    \caption{Pareto front approximations for the UZDT1 and UZDT3 problems obtained after 200 and 400 generations of NSGA-II incorporating different comparison operators. Each case is run for 10 times and the median of the results is plotted. Shown are the mean values of the uncertain objectives $f_1$ and $f_2$.}
  \label{fig:pareto_one}
\end{figure*}
Shown are the reference Pareto front and two approximated fronts achieved after 200 and 400 generations considering only the mean value of the uncertain objectives $f_1$ and $f_2$.
To obtain reliable results, each optimization run is repeated 10 times, the results are sorted with respect to the well-known $\epsilon$-dominance criterion~\cite{ZTLFD03}, and the run which provides the median $\epsilon$-dominance is chosen.
The results show that, the first three comparison operators outperform the mean-based approach in most of the cases.
However, the proposed comparison operator performs slightly better in converging to the Pareto front, while covering diverse regions in the objective space.

Figure~\ref{fig:dom_div} depicts the $\epsilon$-dominance~\cite{ZTLFD03} and \ac{DCI}~\cite{LYL14} calculated from the mean values of each uncertain objective for various instances of UZDT and UDTLZ problems.
Each result is calculated as the median of 10 optimization runs, calculated at each generation of NSGA-II.
$\epsilon$-dominance is calculated using the approach in~\cite{LGHT08} where the value of $\epsilon$ specifies how much the quality of candidate solutions found in an optimization run should be scaled so that these solutions weakly dominate the true Pareto front.
If the true Pareto front cannot be accurately described, which is the case for the UZDT and UDTLZ problems, it is replaced by a reference set of solutions.
This reference set contains all solutions that turn out to be non-dominated after combining the approximated Pareto fronts of all runs of every optimization technique in the experiments.

\begin{figure*}
    \begin{center}
    \addlegends
    \vspace{0.3\baselineskip}

    \begin{subfigure}{0.42\textwidth}
    \dominanceDiversityUZDTone
    \caption{\label{fig:udtlz2}UZDT1}
    \end{subfigure}\qquad
    \begin{subfigure}{0.42\textwidth}
    \dominanceDiversityUZDTthree
    \caption{\label{fig:udtlz3}UZDT3}
    \end{subfigure}\\[0.2\baselineskip]
    \begin{subfigure}{0.42\textwidth}
    \dominanceDiversityUDTLZtwo
    \caption{\label{fig:udtlz5}UDTLZ2}
    \end{subfigure}\qquad
    \begin{subfigure}{0.42\textwidth}
    \dominanceDiversityUDTLZseven
    \caption{\label{fig:udtlz6}UDTLZ6}
    \end{subfigure}
    \end{center}
    \vspace*{-12pt}
    \caption{Resulting $\epsilon$-dominance and \ac{DCI} for the UZDT1, UZDT3, UDTLZ2 and UDTLZ6 problems. The results are obtained using optimizations that incorporate different comparison operators. For a reliable interpretation of the results, each case is repeated 10 times and at each generation, the median $\epsilon$-dominance and diversity values of all 10 runs are calculated.}
  \label{fig:dom_div}
  \vspace*{-10pt}
\end{figure*}

To calculate \ac{DCI}, the objective space is partitioned into $m$-dimensional hypercubes of identical volume.
This allows to define a new coordinate system where solutions that lie within the same hypercube are treated as equal.
Given the coordinates of two sets of hypercubes that enclose the reference and an approximated Pareto front,
the work in~\cite{LYL14} first calculates the sum of the Euclidean distances between each hypercube of the approximated front and the nearest hypercube of the reference front.
Then it normalizes the result to a scale between 0 and 1 that respectively indicate the minimum and maximum possible degrees of diversity.

Figure~\ref{fig:dom_div} shows the development of $\epsilon$-dominance and \ac{DCI} through the optimization, which indicates that the proposed comparison operator allows for a fast convergence to the Pareto front.
The fluctuations in the results are due to the fact that the solutions are evolved with the help of elitism and crowding distance approaches~\cite{DAPM00} which do not differentiate candidate solutions in similar ways as in the post-optimization algorithms used to calculate $\epsilon$-dominance and \ac{DCI}.
Moreover, while the former approaches employ uncertainty-aware comparison operators which consider various characteristics for the uncertainty distribution of objective values, the latter algorithms always make decisions based on the mean values only.
It should be noted that the intensity of these fluctuations also depend on the extent of uncertainty in the objective functions such that for larger uncertainties, objective values of many candidate solutions would overlap.
This may affect the dominance relation between these candidate solutions and give rise to possible disagreements among the aforementioned approaches.

\subsection{Robustness}
In the following, the proposed comparison operator $C_{\it reduce}$ is compared with the techniques in~\cite{KBT18},~\cite{KMGT15} and~\cite{MAG12} in terms of robustness of the found solutions for the UZDT and UDTLZ problems.
The robustness is measured in this paper as the diagonal distance between worst- and best-case objective values such that a solution is said to be more robust if it delivers a smaller value for diagonal distance.
Figure~\ref{fig:robustness} shows the results obtained for all evaluated comparison operators and all UZDT and UDTLZ problem instances.
In each case, the diagonal distance is calculated for all solutions in the Pareto front approximations of 10 optimization runs, and averaged.

\begin{figure*}[t]
  \hspace*{-8pt}\distances
  \caption{Average diagonal distance of the solutions found for all UZDT and UDTLZ problems using optimizations that incorporate different comparison operators. Each case is averaged over the results of 10 independent optimization runs.}
    \vspace*{-3pt}
  \label{fig:robustness}
\end{figure*}

The results in Figure~\ref{fig:robustness} show that none of the comparison operators performs considerably better in finding robust solutions.
This can be justified by the fact that the proposed, the histogram-based and the mean-based comparison operators do not consider the deviation of uncertain objective values as a comparison criterion.
The three-stage comparison operator takes this deviation into account in its third stage.
However, we have observed that for all tested problems, in at least 94\% of cases, objective values can be distinguished in the first two stages, \mbox{i.\,e.,}\xspace the deviation criterion is used in less than 4\% of all comparisons.
Also, in more than 92\% of the extended comparisons, the compared uncertain objective values are considered equal as their deviations are not noticeably different.
This indicates that although a deviation criterion is employed, it is not likely to find solutions with significantly smaller uncertainties in their objective values.
Note that the diagonal distance does not reflect the accuracy of the comparison operators in distinguishing differently distributed uncertain objective values.

\section{Introduction}\label{sec:introduction}
Real-world problems typically demand solutions that are optimized with respect to multiple criteria called objectives.
In these so-called \emph{multi-objective optimization problems}, the objectives often conflict with each other such that no single solution can be found to be optimal in all objectives.
Instead, one usually searches for a set of non-dominated solutions known as \emph{Pareto front} or \emph{Pareto set} that provide decent trade-offs among objectives.
A solution is said to \emph{dominate} another if it is as good as the other in all objectives and is better with respect to at least one objective.
While exact optimization methods such as \emph{integer linear programming} may not be applicable to complex optimization problems,
population-based \emph{meta-heuristics} such as \emph{evolutionary algorithms} enable a fast approximation to the Pareto front of problems with several objectives and large search spaces~\cite{JBT09}.

However, multi-objective optimization problems are often characterized by wide ranges of uncertainties including noise, approximation errors or time-dependent variation in their objective functions and perturbations in their decision variables~\cite{JB05}.
Any optimization algorithm that neglects the effects of uncertainty might prefer actually inferior solutions while traversing the search space.
As a remedy, \emph{robust optimization} techniques have been proposed to enable an accurate comparison of objectives in the presence of uncertainty.
Existing techniques typically model uncertain objectives using instances of uniform~\cite{Teich01} and Gaussian~\cite{Hughes01,FE05} distributions, intervals specified by best and worst cases~\cite{Limbourg05, EM13}, or sampled data~\cite{MAG12,KMGT15}.
While the first two groups fail to deal with various, and possibly non-standard uncertainty distributions, the third group of techniques enable the comparison of candidate solutions with arbitrarily distributed uncertain objectives.
However, these techniques rely on estimated statistics such as mean value and variance and do not take the actual uncertainty distributions into account.
This problem is addressed in a previous work of the authors~\cite{KBT18}, which enables calculating the probability that an uncertain objective of one solution is greater or smaller than that of another solution, for any arbitrary distribution given as a closed-form function or sampled data.
This probability is calculated through partitioning the probability distribution of objectives into small intervals of the same size and applying rectangle integration, \mbox{i.\,e.,}\xspace Riemann sum, while assuming a uniform distribution within each interval.
Therefore, the optimization algorithm can differentiate instances of each uncertain objectives of two candidate solutions under comparison, and determine whether one solution dominates the other or not.

This paper extends the comparison operator in~\cite{KBT18} by introducing a new method for the calculation of the probability that an instance of uncertain objective is greater than another instance of the same objective.
This method is based on obtaining the \ac{CDF} of each uncertain objective, and partitioning the probability space into intervals of the same size.
It uses an iterative approach similar to~\cite{KBT18}, except that its accuracy is not impaired by the assumption of uniform distribution within intervals.

Moreover, this paper extends the well-known DTLZ multi-objective benchmark suite~\cite{DTLZ02} to consider the effects of various uncertainties.
It integrates the proposed comparison operator and the extended benchmark into the multi-objective optimization framework Opt4J~\cite{opt4jpaper} that incorporates several optimization algorithms such as evolutionary algorithms~\cite{DAPM00} and particle swarm optimization~\cite{SC06}.
Experiments show that compared to the existing techniques, the proposed approach enables comparing uncertain objectives more efficiently and achieves higher optimization quality.

The rest of this paper is structured as follows:
Section~\ref{sec:relatedwork} reviews the state-of-the-art techniques for robust multi-objective optimization.
Sections~\ref{sec:proposed} and~\ref{sec:benchmark} respectively introduce the proposed comparison operator and uncertain multi-objective optimization benchmark.
Section~\ref{sec:experiments} presents the experimental setup and evaluation results, and in the end, Section~\ref{sec:conclusion} concludes this work.

\section{Proposed Robust Multi-Objective Optimization}\label{sec:proposed}
A multi-objective optimization problem includes a vector of $n$ decision variables $x = (x_1, x_2,$ $\dots, x_n)$ and a vector of $m$ objective functions $f(x) = \big(f_1(x), f_2(x), \dots, f_m(x)\big)$.
The former describes a feasible solution in the constrained search space of the problem, and the latter evaluates this solution with respect to different objectives, \mbox{i.\,e.,}\xspace quality metrics, that are to be maximized or minimized.
Finding a solution that is optimal in all objectives is often impossible due to the conflict between different objectives.
Therefore, multi-objective optimization algorithms typically search for a set of non-dominated solutions that offer decent tradeoffs for the conflicting objectives.
A solution $A$ dominates another solution $B$, \mbox{i.\,e.,}\xspace $A \succ B$, if and only if $A$ is as good as $B$ for all objectives and there is at least one objective for which $A$ is better than $B$.
In a maximization problem, a multi-objective dominance criterion can be defined as follows:
\begin{align}
  A \succ B \iff \mathop{\forall}_{i=1}^{m} f_i(A) \geq f_i(B) \wedge \mathop{\exists}_{j=1}^{m} f_j(A) > f_j(B) \label{eq:uncertainty_optimization_dominance}
\end{align}
where $\wedge$ denotes the logical AND operation.
In the presence of uncertainty, the standard comparison operators cannot properly distinguish objective values.
Therefore, they should be substituted by operators that incorporate the existing uncertainty into the comparison.
This section investigates various comparison operators that are based on \emph{probabilistic dominance} between instances of uncertain objectives, with a special focus on two novel operators.

Probabilistic dominance is originally defined to describe the probability that a solution $A$ dominates another solution $B$, see~\cite{Teich01} and~\cite{Hughes01}.
This probability is calculated as the product of all probabilities that an objective from $A$ is more favorable than the same objective from $B$.
For a maximization problem, this probability can be calculated as follows:
\begin{align}
  \Pr(A \succ B) = \prod_{i=1}^{m} \Pr\big(f_i(A) > f_i(B)\big)\enspace. \label{eq:probabilistic_dominance}
\end{align}
A threshold value can be used to determine if the resulting probability is significant enough to assume $A$ dominates $B$.
The main limitation of this approach is that Equation~\eqref{eq:probabilistic_dominance} can only be applied if all $m$ objective functions are statistically independent, which is most often not true because objective functions usually have common decision variables in their inputs.
To overcome this limitation, we proposed in~\cite{KBT18} to calculate the probability $\Pr\big(f_i(A) > f_i(B)\big)$ for each objective $f_i$ separately, to distinguish $f_i(A)$ and $f_i(B)$ using a threshold value, and then to determine dominance between $A$ and $B$ according to Equation~\eqref{eq:uncertainty_optimization_dominance}.

For arbitrary distributions of $f(A)$ and $f(B)$, as closed-form \acp{PDF} or sample data, the probability that $f(A)$ is greater than $f(B)$ can be calculated as follows\footnote{Since the approach is applied to each objective function $f_i(\cdot)$ separately, from here on this notation is simply written as $f(\cdot).$} (see also~\cite{SI09}):
\begin{align}
\Pr\big(f(A) > f(B)\big)=
\int_{\underline{f(A)}}^{\overline{f(A)}}\Pr\big(f(A)=a\big)\Pr\big(f(B)<a\big)\mathrm{d}a\enspace. \label{eq:probabilistic_dominance_i}
\end{align}
where $\underline{f(A)}$ and $\overline{f(A)}$ denote the lower and upper bounds on $f(A)$, respectively.
While $\Pr\big(f(A)=a\big)$ is obtained from the \ac{PDF} of $f(A)$, the second probability $\Pr\big(f(B)<a\big)$ can be calculated as follows:
\begin{equation}
  \Pr\big(f(B)<a\big)=\int_{\underline{f(B)}}^{a}\Pr(f(B)=b)\mathrm{d}b\enspace. \label{eq:probabilistic_dominance_ii}
\end{equation}
Figure~\ref{fig:example_distribution} shows an example of \acp{PDF} for $f(A)$ and $f(B)$.
The filled area under the curve of $f(B)$ in this figure amounts to the probability $\Pr\big(f(B)<a\big)$ for a given value of $a$.
The exact calculation of these integrals is tedious if the \acp{PDF} of $f(A)$ and $f(B)$ are not available or if they do not follow instances of Uniform and Gaussian distributions.
The following subsections first describe various approaches for the estimation of $\Pr\big(f(A) > f(B)\big)$ as well as other robust comparison operators.
Then, these are evaluated with respect to estimation error and execution time.

\begin{figure}[t]
  \centering
  \input{figures/example_distribution.tikz}
  \caption{Example of \acp{PDF} of one uncertain objective of two candidate solutions $A$ and $B$ under comparison.}
  \vspace*{-10pt}
  \label{fig:example_distribution}
\end{figure}

\subsection{Robust Comparison Operators}\label{sec:prob_dom_operators}
This subsection describes various comparison operators reviewed in Section~\ref{sec:relatedwork} along with two novel approaches to enable distinguishing instances of an uncertain objective.
These operators require the distribution of uncertain objective values $f(A)$ and $f(B)$ given as \acp{PDF} or sample data.
If the \acp{PDF} are known, different statistics of the distributions can be obtained, including the expected values ($\E{f(\cdot)}$), variances ($\Var{f(\cdot)}$), standard deviations
($\sigma[f(\cdot)]$), $p$-th quantiles ($\q{p}{f(\cdot)}$) and values of the \acp{CDF} ($\CDF{a}{f(\cdot)}=\Pr\big(f(\cdot) \leq a\big)$).
If the \ac{PDF} of an uncertain objective value is not available, it can be represented by a population of samples where each sample is an outcome of evaluating the corresponding objective function.
Given a sequence $S$ of $N$ independent samples $(s_1, s_2, \ldots, s_N)$, the sample mean and the (unbiased) sample variance and standard deviation can be estimated as follows:
\begin{equation}
    \EEst{S}=\frac{1}{N}\sum_{i=1}^{N}s_i\enspace,
    \label{eq:sample_mean}
\end{equation}
\begin{equation}
    \VarEst{S}=\frac{1}{N-1}\sum_{i=1}^{N}\big(s_i-\EEst{S}\big)^2\enspace,
\end{equation}
\begin{equation}
    \sigmaEst{S}=\sqrt{\VarEst{S}}\enspace.
\end{equation}
Also, the $p$-th quantile of a population can be calculated by the inverse empirical distribution function that traverses the population in the ascending order of samples and returns the very first sample after the $p\cdot N$ smallest
samples.
More details on the aforementioned statistics can be found in \cite{durrett2010probability}.

\subsubsection{Pairwise Comparison ($C_{\it pw}$)}
The most straightforward approach to estimate the probability $\Pr\big(f(A) > f(B)\big)$
is to generate $N$ samples $(a_1,a_2,\ldots,a_N)$ from the distribution of $f(A)$ and
$(b_1,b_2,\ldots,b_N)$ according to the distribution of $f(B)$, and to calculate the proportion of pairs
$(a_i,b_i)$ where $a_i>b_i$:
\begin{equation}
  \Pr\big(f(A)>f(B)\big)\approx\frac{1}{\,N\,}\,\big\vert\{i\mid a_i>b_i\}\big\vert\enspace. \label{eq:pairwise}
\end{equation}
The absolute approximation error of this comparison operator can be determined as follows:
\begin{equation}
  e_{\it pw} = \Big\vert \Pr\big(f(A)>f(B)\big) - \frac{1}{\,N\,}\,\big\vert\{i\mid a_i>b_i\}\big\vert\Big\vert\enspace. \label{eq:pairwise_error}
\end{equation}
The probability that this error is larger than a constant tolerance $0<\delta <1$ tends exponentially to zero for large values of $N$ according to Chernoff bounds:
\begin{equation}
  \Pr\left(e_{\it pw}>\delta\right) \leq \mathrm{e}^{-c\cdot\delta^2\cdot N}\enspace,
  \label{eq:pairwiseError}
\end{equation}
where $c$ is a positive constant.

The time complexity of the comparison is of the order $O(N)$.
Note that if the evaluation of objective functions is time-consuming, it is better to maintain a population of samples for each objective of each candidate solution throughout the optimization.
Otherwise, new samples can be generated whenever a comparison takes place.

\subsubsection{Uniform Approximation ($C_{\it uni}$)~\cite{Teich01}}
Another approach to estimate $\Pr\big(f(A) > f(B)\big)$ is based on the assumption that $f(A)$ and $f(B)$ can be approximated by uniform distributions $\mathcal{U}(\underline{f(A)}, \overline{f(A)})$ and $\mathcal{U}(\underline{f(B)}, \overline{f(B)})$, respectively.
This approach is denoted by $C_{\it uni}^1$.
Assuming uniform distributions, $\Pr\big(f(A) > f(B)\big)$ can be calculated as follows according to the law of total probability\footnote{For cases $\overline{f(A)}\leq \underline{f(B)}$ and $\overline{f(B)}\leq \underline{f(A)}$ there is no need for applying Equation~\eqref{eq:dominanceUniform} as the probability $\Pr\big(f(A)>f(B)\big)$ amounts to 0 and 1, respectively.}:
\begin{align}
\Pr\big(f(A)>f(B)\big) =\Pr\big(f(A)>f(B) \wedge f(A) \leq \overline{f(B)}\,\big)
+ \Pr\big(f(A)>\overline{f(B)}\,\big)\enspace,\label{eq:dominanceUniform}
\end{align}
where
\begin{align}
\Pr\big(f(A)>f(B) \wedge f(A) \leq \overline{f(B)}\,\big) &=
\frac{\Big(\max\{\underline{f(A)},\underline{f(B)}\}\Big)^2-\Big(\min\{\overline{f(A)},\overline{f(B)}\}\Big)^2}{2\,\Big(\overline{f(A)} - \underline{f(A)}\Big)\,\Big(\overline{f(B)} - \underline{f(B)}\Big)}  \nonumber \\
&+\frac{\Big(\max\{\underline{f(A)},\underline{f(B)}\}-\min\{\overline{f(A)},\overline{f(B)}\}\Big)\,\underline{f(B)}}{\Big(\overline{f(A)} - \underline{f(A)}\Big)\,\Big(\overline{f(B)} - \underline{f(B)}\Big)} \enspace,\label{eq:dominanceUniformII}
\end{align}
and
\begin{align}
\Pr\big(f(A)>\overline{f(B)}\,\big)=
\frac{\overline{f(B)}-\min\{\overline{f(A)},\overline{f(B)}\}}{\overline{f(A)} - \underline{f(A)}}\enspace.
\end{align}

This comparison operator requires the distribution of an uncertain objective value to have finite bounds.
Alternatively, these lower and upper bounds can be calculated as $\E{f(\cdot)}-\sqrt{3\Var{f(\cdot)}}$ and $\E{f(\cdot)}+\sqrt{3\Var{f(\cdot)}}$, respectively, as the variance of a uniformly distributed
random variable $f(\cdot)\sim\mathcal{U}\big(\,\underline{f(\cdot)},\overline{f(\cdot)}\,\big)$ equals $\big(\,\overline{f(\cdot)}-\underline{f(\cdot)}\,\big)^2/12$.
This approach is denoted as $C_{\it uni}^2$.
Note that treating an arbitrary distribution as a uniform distribution can impose a significant estimation error in the calculation of $\Pr\big(f(A)>f(B)\big)$, but if the distributions are actually uniform then both versions accurately represent the given distributions.
If the lower and upper bounds or alternatively the expectation and variance have to be derived from a population of samples, the time needed for sample generation, \mbox{i.\,e.,}\xspace objective function evaluation, which has the order $O(N)$ can increase the execution time substantially.
Otherwise, the execution time is solely spent for performing the comparison which is of order $O(1)$.

\subsubsection{Gaussian Approximation ($C_{\it{gauss}}$)~\cite{Hughes01}}
This approach assumes that the uncertain objective values $f(A)$ and $f(B)$ follow instances of Gaussian distributions, denoted as
$\Gauss{\E{f(A)}}{\Var{f(A)}}$ and $\Gauss{\E{f(B)}}{\Var{f(B)}}$, respectively.
Thus, the probability of $f(A)$ being greater than $f(B)$ can be estimated as follows:
\begin{align}
    \Pr\big(f(A)>f(B)\big)&=\Pr(f(B)-f(A)<0) \nonumber \\
    &=\frac{1}{2}\left( 1+\erf\left( \frac{\E{f(A)}-\E{f(B)}}{\sqrt{2\,(\Var{f(A)}+\Var{f(B)}\,)}} \right) \right)\,.\label{eq:dominanceGauss}
\end{align}
Here, $\big(f(B)-f(A)\big)\sim\mathcal{N}\big(\E{f(A)}-\E{f(B)}, \Var{f(A)}$ $+\Var{f(B)}\big)$, and $\erf$ is the error function
\begin{align}
\erf(x)=\frac{2}{\sqrt{\pi}}\int_{0}^{x}\mathrm{e}^{-t^2}\mathrm{d}t\enspace.\label{eq:erf_gauss}
\end{align}
Different approximations of the error function can be used in this comparison operator.
Some useful and fast approximations can be found in~\cite{AS64}.

If the distribution of an uncertain objective value is not available, its expected value and variance can be derived by generating a population of samples and applying the respective estimators to this population.
This increases the time complexity from $O(1)$ to $O(N)$.
Note that this comparison operator may be subject to a noticeable approximation error since the actual distributions might be non-Gaussian.

\subsubsection{Histogram Approximation ($\AHistogram{\omega}$)~\cite{KBT18}}
This approach is based on representing uncertain objective values using histogram-based distributions.
Such a distribution partitions the actual \ac{PDF} into intervals and considers a uniform distribution within each interval.
A column is considered in each interval such that the area covered by the column equals the probability of the actual distribution within the corresponding interval.
Given a uniform distribution within each interval, histogram-based distributions are linear combinations --- or more precisely affine combinations --- of uniform distributions.
Therefore, $\Pr\big(f(A)>f(B)\big)$ can be calculated by an affine combination of Equation~\eqref{eq:dominanceUniform}.
To reduce computational overhead, we proposed in~\cite{KBT18} to use histograms with a fixed interval width $\omega$.
Also, the columns are aligned to the intervals $I_{\omega,k}:=[k\cdot \omega,(k+1)\cdot \omega[$ for any integer $k$ such that the columns of different histograms either perfectly overlap or are disjoint.

Given histogram-based distributions of uncertain objective values $f(A)$ and $f(B)$,
\begin{align}
    \Pr\big(f(A)>f(B)\big) =
    \sum_{k}\bigg(&\frac{1}{2}
      \Pr\big(f(A)\in I_{\omega,k}\big)\cdot \Pr\big(f(B)\in I_{\omega,k}\big) \nonumber \\
    &+\Pr\big(f(A)\in I_{\omega,k}\big)\cdot\Pr\big(f(B)<k\cdot \omega\big)\bigg)\enspace ,
  \label{eq:dominanceHistogram}
\end{align}
where
\begin{align}
\Big\lfloor \underline{f(A)}/\omega\Big\rfloor \leq k \leq \Big\lceil \overline{f(A)}/\omega\Big\rceil\enspace .
\label{eq:histogram_column_index_range}
\end{align}
The values for $\Pr\big(f(\cdot)\in I_{\omega,k}\big)$ can be accurately evaluated if the actual \ac{PDF} is available or they can be estimated as the proportion of samples that lie within $I_{\omega,k}$.
Therefore, a histogram-based distribution can be constructed by calculating $\big\lceil\overline{f(\cdot)}/\omega\big\rceil - \big\lceil\underline{f(\cdot)}/\omega\big\rceil + 1$ probabilities, one for each interval, or by generating $N$ samples and calculating this proportion for each interval.
Note that the number of intervals can be different when sampling is used, especially for small values of $N$, because not all potential intervals might be filled with one or more samples.

The probabilities $\Pr\big(f(\cdot)\in I_{\omega,k}\big)$ and $\Pr\big(f(\cdot)<\omega\cdot k\big)$ can be pre-calculated without increasing the time complexity of histogram preparation.
The calculation of $\Pr\big(f(A)>f(B)\big)$ according to the Equation~\eqref{eq:dominanceHistogram} has then a time complexity of $O\big(\big\lceil( \overline{f(A)}-\underline{f(A)})/\omega\big\rceil\big)$.
The error of calculating this probability can be bounded by the sum
\begin{align}
  \frac{1}{2}\cdot\sum_{k}
  \Pr\left(f(A)\in I_{\omega,k}\right)\cdot \Pr\left(f(B)\in I_{\omega,k}\right)\enspace,
\end{align}
where $k$ ranges according to Equation~\eqref{eq:histogram_column_index_range}.
This error is due to the loss of information on the exact distribution within the intervals.
Note that if a column from the histogram of $f(A)$ perfectly overlaps with a column from the histogram of $f(B)$,
then the values in each of the columns are assumed to be greater than the values from the other column half the time --- resulting in a probability of $0.5$.
While depending on the distribution of uncertain objective values within the shared interval, one distribution can always offer greater values which yields a probability of $1$.
This can be the case especially when $\omega$ is sufficiently large.
Furthermore, if the probabilities in Equation~\eqref{eq:dominanceHistogram} are estimated by samples, the comparison is subject to an additional error that is due to the difference between the proportion of samples within an interval and the actual probability to have a value within that interval.
The probability that this additional error is equal to or greater than $\delta$ is bounded by $e^{-c\cdot\delta^2\cdot N}$ for large values of $N$ and $0<\delta<1$, where $c$ is a positive constant and $N$ is the number of used samples.
This result can be obtained by application of the Dvoretzky-Kiefer-Wolfowitz inequality~\cite{DKW56} comparing the \acp{CDF} of the two histograms constructed according to the actual distribution and through sampling.

The execution time and accuracy of this approach significantly depend on the chosen value of $\omega$ such that shrinking $\omega$ reduces the approximation error but increases $\big\lceil( \overline{f(A)}-\underline{f(A)})/\omega\big\rceil$ which in turn prolongs the comparison, while expanding $\omega$ speeds up the comparison at the cost of increased approximation error.
In most cases, a rough idea on how the objective values are distributed can be established, and thus, a good choice for $\omega$ can be made, at least after some prior experiments.

\subsubsection{Proposed Empirical Distribution-based Approach ($C_{\it emp}$)}
The distribution of uncertain objective values often cannot be fitted to a closed-form \ac{PDF}, and therefore, must be represented by a population of samples.
To achieve a good tradeoff between the accuracy and execution time of the comparison, we propose here an approach that is based on the empirical distribution of uncertain objective values.
Given the population of samples $S = (s_1, s_2, \ldots, s_N)$, a random variable $X$ which is distributed according to the corresponding empirical distribution has a \ac{CDF}
\begin{equation}
\CDF{y}{X}=\Pr(X\leq y)=\frac{1}{N}\big\vert\{i\mid s_i\leq
y\}\big\vert\enspace,
\end{equation}
which amounts to the proportion of samples being smaller than or equal to $y$.

Let $f(A)$ and $f(B)$ follow two empirical distributions that are specified by populations of samples $(a_1, a_2, \ldots, a_N)$ and $(b_1, b_2, \ldots, b_M)$, respectively.
A naive approach to compare these uncertain objective values would then be
\begin{equation}
  \Pr(f(A) > f(B))=\frac{\vert\{(i,j)\mid a_i>b_j\}\vert}{N\cdot M}\enspace.
  \label{eq:empirical}
\end{equation}
Unlike the pairwise comparison $C_{\it pw}$, each sample in the first population is not only compared to the corresponding sample but also to all other samples in the second population.
The desired probability is then estimated as the proportion of pairs wherein the sample from the first population is greater than the sample from the second population.

The time complexity of this approach is of order $O(N\cdot M)$, or $O(N^2)$ if $M=N$, which is usually the case.
To reduce this complexity, we propose to first sort each population of samples in ascending order and then to apply the function shown in Algorithm~\ref{alg:compareEmpirical}.
This function receives sorted lists $(a_1,\ldots,a_N)$ and $(b_1,\ldots,b_M)$ that respectively represent the uncertain objective values $f(A)$ and $f(B)$, and returns the probability $\Pr\big(f(A)>f(B)\big)$.
It uses two indices $i$ and $j$ to traverse these lists in ascending order.
It stores the proportion of pairs wherein the sample $a_i$ is greater than the sample $b_j$ in a variable named $num\_pairs$.
For each $a_i$, it adds $j-1$ to $num\_pairs$ when $b_j$ is the first element in its list that is not smaller than $a_i$.
Since each list is traversed only once, the comparison is performed in linear time $O(N+M)$, or simply $O(N)$ if $M=N$.
Also, the condition and the body of the while loop are evaluated at most $N+M$ and $M$ times, respectively.
Note that generating and sorting the samples have the time complexities of respectively $O(N+M)$ and $O(N\log N + M\log M)$.

\begin{algorithm}[t]
  \SetKwInOut{Input}{Input}%
  \SetKw{KwTo}{to}%
  \SetKw{KwAnd}{and}%
  \caption{Comparing sorted lists of samples}
  \label{alg:compareEmpirical}
  \Input{Sorted lists $(a_1,\ldots,a_N)$ and $(b_1,\ldots,b_M)$}
  $j:=1$\;
  ${\it num\_pairs}:=0$\;
  \For{$i:=1$ \KwTo $N$}
  {
    \While{$j<=M$ \KwAnd $a_{i}>b_{j}$}
    {
      $j:=j+1$\;
    }
    ${\it num\_pairs}:={\it num\_pairs}+j-1$\;
  }
  \Return ${\it num\_pairs}/(N\cdot M)$\;
\end{algorithm}

If the \acp{PDF} of $f(A)$ and $f(B)$ are available, the proposed approach can be extended to derive $N$ quantiles that partition the actual distribution into intervals of equal probabilities.
This enables to construct empirical distributions that achieve better approximations of the actual distributions.
Figure~\ref{fig:example_cdf} shows the exact \acp{CDF} of the distributions shown in Figure~\ref{fig:example_distribution} as well as approximations of these \acp{CDF} using 10 quantiles.
Using the quantiles such that $a_i=\q{\frac{2i-1}{2N}}{}$, the difference between the \acp{CDF} of the empirical and the actual distributions are restricted to $\frac{1}{2N}$.
Considering this difference for both $f(A)$ and $f(B)$ results in a maximum estimation error of $1/N$ for the calculation of $\Pr\big(f(A)>f(B)\big)$.
Let $X_A$ and $X_B$ be random variables distributed according to the empirical distributions of $f(A)$ and $f(B)$, respectively.
Then
\begin{align}
  \Pr\big(f(A)>f(B)\big)
  &=\int_{\underline{f(A)}}^{\overline{f(A)}}\Pr\big(f(A)=a\big)\Pr\big(f(B)<a\big)\mathrm{d}a \nonumber \\
  &=\int_{\underline{f(A)}}^{\overline{f(A)}}\Pr\big(f(A)=a\big)\big(\Pr(X_B<a) + \varepsilon(a)\big)\mathrm{d}a \nonumber \\
  &=\varepsilon+\int_{\underline{f(A)}}^{\overline{f(A)}}\Pr(f(A)=a)\Pr(X_B<a)\mathrm{d}a \nonumber \\
  &=\varepsilon+\int_{\underline{f(B)}}^{\overline{f(B)}}\Pr(f(A)\geq b)\Pr(X_B=b)\mathrm{d}b \nonumber \\
  &=\varepsilon+\int_{\underline{f(B)}}^{\overline{f(B)}}(\Pr(X_A\geq b)+ \varepsilon'(b))\Pr(X_B=b)\mathrm{d}b \nonumber \\
  &=\varepsilon+\varepsilon'+\int_{\underline{f(B)}}^{\overline{f(B)}}\Pr(X_A\geq b)\Pr(X_B=b)\mathrm{d}b \nonumber \\
  &=\varepsilon+\varepsilon'+\Pr(X_A>X_B)\enspace,  \label{eq:errorbound}
\end{align}
where $\varepsilon(x),\varepsilon,\varepsilon'(x),\varepsilon'\in[-1/(2N),1/(2N)]$ for all $x \in [\min\{\underline{f(A)},\underline{f(B)}\}, \max\{\overline{f(A)},\overline{f(B)}\}]$.
Note that the quantiles are already sorted in the ascending order, and the time needed for deriving $N$ quantiles is of order $O(N)$.

\begin{figure}
  \centering
  \cdfExample
  \caption{Exact and approximated \acp{CDF} for the distributions shown in Figure~\ref{fig:example_distribution}: the approximation uses 10 quantiles.}
  \label{fig:example_cdf}
\end{figure}

Similar to the pairwise comparison $C_{\it pw}$, the error of the empirical distribution can be bounded.
Let $X$ be a random variable that follows the empirical distribution of $f(\cdot)$ produced from a population of $N$ samples.
Then, according to the Dvoretzky-Kiefer-Wolfowitz inequality~\cite{DKW56}
\begin{equation}
\Pr\left(\sup_{x\in {\mathbb{R}} }\big\vert\CDF{x}{f(\cdot)}-\CDF{x}{X}\big\vert>\delta\right)\leq \mathrm{e}^{-c\cdot\delta^2\cdot N}
\label{eq:empiricalerrorprobability}
\end{equation}
for large values of $N$ and $0<\delta<1$, where $c$ is a positive constant.
The constants used here are worse than those used for $C_{\it pw}$ as the result for the pairwise comparison of random variables $f(A)$ and $f(B)$ can be seen as the value of the \ac{CDF} of their difference at the point zero $\CDF{0}{f(B)-f(A)}$.
Here, instead, the maximal error among all input values is bounded.
The actual error of estimating $\Pr\big(f(A)>f(B)\big)$ is then bounded by the sum of the two errors introduced by the empirical distributions of $f(A)$ and $f(B)$.
Nevertheless, the experiments in Section~\ref{sec:evaluation_comparison_operators} imply that this estimation is much more accurate.

\subsubsection{Proposed Reduced Empirical Distribution-based Approach ($C_{\it reduce}$)}
The proposed empirical distribution-based comparison operator can be further improved for the case where the \acp{PDF} of the uncertain objective values are unknown and can only be approximated through sampling.
This approach is similar to the histogram approximation ($\AHistogram{\omega}$) with the exception that the partitioning is performed evenly in the probability domain instead of the domain of the objective values.
It is used only if at least one of the uncertain objective values is given as a population of samples rather than a \ac{PDF}.
Otherwise, we use quantiles as described for the empirical distribution ($C_{\it emp}$) to provide a better approximation.

If we have a sorted list of samples $(s_1,\ldots,s_N)$, then we will use only
$N'=\Theta(\sqrt{N})$ data points $(y_1,\ldots,y_{N'})$, where $y_i=s_{\lceil (i-1/2)\cdot
N/N'\rceil}$ to produce the empirical distribution.
In our experiments, we will use
exactly $\lceil \sqrt{N}\,\rceil$ data points.
One could use pivot-based sorting algorithms like quicksort for sorting the
samples $(s_1,\ldots,s_N)$ and stop sorting if none of the needed indices are
available in an interval which has to be sorted currently, but as this does
not change the (expected) complexity of $O(N\log N)$ for sorting, we do not
elaborate on this improvement.
Therefore, for initialization, we still have the same complexity as for $C_{\it emp}$
with $N$ sample generations plus $O(N \log N)$ for sorting. The comparison of
two uncertain objective values is then much faster.  Let $(a_1,\ldots,a_{N'})$,
$(b_1,\ldots,b_{M'})$ be the reduced list of sorted samples from uncertain objective values $f(A)$ and $f(B)$, where $N'=\Theta(\sqrt{N})$ and $M'=\Theta(\sqrt{M})$,
then we can use again Algorithm~\ref{alg:compareEmpirical} to obtain the
complexity $O(\sqrt{N}+\sqrt{M})$.

The error caused by approximation with an empirical distribution using only $\sqrt{N}$ samples is then automatically in $O(1/\sqrt{N})$ which can be obtained by an equation analogous to Equation~\eqref{eq:errorbound}.
Now we analyze the situation when we pick a specific position $z$ and query the
probability of a random variable $X$ to be less than $z$.
Let $p$ be the actual
probability. For $N$ samples the relative number of samples less than $z$ is exactly
the value of the empirical distribution at $z$. This value multiplied by $N$ has
a binomial distribution with parameter $p$ and its variance scales with ${N}$.
Consequently the variance of the empirical distribution evaluated on any
position scales with $1/N$ and the standard deviation scales with $1/\sqrt{N}$.
Therefore, the order of the error does not change if the number of data points
is reduced to only the square root of the number of evaluated samples as
suggested.

Table~\ref{tab:complexity} summarizes the analyzed time complexities of the initialization, \mbox{i.\,e.,}\xspace approximating distributions, and comparison for all comparison operators investigated in this section.
It also reports if the result of the respective comparison operator converges to the exact probability $\Pr\big(f(A)>f(B)\big)$.
Note that the initialization is applicable only if the actual \ac{PDF} of at least one objective function is not available.
Also, in the case of the proposed empirical distribution-based approach ($C_{\it emp}$), $N$ denotes either the number of quantiles or the number of samples, depending on whether the actual \acp{PDF} are available or not.

\begin{table}
  \caption{Time complexity for initialization and comparison of the investigated comparison operators, and the information whether their approximation error tends to zero if the number
of samples $N$ is increased.}
\vspace*{-10pt}
  \begin{center}
  \renewcommand{\arraystretch}{1.4}
  \small
  \begin{tabular}{ l | c | c | c }
         &initialization        &comparison         &error$\rightarrow 0$\\
         \hline
         $C_{\it pw}$                 & ---               & $O(N)$ & \cmark\\
         $C_{\it uni}^{1/2}$      & $O(N)$            & $O(1)$ & \xmark\\
         $C_{\it{gauss}}$              & $O(N)$            & $O(1)$ & \xmark\\
         $\AHistogram{\omega}$  & $O\big(N + \big\lceil\frac{\overline{f(\cdot)}-\underline{f(\cdot)}}{\omega}\big\rceil\big)$ & $O(\big\lceil\frac{\overline{f(\cdot)}-\underline{f(\cdot)}}{\omega}\big\rceil\big)$ & \xmark\\
         $C_{\it emp}$          & $O(N\log N)$      & $O(N)$ & \cmark\\
         $C_{\it reduce}$            & $O(N\log N)$      & $O(\sqrt{N})$ & \cmark
  \end{tabular}
  \label{tab:complexity}
\end{center}
\vspace*{-5pt}
\end{table}

In the following, we evaluate the investigated comparison operators in terms of approximation error and execution time.
For the evaluation, five scenarios of two random variables $(X_1, X_2)$ with different distributions are selected, see the left column of Figure~\ref{fig:scenarios}.
\begin{figure}
  \begin{center}
	  \begin{tabular}{@{}c@{}}
    \hspace*{-2mm}
    \distI
    \hspace*{-2mm}
    \compQualityI
    \hspace*{-2mm}
    \compQualityLegendI
    \\[0.5\baselineskip]
    \hspace*{-2mm}
    \distII
    \hspace*{-2mm}
    \compQualityII
    \hspace*{-2mm}
    \compQualityLegendII
    \\[0.5\baselineskip]
    \hspace*{-2mm}
    \distIII
    \hspace*{-2mm}
    \compQualityIII
    \hspace*{-2mm}
    \compQualityLegendIII
    \\[0.5\baselineskip]
    \hspace*{-2mm}
    \distIV
    \hspace*{-2mm}
    \compQualityIV
    \hspace*{-2mm}
    \compQualityLegendIV
    \\[0.5\baselineskip]
    \hspace*{-2mm}
    \distV
    \hspace*{-2mm}
    \compQualityV
    \hspace*{-2mm}
    \compQualityLegendV
  \end{tabular}
\end{center}
\vspace*{-15pt}
    \caption{Left: Five scenarios of two random variables with different distributions introduced for evaluating different comparison operators.
    Right: The absolute error of comparison operators for each scenario and for different numbers $N$ of samples.
    The comparison operators in the legend are ranked according to their absolute error bound with the maximal number of samples $N=10^6$.}
  \label{fig:scenarios}
\end{figure}
These scenarios combine instances of various distributions including uniform $\mathcal{U}(\underline{X}, \overline{X})$, Gaussian $\Gauss{\E{X}}{\Var{X}}$ and beta
$\BetaDistribution{\alpha}{\beta}$ distributions.
The first four scenarios are adopted from~\cite{KBT18}.
These scenarios vary in their statistical properties and pose different challenges to the comparison operators.

\subsection{Approximation Error Analysis of Comparison Operators}\label{sec:evaluation_comparison_operators}
The right part of Figure~\ref{fig:scenarios} displays the quality of
different comparison operators for each scenario.  For each scenario and each
comparison operator one can see the development of the absolute error -
absolute difference of the evaluated value and the correct value - while the
number of samples is increased.
To be more precise the presented graphs represent an error bound such that
$99\%$ of comparisons by the respective comparison operator comply with that
error bound. This limitation is necessary because even with large numbers of
samples all samples of one random variable can be smaller than all samples of
the other random variable even if the first random variable dominates the
second random variable with probability greater than $50\%$. In such cases all
comparison operators will fail badly but fortunately, such an event will most
likely not happen.
To obtain additional information we also provide in
Table~\ref{tab:differences} the differences between the actual dominance values
and the evaluated dominance values by each comparison operator if infinitely
many samples would have been used. $C_{\it pw}$, $C_{\it emp}$ and $C_{\it reduce}$ are
omitted as they always converge to the actual dominance values.

\begin{table}
  \caption{For the scenarios in Figure~\ref{fig:scenarios}, the actual values $\Pr(X_1>X_2)$
	and the values calculated by each comparison operator are reported. Additionally, the respective
difference to the actual values if the number of samples tends to infinity is shown.}
\vspace*{-10pt}
\begin{center}
\small
${\arraycolsep=3pt
  \begin{array}{c| l | l | l | l | l}
    & 1 & 2 & 3 & 4 & 5\\
    \hline
	\Pr(X_1\!\!>\!\!X_2) &   0.7978  &     0.6000 &  0.6092 &  0.5923 &     0.9727\\
    \hline
    C_{\it uni}^1      & \begin{array}{@{}l@{}} 0.5000 \\   -0.2978\end{array}
                     & \begin{array}{@{}l@{}} 0.6000 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.5000 \\   -0.1092\end{array}
                     & \begin{array}{@{}l@{}} 0.5000 \\   -0.0923\end{array}
                     & \begin{array}{@{}l@{}} 0.9727 \\\pm 0.0000\end{array} \\
\hline
    C_{\it uni}^2      & \begin{array}{@{}l@{}} 0.7700 \\   -0.0278\end{array}
                     & \begin{array}{@{}l@{}} 0.6000 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.5962 \\   -0.0130\end{array}
                     & \begin{array}{@{}l@{}} 0.6461 \\   +0.0538\end{array}
                     & \begin{array}{@{}l@{}} 0.9727 \\\pm 0.0000\end{array} \\
\hline
    C_{\it{gauss}}          & \begin{array}{@{}l@{}} 0.8042 \\   +0.0064\end{array}
                     & \begin{array}{@{}l@{}} 0.6261 \\   +0.0261\end{array}
                     & \begin{array}{@{}l@{}} 0.6092 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.6922 \\   +0.0999\end{array}
                     & \begin{array}{@{}l@{}} 0.9669 \\   -0.0058\end{array} \\
\hline
    \AHistogram{0.1} & \begin{array}{@{}l@{}} 0.7902 \\   -0.0076\end{array}
                     & \begin{array}{@{}l@{}} 0.6000 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.6040 \\   -0.0052\end{array}
                     & \begin{array}{@{}l@{}} 0.5936 \\   +0.0013\end{array}
                     & \begin{array}{@{}l@{}} 0.8000 \\   -0.1727\end{array} \\
\hline
    \AHistogram{0.05}& \begin{array}{@{}l@{}} 0.7959 \\   -0.0019 \end{array}
                     & \begin{array}{@{}l@{}} 0.6000 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.6079 \\   -0.0013 \end{array}
                     & \begin{array}{@{}l@{}} 0.5926 \\   +0.0003\end{array}
                     & \begin{array}{@{}l@{}} 0.8721 \\   -0.1006 \end{array} \\
\hline
    \AHistogram{0.01}& \begin{array}{@{}l@{}} 0.7977 \\   -0.0001\end{array}
                     & \begin{array}{@{}l@{}} 0.6000 \\\pm 0.0000\end{array}
                     & \begin{array}{@{}l@{}} 0.6092 \\   -0.00005\!\!\end{array}
                     & \begin{array}{@{}l@{}} 0.5923 \\   +0.00001\!\!\end{array}
                     & \begin{array}{@{}l@{}} 0.9683 \\   -0.0044\end{array} \\
  \end{array}
}$
\end{center}
\vspace*{-5pt}
  \label{tab:differences}
\end{table}

These results visualize that the comparison operators which assume uniform or
Gaussian distributions ($C_{\it{gauss}}$, $C_{\it uni}$) outperform other comparison
operators if the random variables actually follow instances of the assumed distribution but they can
also have quite large errors if this is not the case.

Also the histogram based comparison operator ($\AHistogram{\omega}$) has this problem
as it assumes piecewise constant densities. Also if the positions where
densities of the histogram can change do not match the positions where the actual distributions
change, a significant error can be received. This is most explicitly tracked in
the last scenario. But if the width $\omega$ of the columns in the histograms is suitably
chosen then it leads to similar results as for the proposed comparison operators based on empirical distributions.

The remaining comparison operators ($C_{\it pw}$, $C_{\it emp}$ and $C_{\it reduce}$)
finally converge to the actual dominance value if the number of samples is
increased. The comparison operator using empirical distributions
($C_{\it emp}$) and its reduced version ($C_{\it reduce}$) converge in all cases
faster than the pairwise comparison ($C_{\it pw}$). For larger sample sizes, there is
almost no difference between $C_{\it emp}$ and $C_{\it reduce}$. Only for very small
sample sizes $C_{\it reduce}$ has a larger error due to too few data points in the
reduced empirical distribution. Also for $C_{\it pw}$, the variance is quite large if
the sample size is small.

Here the presented experiments confirm the advantages and disadvantages
predicted by theoretical evaluation in the previous section.

\begin{figure*}
  \resizebox{\textwidth}{!}
{
  \input{figures/timingcomparatorsinitialization.tex}
}
  \resizebox{\textwidth}{!}
{
  \input{figures/timingcomparatorscomparison.tex}
}
   \caption{For each comparison operator, we display the execution times of the
    initialization of one object for comparison (once for
    each object) and the execution times of the comparison of two already initialized
  objects in $\mu s$ for $10$, $100$ and $1\,000$ samples per object
(Programming language: Java, executed on an Intel i7-4790 with an Ubuntu operating system).}
  \label{fig:timing}
\end{figure*}

\subsection{Execution Time Analysis of Comparison Operators}
In addition to the comparison operators presented in Section~\ref{sec:prob_dom_operators}, there exist other comparison operators which leave out the detour via probabilistic dominance.
These comparison operators do not calculate the dominance value and evaluate the preference of an uncertain objective value in comparison to another by different characteristics.
Two well established comparison operators of that kind are the mean-based approach~\cite{MAG12} and the three-stage comparison operator~\cite{KMGT15} which are briefly introduced in the following:

\subsubsection{Mean-based Comparison Operator ($C_{\it Mean}$)~\cite{MAG12}}
This comparison operator simply decides whether $f(A)$ or $f(B)$ is better by comparing their mean values.
The initialization takes linear time as the mean has to be estimated according to Equation~\eqref{eq:sample_mean} and the comparison takes only constant time.

\subsubsection{Three-Stage Comparison Operator ($C_{\it ThreeStage}$)~\cite{KMGT15}}
This comparison operator performs the comparison in three stages.
In the first stage, it checks whether the domains, \mbox{i.\,e.,}\xspace the interval between the lower and upper bound, of the two uncertain objective values overlap.
If not, then the value in the better region is decided to be better.
If the domains do overlap, the next stages apply.
In the second stage, it checks whether the mean of one uncertain objective value is significantly better than that of the other value.
If this is true, then the means determine whether $f(A)$ or $f(B)$ is better.
Otherwise the third stage is applied where the uncertain objective values are compared with respect to their $95\%$ quantile interval, \mbox{i.\,e.,}\xspace the interval between the $0.025$-th and $0.975$-th quantile points.
By comparison of quantiles it is checked whether $f(A)$ or $f(B)$ shows a considerably smaller deviation and delivers a more robust objective value.
If the lengths of these quantile intervals are very similar, it is decided that $f(A)$ and $f(B)$ are equally good.
To prepare quantiles we have the complexity $O(N \log N)$ for sorting $N$
samples during initialization time of each object. The comparison is then
evaluated in constant time.

Now we relate these two comparison operators with the six probabilistic dominance-based comparison operators introduced in Section~\ref{sec:prob_dom_operators}.
For this purpose, we interpret the dominance values calculated by a comparison operator from Section~\ref{sec:prob_dom_operators} to obtain the answer for two solutions $A$ and $B$ whether $f(A)$ is greater/smaller than or as good as $f(B)$.
To accomplish this for two uncertain objective values $f(A)$ and $f(B)$, we can define a threshold $\gamma\geq
0.5$ to determine whether the probability that $f(A)$ is greater/smaller than $f(B)$ is significant enough to reach a conclusion.
For example, in case of a maximization problem we say that $f(A)$ is better than $f(B)$ if
$\Pr\big(f(A)>f(B)\big)>\gamma$, $f(B)$ is better than $f(A)$ if $\Pr\big(f(B)>f(A)\big)>\gamma$ and otherwise
they are equally good, see~\cite{KBT18}.
This probabilistic dominance criterion can incorporate any of the previously introduced comparison operators in Section~\ref{sec:prob_dom_operators} to enable treating uncertainty in multi-objective optimization.

All the previously introduced comparison operators (the six comparison operators introduced in Section~\ref{sec:prob_dom_operators} and the two well established comparison operators introduced in this section) have been implemented in Java and experimented on an Intel i7-4790 with an Ubuntu operating system.
The average execution times for the initialization and comparison of each comparison operator for the scenarios in Figure~\ref{fig:scenarios} are visualized in Figure~\ref{fig:timing}.
The measured results comply with the presented complexities.
The only surprise might be that $C_{\it{gauss}}$ has such a slow comparison time which is mainly spent for the approximation of the error function in Equation~\eqref{eq:erf_gauss}.

\section{Uncertain Multi-Objective Optimization Benchmark}\label{sec:benchmark}
This section presents a number of uncertain multi-objective optimization problems to evaluate the proposed comparison operator alongside the state-of-the-art approaches.
These problems aim to incorporate various uncertainties in six multi-objective optimization problems of the DTLZ benchmark suite~\cite{DTLZ02}.
Each of these problems are specified by a vector of $m$ objective functions $(f_1, f_2, \dots, f_m)$,  that receive a vector of $n$ shared decision variables $x=(x_1, x_2, \dots, x_n)$ where $\forall i \in \{1, 2, \dots, n\}: 0 \leq x_i \leq 1$.

Table~\ref{tab:problem} shows the proposed uncertain DTLZ (UDTLZ) problems.
The considered uncertainties include perturbation in decision variables as well as noise and approximation error in objective functions\footnote{The uncertainty due to time-varying objective functions is not considered in the proposed uncertain problems. However, the proposed comparison operator can deal with any uncertainty that leads to probabilistic representations of objective values.
}.

The UDTLZ1 problem adds uncertainties specified by instances of Beta distribution $\mathcal{B}$ with different shape parameters onto the decision variables of the DTLZ1 problem.
The beta distribution generates samples $\bm{u}$ in the interval $[0,1]$ which are scaled to $[0,0.001]$ and added to the expected values $x_i$.
The resulting value $\bm{x_i}$ is then bounded to 1 which is the upper bound of $x_i$.
Similarly, UDTLZ6 incorporates perturbations following instances of Gaussian distribution $\mathcal{N}$ in the decision variables of DTLZ6.
The Gaussian distributions in this problem generate samples around an expected value of $0$ with different standard deviations.
Since the added variations may be negative or positive, the resulting value is then bounded to $[0,1]$.

\input{body/udtlzbenchmarks_arXiv.tex}

UDTLZ2, UDTLZ3, UDTLZ4 and UDTLZ5 use both function noise and approximation error to add uncertainty to their corresponding DTLZ problems.
Function noise $\bm{u_1}$ describes samples from a Gaussian distribution with an expected value of $0$ and a standard deviation of $0.005$ across all
of these problems.
Also, function approximation error is added to this problems by replacing each trigonometric or exponential function with its corresponding Maclaurin series where
the number of terms in series follows discrete uniform distributions $\Uniform{3}{12}$ for UDTLZ2, UDTLZ4 and UDTLZ5, and $\Uniform{12}{19}$ in the case of UDTLZ3.

\section{Related Work}\label{sec:relatedwork}
The objective functions and decision variables of multi-objective optimization problems are often subject to various uncertainties.
In the context of probabilistic risk assessment, these uncertainties are categorized with respect to their origin as \emph{aleatory} and \emph{epistemic} uncertainties.
Aleatory uncertainty refers to ``the inherent variation associated with the physical system or the environment under consideration'',
whereas epistemic uncertainty describes ``any lack of knowledge or information in any phase or activity of the modeling process''~\cite{OHJWF04}.

The probability distribution of a decision variable or an objective function with aleatory uncertainty can be estimated through sampling or iterative function evaluation, respectively.
However, obtaining the exact value or probability distribution of a variable or function with epistemic uncertainty is usually impracticable or unaffordable.
In fact, only limited characteristics such as confidence intervals may be available.

To deal with epistemic uncertainty in the context of multi-objective optimization, the work in~\cite{Limbourg05} proposes to represent uncertain objective values using lower and upper bounds rather than single point estimates.
It also extends the weak and strong dominance criteria of a multi-objective evolutionary algorithm.
To balance the accuracy and execution time of the process of comparing candidate solutions, it integrates the weak dominance criterion into the process of parent selection and the strong dominance criterion into the process of updating the solution archive.
The authors in~\cite{ERM12} model uncertainty as the lack of knowledge about the exact effects of decision variables on objective values using a triangular fuzzy representation.
They incorporate the pessimistic, anticipated, and optimistic values of uncertain objectives into the comparison of different solutions.
However, the proposed dominance criterion may fail to properly distinguish objective values when the intervals between pessimistic and optimistic values do not overlap, which reduces the quality or robustness of optimization.
Since the distribution of uncertainty in objective functions and decision variables are not available in the case of epistemic uncertainty, the rest of this paper focuses on dealing with aleatory uncertainty, where there is a stronger demand for its effective and efficient handling.

Another common classification of uncertainties can be found in~\cite{JB05} where the uncertainties are categorized with respect to their manifestations into four groups.
The first group includes noise in objective functions, \mbox{i.\,e.,}\xspace variations in the results of different evaluations of an objective function with unchanged input variables.
The second group takes perturbations in decision variables, that are the input variables of objective functions, into consideration.
The third group describes the error of approximate objective functions which is the case when the exact evaluation of an objective function is costly or infeasible, and is therefore substituted by simulations.
The last group models time-varying objective functions where evaluating a function with the same inputs and parameters at different points in time delivers different outputs, while the output is deterministic at any fixed point in time.
An existing uncertainty, regardless of what category it belongs to, results in objective values that should be represented by probability distributions instead of single values.
The resulting distribution may be an instance of a standard distribution such as Gaussian, or might follow any arbitrary distribution given as a \ac{PDF} or sampled data.

To enable handling uncertainty in multi-objective optimization, a group of studies~\cite{Teich01, Hughes01, FE05} propose techniques to determine \emph{probabilistic dominance} which describes the probability that one
candidate solution dominates the other.
This probability is calculated as the intersection of all probabilities that an uncertain objective value from the first solution is more favorable than the same objective of the other solution.
The techniques in~\cite{Teich01, Hughes01, FE05} are based on the simplistic assumption that different uncertain objectives are statistically independent.
Therefore, they calculate the joint probability as the product of all individual probabilities.
Teich~\cite{Teich01} provides a mathematical approach for the calculation of probabilistic dominance given all objectives follow instances of continuous uniform distributions.
This approach can be effortlessly extended to treat uncertain objective values with any discrete distributions.
The work in~\cite{Hughes01} assumes that each uncertain objective is affected by a Gaussian noise with known variance.
The authors in~\cite{FE05} extend this technique to enable the calculation of probabilistic dominance when instances of the same uncertain objective have the same, but unknown variance.
They propose a learning technique to reduce the number of objective function re-evaluations needed to estimate this variance.
However, the main drawback of the techniques in~\cite{Teich01, Hughes01, FE05} is that they require all uncertain objectives of a solution to be statistically independent instances
of specific distribution types.
In fact, an uncertain objective value may follow an arbitrary distribution that combines the uncertain characteristics of different decision variables.
Also, two different objective functions sharing one or more uncertain decision variables would have statistically dependent uncertainty distributions.

The work in~\cite{EGB07} compares candidate solutions with respect to the mean values of their uncertain objectives using a strict dominance criterion.
It proposes to deal with Gaussian noise in objective values while ranking the dominated solutions in the process of parent selection of a genetic algorithm.
For each dominated solution, a strength value is calculated which is the sum of the probabilities that this solution dominates any other solution from the population.
These probabilities are calculated similar to the approach in~\cite{Hughes01}.
Each dominated solution is then ranked with respect to the difference between the sum of strength values of all solutions it dominates and that of all solutions dominating it.
The calculation of this criterion is very time-consuming and the main dominance criterion does not incorporate uncertainty in the comparison of candidate solutions.

Another group of studies in~\cite{MAG12} and~\cite{TC11} proposes to replace each uncertain objective by one or more single-valued objectives, each representing a unique statistic such as mean or variance of the original objective.
This eliminates the need for incorporating the effects of uncertainty in the comparison operator or dominance criteria of optimization algorithms.
As an example, the work in~\cite{TC11} adopts the mean-variance model~\cite{Markowitz52} to replace the uncertain objective in a single-objective optimization with two separate objectives representing its mean and variance.
It then uses integer programming for maximizing the mean and minimizing the variance.
The main disadvantage of this approach is that it may recognize a solution with a significantly inferior mean but a slightly better variance as non-dominated, which can crowd the solution archive and slow down the optimization.
On the other hand, the technique in~\cite{MAG12} represents each uncertain objective with a single statistic which is selected based on the criticality of the objective.
For example, it uses the fifth percentiles for critical objectives which demand a high degree of robustness and mean values for the non-critical ones.
However, this technique often fails to accurately compare uncertain objectives because a single statistic cannot describe all properties of the underlying probability distributions.

The authors in~\cite{KMGT15} and~\cite{MTF14} propose to extend the operators used in existing multi-objective optimization techniques to enable coping with uncertain objectives.
The work in~\cite{MTF14} assumes that the uncertain objectives are specified by mean values and confidence intervals.
It checks whether the confidence intervals of none of the uncertain objectives in two candidate solutions are overlapping.
In this case, it can be easily determined if one solution dominates the other or if the solutions are incomparable.
However, if the confidence intervals overlap for at least one objective, it performs an iterative reduction of confidence intervals by re-evaluating the corresponding objective functions.
This process is continued until the intervals are no longer overlapping or no further reduction is possible.
In the latter case, the overlapping intervals are simply compared with respect to their mean values.
The work in~\cite{KMGT15} proposes to compare instances of each uncertain objective in a three-stage algorithm.
To compare two instances of an uncertain objective, this algorithm first checks if the worst-case of one is better than the best-case of the other.
If no preference can be found, it prefers the objective value which is significantly better with respect to the mean values.
If the mean values are not sufficiently different, the algorithm checks if one objective value has a noticeably smaller deviation.
Two objective values that cannot be differentiated by any of these three comparisons are considered equal.
The comparison operators in~\cite{KMGT15} and~\cite{MTF14} enable comparing arbitrarily distributed uncertain objectives.
Moreover, the optimization algorithms employing these operators compare instances of each objective individually, which implicitly takes possible statistical dependencies among objectives into consideration.
Nonetheless, these techniques depend on estimated statistical properties and do not reflect the probability that one candidate solution (or objective value) dominates the other.

A different approach is proposed in~\cite{HYY18} which proposes to first solve the optimization problem without the consideration of uncertainty using the multi-objective evolutionary algorithm presented in~\cite{DAPM00}.
It performs the decomposition proposed in~\cite{ZL07} to partition the objective space and represent each sub-space by a weighted sum of objectives.
Then, it maps each solution to a weighted sum such that the distance between each solution and its corresponding weighted sum is minimized.
To deal with uncertainty, it iteratively evaluates the optimal solution of each weighted sum and derives the mean and worst-case objective values.
In the end, it removes the non-optimal solutions that are dominated by this worst-case objective value and looks for robust solutions in the neighborhood of the optimal solution of each weighted sum.
Although this technique helps identifying the robust regions~\cite{DG06} in the search space, it lacks efficiency and treats uncertainty as worst-case objective values.

To overcome the aforementioned limitations of existing uncertainty-aware optimization techniques, a histogram-based comparison operator has been proposed in~\cite{KBT18}.
It first partitions the probability distribution of uncertain objectives into intervals of identical width.
Then, considering a uniform distribution within each interval, it calculates the probability that an instance of an uncertain objective is greater or smaller than another instance of the same objective, and enables to differentiate the two solutions with respect to this uncertain objective.
Similar to the techniques in~\cite{KMGT15} and~\cite{MTF14}, it is capable of handling problems with statistically dependent uncertain objectives because it compares instances of each objective separately.
However, at a reasonable performance overhead, it allows for considering the entire probability distribution rather than a certain number of statistics.

In this paper, we propose an extension to the approach in~\cite{KBT18}, aiming at improving its comparison accuracy and execution time.
To represent the probability distribution of an uncertain objective, it uses a \ac{CDF} rather than a histogram.
Given a set of samples obtained from iterative evaluation of an uncertain objective, it constructs the \ac{CDF} of the corresponding uncertain objective value by sorting the samples.
For a given value of the distribution, its cumulative probability equals the proportion of samples smaller than this value to the total number of samples.
We then introduce a fast algorithm to compare \acp{CDF} of two instances of an uncertain objective in order to calculate the probability that one is greater than the other.
Moreover, we propose an approximate representation of \acp{CDF} which helps to significantly reduce the time complexity of this algorithm.

\section{Conclusion}\label{sec:conclusion}
Multi-objective optimization problems are often subject to various sources and forms of uncertainty in their decision variables and objective functions.
A wide range of approaches has been proposed in the literature to distinguish candidate solutions with objective values specified by confidence intervals, probability distributions or sampled data.
However, these approaches typically fail to treat uncertainties with non-standard distributions accurately and efficiently.
In this paper, we investigated a variety of techniques that offer the comparison of uncertain objective values, and proposed novel techniques for the calculation of
the probability that an uncertain objective of one solution is more favorable than the same objective of the other solution.
To enable discovering robust solutions to problems with multiple uncertain objectives, we incorporated the proposed comparison operator into existing optimization techniques such as evolutionary algorithms.
We also extended well-known multi-objective problems with various uncertainties and proposed a benchmark for evaluating robust optimization techniques.
The proposed comparison operators and benchmark suite have been integrated into an existing optimization framework that provides a selection of multi-objective optimization problems and algorithms.
Experiments show that compared to existing techniques, our enhanced comparison operator achieves higher optimization quality and imposes lower overheads to the optimization process.

\section*{Notation}

\begin{description}
 \item[$A$] a candidate solution
 \item[$B$] a candidate solution
 \item[$\mathcal{B}$] beta distribution
 \item[$C$] comparison operator
 \item[$c$] positive constant
 \item[$E$] expected value
 \item[$e$] approximation error
 \item[$f$] objective function
 \item[$\mathcal{N}$] Gaussian distribution
 \item[$N$] number of samples or quantile cuts
 \item[$n$] number of decision variables
 \item[$m$] number of objective functions
 \item[$S$] sequence of samples
 \item[$s$] sample from an uncertain objective's distribution
 \item[$\mathcal{U}$] uniform distribution
 \item[$\bm{u}$] uncertainty added to an optimization problem
 \item[$Var$] variance
 \item[$X$] random variable
 \item[$x$] decision variable
 \item[$\gamma$] comparison threshold
 \item[$\delta$] tolerance (bound on an error)
 \item[$\sigma$] standard deviation
 \item[$\omega$] interval width in a histogram
\end{description}

redpajama_set_name: RedPajamaArXiv


text:
February 15, 2019 Manish Vats LifeLines 0
Haruki Murakami, a 29-year-old Japanese man was just an ordinary jazz club owner until he decided to write. The Kafka award winner was inspired to write his first novel, Hear The Wind Sing in his mid-twenties while watching a baseball game. Since then, the Japanese novelist has never looked back.
Who would have thought that the person with no prior writing experience will take the readers to the whole new fictional dimension with his writings one day?
Murakami has written 14 novels in the genre of fantasy, fiction, realism, magical realism, and surrealism. Along with novels, the famous Japanese prodigy has produced numerous short stories which have been best sellers in Japan as well as in the international market.
Standing in the list of The Guardian's "Great Living Novelist" for his works and achievements, we are here to list some of his greatest works he has written. You can tryout finding best books online with amazing discounts via Amazon Offers on Books available Online.
Here, we are with the list of best Haruki Murakami's books you should read in your lifetime:
A Wild Sheep Chase
In 1982, Haruki Murakami published his third novel, A Wild Sheep Chase which turned out to be a critical success in his writing career. The author dangling between modern Japanese and American middle classes hit the right chords at that time.
Murakami's A Wild Sheep Chase turned out to be a real success not because of its youthfulness but also the book explored Japanese cultural identity which made this book somewhat political too.
The book binds you till the end and will keep you on the edge of the seat as you turn the pages. Also, Murakami was awarded Noma Literary Newcomer Award for A Wild Sheep Chase in 1982 after the release of the book.
It was the year 1987 when Haruki came with another masterpiece which might have made his popularity gone wider and wilder at the same time. The Japanese author came up with a really tight and compelling story of Watanabe Tōru.
While listening to Beatles cover, "Norwegian Woods" in a bar of Germany, Toru reminisces about his past when he develops relationships with two entirely different women. The nostalgic story of loss and sexuality, Norweign Woods is set in the late 1960's Tokyo.
This book made Haruki Murakami an overnight star- Another power of his writings among youth.
The Wind-Up Bird Chronicle
Known as the first-person narrative writer, Murakami came up with a fusion of realism and fantasy in The Wind-Up Chronicle.
Unlike previous ones, this book created more buzz socially as the book kissed the parts of war crimes in Manchukuo, North East China. The work is a study of sex, violence, and collective memories lost and regained.
The novel was not only applauded by the critics but also Murakami was awarded the Yomiuri Literary Award by his former harshest critic, Kenzaburō Ōe. You can buy The Wind-up Bird Chornicle with Flipkart Offers on books Online.
Following to the previous success, Sputnik Sweetheart, Murakami came up with a total metaphysical mind-blender- Kafka On The Shore.
This time the author played between two distinctive stories with interrelated plots moving along with each other side by side.
The story runs back and forth between Kafka, "the world's strongest fifteen-year-old boy" and a mentally ill old man, Nakata. Kafka On The Shore leaves you stunned at times when you read about Nakata talks to cats and predicts about raining fishes and leeches. The book is containing several riddles and Murakami says that the secret to understanding the novel lies in reading it several times.
Along with the critical and social success, this book has been awarded World Fantasy Award for Novels in 2008 and the book holds a spot The New York Times 10 Best Books of 2005.
When Murakami's 1Q84 hit the bookstores, the surreal narrative first printing was sold within a day itself and a million copies in a month.
1Q84 is a three-volume dystopian novel which covers the fictionalized the year of 1984 in parallel with a "real" one.
The book is a complex and tight set which looks hard at the tension between political and religious ideology and the inner soul of the individual.
Like his previous works, this novel too entertains his first-person narrative which grips its readers with contemporary Japanese culture.
The book also was placed on number 2 spot in November on Amazon's top books of the year. Along with that, the book was later longlisted for Man Asian Literary Prize in 2011.
Colour me tradition! Evolution of Holi from the folklores to the modern day
The New Retreat : Man Lives in Sandcastle to Avoid Rent, for REAL!
The mountains are echoing "help!"
The Probashi's Puja: Festive vibes flourishing everywhere