text:
Local Production Manager Hired
Hot off the heels of learning that "The Witcher" would be filmed in Hungary, we are now learning that a production manager in the area has been hired, and a production lot has likely been located. According to IMDB, Tamas Gal has been hired as the production manager for the show. Gal has local experience in Hungary, and has worked on other major Hollywood projects such as "Marco Polo" and "The Martian."
Additionally, user gravemaster7 on Reddit has brought up some very interesting theories on production locations. Korda Studios in Hungary has a multitude of great locations and space- not to mention an already constructed backlot of a 13th-14th century village. Could "The Witcher" be moving in? Time will tell.
Let us know your thoughts in the comments below and in our forum!


text:
Inside Whistleblower Exposes Biden's Darkest Military Secrets
If these reports are true, and they likely are, Biden should be impeached immediately and tried as a war criminal. His negligence and dereliction of duty has resulted in the lives of numerous U.S. servicemen and countless innocent Afghans, both of which remain in immense danger.
It's been reported that the Taliban initially offered Biden U.S. control of Kabul and the airport as they ascended to power. Biden turned down their offer, essentially handing off the task of protecting American citizens, soldiers, and military assets in the region.
The resulting tragedies have been widely reported and are still ongoing.
According to a military insider, as the Taliban began taking control of Afghanistan, senior U.S. military leaders met with Taliban political leader Abdul Ghani Baradar in Doha, Qatar, where an offer was made for the United States to have control of Kabul until all U.S. troops had withdrawn; the offer was declined.
Here are some highlights from the report:
In a hastily arranged in-person meeting, senior U.S. military leaders in Doha – including McKenzie, the commander of U.S. Central Command – spoke with Abdul Ghani Baradar, head of the Taliban's political wing.
"We have a problem," Baradar said, according to the U.S. official. "We have two options to deal with it: You [the United States military] take responsibility for securing Kabul or you have to allow us to do it."
Throughout the day, Biden had remained resolute in his decision to withdraw all American troops from Afghanistan. The collapse of the Afghan government hadn't changed his mind.
McKenzie, aware of those orders, told Baradar that the U.S. mission was only to evacuate American citizens, Afghan allies and others at risk. The United States, he told Baradar, needed the airport to do that.
On the spot, an understanding was reached, according to two other U.S. officials: The United States could have the airport until Aug. 31. But the Taliban would control the city.
As the evacuation mission proceeded, security in the Kabul airport broke down completely, becoming increasingly hostile and chaotic. As a result, ISIS-K terrorists were successful in breaking Taliban security checkpoints and unleashing attack that culminated in the deaths of 13 American servicemen and 160 innocent Afghans.
In fact, Biden and his crack defense team trusted the Taliban to such a degree, they even provided them with a list containing names of American citizens, green card holders, and Afghan allies in the region to prioritize for evacuation.
Biden did not deny the existence of said list when pressed in the disturbing press conference:
Joe Biden: "there may have been" a list of American and ally names given to the Taliban pic.twitter.com/X28iSuuaUh
Following the report, Republican lawmakers have blasted the Biden administration for the harrowing decisions made that resulted in the lives of hundreds:
If the reports are true, and the Biden administration willfully gave control of Kabul to the Taliban, they have yet again shown their complicity for a terrorist takeover. https://t.co/qmrWSgJpmn
— Sen. Marsha Blackburn (@MarshaBlackburn) August 29, 2021
Want to know what incompetence looks like ? Read this? We could have kept Kabul..we only asked for airport, It was Austins decision to leave Embasey – not even a threat- all have to fired..Sec'y of state in Hamptons as war was lost!! https://t.co/E4cfnoYOaN
— Brian Kilmeade (@kilmeade) August 29, 2021
The foreign policy decisions made by Biden and his team will likely change foreign policy in the region for years to come. The positive strides made by the Trump administration essentially vanished once Biden took the helm. Instead of bombs, Trump chose to use his strengths as a negotiator to ensure American safety from Taliban forces. Instead of idle threats, Trump used diplomacy to broker peace in the region.
Still, the Presidential Medal of Freedom has hung around Biden's neck for years, and Mainstream Media outlets will continue to applaud his efforts despite the death and destruction.
Author: Nolan Sheridan

redpajama_set_name: RedPajamaCommonCrawl


text:
Is Uber's London Ban Really Shunning Innovation?
With TFL revoking Uber's license, are they saying no to innovation?
Chief Innovation OfficerInnovationInnovation Culture
On Friday September 22nd TFL (Transport For London) ruled to remove Uber's operating license for London, throwing 40,000 drivers and millions of users in the UK capital into a tailspin. In the 5 years since it began operating in the city, Uber has become ubiquitous with quick journeys around the city, even becoming a verb for travel amongst many of its users. It is also considerably cheaper than its competitors in the city, with a study by the Daily Telegraph finding that a trip from Marble Arch to Westminster was around £6, compared to £10 for a Black Cab and £9 for Addison Lee (the largest traditional private taxi company in the city). It is little wonder then that the potential ban on the company has caused such uproar amongst the city's population.
Reasons for London banning Uber are many, but the 4 points outlined in the press release from TFL are:
- Its approach to reporting serious criminal offences.
- Its approach to how medical certificates are obtained.
- Its approach to how Enhanced Disclosure and Barring Service (DBS) checks are obtained.
- Its approach to explaining the use of Greyball in London, software that could be used to block regulatory bodies from gaining full access to the app and prevent officials from undertaking regulatory or law enforcement duties.
Although these are certainly some of the clearer issues that TFL have with the service, there are many that are not mentioned, such as working conditions, the employment status of Uber drivers, and even claims the success of the app has led to an increase in congestion in the city. However, the impact goes beyond Uber and its users - many claim that the optics are a hinderance to innovation.
A headline in the Daily Telegraph read 'In revoking Uber's licence, Khan's message is that London is shut to innovation and business' and this kind of sentiment has been common throughout the media since Friday's decision. However, is the decision really limiting innovation or is it simply controlling innovation to make it safer for the public?
One thing that has become clear over the past year is that Uber is an innovative company, but its innovation has clearly been at the expense of CSR. It has frequently bent or bypassed the rules in order to 'push innovation', but in reality this has been to crush the competition and maximize their profit whilst driving down their costs by putting the burden on individual drivers. There is little doubt that this is a boon to consumer pricing, with fares considerably lower than their competitors due to the decreased cost of leasing the cars needed, the wage paid to drivers and the economies of scale with the number of drivers available. However, much of this has impacted on customer safety as their vetting process is purposefully kept lax in order to get people started as quickly as possible. With 32 allegations of rape and sexual assault against its drivers between February 2015 and 2016 by Uber drivers in the UK alone, it is clear that this isn't working.
The company has also frequently failed to follow rules from regulators across the world, bringing them into conflict with local authorities. For instance, Uber began testing self-driving cars on California public roads without seeking the relevant permits, which led to widespread condemnation. Other companies like Mercedes, Google, and Ford had all needed to pass extensive tests and checks to gain these permits, but Uber believed that they could just do it without them. Anthony Levandowski, vice president of Uber's advanced technologies group, argued that 'Tesla's autopilot technology does not need them' so why should Uber? Which is slightly disingenuous. They also flouted a ban in Germany and continued to operate despite the service being barred. When the company is willing to break fairly important laws like these, it should not be a surprise that the eyes of other regulators falls more harshly on them than other companies.
The key to their success has been innovation, doing things before others. However, much of the early adopter advantage they've held has come from ignoring the rules and giving themselves the best opportunity to build a following that meant stopping would have a negative impact on those making the decisions. In London it is clear that Uber has done a large number of questionable things, such as the Greyball software and a lack of effective driver vetting, which for most companies would cause uproar, but due to the prevalence of Uber most customers don't care or their concern doesn't outweigh the benefits. This is why over 750,000 of Uber's London customers have signed a petition against the ban - the service is great for them, even if its bad for many others. This kind of public pressure makes it difficult for regulators and government leaders to come out against Uber, they're so popular they could easily sway elections. Given that Sadiq Khan's winning margin in the last Mayoral election was 315,529 the number of people signing this petition is more than double and could therefore be a decisive issue in a mayoral election in the future.
Although regulations do often exist purely to protect industry incumbents, this is not the case in many of the situations that Uber finds itself, where regulations exist for important reasons. They are not anti-innovation, they are pro-safety. Innovation and disruption are a positive force in the world, but only when they're done properly and obey the rules in place to make sure they have a positive outcome. This decision has not been made to tell the world London is closed to innovation, what is saying is that companies cannot run free. They are saying innovate, but don't cut corners.
How Idea Management Drives Tangible Employee Engagement
How to Create a Strategy to Retain Existing Clients
Could the Growth of 5G Finally Trigger the Extinction of Analog Phones?
Amazon latest delivery drone demoed in Las Vegas
Why companies should take advantage of mobile technology
Chevron and Occidental invest in CO2 tech
Rolls-Royce partners with AI company to support its aero-fleet engines
Intel's new AI wheelchair technology helps individuals regain autonomy
Amazon aims high with its cashierless tech system
Infographic: Payment trends and disruptors
Xiaomi IoT platform to support IKEA's smart light products
AWS ML training courses to be made free to access
UK government's attitude to RPA adopts "massive shift" toward implementation
Innovation Is As Much About Environment As Ideas


text:
U Przybyszewskiego
Stanisława Przybyszewskiego, Krowodrza
Apartments for rent - U Przybyszewskiego
U Przybyszewskiego:
Rental (1) Sale (1)
List your property in U Przybyszewskiego
About investment U Przybyszewskiego is an intimate investment that provides comfort and a sense of privacy. The area of the estate is available only to residents, and additional securi...
About investment
U Przybyszewskiego is an intimate investment that provides comfort and a sense of privacy. The area of the estate is available only to residents, and additional security is ensured by 24-hour monitoring. There are also attractive common areas, such as a TV room, table football, darts and billiards, as well as a private gym, sauna and a safe playground for children.
The advantage of the investment is its location close to the center, with a quiet environment in a quiet and green part of the city at the intersection of prestigious districts - Krowodrza, Wola Justowska and Stare Bronowice. In close proximity to the estate, there are modern office buildings and a business center, universities, sports facilities and places for recreation. Nearby, there are numerous public transport stops, as well as Błonia, Jordana Park, ZOO and the AGH swimming pool. Nearby, there are also walking areas with lush greenery and beautiful views - Wolski Forest and Kościuszko Mound.
The nearby Armii Krajowej Street allows you to quickly get to the center of Krakow, as well as from the city to the A4 motorway or Balice airport. Around the estate, there is a full range of service and catering premises, medical facilities, as well as kindergartens and schools. In the vicinity available, among others Galeria Bronowice and a modern fitness club.
The buildings have three floors, with apartments ranging from studios to 4-room apartments. The apartments include balconies or gardens, as well as storage rooms. In the U Przybyszewskiego complex, each resident will find his own quiet and convenient place to live.
Stanisława Przybyszewskiego
Krowodrza
Krowodrza, Stanisława Przybyszewskiego
240 PLN/month
Guide to Krowodrza
Krowodrza - an atmospheric district beloved by young residents, full of intimate, modernist buildings.
Explore the district that surrounds the Old Town from the north and connects to the green West-side through the spectacular National Museum.


text:
About N-IUSSP
Alaka Basu
Alaka M. Basu is Professor, Development Sociology, Cornell University, and a member of the Guttmacher-Lancet Commission on Sexual and Reproductive Rights and Health Alessandro Rosina
Professor of Demography and Director, Center for Applied Statistics in Business and Economics, Università Cattolica del Sacro Cuore Andrea Brandolini
Head of Statistical Analysis Directorate, Bank of Italy
Bruno Masquelier
Professor of Demography, University of Louvain, Belgium Cheikh Mbacké
Associate Professor, Sociology department, Laval University Cinzia Conti
Researcher at Istat, Head of Unit on Foreign Presence and Social Dynamics
Corrado Bonifazi
Director of the Institute for Research on Population and Social Policies, National Research Council, Rome Italy Ernestina Coast
Associate Professor of Population Studies, London School of Economics Feng Wang
Professor of Sociology at the University of California, Irvine, USA, and Professor at Fudan University, Shanghai, China fwang(at)uci.edu
Francesco Billari
Professor of Sociology and Demography, Department of Social and Political Sciences, Bocconi University. • Personal webpage • francesco.billari(at)unibocconi.it Gilles Pison
Professor at Museum National d'Histoire Naturelle and Director of Research at the French National Institute for Demographic Studies (INED) (Paris) Gustavo De Santis
Professor of Demography, University of Florence, Italy
Jacques Vallin
Emeritus Research Director at INED, Paris; Honorary President of IUSSP John Knodel
Research Professor Emeritus, Population Studies Center, Institute for Social Research, University of Michigan (USA) and International staff, College of Populations Studies, Chulalongkorn University (Thailand) Letizia Mencarini
Associate professor of Demography, Bocconi University - Dondena Centre for Research on Social Dynamics and Public Policy & Collegio Carlo Alberto; P.I. ERC P.I. ERC project n. 313617 (2013-2018) SWELLFER http://swellfer.wordpress.com
Letizia Tanturri
Associate Professor of Demography, University of Padova, Italy Massimo livi Bacci
Emeritus Professor of Demography, University of Florence, Italy Monica Das Gupta
Research Professor of Sociology, University of Maryland, USA
Paula Miranda-Ribeiro
Professor, Demography Department and Cedeplar, Universidade Federal de Minas Gerais, Brazil. Peter McDonald
Professor of Demography in the Australian National University. Honorary President of IUSSP and winner of the Irene B. Taeuber Award
Roberto Impicciatore
Università di Bologna. Salvatore Strozza
Professor of Demography, University Federico II, Naples (Italy) Stefano Molina
Senior Program Officer, Giovanni Agnelli Foundation, Italy
N-IUSSP
N-IUSSP is a new IUSSP news magazine, which will disseminate scientific findings from demographic research carried out all over the world. The practical implications of current trends, the risks and potentialities of emerging situations, the pros and cons of specific laws are discussed in rigorous but plain language.
Everybody is free to reproduce our articles, for free, provided the original source is cited.
You are invited to contribute to this new publication: please check our guidelines and submit your 1000 word contribution to contact@niussp.org
March 23, 2020 article
Why has migrant fertility in Norway declined?
Comment expliquer le recul de la fécondité des migrants en Norvège ?
Marianne Tønnessen
Fertility among immigrants has declined in many Western countries. In Norway, Marianne Tønnessen finds that the successful integration of immigrants is not the main driver of the decline since 2000, but rather decreased fertility in origin areas.
In many Western countries, the total fertility rate (TFR) of immigrant women has declined over the last decades (Figure 1). In Norway, for instance, immigrant TFR fell from 2.6 children per women in 2000 to less than 2.0 in 2017. This may be due to several factors, such as immigrants' adaptation to the fertility norms at destination, changing composition of immigrant women by area of origin, and other factors.
In a recent study, I try to disentangle the different possible causes of this decline (Tønnessen, 2019). Like many other Northern European countries, Norway has relatively high fertility (ranging between 1.98 and 1.62 in the study period) and an increasing share of immigrants (from 5.3 per cent in 2000 to 13.8% in 2017) from all over the world and, in the last decade, from Eastern EU countries like Poland and Lithuania in particular (Statistics Norway 2020a,b).
Newly arrived family migrants explain a lot
The results show that in line with findings from many other countries (see for instance Sobotka 2008), immigrant women from regions with high fertility (such as Africa and parts of Asia) often have higher fertility in Norway than women from low fertility countries. Also, immigrant women's fertility often declines with time since arrival in the destination country, probably because of integration, so that, everything else equal, women who have been living in Norway for a long time have lower fertility than those who arrived only recently. For instance, in 2000, newly arrived immigrants (0-2 years since migration), from Africa and Asia had a TFR above 4 children per women, while women from the same regions who had entered the country more than ten years earlier had a TFR of around 2. These trends are not particularly surprising, of course, nor is the considerably lower fertility – about 2 children per woman – of women from the European Union or the US.
However, the decline in immigrant TFR in Norway from 2000 is due neither to an increased share of immigrant women from low fertility countries, nor to an increased share of women with long duration of stay. It is linked, instead, to the fact that the fertility of recent immigrants to Norway is considerably lower than it was 20 years ago. For instance, while in 2000 the fertility of newly arrived women from Eastern and South Eastern Asia was above 4 children per women, in 2017 it was below 2. The same holds for newly arrived women from Latin America (from 3.5 in 2000 to 2.0 in 2017) and from Western and Southern Asia (4.5 to 3.5).
Decomposition methods make it possible to quantify the effect of changes in composition by duration of stay and area of origin, and the effect of changing fertility within subgroups (by duration of stay and reason for migration). The results (Table 1) show that lower fertility among newly arrived immigrants explains 93% of the overall decline, and that this is mainly driven by women from Asia.
I further analyzed this fertility decline among newly arrived women by reason for migration. Women who migrate for family reasons appear to provide a key: their share among all newly arrived immigrant women declined in this period – for instance, from above 80% in 2000 to 55% in 2017 among women from Eastern and Southern Asia – and so did their fertility (see Figure 2). Among newly arrived family migrants from Asia, TFR fell by more than two births per woman from 2000 to 2017.
Further decompositions show that the lower TFR of newly arrived family migrants accounts for a large share of the total TFR decline among all immigrants in Norway. Among newly arrived immigrants from Western and Southern Asia only, lower fertility among family migrants accounts for 10% of the overall TFR decline, and among those from Eastern and South East Asia it accounts for 8%. Hence, decreased fertility among these two groups alone accounts for 18% of the TFR decrease of all immigrant women in Norway. This is a large effect from a relatively small group; at end-2017, they represented just 3% of all immigrant women of childbearing age (5% in 2000).
A large part of this fertility decline among newly arrived family migrants from non-western parts of the world may reflect fertility declines in origin due to factors such as family planning programs or more education for women. Such factors may, in turn, affect both the share of women who migrate for family reasons and the fertility of the family migrants. And although migrant women may be a select group compared to those who remain in origin, they may have been affected by the same secular changes in fertility norms and behaviors.
Fertility has declined in many origin areas
Fertility has declined substantially in many non-western countries. Such origin area trends are sometimes overlooked in studies of immigrant fertility. Although the newly arrived immigrant women grew up in the same origin areas as those who moved to Norway one or two decades ago, they grew up in a different time, with different fertility norms and patterns.
The fact that the decreased immigrant TFR in Norway is mainly driven by lower fertility among newly arrived women, possibly reflecting declining fertility trends in their countries of origin, may remind migration researchers to look for explanations of changes in immigrant fertility beyond the destination country and the characteristics of individual migrants.
This can also be a reminder for policy makers and others not to draw too hasty conclusions about the effect of domestic policies by looking at trends in immigrant TFR. Although an immigrant woman's fertility often declines with her duration of stay, thanks to successful integration for instance, this does not necessarily translate into a declining TFR for all immigrants, unless the share with long duration of stay increases.
The results of my research also point to the future: if fertility change in origin areas is a driver of fertility decline among many non-western newly arrived migrants, and if fertility continues to fall in important origin areas – as projected by the United Nations for high-fertility parts of the world (United Nations 2019) – we may expect further fertility declines among immigrants from these areas. Moreover, policies affecting fertility preferences in high-fertility parts of the world may, in turn, affect the fertility of Western countries' own immigrant populations.
Sobotka, T. (2008). The rising importance of migrants for childbearing in Europe. Overview Chapter 7. Childbearing Trends and Policies in Europe. Demographic Research, Special Collection, 7, 225-248.
Statistics Norway (2020a). Births. Statistics from retrieved February 2020.
Statistics Norway (2020b). Immigrants and Norwegian-born to immigrant parents. Statistics from retrieved February 2020.
Tønnessen, M. (2019). Declined Total Fertility Rate Among Immigrants and the Role of Newly Arrived Women in Norway. European Journal of Population.
United Nations (2019). World population prospects: The 2019 revision. New York: Population Division of the Department of Economic and Social Affairs of the United Nations Secretariat.
Demografische Forschung (German)
Demos (Dutch)
Demoscope Weekly (Russian)
Neodemos (Italian)
Population and Societies (English and French )
Population Europe (English)
PRB Data Sheet (English)
The rise of divorce, separation, and cohabitation in the Philippines
La montée des divorces, séparations et cohabitations aux Philippines
Jeofrey B. Abalos
Les politiques démographiques sont-elles efficaces ?
On the dubious effectiveness of population policies
The demographic dividend is driven by education, not changes in age structure
Le dividende démographique est impulsé par l'éducation et non par l'évolution de la structure par âge
Wolfgang Lutz
The impact of migration on the fertility of receiving countries
L'impact de la migration sur la fécondité des pays d'accueil
Christos Bagavos
Modernizing the census in Europe: traditional and new methods for the 2020 round
Modernisation du recensement en Europe : méthodes classiques et nouvelles pour la série 2020
Paolo Valente
Breaking down the decline in fertility and births in China
Une décomposition du recul de la fécondité et de la natalité en Chine
Quanbao Jiang
Demographic change and aging without family: a global perspective
Vieillir sans famille dans un contexte d'évolution démographique : perspectives mondiales
Rachel Margolis
The world's next 4 billion people will differ from the previous 4 billion
Quatre milliards de personnes en plus. Mais différentes
David Lam
In Australia fertility is falling only for low educated women
En Australie, la fécondité baisse uniquement chez les femmes peu scolarisées
Peter McDonald
Demography and development in the maze of Indian districts
Démographie et développement des districts indiens
Sanjay K. Mohanty
REPRODUCTION POLICY
If you want to get in touch with us write an email to contact@niussp.org
ISSN 2704-7067 – N-IUSSP – www.niussp.org
Women's employment 1996-2016: US vs Europe]Emploi des femmes 1996-2016: les États-Unis comparés à l'Europe
What becomes of marriage when non-marital childbearing is the norm?]Quel avenir pour le mariage lorsque les naissances hors mariage deviennent la norme ?
Female genital mutilation/cutting in Africa: ineffective policies and persistent inequalities]Mutilations génitales féminines / excision en Afrique : politiques inefficaces et inégalités persistantes
Migrant children's schooling in urban China]Scolarité des enfants migrants en Chine urbaine
Grandmotherhood and retirement in Italy]Devenir grand-mère et prendre sa retraite en Italie
Caterina Livi Bacci
Editing – graphics – web design
Filippo Bonechi & Francesco Cipriani
Web developer – database project
Editore – Neodemos, via A. Baldesi 18-20, 50131 Firenze
© 2019 N-IUSSP
Powered by WordPress | Theme Designed by: website | Thanks to website, find out more and website


1. The number of recorded incidents relating to Knives or other sharp implements.
2. I would like this report broken down into months for 2016, 2017 and to date in 2018.
Data has been collated through a search of all crimes created between 01/01/2016 and 15/04/2018 where the weapon attached to the crime is one that is sharp and capable of piercing the skin. Weapons included are in accordance with Home Office counting rules for Knife Crime.


text:
NCAA Tournament Thoughts
Well... this was easily my worst year in projecting the field (something I thought might happen as I evaluated teams this week). I do not think the selection committee did a great job this year. As I have always said to me the most important quality in an NCAA tournament team is a proven ability to beat other tournament teams. Cal, Utah State and UTEP all failed to prove they could that, while Illinois proved they could by beating 4 top 25 RPI teams, 3 of which were away from home. To me that is an NCAA tournament team. My other major issue was Florida getting in over Mississippi State team that beat them on a neutral floor a two days earlier, then beat Vandy, then took UK to OT and really lost on a bad bounce. Now I would have been fine with Mississippi State not getting in, but Florida getting in over them does not make any sense at all. Their resume's are essentially the same, but Mississippi State was better late and again beat them on a neutral floor.
I will say though that no one that missed really has anyone to blame but themselves. Mississippi State did not do enough throughout the year, Illinois had some bad losses, VT had an awful non-conference schedule. In the end I may disagree with the committee but we are not debating teams that would have a shot to win the whole thing... so I am good with how it is.
The biggest issue I had was Duke being treated like the number 1 overall seed. Duke was easily the weakest of the 1 seeds and did not have as good a resume as WVU. Duke was given the weakest 2 seed in their region. They were also given a 4 seed missing their best player. Duke does draw the toughest 8/9 winner (unless Texas suddenly starts playing to their ability), but other than that they were given a great draw.
WVU who had 5 top 25 RPI wins to Duke's 1 and was 7-4 on the road vs. Duke who was 5-5 ended being a 2 seed and was put into Kentucky's region making a final four run that much more difficult. Had they put WVU in Duke's region they'd be a favorite to make the final four. The good news for WVU is they may have gotten some favors at the bottom of the region being matched up with New Mexico as the three seed (a team they should beat if New Mexico were to get that far). It seems that the committee felt it was more important to give them the Buffalo/Syracuse locations than to match them with the weakest 1 seed.
Another team I think got jobbed was Ohio State. Why did Ohio State get put in the bracket of the top 1 seed? I guess it is as simple as keeping OSU in St Louis for the the regional finals instead of sending them to Houston...but if you are Ohio State would you rather be in St Louis in a Region with Kansas and Georgetown (probably the best 1 and 3 seeds respectively) or be in Houston with Duke and Baylor as your chief rivals. On top of that the Oklahoma State v Georgia Tech winner is going to be an absolutely rough second round match-up for the Buckeyes. To me they have far and away the toughest path of any of the high seeds to the final four.
Now lets take a look at the first round match-ups I am most looking forward to:
Thursday 12:20, 7 BYU vs. 10 Florida- 7/10 games always tend to be close. I really like this BYU team. Florida has struggled a bunch down the stretch. I do not think they should have made the tournament, that being said they could easily win this game.
Thursday 12:25, 6 Notre Dame vs. 11 Old Dominion- Notre Dame was one of the hottest teams in the country to end the season. Old Dominion won a very competitive Colonial Athletic Association. Four years ago George Mason made an historic final four run from a similar position. Obviously that is a huge unlikelihood, but a Sweet 16 run would not be surprising at all.
Thursday 2:30, 4 Vanderbilt vs. 13 Murray State- Really all the 13/4 games have a certain level of intrigue to them this year. I think this is one of the better sets of 13 seeds I can remember in the tournament and the four seeds they have been matched with are not really teams that seem to blow others out. Murray State was absolutely dominant in the Ohio Valley this year (big surprise I know they've been doing it for over 2 decades). Vanderbilt had a great year. They are well coached and very skilled, but I think Murray States quickness could give them trouble. It should be a very interesting matchup and maybe the first big upset of the tournament.
Thursday 4:45, 5 Butler vs. 12 UTEP- I did not pick a single 5/12 upset this year...which will probably prove to be foolish as they happen like clockwork. I think Butler is too good a team for UTEP, but I guess we will see. Anyways the 5/12 games are always worth watching.
Thursday 7:20, 6 Marquette vs. 11 Washington- Marquette is another example of how much respect the Big East got this year. I think Marquette is probably a bit over-seeded. They really did overachieve with the roster they have in place. I think they are better than Washington, but Washington has been hot. I think the winner of this game will likely beat New Mexico and find their way to the sweet 16.
Thursday 7:25, 3 Georgetown vs. 14 Ohio- I think this will probably be a blowout, Georgetown is incredibly dangerous in this tournament. That being said, always look forward to watching the Bobcats.
Thursday 9:45, 6 Tennessee vs. 11 San Diego State- San Diego State is red hot moving into the tournament. I like their chances against Tennessee, but in the end I'm not betting against Bruce Pearl.
Friday 12:25, 6 Xavier vs. 11 Minnesota- I personally think Xavier is much better than Minnesota. However, I also think Tubby Smith is a very good game to game coach and he will have them ready. I thought Xavier got a nice draw for getting back to the sweet 16, but they have to get the first one.
Friday 12:30, 5 Temple vs. 12 Cornell- The Atlantic 10 is very much featured in the 12:30 games. I thought Temple was under-seeded in the tournament this year. I know Cornell is a trendy pick here but I think Temple is a much more athletic and a generally better team. I personally like the Owls to make the sweet 16.
Friday 2:30, 4 Purdue vs. 13. Siena- It is the second of the very intriguing 4/13 match-ups in this year's tournament. Purdue has looked really bad in their two games against tournament competition without Robby Hummel. They have the talent to win without him, but will they have it together by then? Siena brought back four starters from the team that upset Ohio State last year. They are very good and tournament tested. This will be a really good game.
Friday 4:45, 5 Texas A&M vs. 12. Utah State- I think this is the most likely of the 12/5 upsets. Utah State had been very hot before the tournament. Texas A&M is better and should win, but we have seen bigger upsets in 12/5 games before.
Friday 7:15, 7 Oklahoma State vs. 10 Georgia Tech- These are two very talented teams who will give Ohio State a hell of a time in the second round. I will be very anxious to see who wins.
Friday 7:20- 5 Michigan State vs. 12 New Mexico State- I thought New Mexico State was probably seeded to high and it is dumb to bet against Tom Izzo in the Big Dance, but Michigan State has looked very vulnerable and we all know about 5/12.
Friday 9:40- 4 Maryland vs. 13 Houston- The last of the 4/13 match-ups I am very interested in. Houston impressed me in the Conference USA title. Tom Penders I a good coach. Maryland has not been great away from home this year. I think Maryland wins, but it should be a good game.
Friday 9:45- 8 Cal vs. 9 Louisville- The winner of this game is going to be a real threat to Duke. Just from that level of interest I will be paying attention.
Additional Thoughts on the Tournament:
15/16 team that is most likely to keep the game close- Winthrop (they are in the play in game, but screw it I went to school their for a year I'm going with them)
Toughest Draw for a 1-4 seed- Ohio State
Potential Second Round Games to Look Forward to-
Duke vs. UofL/Cal winner
Ohio State vs. Oklahoma State/Georgia Tech winner
Georgetown vs. Tennessee
12 seed or lower that could make the sweet 16- Murray State
Will the Big East get two in the final four again? Yes, out of Syracuse, WVU, Georgetown and Villanova two will find their way to the final four.
Outsider most likely to crash the final four- Baylor (and if you do not consider them an outside because they are from a power conference... Temple)
Biggest X factor in the tournament- The health of Arinze Onuaku.
Texas A& M
Biggest Game
Kansas vs. Syracuse (the winner will be Champion)
Championship Game
Syracuse over West Virginia (I'm assuming Onauku will be good to go...if not...well I'll be wrong)
An Open Letter to Mike Thomas
Final Bracket Selection Sunday
Championship Saturday 2:30 Bracket
Bearcatmark's First Mock Bracket
Championship Week Bubble Look
Ranking the Bubble Part 1


The goal of A.W.A.K.E is to educate and spread awareness for sleep disorders including Sleep Apnea and Snoring to members of our community.
have honored Dian Olah and her team of sleep experts as the Sleep Center for treatment and education in the West Los Angeles area.
If you or anyone you know is suffering from a sleep disorder call us now to reserve your seat at one of our monthly meetings.


Conceived in New Jersey from its emerging DIY scene from the late 90s, cult hop-hop trio DÄLEK were known as pioneers over the last fifteen years or so of an intriguing sound which fused a blend of progressive hip-hop artistry supported by a rhythmic backdrop consisting of sonic experimentation through noise music and dark ambient (they would be dubbed by journalists as what would happen when you mix Public Enemy and My Bloody Valentine). Their singularity would be highlighted in four defining milestone full-length albums released on Mike Pattons Ipecac Recordings (one of them being their Absence masterpiece LP from 2005), along with live actions where the band would support the likes of Tool and The Melvins, and even collaborate with the likes of artists such as Faust. In Spring 2015 mastermind MC Dälek (a.k.a. Will Brooks, with the blessing of ex-member/co-producer Oktopus) revived DÄLEK after a hiatus with a new lineup which now includes DJ rEk and Mike Swarmbots and are now preparing to release their new album Asphalt For Eden which will be the first official DÄLEK release since their last full-length, 2009 s Gutter Tactics . Asphalt For Eden sees a new creative peak for DÄLEK and will bring a new awareness for this cult hip-hop entity now resurrected once again and ready to show a formidable display of true hip hop artistry and experimentation through their new album and the plethora of touring the band have planned in support of it. A band which many see as one of the most genuine underground DIY hip hop iconoclasts.


text:
India, Bahrain call on global community to reject use of terrorism against other countries
In a veiled attack on Pakistan, India and Bahrain on Sunday called on the international community to reject the use of terrorism against other countries as both sides agreed to further enhance cooperation in the field of security and counter-terrorism. During his two-day visit to Bahrain, the first by an Indian prime minister, that concluded on Sunday, Modi held bilateral meetings with King Hamad bin Isa Al Khalifa and prince and Prime Minister Khalifa bin Salman Al Khalifa.
Manama Published on: August 25, 2019 16:48 IST
Prime Minister Narendra Modi with Bahraini Prime Minister Prince Khalifa bin Salman Al Khalifa, in Bahrain on Saturday (August 24, 2019)
In a veiled attack on Pakistan, India and Bahrain on Sunday called on the international community to reject the use of terrorism against other countries as both sides agreed to further enhance cooperation in the field of security and counter-terrorism.
During his two-day visit to Bahrain, the first by an Indian prime minister, that concluded on Sunday, Modi held bilateral meetings with King Hamad bin Isa Al Khalifa and prince and Prime Minister Khalifa bin Salman Al Khalifa.
The two sides exchanged views on bilateral, regional and multilateral issues of mutual interest during the meetings, according to a joint statement.
"Both sides called on all states to reject the use of terrorism against other countries; dismantle terrorism infrastructures where they happen to exist and to cut off any kind of support and financing to the terrorists perpetrating terrorism from all territories against other states; and bring perpetrators of acts of terrorism to justice," the statement said without referring to any country.
India has been pressing Pakistan to end its state policy of backing cross-border terrorism and its support to terror groups on its soil.
India and Bahrain also discussed ways and means to further promote cooperation in cyber security, including prevention of use of cyber space for terrorism, radicalisation and for disturbing social harmony.
The two sides agreed to further enhance cooperation in the field of security, counter-terrorism and the exchange of intelligence and information, it said.
The leaders also noted the "need for concerted action by the international community against terrorism" and "underlined the importance of comprehensive sanctioning of terrorists and their organisations by the UN".
They also said that regional connectivity projects should be based on international law, including respect for sovereignty and territorial integrity of states, in an apparent reference to China's push for its Belt and Road Initiative (BRI).
During the meetings, the two sides also noted the increasing trend of bilateral trade and existing potential for two-way investments and reaffirmed their desire to provide favourable environments for trade and investments.
The two countries agreed to focus on enhancing cooperation in energy, covering the areas of joint exploration and training of human resources.
The two leaders agreed to enhance cooperation to strengthen maritime security in the Gulf region, vital for the security and safety of sea lanes.
The two sides underlined the importance of enhancing high-level political exchanges, defence and higher education cooperation, trade and economic relations and people-to-people linkages.
The leaders also noted the increasing trend of bilateral trade and existing potential for two-way investments and reaffirmed their desire to provide favourable environments for trade and investments.
India also welcomed the interest of Bahrain to join the International Solar alliance (ISA), recognising the importance of it as international intergovernmental organisation, which has become an effective platform to facilitate the deployment of solar energy.
Both the sides also emphasised on the urgent need to pursue UN reforms, including the Security Council through an expansion of both categories of its membership, to make it more representative and effective in tackling global challenges.
The two countries also signed MoUs on cultural exchange programme, space technology, solar energy, the statement said.
An MoU was also signed between BENEFIT, a Bahrain company handling ATM and POS transactions among others, and National Payments Corporation of India (NPCI) for launch of RuPay card in the Gulf Kingdom, it said.
During the official meetings, the two sides also underlined their close bilateral ties, deep-rooted in shared history and cultural affinities, sustained and nourished through growing economic linkages and close people-to-people contacts.
AUS vs IND | Mohammed Siraj racks up milestones with maiden Test five-wicket haul - In Numbers
Horoscope Today, Jan 18: Scorpio people will get full support of luck, know about other zodiac signs
Oppo Reno5 Pro 5G with 8GB RAM launched in India: Price, specifications
Saif Ali Khan starrer 'Tandav' makers apologize for allegedly hurting religious sentiments
Subhash Ghai announces his next production venture '36 Farm House'
UK shoot of Neena Gupta & Kalki Koechlin's 'Goldfish' postponed due to new COVID strain
Bengal polls 2021: Mamata Banerjee to contest from Suvendu's stronghold Nandigram
Farmer unions bury differences ahead of 10th round of talks, Chaduni to be part of delegation
Stones pelted at BJP roadshow in Kolkata; 'these tactics won't work', says Suvendu Adhikari
Rafale to feature in Republic Day parade for first time, will carry out 'Vertical Charlie' formation
Will Vijay Mallya be extradited soon? Here's what Centre told SC
Gold Rate Today: Gold price rises Rs 117; silver jumps Rs 541
Sensex tanks 470 points; Nifty drops below 14,300
Drawn series will be worse than loss to India a couple of years ago: Ponting
AUS vs IND | Starc gives Australia injury scare ahead of final day in Brisbane
Australian Open: Tennis players getting on with life in Melbourne
Hope my father could have been there to watch this day: Emotional Siraj after maiden fifer
XAT Result 2021 Declared. Direct link to download
Education Minister's big hint for board examinations, JEE, NEET
ICSI CS Foundation Result 2020 Declared; CSEET January released. Get direct link
Scholastic India launches new language improvement module for kids
JEE Main 2021: NTA extends last date for registration till January 23. Check new deadline
Vastu Tips: Get stairs, elevators constructed in THESE directions in hotel


SKU: 636391205986. Availability: 48 in stock Size: N/A Shipping Weight: 0.2000 lbs Categories: By Shop, Intrauterine Insemination Kits.
The TenderNeeds Fertility Professional Series Insemination Device is our premium device. We are the exclusive provider of this Professional Series device in the United States, United Kingdom, Australia, and Canada. The Professional Series Semi-Flexible/Rigid Intrauterine Insemination (IUI) Device is intended for human use, unlike many of our competitors whom only pass veterinarian use devices for women to use. Our Professional Series device is designed and intended for a woman to use. This kit is user friendly, using a semi-rigid shaft to assist getting the softer tip to the cervix. It also has a safety stop to prevent or reduce over insertion. If you are concerned about how far to insert the insemination device, this professional series kit is fairly foolproof and comfortable for you to use.


text:
\section{Introduction}
Deep reinforcement learning over the years has made considerable advancements with applications across a variety of domains -- from learning to play Atari 2600 suite from raw visual inputs \cite{mnih2015human}, mastering board games \cite{silver2017mastering,schrittwieser2019mastering}, learning locomotion skills for robotics \cite{schulman2015high,schulman2015trust,lillicrap2017continuous}, mastering Starcraft \cite{vinyals2019grandmaster}, the development of Alpha Fold \cite{evans2018novo} to predict the 3D structure of a protein and most recently, the improvements in model-based \cite{kaiser2019model} and off-policy reinforcement learning \cite{hessel2018rainbow,haarnoja2018soft}
\\
Nonetheless, it is a challenging task to create a single agent that performs well, is sample efficient and is robust. There are considerable hurdles surrounding training and optimization of deep reinforcement learning algorithms such as the sensitivity  to hyper-parameters, the high variance associated with the learning process and high sample complexity. In order to overcome these problems, we exploit the well-known concept of ensemble learning \cite{dietterich2000ensemble} and adapt it for reinforcement learning in a novel way. Traditionally, the idea of using ensembles in reinforcement learning settings is associated with combining multiple value functions or policies from different agents. These agents could be the same algorithm trained across different hyper-parameter settings, generated by different algorithms altogether \cite{wiering2008ensemble,duell2013ensembles,fausser2015neural} or by training multiple networks of an agent in parallel \cite{lee2020sunrise}. Training multiple such agents is an approach that cannot be used in practice owing to high sample complexity and computational expense.

We tackle this problem by creating sufficiently diverse policies from a single training run. The policies are generated in a serial manner, one after the other, where the subsequent policies take into account the diversity from earlier policies. Our framework is seen to achieve state-of-the-art (SOTA) performance on popular reinforcement learning domains.
\\
 Our approach to sequentially generate  policies is inspired by the recent developments  in the deep learning literature studying the effects of learning rate schedules and their impact on generalization \cite{li2019towards,nakkiran2020learning}. It is shown that learning rate annealing generalizes better than using a small constant learning rate and high initial learning rate impacts the model's selection of the local minimum to converge \cite{li2019towards, jastrzebski2020break}. We leverage these properties of neural network training to learn a diverse ensemble of policies.
The diversity among the policies are obtained by the directed perturbation of the model weights at regular intervals. The directed perturbation is induced by sudden and sharp variations in the learning rate, and for doing so, we employ cyclical learning rates \cite{loshchilov2016sgdr}. When the model weights are perturbed using larger learning rates,  the directed motion along the gradient direction prevents the optimizer from settling in any sharp basins and directs the model into the general vicinity of a local minima \cite{li2019towards}. Annealing the learning rates during training leads the optimizer to converge to some local minima and improves generalization \cite{nakkiran2020learning}. We leverage the diversity of the policies learned at these different local minima for the ensemble. We also show through experimentation that directed perturbation and not random perturbation is necessary for obtaining diverse policies. We also empirically show that an extremely diverse set of policies do not form a good ensemble.
\\
In order to prevent bias from sub-optimal policies in the ensemble, we introduce a novel framework that selects the best subset of policies to include in the ensemble. Our approach uses the trajectories obtained during training the policies to find this subset.
\\
Since we use models from a single training run instead of training $M$ different models independently from scratch, we refer to our approach as Sample Efficient Ensemble Reinforcement Learning (SEERL).
 To summarize, our main contributions are:
\begin{itemize}
    \item A sample efficient framework for learning $M$ diverse models in a serial manner from a single training run with no additional computation cost. The framework can adopt any existing reinforcement learning algorithm to create an ensemble
    \item A novel optimization framework to select the best subset of policies for the ensemble. It acts as a filter to choose the best performing and diverse subset of policies without using any additional samples from the environment.
    \item Evaluation of various ensemble strategies for discrete and continuous action spaces with SOTA performance across multiple environments
\end{itemize}
We demonstrate the effectiveness of SEERL for discrete (Atari 2600 \cite{bellemare2013arcade}) and continuous control benchmarks (Mujoco \cite{todorov2012mujoco}). Our experiments show that SEERL consistently outperforms popular model-free and model-based  reinforcement learning methods including those that use ensembles.

\section{Related work}
There has recently been a small body of work on using ensembles for reinforcement learning, primarily the use of ensembles during the training phase to reduce the variance and improve robustness of the policy.
\\
Value function based methods such as Averaged DQN \cite{anschel2017averaged} train multiple Q networks in parallel with different weight initialization and average the Q values from all the different networks to reduce variance. It results in learning policies that are much more stable. However, the approach requires training multiple networks simultaneously and a possibility that the model might diverge if either of the Q values being averaged is biased.
\\
Bootsrapped DQN \cite{osband2016deep} uses an ensemble of Q functions for efficient exploration but does not enforce diversity in the Q networks. The Q head is chosen at random and used to generate actions for that episode. The multiple heads are also trained in parallel while we train our models in a serial manner.
\\
SUNRISE \cite{lee2020sunrise} also uses an ensemble of Q networks that are randomly initialized and each head trained with a different set of samples to stabilize learning and enforce diversity. The actions are selected by examining the Upper Confidence Bound(UCB) of the Q values and choosing the action with the highest upper confidence bound to allow for exploration. However, the framework does not account for bias from the different Q heads which in turn would affect the UCB. Our policy selection framework prevents such biases by filtering out such models during evaluation. Additionally, SUNRISE works only where the base learner is an off policy algorithm whereas our framework works with both on-policy and off-policy methods.
\\
Earlier works \cite{wiering2008ensemble,duell2013ensembles,fausser2015neural,fausser2015selective} explore the idea of value function ensembles and policy ensembles during evaluation phase. However, value function ensembles from different algorithms trained independently could degrade performance as they tend to converge to different estimates of the value function that are mutually inconsistent. An alternative method \cite{marivate2013ensemble} tries to tackle this problem by having a meta-learner linearly combine the value functions from different algorithms during training time to adjust for the inherent bias and variance. Although training multiple policies or value functions in parallel is sample efficient, it tends to reduce the diversity among them and is computationally expensive.
Our method combines the best of both approaches and improves the performance of the algorithm by balancing sample complexity with the computational expense.
Training deep neural network architectures with cycling learning rates \cite{smith2015no,loshchilov2016sgdr} and using ensembles in supervised learning settings, \cite{huang2017snapshot} have shown to be useful. The authors \cite{huang2017snapshot} show that in each cycle, the models obtained are comparable with those learned using traditional learning rate schedules. Even though the model is seen to degrade in performance temporarily, the new model surpasses the previous one, as the learning rate anneals.

\section{Preliminaries}
Reinforcement learning is associated with sequential decision making and involves the interaction of an agent with an environment. In this paper, we consider a discrete-time finite-horizon Markov Decision Process(MDP) defined by (${S}$, ${A}$, ${P}$, ${r}$, $\gamma$, $\rho$), where ${S}$ denotes the state space, ${A}$ denotes the action space, ${P}\equiv P(s'|s,a) $ the transition function, $\rho$, the probability distribution over the initial states, $r(s,a)=\mathbb{E}[R_{t+1}|s_t,a_t]$, the reward function with $R_{t+1}$ being the scalar reward at time $t+1$ when the agent in state $s_t$ took the action $a_t$,  and $\gamma \in (0,1)$ the discount factor. The policy dictates the behavior of an agent at a particular state in an environment. More formally, a policy is defined by $\pi :{S} \rightarrow \mathcal{P}({A})$ where $\mathcal{P}({A})$ denotes the probability distribution over actions $a$ $\epsilon$ ${A}$. The objective of the agent is to maximize the discounted return $G_t = \Sigma_{i=t}^{T}$  $\gamma^{i-t}$ $R_i$

\section{SEERL}
In this paper, we propose SEERL, a novel framework for learning and selecting an ensemble of diverse policies obtained from a single training instance. Unlike supervised learning, where the dataset can be reused for training different models, training multiple RL agents on the same trajectories can result in a lack of diversity among the policies. Instead, multiple RL agents can be trained independently for an ensemble but would suffer from high sample complexity.
If each agent requires $N$ number of samples and the computational expense for training a single agent is $C$, then training $M$ agents independently require $M \times N$ samples and $M \times C$ computational cost. If trained in parallel, only $N$ samples are required, but the computational cost remains at $M \times  C$.
Though training agents in parallel is a possible solution to tackle sample complexity, it is computationally expensive and limits the diversity among the learned policies since every policy observes the same state.
\\
SEERL follows a two-step process-learning policies and policy selection, summarized in Algorithm \ref{alg:algorithm}.
Learning policies involves saving $M$ policies during training at periodic intervals when the learning rate anneals to a small value.
This is followed by policy selection that finds the best subset of $m$ policies from the entire set $M$. Policy selection uses an optimization framework that uses the $N$ samples obtained during training to find the $m$ policies.
These policies are then used as an ensemble during the evaluation phase.
Thus SEERL produces $m$ models for the ensemble, requiring only $N$ number of samples and a computational expense of $C$. In the later sections, we empirically verify that the policies from different local minima are diverse in nature.

\begin{figure}[t]
\centering
    \includegraphics[width=0.8\linewidth]{figures/clr.png}
    \caption{ Cyclical cosine annealing learning rate schedule. $\alpha_0$ is set at $0.05$, number of models $M=5$ and training timesteps $T=1000000$ }
    \label{clr}
    \Description{Cyclical cosine annealing learning rate schedule}
\end{figure}

\subsection{Learning policies}
 Learning rate annealing \cite{nakkiran2020learning, li2019towards} is shown to improve generalization and the use of high initial learning rate \cite{jastrzebski2020break} determines the local minima for the model.
 We use cosine cyclical annealing learning rate schedule \cite{loshchilov2016sgdr} to introduce learning rate annealing and learn multiple policies, as shown in Figure \ref{clr}. Depending on the number of time-steps needed to train the agent, and the number of models needed for the ensemble, the learning rate schedule can be calculated.
\\
As the learning rate anneals to a small value, the model converges to a local minimum, and we obtain the first policy. By increasing the learning rate, the model is perturbed along the gradient direction and dislodged from its local minima. In other words, if $M$ models are required, we split the training process into $M$ different training cycles where in each cycle the model starts at a high learning rate and anneals to a small value. The high learning rate is significant as it provides energy to the policy to escape the local minima and the learning rate annealing traps it into a well behaved local minima. The annealing helps in generalization and converging to a local minima while the directed perturbation re-orients the model to converge to a different local minima. The formulation is as follows:
\begin{eqnarray}
\label{eqn 1}
\alpha(t)=\frac{\alpha_0}{2}\left(\cos\left({\frac{\pi \mod{(t-1,\lceil T/M \rceil)}}{\lceil T/M \rceil}}\right)+ 1\right)
\end{eqnarray}%
where $\alpha_0$ is the initial learning rate, $t$ is the time-step, and $T$ is the total number of time steps for which the agent is trained, and $M$ is the number of models.

\subsection{Policy selection}
\label{section:4.2}
In order to avoid bias from the more inferior policies, the best $m$ policies should ideally be selected for the ensemble.
At the same time, the policies also need to be diverse to obtain good performance in different parts of the state space when used in an ensemble. Near identical policies would not yield much improvement in an ensemble.

\subsubsection{\textbf{Framework} :}
We propose an optimization framework to select the best subset of policies. The formulation has two parts, a policy error term and a Kullback-Liebler (KL) divergence term indicative of diversity. Only optimizing for the policy error term would result in the selection of policies with excellent performance. The addition of the KL divergence term helps balance the requirements of performance and diversity.  We have a hyper-parameter $\beta \in [1,2)$, that balances between diversity and performance. The KL divergence is calculated based on the action distribution between the two polices over the state space. The formulation is as follows :
\begin{equation}
\label{eqn 2}
J(w)=\sum_{s \in S}{\Bar{P}(s)}\left[\sum_{i \in M} w_i B_{i}(s) \right]^2
\end{equation}
\begin{equation}
\label{eqn 3}
\resizebox{0.91\linewidth}{!}{$
    \displaystyle
B_{i}(s)=\left(\sum_{a \in A} L(s,a) -\frac{\beta}{M-1} \sum_{k \in M, k \neq i} \text{KL}(\pi_i (s) || \pi_k(s))\right)
$}
\end{equation}
with the following constraints $\sum_{i \in M}w_{i} =1, w_{i}\geq 0   \forall i $.
$S$ is the set of states and $\Bar{P}(s)$ is the probability of observing a particular state. $L(s, a)$ is the weighted error associated with the model and is indicative of the performance of the model. We weigh the loss in a manner that we give more weight when the loss is above a certain threshold value, and the action taken by the model, $a$, matches the action taken by the ensemble $a_e$. We formalize $L(s,a)$ as follows :
\begin{equation}
\resizebox{0.91\linewidth}{!}{$
\label{eqn 4}
    \displaystyle
  L(s,a)=\left\{
  \begin{array}{@{}ll@{}}
    1, & \text{if} \ |L'(s,a)| \geq T_{err}\ \textbf{and}  \ |a-a_e|<\epsilon,\ \text{a is continuous}  \\
        1, & \text{if} \ |L'(s,a)| \geq T_{err}\ \textbf{and}  \ a=a_e, \ \text{a is discrete} \\
    0, & \text{otherwise}
  \end{array}\right.
  $}
\end{equation}
$L'(s, a)$ is the total error for a particular state-action pair. While the overall process is algorithm agnostic, the details of the total error depend on the actual RL algorithm adopted. $T_{err}$ is the threshold value that is used to distinguish the right actions from the wrong. If $L'(s, a)$ is above the threshold error and the actions match, then we would like to consider that action from the policy as a wrong one.
\\
In a discrete action space, it is relatively easy to determine if the action taken by the model and ensemble are the same.
However, in continuous action space, the final ensemble action and the action taken by the policy might not coincide. Therefore, we introduce a  $\epsilon$ bound on the ensembled action. If the action from the model is within a $\epsilon$ distance from the ensembled action, we consider it as a match. $\epsilon$ ranges between 0.005 to 0.01, depending on the environment.
\\ We use a squared loss formulation to capture the inter-dependencies among the policies. Instead, if the degree were 1, the objective function would be the weighted sum of the loss, and the one with the lowest error would be the best policy. By having a higher degree, we are capturing the dependencies among the policies. The dependencies arise because the initialization of a new policy is  at the termination of the previous policy and the policy network weights are shared.
\\
Let us consider the computation of the total error for the case of A2C, the total error, $L'(s, a)$,  for a state action pair is the weighted sum of the policy gradient loss and the value function loss. $V(s)$ and $A(s,a)$  is the value function and the advantage function respectively at state $s$ obtained by using policy $\pi(a|s)$ .
\begin{equation}
\label{eqn 5}
V_{loss}(s) = r(s,a)+\gamma V(s')-V(s)
\end{equation}
\begin{equation}
\label{eqn 6}
\pi_{loss}(a|s)=-log(\pi(a|s)) A(s,a)
\end{equation}
\begin{equation}
\label{eqn 7}
L'(s,a)=\pi_{loss}(a|s) + V_{loss}(s) * C_v
\end{equation}
\\
$\gamma$ is the discount factor, $C_v$ is the coefficient used for weighting the value function against the policy gradient loss.
\\
By minimizing this objective function, we obtain the values of $w_i$, the Lagrange multipliers to this optimization framework. To choose the best ensemble, we select the $m$ policies with the highest corresponding lagrange multipliers.

\subsubsection{\textbf{Implementation detail} :}
In order to solve the optimization problem, we can re-frame it as a  quadratic programming problem with box constraints as follows:
\begin{equation}
\label{eqn 8}
J(w)=w^{T}Bw
\end{equation}
and,
\begin{equation}
\label{eqn 9}
B_{ij}=\sum_{s \in S}{\Bar{P} (s)}b_{i}b_{j}
\end{equation}
where,
\begin{equation}
\label{eqn 10}
\resizebox{0.91\linewidth}{!}{$
    \displaystyle
b_{i}=\left[\sum_{a \in A} L(s,a) -\frac{\beta}{M-1} \sum_{k \in M, k \neq i} \text{KL}(\pi_i (s) || \pi_k(s))\right]
$}
\end{equation}
The formulation is still subjected to all the linear constraints as earlier. The matrix $B$ is a positive definite matrix since it is an inner product of the loss terms. This results in a convex objective function for which the global minimum can be found.
In order to run this optimization, we select the states from multiple trajectories in the training samples. This framework efficiently reuses data and is better in comparison to evaluating the policies directly in the environment.
Without this framework, the method to find the best ensemble subset would be to evaluate all possible combinations.
For comparison, if each ensemble model is evaluated using $N'$ samples from the environment, a total of $m \times N'$ samples will be used up in selecting the $m$ policies.

\begin{algorithm}[tb]
\caption{SEERL}
\label{alg:algorithm}
\begin{algorithmic}[1]
\STATE {\bfseries Input} Initialize a policy $\pi_\theta$, training time-steps $T$, evaluation time-steps $T'$, number of policies $M$, maximum learning rate $\alpha_0$, number of policies to ensemble $m$, ensemble strategy $E$\\
\STATE {\bfseries Output} Average reward during evaluation\\
\STATE {\bfseries Training}
\WHILE{$t \leq T$}
\STATE Calculate the learning rate, based on the inputs to the cosine annealing learning rate schedule $f$
\STATE $\alpha(t)=f( \alpha_0, t, T, M)$ //Equation \ref{eqn 1}
\STATE Train the agent using $\alpha(t)$
\IF {t $\bmod$ (T/M)}
\STATE Save policy $\pi_\theta^i$ for $i={1,2\dots ,M}$
\ENDIF
\ENDWHILE
\STATE {\bfseries Evaluation}

\STATE Select the $m$ policies using the Policy selection process //Section \ref{section:4.2}
\STATE Select an ensemble strategy $E$
\WHILE{$t \leq T'$}
\STATE Collect actions from the $m$ policies, ${a_1,a_2\dots, a_m}$ for environment state $s_t$
\STATE Find the optimal action, $a^*$ using $E$ //Section \ref{section:4.3}
\STATE Perform action $a^*$ on the environment
\STATE Obtain cumulative reward for the episode, $r_t$ and the next state $s_{t+1}$
\ENDWHILE
\STATE \textbf{return} Average reward obtained during evaluation
\end{algorithmic}
\end{algorithm}
\subsection{Ensemble techniques}
\label{section:4.3}
Once the $m$ policies are chosen, depending on the complexity of the action space, discrete or continuous, there are multiple strategies to ensemble the actions in the environment.
We divide the ensemble strategy into two categories, for discrete and continuous action spaces.
\subsubsection{\textbf{Ensemble in discrete action spaces :}}
In discrete action spaces, we consider majority voting as a good solution.
Due to different fixed point convergences of value functions of algorithms trained independently, it is not possible to compare actions by their $Q$ values.
\begin{equation}
\label{eqn 11}
\pi(a|s)=\mathop{argmax}_{a \in A(s)}\left[\sum_{i \in m} N_i(s,a)\right]
\end{equation}%
where $ N_i(s,a)$ is one if the agent $i$ takes action $a$ in state $s$, else zero.
It the case of a tie, random action is chosen among the set of actions having the tie.
\begin{figure*}[t]
        \centering
        \begin{subfigure}{0.48\textwidth}
            \includegraphics[width=\textwidth]{breakout_a2c.png}
            \caption{Training performance of SEERL vs A2C on Breakout }
            \label{fig:gull1}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.48\textwidth}
            \includegraphics[width=\textwidth]{half_cheetah_sac.png}
            \caption{ Training performance of SEERL vs SAC on Half-Cheetah}
            \label{fig:gull2}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.3\textwidth}
            \includegraphics[width=\textwidth]{sota/breakout_eval.png}
            \caption{Comparison between SEERL(A2C), SOTA and baseline ensembles B1(A2C), B2(A2C and ACER) on Breakout }
            \label{fig:gull3}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.3\textwidth}
            \includegraphics[width=\textwidth]{sota/halfcheetah.png}
            \caption{Comparison between SEERL(SAC), SOTA and baseline ensembles B1(SAC), B2(SAC and DDPG) for Half Cheetah }
            \label{fig:gull4}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.3\textwidth}
            \includegraphics[width=\textwidth]{ps/Figure_1.png}
            \caption{Performance of each subset of policies , an ensemble of 3 policies selected from a set of 5, on Breakout }
            \label{fig:gull5}
        \end{subfigure}
\end{figure*}

\subsubsection{\textbf{Ensemble in continuous action spaces :}}
In continuous action spaces, \cite{duell2013ensembles} proposes multiple strategies to find the optimal action. However, the performance comparison of the strategies is not provided and environments considered are too simple. The different strategies are as follows:
\begin{itemize}
    \item Averaging: We take the average of all the actions as part of the ensemble. This strategy could fail in settings where one or more of the actions are extremely biased and thereby shifts the calculated value away from the true mean value.
    \item Binning: This is the equivalent of majority voting in a continuous action space setting. We discretize the action space into multiple bins of equal size and average the bin with the most number of actions. The average value obtained is the optimal action to take. Through this method, we have discretized the action space, sorted the bins based on its bin-count, and calculated the mean of the bin with the highest bin-count. The hyper-parameter to be specified here is the number of bins. We use five bins in our experiments
    \item Density-based Selection(DBS): This approach tries to search for the action with the highest density in the action space. Given $M$ action vectors, $a$, each of $k$ dimensions to be ensembled, we calculate the density of each action vector using Parzen windows as follows:
    \begin{equation}
    \label{eqn 12}

    d_{i}=\sum_{j=1}^{M}exp{-\frac{\sum_{l=1}^{k}(a_{il}-a_{jl})^{2}}{h^{2}}}
    \end{equation}%

     The action with the highest density, $d_i$, is selected as the final action. The only parameter to be specified is $h$, the window width, and we have chosen $h=0.0001$ in our experiments.
    \item Selection through Elimination(STE): This approach eliminates action based on the Euclidean distance. We calculate the mean of the action vectors and compute the euclidean distance to each action from the mean. The action with the largest euclidean distance is eliminated, and the mean is re-computed. The process is repeated until two actions remain. The final action is chosen as the average of the two actions.
\end{itemize}

\begin{table*}[t]
\centering
\caption{Evaluation scores across Atari 2600 games averaged over 100 episodes. We report the published scores for DQN \cite{mnih2015human}, Distributional DQN\cite{bellemare2017distributional} and Rainbow DQN. SEERL uses A2C as base learner is the best performing of the methods, topping in 11 of the 15 games. All results represent the average of 5 random training runs. We run each game for 20M environment time steps.}
\begin{tabular}{rlllll}\toprule
{Game}&  {DQN} & A2C &Distrib. DQN  &  Rainbow DQN  & SEERL  \\\midrule
Alien &  634 & 518.4 & 1997.5 & \textbf{6022.4} & 1924\\
Assault &  178.4 & 263.9 & 237.7 & 202.8 & \textbf{341.2}\\
Bank Heist &  312.7 & 970.1 & 835.6 & 826 & \textbf{1124.1}\\
Battle zone &  23750 & 12950 & 32250 & \textbf{52040} & 28580\\
Breakout &  354.0 & 681.9 & 584.9 & 379.5 & \textbf{821}\\
Freeway &  26.9 & 0.1 & 28.8 & 29.1 & \textbf{33.1}\\
Frostbite&  496.1 & 190.5 & {2813.9} & \textbf{4141.1} & 1226\\
Krull &  6206.0& 5560.0 & 6757.8 & {6715.5} & \textbf{6795.2}\\
MsPacman &  1092.3 & 653.7 & 2064.1 & {2570.2} & \textbf{2614.2}\\
Pong &  18 & 5.6 & 18.9 & 19.1 & \textbf{19.8}\\
PrivateEye &  207.9 & 206.9 & \textbf{5717.4} & 1704.4 & {722.1}\\
Qbert &  9271.5 & 15148.8 & 15035.9 & 18397.6 & \textbf{18834.2}\\
Road Runner &  35215 & 34216 & {56086} & 54261 & \textbf{58624.2}\\
Robotank &  58.7 & 32.8& 49.8 & 55.2 & \textbf{61.2}\\
Seaquest &  4216.7 & 2355.4 & 3275.4 & \textbf{19176.0} & {4811.2}\\
\bottomrule
\end{tabular}
\label{tab1}
\end{table*}

\begin{table}[t]
\centering
\caption{Evaluation scores across Mujoco environments averaged over 100 episodes. SEERL using SAC as base learner is seen to outperform across 5 of the 6 environments. All results represent the average of 5 random training runs.}
\begin{tabular}{rlllll}\toprule
Environment &  TRPO  & PPO  &DDPG  &  SAC & SEERL  \\
\midrule
Ant &  2342.2 & 962 & 342.4 & 1958 & \textbf{2564}\\
Half Cheetah &  4233 & 1820 & 5440 &7269 & \textbf{11658}\\
Hopper & 2252 & 1112 & 1233 & \textbf{3379.2} & 3156\\
Humanoid &  3882 & 735 & 101.4 & 4380 & \textbf{4845}\\
Swimmer &  121.2& 42.4 & 43.4 & 44.6& \textbf{162.4}\\
Walker 2D&  3215 & 1892 & 782 & 2112 & \textbf{4366}\\
\bottomrule
\end{tabular}

\label{tab2}
\end{table}

\begin{table*}[t]
\centering
\caption{Evaluation scores across Atari 2600 games averaged over 10 episodes using Rainbow DQN \cite{hessel2018rainbow} as base learner for SEERL at 100K interactions. We report the published scores for SimPLe \cite{kaiser2019model}, CURL\cite{srinivas2020curl} and SUNRISE\cite{lee2020sunrise} at 100K interactions. All results represent the average of 3 random training runs.}
\begin{tabular}{rlllll}\toprule
Game &   Rainbow DQN  & SimPLe  &CURL  &  SUNRISE  & SEERL \\
\hline
Alien         &  789   & 616.9 & 558.2 & \textbf{872} & 800\\
Amidar        &  118.5 & 88.0  &{142.1} & 122.6 &  \textbf{208.3}\\
Assault       &  413.0 & 527.2 & {600.6} & 594.8 & \textbf{627.7}\\
BankHeist     &  97.7  & 34.2 & 131.6 & 266.7 & \textbf{508.0}\\
BattleZone    &  7833.3 & 5184.4 & 14870.0 & 15700.0 & \textbf{19400.0}\\
Breakout       &  2.3  & \textbf{16.4} & 4.9 & 1.8 & {3.2}\\
Freeway       &  28.7 & 20.3 & 26.7& 30.2 & \textbf{31.3}\\
Frostbite     &  1478.3 & 254.7 & 1181.3 & \textbf{2026.7} & 2010.0\\
Krull         &  3282.7 & {4539.9} &4229.6 &3171.9 & \textbf{3203.1}\\
MsPacman      &  1118.7 & 1480 & 1465.5 & 1482.3 & \textbf{1838.0}\\
Pong          &  -16.9 & \textbf{12.8} & -16.5 & -13.8 & {-14.2}\\
PrivateEye    &  97.8 & 58.3 & \textbf{218}.4 & 100 & {100}\\
Qbert         &  646.7 & 1288.8 & 1042.4 & {1830.8} & \textbf{2125}\\
RoadRunner    &  9923.3 & 5640.6 &5661.0 & {11913.3} & \textbf{15290.0}\\
Seaquest      &  396.0 & \textbf{683.3} & 384.5 & {570.7} & {458.0}\\
\bottomrule
\end{tabular}
\label{SUNRISE}
\end{table*}

\section{Experiments}
Through our experiments, we answer the following questions:
\begin{itemize}
    \item RQ1 : How does SEERL compare against traditional ensembles and SOTA reinforcement learning algorithms in terms of sample complexity and performance?
    \item RQ2 : How does the diversity among policies contribute to the final performance?
    \item RQ3 : How does the policy selection framework help to find the best subset of policies?
    \item RQ4 : How are the policies obtained from SEERL any different from those obtained through random perturbation?
\end{itemize}
In the following sections, we describe the setup used for experimentation, the training procedure, the evaluation procedure and analysis of the results obtained. We answer the questions posed above in detail in the analysis section.
\subsection{Setup}
In order to answer the above questions, we consider the environments from the Atari 2600 game suite \cite{bellemare2013arcade} for its discrete action space and Mujoco \cite{todorov2012mujoco} for its continuous action space. We conduct three sets of experiments to validate our framework :
\begin{itemize}
    \item The first set uses the following algorithms as the underlying reinforcement learning method (base learner) for SEERL - A2C \cite{DBLP:journals/corr/MnihBMGLHSK16}, ACER \cite{DBLP:journals/corr/WangBHMMKF16}, ACTKR \cite{DBLP:journals/corr/abs-1708-05144}, DDPG \cite{lillicrap2017continuous}, SAC \cite{haarnoja2018soft} and TRPO \cite{schulman2015trust}. The goal is to show that SEERL can be adapted to any reinforcement learning algorithm, both off-policy and on-policy.
    \item The second set compares SEERL with three baseline ensemble methods that we have created. The three baselines are as follows:
    \begin{itemize}
    \item B1 : Ensembles of policies trained independently from a single algorithm. E.g., Five models of A2C.
    \item B2 : Ensembles of policies trained independently from different algorithms. E.g., two models of A2C, two models of ACER and one model from ACKTR.
    \item B3 : Ensembles of policies generated from random perturbation of model parameters at regular intervals. E.g., Five models of A2C, each of which has been obtained by perturbing at regular intervals and saving the parameters
    \end{itemize}
    The goal is to understand how the traditional ensembles (B1 and B2) compare to SEERL and the role of diverse policies in an ensemble. Comparison with B3 is intended to show the importance of directed perturbation against random perturbation.

    \item The third and final set compares SEERL with SOTA reinforcement learning algorithms that aim to be to sample efficient. We compare SEERL with SimPLE \cite{kaiser2019model}, CURL \cite{srinivas2020curl}, Rainbow DQN \cite{hessel2018rainbow} and SUNRISE \cite{lee2020sunrise}. In these experiments SEERL uses Rainbow DQN as the base learner in order to make a fair comparison with SUNRISE. The goal is to understand whether SEERL can be competitive with these algorithms that are sample efficient and be trained using the limited interactions it has with the environment. Comparison with SUNRISE is intended to understand whether SEERL is competitive
    with a method that trains multiple critics in parallel.
\end{itemize}
\subsection{Training}
SEERL uses a base learner to learn an ensemble of policies. This base learner can be any reinforcement learning algorithm such as A2C, SAC or Rainbow DQN. After selecting the base learner, SEERL is trained using the same hyper-parameters configurations as in the original implementation of the base learner.  The ambiguity that SEERL will lead to poor convergence as a result of shifting from zero to the maximum learning rate multiple times is mitigated through our results. SEERL performance during training is at least at par or better than the base learner, as shown in Figure 2(a, b). We train the models across different values of $M$ ranging from 3 to 9.\\
In order to make the comparison of SEERL with the base learner fair, we train both models on the same number of timesteps as specified in the original paper of the base learner. However, for comparison with SUNRISE, SimPLe, CURL and Rainbow DQN, we train SEERL using Rainbow DQN as base learner for 100k timesteps for a fair comparison.  Additional training results on the environments are presented in the supplementary under section 1.
\subsection{Evaluation}
During evaluation, the policy selection framework is used to select the best subset of policies for the ensemble with $m=5$. All $m$ policies are loaded in parallel and provided with an observation from the environment. Based on the observation, every policy outputs an action, and the ensemble strategy decides the final action to be used on the environment. We perform this evaluation process for 100 episodes, and the average reward over these 100 episodes is reported in Table \ref{tab1} and Table \ref{tab2}. We use the SOTA scores from Rainbow DQN \cite{hessel2018rainbow} for benchmarking Atari 2600, and SEERL is seen to outperform it in many games, as seen in Table 1. For Mujoco environments, SEERL is seen to outperfom baselines by a considerable margin as shown in Table 2. Comparison of SEERL with baselines, B1 and B2, for Breakout and Half cheetah is shown in Figure 2(c,d). For comparison with SUNRISE, SimPLe, CURL and Rainbow DQN, the evaluation process is for 10 episodes and reported in Table \ref{SUNRISE}.

\begin{figure*}[t]
        \centering
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/heatmap_1.png}
            \caption{Divergence between independently trained policies used in the baseline, B1}
            \label{fig:gull6}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/heatmap_seerl.png}
            \caption{Divergence between SEERL policies}
            \label{fig:gull7}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/heatmap_random.png}
            \caption{Divergence between policies obtained using random perturbations}
            \label{fig:gull8}
        \end{subfigure}

\end{figure*}

\begin{figure*}[t]
        \centering
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/M.png}
            \caption{Training performance of SEERL as $M$ varies between 3 to 9}
            \label{fig:gull9}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/alpha.png}
            \caption{Training performance of SEERL as maximum learning rate $\alpha_0$ varies}
            \label{fig:gull10}
        \end{subfigure}
        \hfill
        \begin{subfigure}{0.31\textwidth}
            \includegraphics[width=\textwidth]{figures/seerlvsrand.png}
            \caption{Training performance between SEERL and randomly perturbed model, with multiple perturbations in a sequence}
            \label{fig:gull11}
        \end{subfigure}

\end{figure*}

\subsection{Analysis}
We try to understand why and how SEERL gives such superior performance in comparison to baselines. We analyze the performance and sample efficiency of SEERL, the diversity among the policies, and finally, the comparison between a randomly perturbed model and SEERL. Analysis on individual performance of the SEERL policies and the dominance among policies in the ensemble are presented in the supplementary material under section 2.
\subsubsection{\textbf{RQ1 : Performance and sample efficiency of SEERL}}
The training curve in Figure 2(a,b) illustrates the performance of SEERL with the base learner and is observed to consistently outperform it. In Figure 2(a), it can be observed that SEERL achieves the same performance in 6 million timesteps that which requires the base learner to achieve in 20 million timesteps. Figure 2(c) and 2(d) illustrates the performance of SEERL in comparison with B1, B2 and is seen to outperform both baselines. Table \ref{tab1} shows that SEERL outperforms the  SOTA score from Rainbow DQN on most games. Similarly Table \ref{tab2} shows that SEERL outperforms across 5 of the 6 environments. The results in Table \ref{SUNRISE} illustrate comparison of SEERL with Rainbow DQN, SimPLe, CURL and SUNRISE at 100k timesteps. We observe that SEERL outperforms SUNRISE in 11 of the 15 Atari games used for comparison and achieves SOTA score for 100k interactions in 9 games. The performance of SEERL trained on 100k interactions highlights its sample efficiency. It is also observed that SEERL is 4-6 times faster than SUNRISE since it only uses a single network to sequentially train the policies while SUNRISE has multiple networks in parallel. As the number of networks grows in SUNRISE, the training time is longer while for SEERL, it remains the same. To summarize, SEERL outperforms most methods whether it is trained on millions of interactions or just 100k.
\subsubsection{\textbf{RQ2 and RQ3 : Diversity of Policies and Policy Selection in SEERL}}
We establish the diversity among the individual policies concretely by computing the KL divergence between them using the action distribution across a diverse number of states. The higher the KL divergence between the policies, the more diverse the policies are. From Figure 3(b), we can observe that the SEERL policies are diverse, and diversity continues to exist as new models are formed. Conversely, for the baseline models, B1 and B3 (Figure 3(a) and Figure 3(c)), the KL divergence between the policies is substantial. This observation can be used to explain why the baseline ensembles failed to perform. The larger KL divergence among the baseline policies indicate that the policies do not have much overlap in the action space, and hence ensemble techniques such as majority voting were unable to find a good action.
\\
Additionally, the analysis of the different subsets of policies in Figure 5 and their performance in Figure 2(e) confirms that diversity among policies is required, but too much diversity does not help. This empirically proves that extremely diverse models are not that helpful in an ensemble.
\\
We can, therefore conclude that SEERL can generate policies with sufficient diversity for a good ensemble.

\subsubsection{\textbf{RQ4: Random perturbation vs. SEERL}}
To emphasize that directed perturbation of the weights will lead to better models, we show the comparison between SEERL and a randomly perturbed model in Figure 4(c). This model has been perturbed with random values at regular intervals similarly to SEERL. The perturbation is done by backpropagating  random values of gradients instead of the actual values. We observe that doing a single perturbation or, multiple ones sequentially for a small period, does not improve performance. We can, therefore, establish that directed perturbation along the gradient direction is necessary to obtain a better model. In this experiment, all the hyper-parameters have been kept identical to that used in SEERL.

\begin{figure}[t]
        \centering
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_1_2.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_1_3.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_1_4.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_2_3.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_2_4.png}
        \\
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_0_3_4.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_1_2_3.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_1_2_4.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_1_3_4.png}
                \includegraphics[width=0.19\linewidth]{ps/heatmap_new_data_2_3_4.png}

    \caption{ Divergence between policies in a subset. Each subset consists of 3 policies chosen from a larger set of 5. The policies are trained on Breakout using A2C }
\end{figure}
\subsection{Ablation studies}
\subsubsection{\textbf{Effect of varying the number of cycles}}
The performance of SEERL is affected by the selection of $M$. For a fixed training budget, if the value of $M$ chosen is very large, the performance is seen to degrade. With larger $M$, the training cycle for each policy is reduced, thereby reducing the chance for the policy to settle at a good local minimum before it is perturbed again. In practice, we find that setting the value of $M$ between $3$ to $7$  works reasonably well. Figure 4(a) compares the performance of SEERL with varying $M$ values between $3$ and $9$.
\subsubsection{\textbf{Effect of varying maximum learning rate}}
The maximum leaning rate value influences the performance of the policies and therefore affects the performance of SEERL. It directly impacts the perturbation of the local minima and hence, the diversity of the policies. In practice, we have seen that having a larger value tends to perform better, owing to the strong perturbation it causes at different local minima leading to reasonably different policies. We have used values ranging between $0.01$ to $0.001$ throughout our experiments. Figure 4(b)  compares the performance of SEERL with different values of $\alpha_0$ with $M=5$
\section{Conclusion and Future work}
In this paper, we introduce SEERL, a framework to ensemble multiple policies obtained from a single training run. We show that the policies learned at the different local minima are diverse in their performance and our policy selection framework helps to select the best subset of policies for the ensemble during evaluation. SEERL outperforms the three baseline methods and beats SOTA scores in complex environments having discrete and continuous action spaces. We show our results using both off-policy and on-policy reinforcement learning algorithms and therefore showcase the scalability of the framework. Our analysis shows that, in comparison to baselines, SEERL achieves comparable, and sometimes better performance using a significantly low number of samples, making it an extremely sample efficient algorithm.
Future work will explore how to combine the learned policies during training time as a growing ensemble to stabilize training and increase diversity.

\newpage
\bibliographystyle{ACM-Reference-Format}
\balance

redpajama_set_name: RedPajamaArXiv


In September 2015, engineering and construction group UGL Kentz JV conducted its regular review of its EWP operator training at the Ichthys MEC-1 Project site in Darwin.
Reinforcing its reputation as the best team in the EWP industry, Genie has shifted its Melbourne operation, which includes its heavy maintenance facility, into a brand-new, purpose-built location in Dandenong South, in the city's south eastern suburbs.
Genie is revolutionising customer service to give your business the best possible outcomes.
Genie recently celebrated the first sale of the Z-33/18 boom lift in Australia and New Zealand. Access Solutions NZ has purchased the first four units through Genie partner Youngman Richardson & Co.
Established in 2005, the Allwest Plant Hire is well-regarded in Western Australia as a hire provider of heavy earthmoving plant and skilled operators to earthworks, major civil infrastructure and mining projects.
Proving to be a true all-rounder, the Genie GS-4047 scissor lift, which combines a small footprint with the ability to reach heights of close to 14 m, is proving popular with customers of Sydney rental firm Uphire.
The recently-authorised dealer for the Genie range of aluminium equipment for Tasmania, Victoria and South Australia, Scott Spackman from SMS Access Sales Pty Ltd, is stocked up and ready to distribute after receiving a significant delivery of the material lifts.
From the first materials lift powered by compressed air to always putting our customers first, Genie is celebrating 50 years of success.


Roxbury Community College now offers students the ability to apply online with no application fee. Simply click on the link in the window below to complete this process. We are currently enrolling students for the spring semester.
Note: Dual Enrollment and International Students DO NOT apply online. Please print out the Dual Enrollment or International Students Application and see the sections on this webpage for more information.
After completing your application, please contact the RCC Admissions Office with any questions or concerns.
We can be reached at 617-541-5310 or 857-701-1200 or Enrollment.management@rcc.mass.edu.
Students that have taken college level courses in the past should request official transcripts from those institutions, if you have not taken college level courses you should contact our testing department at 857-701-1405 or http://www.rcc.mass.edu/future-students/assessment/placement-testing. All new College students are required to complete a placement test in English, Math and Science.


text:
John Morgan Exceeds Signature Goal for $15 Minimum Wage
caitlinw | Labor and Employment | Nov 12, 2019
[Image from April 21, 2017, by Charlie Kaijo of the Tampa Bay Times]
John Morgan has long been involved in the fight for a $15 hourly minimum wage in his home state of Florida. Recently, Morgan announced that he exceeded his signature goal to add a proposal about his minimum wage mandate on Florida's 2020 ballot.
In an interview, Morgan said that "In life, I think that you're supposed to do the most for [those] with the least," which stands very much in line with Morgan & Morgan's "for the people" promise.
According to reporting originally in the Miami Herald, "Florida's minimum wage is currently indexed to inflation, and the increases only began in 2004, when the federal minimum wage was just $5.15 an hour, already below the cost of inflation. In other words, if the state had made an initial adjustment for inflation when the increases kicked in, the state's current minimum wage would be more than $9.50 an hour. It is currently $8.46 an hour; the federal minimum wage is $7.25 an hour."
Morgan intends to make up the difference with this ballot measure, ensuring a Florida minimum wage of $10 per hour with an annual $1 bump until it reached $15/hour. Although he understands the concern over rising costs and the impact to businesses, he reminds that these objections don't override the needs of a working individual.
Currently, the federal minimum wage is $7.25 per hour; the federal tipped minimum wage for people like servers is $2.13 per hour. (The tipped wage assumes that tips compensate for the discrepancy between it and the untipped minimum wage.)
The Economic Policy Institute claims that if we were to gradually raise the minimum wage to $15, it would benefit workers, companies, and the economy. But the minimum wage isn't the only workplace financial issue affecting employees: Wage theft is widely prevalent across all industries, from minimum wage work to upper-level executives.
Wage theft can present itself in a variety of ways. For lower-earning workers, usually, wage theft can be tied to their "independent contractor" role, a day rate that doesn't account for overtime, or money kept by the employer without reason (like withholding tips or not making up the difference if an under-tipped worker earns less than minimum wage). Higher earners may find they're misclassified and are owed overtime the employer hasn't paid.
So what do you look for to know if you're affected?
Signs of Wage Theft
If you're "exempt" from being paid overtime, there are a few reasons why that might be. You should fit under one of the Fair Labor Standard Act's five exemptions from overtime, which include: administrative exemption, executive exemption, outside sales exemption, professional exemption, and computer employee exemption. You can learn more about who those exemptions apply to here.
If you don't fit into one of those exemptions, you are likely entitled to overtime pay if you work more than 40 hours in a week. If you're not getting overtime pay and think you should, you may be a victim of wage theft.
On the other side, sometimes employees are misclassified as "independent contractors" so companies can avoid giving them benefits and/or paying overtime. There's nothing inherently wrong with being an independent contractor, but you need to be mindful that you're not being purposely misclassified. If you, as a worker, perform all the duties that would belong to full-time staff, there's a chance you're misclassified and may be owed compensation for unpaid benefits, wages, and overtime.
Improper Reimbursements
This issue can apply to all workers — you might need to reimburse a business lunch, or maybe the gas in a long-haul truck. However, the former is very easy to track: You either get reimbursed, or you don't. For drivers, there are specific reimbursements they should receive.
The IRS mandates that all drivers using their own vehicle should be reimbursed at least 58 cents per mile. Additionally, those drivers should be reimbursed for normal vehicle wear and tear over the course of their employment.
Tips come into play too — pizza delivery drivers, for example, need to take note of the reimbursement information above when they drive, but need to consider their wage as well. The federal tipped minimum wage, as mentioned above, is $2.13 per hour. However, the tips they receive should compensate for the roughly five dollars (using federal figures) less that they make an hour compared to un-tipped workers.
However, some delivery drivers mention making $1 per delivery or a similar figure rather than an hourly rate subsidized by tips, and that's illegal.
Unfortunately, employers can't always be trusted to treat their employees properly. Anyone who thinks they may be a victim of wage theft should collect evidence and retain a lawyer, who can not only help them recover unpaid wages, but protect them from retaliation as well.
By caitlinw
Supreme Court Rules Discrimination Against LGBT Employees Illegal
Today, the U.S. Supreme Court ruled that employers cannot discriminate against their employees based on their sexual orientation or gender identity. The provision in question was Title VII of the…
Staff | Labor and Employment | Jun 15, 2020
Independent Contractor Or Employee?
It's a common question in the workforce: whether you're an employee or an independent contractor. You might wonder, why does it matter? I'm paid the same either way. However, that's typically not…
Are You Eligible For Paid Leave?
In March, Congress passed a law meant to help workers in the wake of COVID-19. This law is deeply significant for the U.S., which lacks centralized or widespread paid leave programs. But what does…
Staff | Labor and Employment | May 26, 2020
Fill out this form for a Free, Immediate Case Evaluation
By submitting you agree to our
Terms & Privacy Policy.
Employment Rights And Risks Federal Employees Should Know About
Nashville had already suffered an intense 2020, even as Decem
Adam | Catastrophic Events
In Florida, a class-action lawsuit alleging that Ritz-Carlton
Staff | Consumer Alerts
With protesters across the country getting arrested for exerc
Staff | Firm News
5 of This Year's Biggest Data Breaches
Data breaches are <
Understanding The Business Interruption Landscape
If you're a business owner whose business has lost revenue du
sholom | Insurance