We all have compounding daily, weekly and monthly tasks that get pushed back due to the fires you have to put out right now. But, eventually those seemingly insignificant tasks have to be addressed.
For many in healthcare and healthcare technology, preparing for phase II of the HIPAA Audit Program is one of those tasks that gets pushed off until it's too late. Any business that operates within the lawful confines of HIPAA, should already have matters well in order, but it's a complex process that requires vigilance and constant attention to stay up-to-date with regulation.
The Department of Health and Human Services (HHS) started sending out notifications to Covered Entities (CEs) and their Business Associates (BAs) via email in April of 2016.
We shared a few things that would help you prepare for this phase of audits—and the simplest and most critical action to take now is to ensure that you have whitelisted OSOCRAudit@hhs.gov. The last thing you want is to have notices go to your spam folder!
The OCR clearly stated that they expect you to be aware of their first round of notices, as well as any subsequent emails. And because you only have 14 days from initial contact to verify information and ten days from any contact to address audits, time is of the essence. Once you check to make sure that you've whitelisted their email domain, we recommend double checking your spam filters. Then, make sure the contact information you provided the OCR is still valid.
According to the sample letter posted on the HHS site, "If we [OCR] do not receive a response from you, we will use this email address for future communications with this entity. Failure to respond will not shield your organization from selection." The OCR's email notifications include a deadline for responding to their request for verification of your contact information. The deadlines for response may have already passed, so if you have not responded to the OCR's first notification, the default is that they will send future notices to this email address.
The first round of desk audits will start with Covered Entities, followed by Business Associates. Those audits started in 2016 will likely continue into next year; some organizations must endure detailed reviews and are subject to onsite reviews.
Alongside the email notifications that began in April of 2016, the OCR published an audit protocol to help organizations prepare for the audits, regardless of how the audits were triggered. Some were part of phase 2 of the HIPAA Audit Program while others were part of a consumer complaint or breach report. There are 180 areas of scrutiny in the protocol. Be warned: You will need to have your documentation and requested materials organized and ready should you be selected for an audit.
The key takeaway from the HIPAA Omnibus Rule is that BAs and their subcontractors are directly liable for HIPAA compliance, including security breaches. If you outsource your cloud storage or use a managed service provider to create, transfer, or store your HIPAA-regulated data, you'll want to take note. Your provider must be able to prove compliance, and if they are not able to demonstrate this to you, please change providers immediately.
As mentioned, there are approximately 180 different areas that may be addressed in an audit. The OCR's audit protocol specifically notes: "Entities must provide only the specified documents, not compendiums of all entity policies of procedures. The auditor will not search for relevant documentation that may be contained within such compilations." This means that a high-level overview or similar documents are not acceptable.
If you haven't already, it's important that you organize all relevant audit documentation and develop training programs for employees, contractors, and partners to ascertain that they're aware of their role in your organization's compliance. Don't forget to see how your partners and providers will prove they're compliant, too. These are just a few ways to prepare for a compliance audit. Depending on the size and nature of your organization, you may be held to additional requirements.
Non-compliance can result in fines, required remediation, legal costs, loss of reputation, and may even cause your business to permanently close. Even if you've made it to this point without an audit, it's only a matter of time before it's your turn. Keep your documentation and processes up to date and continue to improve upon your compliance.
Feel free to visit the HHS Compliance website for more information.
Source: This article was originally published on the OnRamp blog.


The bungalow sleep six guests only. Additional guests are not allowed. The property is not suitable for young infants. Please bring your own towels. Parking is available at the premises.
The property has resident staff who will welcome you on your arrival. The house keeper keeps the house clean and will do laundry and basic cooking. The gardener maintains the garden and looks after the swimming pool. A cook can be arranged on request for a small daily charge. The night guard is also resident on the property. I am easily contactable through the staff. I hope you enjoy your stay.
Would strongly recommend. An extremely peaceful and therapeutic location with a pretty swimming pool and a great view of the Indian Ocean. Friendly caretakers. A cool breeze off the sea prevents one getting too hot as in many other Kenyan coastal locations.
Malindi has several beaches, Malindi Bay and harbour, Silversands and Malindi Marine Park. The historic town of Malindi has been a Swahili settlement since the 13th Century and has classic examples of Swahili architecture. Vasco Da Gama anchored in the Bay in 1498 before setting sail for India, commemorated by Vasco Da Gama Pillar at the harbour headland. Malindi has numerous restaurants with local Swahili and Italian influence, including the infamous Italian 'I love Pizza' and ice cream cafe on the harbour. Malindi has numerous activities and I would be happy to recommend any operator on request.
Local taxis are reasonable priced, readily available and reliable.


text:
Gima-Ha Eikoku Karate-Do Renmei
儀間派空手道英国連盟
Shihan-Dai
Carl Gigg
Mark Carley
Paul Simpson
Shidoin
Paul Jefferies
United Kingdom Dan Grade Register
NIDAN
DAN Grading Syllabus
NANADAN
Dojo Locations
Shisei-Kai?
Maidstone Shisei-Kai
Woodchurch Shisei-Kai
Seikukan Dojo 誠空館道場
Shihan Makoto Gima
More on the Gima-Ha Lineage!
The 'principles' of M. Gima
'Kara' means 'Empty'
Higaonna Shihan on Makoto Gima.
Gima-Ha U.K. Kata
Archives Select Month December 2022 (1) November 2022 (1) August 2022 (2) March 2022 (8) February 2022 (3) January 2022 (2) July 2021 (1) June 2021 (2) May 2021 (3) October 2020 (2) September 2020 (2)
REIHO, SAHO, KU et al. December 9, 2022
2022 JAPAN TRIP November 29, 2022
ARE SPORT & BUDO COMPLIMENTARY August 8, 2022
LET'S THINK ABOUT KIME August 8, 2022
SOME THOUGHTS ON THE FUTURE OF KARATE March 7, 2022
What is Gima-Ha Karate?
Gima-Ha Karate (formerly called Shōtō-Ryu) is a fusion between mainline Shotokan and the original style from Okinawa.
We encompass both the standardised Shōtō-Kan as seen in Japan Karate Association, Shōtōkan Karate International, All Japan Karate-Do Shōtō-Kan etc.
Our karate differs in our Koryu (old School kata) and the 'Funakoshi Nage-waza' (throwing techniques) which augment the standard curriculum.
(志誠会)
Top L to R: R. Sidoli, M. Carley, S. Nicholson-Root, M. Silvester, C. Gigg, S. Edwards, P. Jefferies. Bottom L to R: J. Murray, K. Boelitz, M. Scott, J. Collier.
GIMA-HA is our National association name, as such it is very important to us all. Gima-Ha is named in honour of the late Saiko Shihan Shinkin (Makoto) Gima, thus the lineage is of huge significance, and the need to keep his name in the arena of Karate-Do, is paramount.
What few know, and the reason why we have retained the name, Shisei-Kai, stems from the association name used by Gima Shihan and his senior student Shihan Ikuo Higuchi prior to the death of the master in 1989. Gima Shihan was the President of the Zen Nihon Karate-Do Shisei-Kai-Kai Karate-Do Rengou Kai (All Japan Karate-Do Shisei-Kai Karate Union).
After Gima Shihan's passing his senior students chose the Gima-Ha Shōtōkan Karate-Do Association to commemorate the great Master; in recognition and honour of the great contributions he made to both Japanese Karate-Do as a whole, and Shōtōkan-Ryu in general.
Our Dojo involvement with the name Shisei-Kai is of a personal note. In 1970 one of the Shisei-Kai instructors from Japan arrived in Canada for a two week vacation. This turned into a ten year stay. One of the students at the very beginning of that era was our own senior instructor, now resident in the United Kingdom. The name Shisei-Kai held significance for him for some 35 years until he was re-united with his sensei.
Shisei-Kai was instrumental in the founding of The National Karate Association (of Canada) British Columbia Branch. The mentor of all Shisei-Kai-Kai Canadian members, and in fact for us here in the United Kingdom was Shihan Hidetada Narumi.
In 1984 the Chief Instructor of the All Japan Karate-Do Shisei-Kai Union, Shihan Ikuo Higuchi, wrote: "… the growth of Shisei-Kai has been a long and difficult road.
As well as being a corporate lawyer, I was Shihan of the Shotokan style for the Okinawa Karate-do Association, in the vigour of my youth. The outdoor Dojo at Ikebukuro (district in Tokyo) with 20 disciples, became the birth-place of Shisei-Kai…
… In Showa 45th (1971) year the foundation of the Metropolitan Karate Organization sent a fresh wind through the karate world. The Vice-President, Y. Fuji, was at that time on the Board of Directors of E. Eriguchi, and one of the instructors Teruo Hayashi of F.A.J.K.O. It became very famous in the karate world, having had opportunities to exchange ideas with the best teachers of the karate world.
Throughout roughly six years, Shisei-kai was proud to form the centre of that growth. The man who really helped to add to our strength was Kobayashi Sensei. He was firm with us, and we continue to feel a great trust and reliance in this man.
During the past eight or nine years, we have been very proud to see Shisei-Kai's development as an association, its developments as a start of the karate world, its growth in Tokyo, Saitama, Chiba, Kanagawa, Iwate, Niigata and even in Canada, where Shisei-Kai has enjoyed a splendid growth and the formation of an association, giving Shisei-Kai an international flavour. For more than ten years we have known no stoppage in the process from hardship to good fortune, and we are greeting a second golden age. …
… The basis of all this is the support given by our teacher, Gima, which has amounted to some twenty years, and which we still continue to receive. I firmly believe that it is because every member receives his enthusiasm and feeling in the same way (that we achieved our success). This year, even the construction of a branch Dojo in Urawa City of Saitama Ken has been scheduled."
Shisei-Kai is a name that is in our hearts and can be taken to have different meanings. 'Kokorozashi' (Shi) can mean the 'warrior's heart', or 'ambition'; Makoto (Sei) means sincerity; and 'Kai' means school. We like to think of our name as the 'School of Sincere Heart', or "Way of the Heart'.
Shihan Higuchi wrote … "For this reason, the value of 'Shisei-Kai' is in aspiring to the way of truth (also sincerity, honesty) through karate."
The Shisei-Kai name is only used by the dojos directly under our lineage. Other members of Gima-Ha in the UK use their own dojo names.
All Shōtōkan groups are welcome to drop in and visit us, but only those with 'heart' should look to Gima-Ha as a future home.

redpajama_set_name: RedPajamaCommonCrawl


text:
/**
 *
 */
package com.systemdesign.dao;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.systemdesign.constants.RoomCapacity;
import com.systemdesign.exception.RoomBookingDaoException;
import com.systemdesign.model.Meeting;
import com.systemdesign.model.MeetingRoom;
import com.systemdesign.model.User;

/**
 * @author kushal
 *
 */
public class RoomBookingDao {

  /**
   * Basically the MeetingRoom will hold a Map<Date,Meeting[48]> (each day is divided into 48 time
   * slices 30 minutes each), a user has to book a room for some multiple of this time slices,
   * whenever a time slice is booked, ll' mark this by adding an entry into Meeting[48] to tell the
   * system that this time slice is booked, while fetching the list of free rooms will check this
   * map to see if the room is available for this given no of given timeSlices
   *
   * @param cityId
   * @param buildingId
   * @param capacity
   * @param date
   * @param startTimeSlice
   * @param timeSlices
   * @return
   * @throws ParseException
   */
  public List<MeetingRoom> getMeetingRooms(int cityId, int buildingId, int capacity, Date date,
      int startTimeSlice, int timeSlices) throws RoomBookingDaoException {

    // Mock list of MeetingRoom's with given cityId, buildingId, capacity and data and available
    // time slices
    Meeting[] meeting1 = {null, null, null, null, null, null, null, new Meeting(), null, null, null,
        null, null, null, null, null, null, null, null, null, null, null, null, null, null, null,
        null, null, null, null, null, null, new Meeting(), null, null, null, null, null, null, null,
        null, null, null, null, null, null, null, null, null, null, null};

    Meeting[] meeting2 = {null, null, null, null, null, null, null, new Meeting(), null, null, null,
        null, null, null, new Meeting(), null, null, null, null, null, null, null, null, null, null,
        null, null, null, null, null, null, null, new Meeting(), null, null, null, null, null, null,
        null, null, null, null, null, null, null, null, null, null, null, null};

    SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
    Date date1 = null;
    Date date2 = null;
    try {
      date1 = sdf.parse("2018-06-28");
      date2 = sdf.parse("2018-07-02");
    } catch (ParseException e) {
      throw new RoomBookingDaoException(e.getMessage());
    }

    Map<Date, Meeting[]> timeSlice1 = new HashMap<>();
    timeSlice1.put(date1, meeting1);
    Map<Date, Meeting[]> timeSlice2 = new HashMap<>();
    timeSlice2.put(date2, meeting2);

    MeetingRoom room1 =
        new MeetingRoom(1, "IWF-01-01", RoomCapacity.BOARD_ROOM.getValue(), 1, timeSlice1);

    MeetingRoom room2 =
        new MeetingRoom(2, "IWF-03-04", RoomCapacity.TWO_SEATER.getValue(), 3, timeSlice1);

    List<MeetingRoom> meetingRooms = new ArrayList<>();
    meetingRooms.add(room1);
    meetingRooms.add(room2);
    // Return the mock data

    return meetingRooms;
  }

  /**
   * Returns the meeting data
   *
   * @param roomId
   * @param userId
   * @param date
   * @param startTimeSlice
   * @param timeSlices
   * @return
   */
  public Meeting bookMeetingRoom(int roomId, int userId, Date date, int startTimeSlice,
      int timeSlices) {

    // Get the meeting room and user data from db
    MeetingRoom room = findById(roomId, MeetingRoom.class);
    User user = findById(userId, User.class);

    // Iterate over the time slices and update the time slices to add an entry for this meeting
    Map<Date, Meeting[]> timeSlice = room.getTimeSlices();
    Meeting[] meetingData = timeSlice.get(date);

    for (int i = startTimeSlice; i <= timeSlices; i++)
      meetingData[startTimeSlice] =
          new Meeting("ABC Meeting", user, room, date, startTimeSlice, timeSlices);

    Meeting meetingDetails =
        new Meeting("ABC Meeting", user, room, date, startTimeSlice, timeSlices);

    return meetingDetails;
  }

  /**
   * Mock method to returns a entity based on the id of the entity and the return class
   *
   * @param id
   * @param returnClass
   * @return
   */
  private <T extends Object> T findById(int id, Class<T> returnClass) {
    return returnClass.cast(new Object());
  }

}

redpajama_set_name: RedPajamaGithub


OUTLINE: Patients undergo standard high-dose rate (HDR) brachytherapy. Approximately 2 hours after brachytherapy, patients undergo catheter-based ultrasound hyperthermia therapy over 60 minutes. Treatment with HDR brachytherapy and hyperthermia therapy repeats within 1-3 weeks. Patients may then undergo 2 additional standard HDR brachytherapy sessions.
Eligible for brachytherapy as determined per clinical standard of care.


Accra, Dec. 3, GNA- An estimated 10,288,382 registered voters out of a population of 18.9 million Ghanaians are expected to cast their ballots in the general elections on Tuesday, December 7 to elect a President and 230 lawmakers.
There are 954 candidates contesting the 230 parliamentary seats while four Presidential candidates representing the People's National Convention (PNC), New Patriotic Party (NPP), National Democratic Congress (NDC) and the Convention People's Party (CPP) are contesting the presidential slot.
Statistics made available to the Ghana News Agency (GNA) by the Electoral Commission (EC) indicate that 1,007,348 voters registered in the Western Region. There are 85 candidates contesting the 22 parliamentary seats.
The Central Region registered 857,067 voters with 19 parliamentary seats being contested by 77 candidates.
The Greater Accra has 2,092,915 voters with 27 parliamentary seats being contested by 125 persons.
The Volta Region has 819,175 voters to decide on 93 contestants in 22 constituencies.
The rest are Eastern Region 1,087,648 voters with 117 candidates vying for 28 parliamentary seats; Ashanti Region 1,891,958 voters to select 39 MPs out of 158 contestants and the Brong Ahafo Region 938,066 voters with 94 candidates contesting 24 seats.
The Northern Region has 892,616 voters with 106 aspiring parliamentary candidates vying for 26 seats; Upper East 425,442 voters to select 13 MPs out of 59 candidates and the Upper West Region 276,147 voters to decide on 10 MPs out of 40 candidates.
The statistics show that the ruling New Patriotic Party (NPP) is contesting in 226 constituencies while the main opposition National Democratic Congress (NDC) is contesting 229 seats.
The Convention People's Party (CPP) is contesting 173 seats, the People's National Convention (PNC) 125 seats, Democratic People's Party (DPP) and the Every Ghanaian Living Everywhere (EGLE) 30 seats each, the Great Consolidated Popular Party (GCPP) 10 seats and the National Reform Party (NRP) five seats.
In 2000 Election, the PNC contested 155 seats, NPP, 199 seats, NDC, 200, CPP, 188, GCPP, three, EGLE, one, UGM, 104, NRP, 173 and Independent candidates, 63 seats.
The regional statistics for Election 2004 indicate that PNC is contesting 11 out of the 22 constituencies in the Western Region; eight out of 19 in the Central Region; 14 out of 27 in the Greater Accra Region; 13 out of 22 in the Volta Region and 13 out of 28 in the Eastern Region.
The rest are 20 out of the 39 in the Ashanti Region; six out of 24 in the Brong Ahafo Region; 19 out of 26 in the Northern Region; 11 out of 13 in the Upper East Region and in all the 10 seats in the Upper West Region.
The NPP is contesting 20 seats in the Western Region; 18 in the Central Region; 27 in the Greater Accra Region; 22 in the Volta Region; 28 in the Eastern Region; 39 in the Ashanti Region; 24 in the Brong Ahafo Region; 26 in the Northern Region; 13 in the Upper East and nine in the Upper West Region.
The NDC is contesting 22 in the Western Region; 19 in the Central Region; 27 in the Greater Accra Region; 22 in the Volta Region; 28 in the Eastern Region; 39 in the Ashanti Region; 23 in the Brong Ahafo Region; 26 in the Northern Region; 13 in the Upper East Region and 10 in the Upper West Region.
The CPP's statistics are as follows: Western 15 seats; Central 15; Greater Accra 25; Volta 13; Eastern 22; Ashanti 33; Brong Ahafo 16; Northern 17; Upper East 12 and Upper West five.
EGLE: Western one seat; Central one; Greater Accra two; Volta eight; Eastern 10; Ashanti three; Brong Ahafo three; Northern two; Upper East nil and Upper West nil.
GCPP: Western two seats; Central three; Greater Accra two; Volta one; Eastern one and Ashanti one. The Party is not contesting in the Brong Ahafo, Northern, Upper East and Upper West regions.
The DPP: Western one seat; Central one; Greater Accra five; Ashanti four; Brong Ahafo 10; Northern five; Upper East two and Upper West two. The DPP is not contesting any of the seats in Volta and Eastern Regions. The NRP is contesting one seat each in the Western, Central and Volta Regions and two seats in the Greater Accra Region.
There are 12 independent candidates contesting seats in the Western Region; 10 in the Central Region; 21 in the Greater Accra Region; 14 each in the Volta and Eastern Regions; 19 in the Ashanti Region; 12 in the Brong Ahafo Region; 11 in the Northern Region; eight in the Upper East Region and four in the Upper West Region.


This course is designed to enable the students to acquire advanced knowledge in special subjects, access latest happenings and research etc. It enables the graduates to successfully overcome every challenge.
Admissions will be made strictly according to the rules and procedures laid down by the University of Pune.
No Candidates shall be admitted to enter the First Year of the B.Com. Degree Course (Revised Structure) unless he/she has passed the Higher Secondary School Certificate Examination of the Maharashtra State Board of Higher Secondary Education Board or equivalent or University with English as a passing subject.
No candidate shall be admitted to the annual examination of the First year B.Com. (Revised Structure) unless he/ she has satisfactorily kept two terms for the course at the college at the college affiliated to this University.
No candidate shall be admitted to the annual examination of the Second Year unless he/she has kept two terms satisfactorily for the course at the college affiliated to this University.
No candidate shall be admitted to the Third year of the B.Com. Degree Course (Revised Structure) unless he/she has passed in all the papers at the First Year B.Com. Examination and has passed in all the papers at the first Year B.Com. Examination and has satisfactorily kept terms for the second year and also two terms for the third year of B.Com. satisfactorily in a college affiliated to this University.


text:
La paroisse de Dufferin est à la fois une paroisse civile et un district de services locaux (DSL) canadien du comté de Charlotte, située au sud-ouest du Nouveau-Brunswick.

Toponyme

La paroisse de Dufferin est nommée ainsi en l'honneur de Frederick Temple Blackwood, marquis de Dufferin et d'Ava (1826-1902), gouverneur général du Canada de 1872 à 1878.

Géographie

Situation
La paroisse de Dufferin est située dans le sud-ouest du comté de Charlotte, à  de route à l'ouest de Saint-Jean et à  au sud-ouest de Fredericton. La paroisse est bordée au sud par le fleuve Sainte-Croix ainsi qu'au nord et à l'est par la baie Oak. L'état américain du Maine se trouve au-delà du fleuve.

La paroisse de Dufferin est limitrophe de Dennis-Weston à l'ouest et de la paroisse de Saint-David au nord. Elle partage également une frontière avec la ville de Saint-Stephen au sud-ouest, par la rivière Dennis. De plus, la ville américaine de Calais se trouve au sud, au-delà du fleuve, alors que Bayside s'élève à l'est, au-delà de la baie Oak. Outre Saint-Stephen, les villes les plus proches sont Saint-Andrews à  au sud-est et Saint-George à  à l'est.

Hameaux et lieux-dits

La paroisse comprend les hameaux de Champlain, Crocker Hill et The Ledge.

Histoire

La paroisse est colonisée en 1784 par une partie de l'Association des Loyalistes de Penobscot. Au plus fort de l'exploitation forestière, entre 1830 et 1850, un village important et un port destiné à l'exportation du bois d'œuvre se développe à The Ledge. Le village est presque abandonné au début du . La municipalité du comté de Charlotte est dissoute en 1966. La paroisse de Dufferin devient un district de services locaux en 1967.

Démographie

D'après le recensement de Statistique Canada, il y avait 548 habitants en 2001, comparativement à 451 en 1996, soit une hausse de 21,5 %. La paroisse compte 235 logements privés, a une superficie de  et une densité de population de 44,2 habitants par kilomètre carré.

Économie

Entreprise Charlotte, membre du Réseau Entreprise, a la responsabilité du développement économique.

Administration

Commission de services régionaux
La paroisse de Dufferin fait partie de la Région 10, une commission de services régionaux (CSR) devant commencer officiellement ses activités le . Contrairement aux municipalités, les DSL sont représentés au conseil par un nombre de représentants proportionnel à leur population et leur assiette fiscale. Ces représentants sont élus par les présidents des DSL mais sont nommés par le gouvernement s'il n'y a pas assez de présidents en fonction. Les services obligatoirement offerts par les CSR sont l'aménagement régional, l'aménagement local dans le cas des DSL, la gestion des déchets solides, la planification des mesures d'urgence ainsi que la collaboration en matière de services de police, la planification et le partage des coûts des infrastructures régionales de sport, de loisirs et de culture; d'autres services pourraient s'ajouter à cette liste.

Comité consultatif
En tant que district de services locaux, Dufferin est administré directement par le Ministère des Gouvernements locaux du Nouveau-Brunswick, secondé par un comité consultatif élu composé de cinq membres dont un président. Il n'y a actuellement aucun comité consultatif.

Budget et fiscalité

Représentation et tendances politiques
: Dufferin fait partie de la circonscription provinciale de Charlotte-Campobello, qui est représentée à l'Assemblée législative du Nouveau-Brunswick par Curtis Malloch, du Parti progressiste-conservateur. Il fut élu en 2010.

: Dufferin fait partie de la circonscription électorale fédérale de Nouveau-Brunswick-Sud-Ouest, qui est représentée à la Chambre des communes du Canada par Gregory Francis Thompson, ministre des Anciens Combattants et membre du Parti conservateur. Il fut élu lors de la  fédérale, en 1988, défait en 1993 puis réélu à chaque fois depuis 1997.

Vivre dans la paroisse de Dufferin
Il n'y a aucune école francophone dans le comté, les plus proches étant à Saint-Jean ou Fredericton. Les établissements d'enseignement supérieurs les plus proches sont quant à eux situés dans le Grand Moncton.

Le bureau de poste et le détachement de la Gendarmerie royale du Canada les plus proches sont à Saint-Stephen.

Les anglophones bénéficient du quotidien Telegraph-Journal, publié à Saint-Jean, et de l'hebdomadaire Saint Croix Courier, publié à Saint-Stephen. Les francophones ont accès par abonnement au quotidien L'Acadie nouvelle, publié à Caraquet, ainsi qu'à l'hebdomadaire L'Étoile, de Dieppe.

Municipalités limitrophes

Notes et références

Notes

Références

Bibliographie

Comté de Charlotte (Nouveau-Brunswick)
District de services locaux au Nouveau-Brunswick
Paroisse civile au Nouveau-Brunswick
Frederick Temple Blackwood

redpajama_set_name: RedPajamaWikipedia


Anders is an Assistant Professor at Harvard Kennedy School. His research focuses on public economics and development economics. He is currently involved in projects in Guatemala, Ghana, and Zambia. He received his PhD in Economics from London School of Economics in 2016, and is a Post-Doctoral Fellow in Long-Run Fiscal Policy at the NBER for the academic year 2016-17.


Wow. Those are some serious thoughts.
The point about aging is interesting. We happen to be around a lot of older people because our church is mostly older and we have some older (but still quite active) neighbors. I don't know how old our next-door neighbor is. He looks like he could be getting close to 80, but he still mows and snowblows his fairly large property himself. And we have a woman in her 80s who still actively manages our church's free lunch program. It's just hard to tell from anecdotes how common that really is. Which I guess was your exact point.


text:
		<link href="pre_style.css" rel="stylesheet" type="text/css" />

Introduction to PHP & AJAX

 Hello WAMADians!

 Today we will be going over PHP:

What is PHP?

Why and How to use PHP?

What PHP looks like?

 Today we will be going over AJAX:

What does AJAX stand for?

Why should anyone care?

How does asynchronous programming works?

What is PHP 1/2

PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. (http://www.php.net/manual/en/intro-whatis.php)

					Things to note:

Scripting Language: pros and cons of scripting languages, but it can also be Object Oriented.

Embedded into HTML: can be inserted into HTML

Preprocessor?... like "#define" in C or "private static final" in Java. But can do more then just preprocess.

					Pros:

Apache and PHP are attached at the hip.

I have seen a version for PHP on every OS that I can think of.

What is PHP 2/2

What PHP looks like:

					<textarea style="font-size:30px; width:100%;" rows="10">
<html>
    <head>
        <title>Example</title>
    </head>
    <body>
        <?php
            echo "Hi, I'm a PHP script!";
        ?>
    </body>
</html></textarea>

Why and How to use PHP?

Why use PHP?

For Server Side Scripting

For Command line scripting

For Writing desktop applications

How to use PHP?

HTML Embedded (preprocessor)

Server-side (procedural/object-oriented)

Hybrid (mixed)

What PHP looks like?

What makes php special

Variables start with a $ sign

Concatenate strings is with . not a + sign.

No explicit variable types like javascript, but there is type casting

Double quotes are not the same as single quotes. Examples below:

PHPs many functions!

Server side function like Database calles/queries, file handling, image manipulation, Session variables, Cyptography, Credit card handling, Compression etc. etc...

					<textarea style="font-size:30px; width:100%;" rows="6">
$var = 5; // auto typeset to integer
$str = "Hello" . ", World"; // results in "Hello, World"
$str .= " Peace"; // Results in "Hello, World Peace"
$double = "$var = $var"; Results in 5 = 5
$single = '$var = $var'; Results in $var = $var
$proper = '$var = ' . $var; Results in $var = 5</textarea>

What does AJAX stand for

Asynchronous JavaScript and XML... that is all

How does AJAX works?

					Figure 1: The traditional model for web applications (left) compared to the Ajax model (right).

					Figure 2: The synchronous interaction pattern of a traditional web application (top) compared with the asynchronous pattern of an Ajax application (bottom).

Why should anyone care?

Well... because!

J.K. But see The AJAX PHP demo to see what I mean!

No need to explain after seeing that

Resources

J. J. Garrett. "Ajax: A New Approach to Web Applications." http://www.adaptivepath.com/ideas/ajax-new-approach-web-applications/. 18 Feb 2005

"What is PHP?" http://www.php.net/manual/en/intro-whatis.php. 4 Nov 2013


The perfect MultiFunction tester for any Electrician or apprentice.
Insulation, Loop/Line, RCD, Phase, Voltage testing all in one !
5 instruments in 1, the CZ20500 Multi-Function Electrical Tester is a must have!
The CZ20500 Multi-Function electrical tester includes some of the latest advances in electrical testing, combining 5 instruments into 1! Insulation, Loop, Phase, RCD, and Voltage testing are all included in the one instrument, making your job of installation testing easier.
The CZ20500 Multi-Function electrical tester is the perfect tool for certification of domestic, commercial, and industrial fixed wiring installations as required by Australian standards.
The CZ20500 is a comprehensive digital multifunctional electrical safety test instrument. It features large-scale integrated analogy circuits, digital circuits and a microcomputer chip combination. The CZ20500 has multiple functions, higher accuracy and a more stable performance than its competitors, as well as a reliable and convenient mode of operation.
It can be used to measure residual current devices, electric leakage protectors and the insulation of various electrical equipment, as well as performing grounding continuity tests.
This Multi-Function Electrical Tester is your ideal choice for the maintenance, testing and inspection of electric protector of various electrical equipment.


To provide a conduit to share his passion for aesthetic dentistry, Dr. Koczarski founded PAC-live, The Northwest Aesthetic Continuum and The Hornbrook Group. Dr. Koczarski has published numerous articles on the topics of aesthetic dentistry and is the editor for the Journal of Cosmetic Dentistry and is on the editorial board for PPAD journal.
Dr. Koczarski will discuss how an intra oral mock-up can provide valuable information to the patient, the lab and the dentist. He will demonstrate a freehand composite mock-up technique on the front four incisors. The process of a mock-up is not only designed to show what can be achieved but also what cannot be achieved. Dr. Koczarski will follow the AACD accreditation standards for the Case 5, 6 or more direct resins veneers but will only be demonstrating the technique on the front four incisors.
Describe treatment options for a patient with short front incisors, old silver amalgams and limited fullness of the buccal corridor.
Discuss smile design utilizing the length, horizon and midline of the smile combining the eyes, commissure and the lower part of the face.
Compare and contrast a wax-up and a mock-up.
Describe the necessary steps to complete an intra oral mock-up.
Simon, Harel and Magne, Pascal. Clinically Based Diagnostic Wax-up for Optimal Estetics: The Diagnostic Mock-up. CDA Journal., Vol 36(5), 2008.
In part two of Dr. Michael Koczarski\'s program on taking a mock-up to completion.


text:
Believers Rally After HGTV Cancels Reality Show Starring Christian Twin Brothers Who Oppose Abortion, Homosexuality, Divorce
CP Current Page: Entertainment | Thursday, May 08, 2014
By Leonardo Blair, Christian Post Reporter | Thursday, May 08, 2014
David (l) and Jason Benham (r). | (Photo: Facebook/Benham Brothers)
Christian viewers of the Home & Garden Television (HGTV) are now up in arms after the network announced Wednesday that they have "decided not to move forward" with a reality show called "Flip It Forward" starring conservative Christian twin brothers David and Jason Benham who are openly opposed to abortion, homosexuality and no-fault divorce.
The episodes of the show were expected to reflect the brothers' experience in flipping houses for profit and helping other families do the same "with a healthy dose of sibling rivalry between the brothers along the way," according to MSNBC.
HGTV made the announcement via social media just one day after Right Wing Watch highlighted their views in a report.
HGTV has decided not to move forward with the Benham Brothers' series.
— HGTV (@hgtv) May 7, 2014
In response to HGTV's decision, the Benham brothers said efforts seeking to paint their views as hateful are misinformed or untrue in a joint statement Thursday.
"The first and last thought on our minds as we begin and end each day is; have we shined Christ's light today? Our faith is the fundamental calling in our lives, and the centerpiece of who we are. As Christians we are called to love our fellow man. Anyone who suggests that we hate homosexuals or people of other faiths is either misinformed or lying," said the brothers.
"Over the last decade, we've sold thousands of homes with the guiding principle of producing value and breathing life into each family that has crossed our path, and we do not, nor will we ever discriminate against people who do not share our views," they explained.
"We were saddened to hear HGTV's decision. With all of the grotesque things that can be seen and heard on television today you would think there would be room for two twin brothers who are faithful to our families, committed to biblical principles, and dedicated professionals. If our faith costs us a television show then so be it," they ended.
Christian fans, however, have not taken HGTV's decision lightly.
Faith Driven Consumer — the group behind the IStandWithPhil.com movement which played a key role in the return of Phil Robertson to Duck Dynasty after he was suspended indefinitely from the show for his views on homosexuality, is now throwing their weight behind the Benham brothers.
"Following the sudden cancellation of the upcoming HGTV series "Flip It Forward' simply because the stars of the show, brothers David and Jason Benham, hold a biblical worldview on the issues of life and marriage, Faith Driven Consumer is announcing a brand new grassroots campaign called #FlipThisDecision to return the show to the network's fall lineup," said the group in a release Thursday.
"This is pure intolerance, discrimination, and bullying toward those who hold to widely held and legitimate views. HGTV's rash actions hold no place in America's rainbow of diversity," said Chris Stone, founder of Faith Driven Consumer and Certified Brand Strategist.
"Whether people agree or disagree with the Beham's faith-driven perspective is beside the point; the Benhams have a right to have those views and to be treated equally with those who hold to other viewpoints. This is the very definition of tolerance," he said.
As news spread of the decision Thursday, Christian viewers flocked to HGTV's Facebook page and peppered it with comments protesting the decision and threatening to boycott the network.
"Not only will I boycott HGTV but I will boycott every sponsor until they drop HGTV. I will email or call each one. If Christians do not stand up to discrimination then those Christians are a BIG part of the problem," wrote Maryanne Linkes.
"Very disappointed in HGTV for allowing all groups to be represented EXCEPT CHRISTIANS! You've lost a viewer because you're more concerned with being politically correct than actually helping people," said Terry Bruce Vogenthaler.
Contact: leonardo.blair@christianpost.comFollow Leonardo Blair on Twitter: @leoblairFollow Leonardo Blair on Facebook: LeoBlairChristianPost


Below you will find oral and poster presentation tips to help with your conference presentation.
Your hands are all clammy, your throat is dry, and you can't decide what is worse, the immense sweat pouring from your body or the abnormal beating of your heart in your chest. Suddenly your name is called its your turn to speak. Panic sets in.
These reactions to public speaking are quite common and one should expect a little nervousness when speaking in public. Public speaking is one of the more anxiety-provoking situations people face. The anxiety is manageable and can be lessened, though not completely eliminated. However, you can be nervous and still do a good presentation.
Take a deep breath before you stand up to approach the podium. Then take another deep breath before you begin speaking.
During the speech, breathe properly by taking deep breaths. The audience appreciates little pauses during the speech, you catch your breath (literally) and they catch their breath (figuratively).
If your face tends to get red during public speaking, wear a coloured shirt to offset color of your face (a red shirt for example). If you think you look pale, wear a lighter colour.
If you think your neck turns red or blotchy when speaking, wear a high neck shirt.
If your hands get sweaty, put a little Kleenex in your pocket to wipe off hands, no one will ever know!
If you feel a rush of nervousness and speak too quickly, try to "feel your feet." This means concentrate briefly on the sensation in your feet, toes, the weight of your shoes, etc., it's a little trick that helps focus attention away from nervous physiological reactions.
Visual – refers to how you compose yourself. Make a good first impression by looking at the people around the room, not only at the computer slide or paper. In most audiences, people want to be supportive, so they will smile and nod if you look at them. Try to omit distracting gestures (like stray hand movements) but feel free to use enhancing gestures. For example, if you say "on one hand…" hold out your hand, and when you say "on the other hand…" hold out your other hand. If you make a sweeping statement, a sweeping gesture with your hand will seem appropriate (e.g., "In all of psychology, no theory has been as influential as …").
Vocal – refers to the sound of your voice. Keep the volume up even at the ends of sentences. We all have a tendency to speak lower and faster when nervous, but try not to fall into this habit. Speaking slowly helps the listeners to follow your points. If you practice your speech 100 times, it is still the first time that the audience is hearing it – give them a chance to process the information. Plan to pause from time to time. Be enthusiastic about your topic, avoid monotone – remember this is your topic that you worked super hard to research so be proud and express yourself! Speak loudly, without shouting. Speak at the volume you would use to talk to a friend sitting in the back row of the room. Breathing deeply helps with keeping good volume. Use a variety of speeds, slow down for emphasis.
Verbal – refers to the words you use and the organization of your presentation. Make sure your presentation has a clear introduction, body and conclusion. Be sure to define your terms early on, and organize them well. Include attention letters such as an interesting picture, quotation, or statistics related to your topic, then state your purpose in one sentence. Talk your audience through the organization of your presentation. "We will discuss three issues, first X, then Y, and end with a discussion of Z." At the end of each section, tell the audience that you are now moving on to the next section. If you use overheads or PowerPoint, use these as aides to what you say. Use them to show organization, define your terms, show pictures, figures, and tables. Don't overload the audience's eyes with information that will distract from what you are saying. The visual supports the verbal, not the other way around. This means that your ideas, your words should be the focus of the audience. Find 2 or 3 things that you want them to remember. "If you remember nothing else, remember these 3 things…" Build your presentation to support those conclusions, provide the most important details of method and theory that are necessary to see your conclusions as credible. Oral presentations cannot be as detailed as a journal article or thesis paper, though the organization can be similar. Be sure to have a clear conclusion, restate your purpose and main points.
If anything goes wrong with the computer equipment, overhead, etc., don't panic. It happens to everyone at one time or another. The problem can probably very easily be fixed, and there will be several people ready to offer assistance. Best of all, the audience will be completely sympathetic to you. Remain calm and proceed as best you can.
Remember: There is always too much text!
Why are these results unique/important?
How does this relate to other research?
Practice your presentation ahead of time, and time it!


text:
Q: Difference between clustered index and partitioning column As I understand partitioning, the column that you partition on is how things are divided into partitions.
But does the partitioning column have anything to do with order on the disk? (Aside from which partition the data goes into.)
Say I have a table that looks like this:
CREATE TABLE [dbo].[Something](
    [SomethingId] [bigint] IDENTITY(1,1) NOT NULL,
    [OtherThingId] [bigint] NOT NULL,
    [CreatedBy] [int] NOT NULL,
    [CreatedWhen] [datetime] NOT NULL,
    [CreatedWhere] [varchar](255) NOT NULL,
     CONSTRAINT [PK_Something] PRIMARY KEY CLUSTERED
    (
        [SomethingId] DESC
        [OtherThingId] DESC,
    )
) ON OtherThingIdPartitionScheme(OtherThingId)

The partition that a new row will be stored on is decided by OtherThingId.
But I setup the clustered index to be first SomethingId, then by OtherThingId.
Does that mean that each partition will be ordered by SomethingId, then by OtherThingId?
(Even though it is partitioned only by the secondary value of OtherThingId.)
(I am setting up a lot of partitioning right now and I want to be sure I fully understand it.)

A:
But does the partitioning column have anything to do with order on the disk?

From Clustered Index Structures :

Clustered indexes have one row in sys.partitions, with index_id = 1 for each partition used by the index. By default, a clustered index has a single partition. When a clustered index has multiple partitions, each partition has a B-tree structure that contains the data for that specific partition. For example, if a clustered index has four partitions, there are four B-tree structures; one in each partition.

From Table and Index Organization:
When a table or index uses multiple partitions, the data is partitioned horizontally so that groups of rows are mapped into individual partitions, based on a specified column. The partitions can be put on one or more filegroups in the database. The table or index is treated as a single logical entity when queries or updates are performed on the data.
The pages in the data chain and the rows in them are ordered on the value of the clustered index key. All inserts are made at the point where the key value in the inserted row fits in the ordering sequence among existing rows. The page collections for the B-tree are anchored by page pointers in the sys.system_internals_allocation_units system view.
If you want to really dig more into how data is layed out then refer to : Inside The Storage Engine: sp_AllocationMetadata
As a side note : SQL Server 2016 CTP2 has TRUNCATE TABLE ... [ WITH ( PARTITIONS ( {  |  }
For automating partitioning switching check out - SQL Server Partition Management utility

A: To add to the answer of Kin Shah, this query might help:
SELECT
 o.type_desc             AS  objtype,
 u.name                  AS [schema],
 o.name                  AS  object,
 i.type_desc             AS  idxtype,
 i.index_id              AS  idxid,
 ISNULL(i.name, '*Heap') AS [index],
 p.partition_number      AS  partitn,
 p.partition_id,
 p.rows,
 p.data_compression_desc AS  compr,
 a.allocation_unit_id    AS  alloc_unit_id,
 a.type_desc             AS  alloc_unit_type,
 8 * a.total_pages       AS  totl_KB,
 8 * a.used_pages        AS  used_KB,
 8 * CASE WHEN i.index_id < 2 THEN a.data_pages
     END                 AS  data_KB,
 CONVERT(varchar(6),  CONVERT(int, CONVERT(binary(2), REVERSE(SUBSTRING(a.first_page,     5, 2))))) + ':' +
 CONVERT(varchar(20), CONVERT(int, CONVERT(binary(4), REVERSE(SUBSTRING(a.first_page,     1, 4))))) AS first_page,
 CONVERT(varchar(6),  CONVERT(int, CONVERT(binary(2), REVERSE(SUBSTRING(a.root_page,      5, 2))))) + ':' +
 CONVERT(varchar(20), CONVERT(int, CONVERT(binary(4), REVERSE(SUBSTRING(a.root_page,      1, 4))))) AS root_page,
 CONVERT(varchar(6),  CONVERT(int, CONVERT(binary(2), REVERSE(SUBSTRING(a.first_iam_page, 5, 2))))) + ':' +
 CONVERT(varchar(20), CONVERT(int, CONVERT(binary(4), REVERSE(SUBSTRING(a.first_iam_page, 1, 4))))) AS first_iam_page
FROM       sys.objects AS o
LEFT  JOIN sys.schemas AS u
 ON  o.schema_id = u.schema_id
LEFT  JOIN sys.indexes AS i
 ON  o.object_id = i.object_id
INNER JOIN sys.partitions AS p
 ON  i.object_id = p.object_id
 AND i.index_id  = p.index_id
LEFT  JOIN sys.system_internals_allocation_units AS a  -- Internal variant of sys.allocation_units.
 ON p.partition_id = a.container_id
WHERE o.is_ms_shipped = 0
 AND  a.total_pages   > 0
ORDER BY o.type, u.name, o.name, i.index_id, p.partition_number

redpajama_set_name: RedPajamaStackExchange


text:
\section{Derivation of VAN}
\label{appendix:derivation_van}

Denote the mean parameters of $q_t(\mbox{$\myvecsym{\theta}$})$ by $\mbox{$\myvec{m}$}_t$ which is equal to the expected value of the sufficient statistics $\mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$})$, i.e., $\mbox{$\myvec{m}$}_t := \mathbb{E}_{q_t} [\mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$})] $. The mirror descent update at iteration $t$ is given by the solution to
\begin{align}
\mbox{$\myvec{m}$}_{t+1}
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}} \left\langle \mbox{$\myvec{m}$}, \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle + \frac{1}{\beta_t}\dkls{}{q}{q_t} \\
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}} \mathbb{E}_q \left[ \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle + \log\left((q/q_t)^{1/\beta_t}\right) \right] \\
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}} \mathbb{E}_q \left[ \log \frac{ \exp \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle q^{1/\beta_t} }{ q_t^{1/\beta_t}} \right] \\
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}} \mathbb{E}_q \left[ \log \left( \frac{q^{1/\beta_t}}{q_t^{1/\beta_t} \exp \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), -\widehat{\nabla}_{m} \mathcal{L}_t \right\rangle } \right) \right] \\
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}} \frac{1}{\beta_t} \,\, \mathbb{E}_q \left[ \log \left( \frac{q}{q_t \exp \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), -\beta_t \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle } \right) \right] \\
&= \mathop{\mathrm{argmin}}_{\boldsymbol{m}}  \frac{1}{\beta_t} \,\,\mathbb{D}_{KL}\sqr{ q \| q_t \exp \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), -\beta_t \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle /\mathcal{Z}_t} .
\end{align}
where $\mathcal{Z}$ is the normalizing constant of the distribution in the denominator which is a function of the gradient and step size.

Minimizing this KL divergence gives the update
\begin{equation}
q_{t+1}(\mbox{$\myvecsym{\theta}$}) \propto q_t(\mbox{$\myvecsym{\theta}$})\exp \left\langle \mbox{$\myvecsym{\phi}$}(\mbox{$\myvecsym{\theta}$}), -\beta_t \widehat{\nabla}_{m} \mathcal{L}_t \right\rangle.
\end{equation}
By rewriting this, we see that we get an update in the natural parameters $\mbox{$\myvecsym{\lambda}$}_t$ of $q_t(\mbox{$\myvecsym{\theta}$})$, i.e.
\begin{equation}
\mbox{$\myvecsym{\lambda}$}_{t+1} = \mbox{$\myvecsym{\lambda}$}_t - \beta_t \widehat{\nabla}_{m} \mathcal{L}_t.
\end{equation}
Recalling that the mean parameters of a Gaussian $q(\mbox{$\myvecsym{\theta}$}) = \mbox{${\cal N}$}(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\mu}$},\mbox{$\myvecsym{\Sigma}$})$ are $\mbox{$\myvec{m}$}^{(1)} = \mbox{$\myvecsym{\mu}$}$ and $\mbox{$\myvec{M}$}^{(2)} = \mbox{$\myvecsym{\Sigma}$} + \mbox{$\myvecsym{\mu}$}\vmu ^T$ and using the chain rule, we can express the gradient $\widehat{\nabla}_{m} \mathcal{L}_t$ in terms of $\mbox{$\myvecsym{\mu}$}$ and $\mbox{$\myvecsym{\Sigma}$}$,
\begin{align}
    \widehat{\nabla}_{m^{(1)}} \mathcal{L} &= \widehat{\nabla}_{\mu} \mathcal{L} - 2\left[\widehat{\nabla}_{\Sigma} \mathcal{L} \right]\mbox{$\myvecsym{\mu}$} \\
    \widehat{\nabla}_{M^{(2)}} \mathcal{L} &= \widehat{\nabla}_{\Sigma} \mathcal{L}.
\end{align}
Finally, recalling that the natural parameters of a Gaussian $q(\mbox{$\myvecsym{\theta}$}) = \mbox{${\cal N}$}(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\mu}$},\mbox{$\myvecsym{\Sigma}$})$ are $\mbox{$\myvecsym{\lambda}$}^{(1)} = \mbox{$\myvecsym{\Sigma}$}^{-1}\mbox{$\myvecsym{\mu}$}$ and $\mbox{$\myvecsym{\lambda}$}^{(2)} = - \frac{1}{2}\mbox{$\myvecsym{\Sigma}$}^{-1}$, we can rewrite the VAN updates in terms of $\mbox{$\myvecsym{\mu}$}$ and $\mbox{$\myvecsym{\Sigma}$}$,
\begin{align}
    \mbox{$\myvecsym{\Sigma}$}_{t+1}^{-1} &= \mbox{$\myvecsym{\Sigma}$}_t^{-1} + 2\beta_t \left[ \widehat{\nabla}_{\Sigma} \mathcal{L}_t \right] \\
    \mbox{$\myvecsym{\mu}$}_{t+1} &= \mbox{$\myvecsym{\Sigma}$}_{t+1} \left[ \mbox{$\myvecsym{\Sigma}$}_t^{-1}\mbox{$\myvecsym{\mu}$}_t - \beta_t \left( \widehat{\nabla}_{\mu} \mathcal{L}_t - 2\left[\widehat{\nabla}_{\Sigma} \mathcal{L}_t \right] \mbox{$\myvecsym{\mu}$}_t \right) \right] \\
    &= \mbox{$\myvecsym{\Sigma}$}_{t+1} \left( \mbox{$\myvecsym{\Sigma}$}_t^{-1} + 2\beta_t \left[ \widehat{\nabla}_{\Sigma} \mathcal{L}_t \right] \right)\mbox{$\myvecsym{\mu}$}_t - \beta_t\mbox{$\myvecsym{\Sigma}$}_{t+1} \left[ \widehat{\nabla}_{\mu} \mathcal{L}_t \right] \\
    &= \mbox{$\myvecsym{\mu}$}_t - \beta_t\mbox{$\myvecsym{\Sigma}$}_{t+1} \left[ \widehat{\nabla}_{\mu} \mathcal{L}_t \right].
\end{align}

\section{Details on the RL experiment}
\label{appendix:rl}

In this section, we give details of the parameter-based exploration task in reinforcement learning (RL). An important open question in reinforcement learning is how to efficiently explore the state and action space. An agent always acting greedily according to the policy results in a pure exploitation. Exploration is necessary to visit inferior state and actions once in while to see if they might really be better.
Traditionally, exploration is performed in the action space by, e.g., injecting noise to the policy output.
However, injecting noise to the action space may not be sufficient in problems where the reward is sparse, i.e., the agent rarely observes the reward of their actions.
In such problems, the agent requires a rich explorative behaviour in which noises in the action space cannot provide.
An alternative approach is to perform exploration in the \emph{parameter space}~\citep{DBLP:journals/paladyn/RuckstiessSSWSS10}.
In this section, we demonstrate that variational distribution $q(\mbox{$\myvecsym{\theta}$})$ obtained using VAN can be straightforwardly used for such exploration in parameter space, $\mbox{$\myvecsym{\theta}$}$.

\subsection{Background}

First, we give a brief background on reinforcement learning (RL).
RL aims to solve the sequential decision making problem where at each discrete time step $t$ an agent observes a state $\mbox{$\myvec{s}$}_t$ and selects an action $\mbox{$\myvec{a}$}_t$ using a policy $\pi$, i.e., $\mbox{$\myvec{a}$}_t \sim \pi(\mbox{$\myvec{a}$}|\mbox{$\myvec{s}$}_t)$.
The agent then receives an immediate reward $r_t = r(\mbox{$\myvec{s}$}_t,\mbox{$\myvec{a}$}_t)$ and observes a next state $\mbox{$\myvec{s}$}_t \sim p(\mbox{$\myvec{s}$}'|\mbox{$\myvec{s}$}_t, \mbox{$\myvec{a}$}_t)$.
The goal in RL is to learn the optimal policy $\pi^*$ which maximizes the expected returns $\mathbb{E}\left[ \sum_{t}^\infty \gamma^{t-1} r_t \right]$ where $\gamma$ is the discounted factor and the expectation is taken over a sequence of densities $\pi(\mbox{$\myvec{a}$}|\mbox{$\myvec{s}$}_t)$ and $p(\mbox{$\myvec{s}$}'|\mbox{$\myvec{s}$}_t, \mbox{$\myvec{a}$}_t)$.

A central component of RL algorithms is the state-action value function or the Q-function $Q^{\pi}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$ gives the expected return after executing an action $\mbox{$\myvec{a}$}$ in a state $\mbox{$\myvec{s}$}$ and following the policy $\pi$ afterwards. Formally, it is defined as follows:
\begin{align}
Q^{\pi}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) = \mathbb{E}\left[ \sum_{t=1}^\infty \gamma^{t-1} r_t | \mbox{$\myvec{s}$}_1 = \mbox{$\myvec{s}$}, \mbox{$\myvec{a}$}_1 = \mbox{$\myvec{a}$}\right].
\end{align}
The Q-function also satisfies a recursive relation also known as the Bellman equation:
\begin{align}
Q^{\pi}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) = r(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) + \gamma \mathbb{E}_{p(\mbox{$\myvec{s}$}'|\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}), \pi(\mbox{$\myvec{a}$}'|\mbox{$\myvec{s}$}')} \left[ Q^{\pi}(\mbox{$\myvec{s}$}', \mbox{$\myvec{a}$}') \right].
\end{align}
Using the Q-function, the goal of reinforcement learning can be simply stated as finding a policy which maximizes the expected Q-function, i.e.,
\begin{align}
\pi^* = \mathop{\mathrm{argmax}}_{\pi} \iint p(\mbox{$\myvec{s}$}) \pi(\mbox{$\myvec{a}$}|\mbox{$\myvec{s}$}) Q^{\pi}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) \mathrm{d}\boldsymbol{s} \mathrm{d}\boldsymbol{a}. \label{eq:rl_q_objective}
\end{align}
In practice, the policy is represented by a parameterized function such as neural networks with policy parameter $\boldsymbol{\theta}$ and the goal is to instead find the optimal parameters $\boldsymbol{\theta}^\star$.

\subsubsection{Deterministic Policy Gradients}

Our parameter-based exploration via VAN can be applied to any reinforcement learning algorithms which rely on gradient ascent to optimize the policy parameter $\boldsymbol{\theta}$.
For demonstration, we focus on a simple yet efficient algorithm called the deterministic policy gradients algorithm (DPG)~\citep{DBLP:conf/icml/SilverLHDWR14}.
Simply speaking, DPG aims to find a deterministic policy that maximizes the action-value function by gradient ascent.
Since in practice the action-value function is unknown, DPG learns a function approximator $Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$ with a parameter $\boldsymbol{v}$ such that $Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) \approx Q^{\pi_{\boldsymbol{\theta}}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$.
Then, DPG finds $\boldsymbol{\theta}^*$ which locally minimize an objective $f(\boldsymbol{\theta}) = -\mathbb{E}_{p(\mbox{$\myvec{s}$})} \left[ Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$}, \pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$})) \right]$ by gradient ascent where the gradient is given by
\begin{align}
\nabla_{\boldsymbol{\theta}} f(\boldsymbol{\theta})
&= -\nabla_{\boldsymbol{\theta}} \int p(\mbox{$\myvec{s}$}) Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$})) \mathrm{d}\boldsymbol{s} \notag \\
&= -\int p(\mbox{$\myvec{s}$}) \nabla_{\boldsymbol{\theta}} Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$})) \mathrm{d}\boldsymbol{s} \notag \\
&= -\int p(\mbox{$\myvec{s}$}) \nabla_{\mbox{$\myvec{a}$}} Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})|_{\mbox{$\myvec{a}$} = \pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$})} \nabla_{\boldsymbol{\theta}} \pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$}) \mathrm{d}\boldsymbol{s}. \label{eq:dpg_update}
\end{align}

The parameter $\boldsymbol{v}$ of $Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$ may be learned by any policy evaluation methods.
Here, we adopted the approach proposed by~\citep{DBLP:journals/corr/LillicrapHPHETS15} which minimizes the squared Bellman residual to the slowly moving target action-value function.
More precisely, $\boldsymbol{v}$ is updated by
\begin{align}
\min_{\boldsymbol{v}} \mathbb{E}\left[ \left( Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) - r(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$}) - \gamma \tilde{Q}_{\tilde{\boldsymbol{v}}}(\mbox{$\myvec{s}$}', \tilde{\pi}_{\tilde{\boldsymbol{\theta}}}(\mbox{$\myvec{s}$}'))\right)^2  \right],
\label{eq:q_residual}
\end{align}
where the expectation is taken over $p(\mbox{$\myvec{s}$})$ and $p(\mbox{$\myvec{s}$}'|\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$.
The $\tilde{Q}_{\tilde{\boldsymbol{v}}}$ and $\tilde{\pi}_{\tilde{\boldsymbol{\theta}}}$ denote target networks which are  separate function approximators that slowly tracks $Q_{\boldsymbol{v}}$ and $\pi_{\boldsymbol{\theta}}$, respectively. The target networks help at stabilizing the learning procedure~\citep{DBLP:journals/corr/LillicrapHPHETS15}.

Overall, DPG is an actor-critic algorithm that iteratively update the critic (action-value function) by taking gradient of Eq.\eqref{eq:q_residual} and update the actor (policy) by the gradient Eq.\eqref{eq:dpg_update}.
However, the crucial issue of DPG is that it uses a deterministic policy and does not perform exploration by itself.
In practice, exploration is done for DPG by injecting a noise to the policy output, i.e., $\mbox{$\myvec{a}$} = \pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$}) + \boldsymbol{\epsilon}$ where $\boldsymbol{\epsilon}$ is a noise from some random process such as Gaussian noise.
However, as discussed about, action-space noise may be insufficient in some problems.
Next, we show that VAN can be straightforwardly applied to DPG to obtain \emph{parameter-based exploration DPG}.

\subsubsection{Parameter-based Exploration DDPG}
To perform parameter-based exploration, we can relax the policy-gradient objective $f(\boldsymbol{\theta})$ by assuming that the parameter $\boldsymbol{\theta}$ is sampled from a distribution $q(\boldsymbol{\theta}) := \mathcal{N}(\boldsymbol{\theta} | \boldsymbol{\mu}, \boldsymbol{\Sigma})$, and solve the following optimization problem:
\begin{align}
   \min_{\boldsymbol{\mu},\boldsymbol{\Sigma}} \,\, \mathbb{E}_{\mathcal{N}(\boldsymbol{\theta}|\boldsymbol{\mu},\boldsymbol{\Sigma})}[f(\boldsymbol{\theta})]
\end{align}
This is exactly the VO problem of \eqref{eq:VO}.
The stochasticity of $\boldsymbol{\theta}$ through $q(\mbox{$\myvecsym{\theta}$})$ allows the agent to explore the state and action space by varying its policy parameters.
This exploration strategy is advantageous since the agent can now exhibits much more richer explorative behaviours when compared with exploration by action noise injection.

   Algorithm \ref{algorithm:dpg_van} outlines \emph{parameter-based exploration DPG via VAN}.
   \begin{algorithm}[t]
      \begin{algorithmic}[1]
   	\WHILE {Not converged}
   		\STATE Observe state $\mbox{$\myvec{s}$}_t$, sample parameter $\boldsymbol{\theta} \sim \mathcal{N}(\boldsymbol{\theta} | \boldsymbol{\mu}, \boldsymbol{\Sigma})$, take action $\mbox{$\myvec{a}$}_t = \pi_{\boldsymbol{\theta}}(\mbox{$\myvec{s}$}_t)$, observe reward $r_t$ and next state $\mbox{$\myvec{s}$}'_t$.
   		\STATE Add $(\mbox{$\myvec{s}$}_t, \mbox{$\myvec{a}$}_t, r_t, \mbox{$\myvec{s}$}'_t)$ to a replay buffer $\mathcal{D}$.
   		\FOR {$i = 1,\dots,K$}
   			\STATE Drawn N minibatch samples $\{(\mbox{$\myvec{s}$}_i, \mbox{$\myvec{a}$}_i, r_i, \mbox{$\myvec{s}$}'_i)\}_{i=1}^N$ from $\mathcal{D}$.
   			\STATE Update $Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$},\mbox{$\myvec{a}$})$ by gradient descent:
   			\begin{align*}
   			\boldsymbol{v} \leftarrow \boldsymbol{v} + \alpha_{\boldsymbol{v}} \nabla_{\boldsymbol{v}} \frac{1}{N} \sum_{i=1}^N
   			\left[ \left( Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$}_i,\mbox{$\myvec{a}$}_i) - y_i\right)^2  \right],
   			\end{align*}
   			where $y_i = r(\mbox{$\myvec{s}$}_i,\mbox{$\myvec{a}$}_i) + \gamma \tilde{Q}_{\tilde{\boldsymbol{v}}}(\mbox{$\myvec{s}$}'_i, \pi_{\tilde{\boldsymbol{\mu}}}(\mbox{$\myvec{s}$}'_i))$.
   			\STATE Update parameter of $q(\boldsymbol{\theta})$ by sVAN in Eq.\eqref{eq:Vand_prec_0}  with
   			\begin{align}
               \widehat{\nabla}_{\sigma} \mathcal{L}(\boldsymbol{\mu}_t, \mbox{$\myvecsym{\sigma}$}_t^2)
   			&= - \frac{1}{NM}\sum_{i=1}^N\sum_{j=1}^M \left[ \nabla_{\sigma}\pi_{\boldsymbol{\theta}_j}(\mbox{$\myvec{s}$}_i) \nabla_{\boldsymbol{a}} Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$}_i,\boldsymbol{a})|_{\boldsymbol{a} = \pi_{\boldsymbol{\theta}_j}(\mbox{$\myvec{s}$}_i)} \right], \\
   			\widehat{\nabla}_{\boldsymbol{\mu}} \mathcal{L}(\boldsymbol{\mu}_t, \mbox{$\myvecsym{\sigma}$}_t^2)
   			&= - \frac{1}{NM}\sum_{i=1}^N\sum_{j=1}^M \left[ \nabla_{\boldsymbol{\mu}}\pi_{\boldsymbol{\theta}_j}(\mbox{$\myvec{s}$}_i) \nabla_{\boldsymbol{a}} Q_{\boldsymbol{v}}(\mbox{$\myvec{s}$}_i,\boldsymbol{a})|_{\boldsymbol{a} = \pi_{\boldsymbol{\theta}_j}(\mbox{$\myvec{s}$}_i)} \right],
   			\end{align}
   			where $\{\boldsymbol{\theta}_j \}_{j=1}^M \sim q(\boldsymbol{\theta})$.

   			\STATE Update target network parameters $\tilde{\boldsymbol{v}}$ and $\tilde{\boldsymbol{\mu}}$ by moving average with, e.g., step size $\tau = 0.001$:
   			\begin{align*}
   			\tilde{\boldsymbol{v}} &\leftarrow (1 - \tau) \tilde{\boldsymbol{v}} + \tau \boldsymbol{v}, \\
   			\tilde{\boldsymbol{\mu}} &\leftarrow (1 - \tau) \tilde{\boldsymbol{\mu}} + \tau \boldsymbol{\mu}.
   			\end{align*}
   		\ENDFOR
   	\ENDWHILE
   	\end{algorithmic}
   \caption{Parameter-based exploration DPG via VAN}
   \label{algorithm:dpg_van}
   \end{algorithm}

\iffalse

\section{Imputation Details}
\label{appendix:imputation}
\subsection{Image Reconstruction}
For this experiment, we measure the model's performance by reconstructing test data. If the model performs well, it should decode and correctly encodes images. The reconstructed images $\widehat{\mbox{$\myvec{y}$}}$ are drawn from the following distribution:
\begin{align*}
	p\rnd{ \widehat{\mbox{$\myvec{y}$}} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{y}$}}
		&= \int p \rnd{ \widehat{\mbox{$\myvec{y}$}}, \mbox{$\myvec{z}$} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{y}$} } d\mbox{$\myvec{z}$}\\
		&= \int p \rnd{ \widehat{\mbox{$\myvec{y}$}} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{z}$} }
			p \rnd{ \mbox{$\myvec{z}$} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{y}$} } d\mbox{$\myvec{z}$}\\
		&\simeq \int p_{\mbox{$\myvecsym{\theta}$}} \rnd{ \widehat{\mbox{$\myvec{y}$}} | \mbox{$\myvec{z}$} }
			q_{\mbox{$\myvecsym{\phi}$}} \rnd{ \mbox{$\myvec{z}$} | \mbox{$\myvec{y}$} } d\mbox{$\myvec{z}$}
 \label{eq:vae_full_rec}
\end{align*}

\subsubsection{Performance Measures}
\label{sec:measures}

The log-loss is the negative log-probability of correctly reconstructing $\mbox{$\myvec{y}$}$:
\begin{align*}
	\text{err}_{\text{log-loss}}
		&= -\log{p\rnd{ \widehat{\mbox{$\myvec{y}$}}=\mbox{$\myvec{y}$}|\mbox{$\myvec{Y}$}_{Tr}}}\\
		&= -\sum^{N}_{n=1} \sum^{D}_{d=1}\log{p\rnd{ \widehat{\mbox{$\myvec{y}$}}_{nd}=\mbox{$\myvec{y}$}_{nd}|\mbox{$\myvec{Y}$}_{Tr}}}\\
		&= -\sum^{N}_{n=1} \sum^{D}_{d=1} y_{nd} \log \widehat{p}_{nd} + (1 - y_{nd}) \log(1 - \widehat{p}_{nd}),
\end{align*}
where $\widehat{p}_{nd} = p\rnd{ \widehat{\mbox{$\myvec{y}$}}_{nd} = 1 | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{y}$}_{nd}} $. We estimate $\widehat{p}_{nd} \simeq \frac{1}{S} \sum^S_{s=1} p_{\theta}(\widehat{\mbox{$\myvec{y}$}}_{nd}|\mbox{$\myvec{z}$}^{(s)}_n)$, sampling $S$ times $\mbox{$\myvec{z}$}^{(s)}_n \sim q_{\phi}(\mbox{$\myvec{z}$}_{n}|\mbox{$\myvec{y}$}_{n})$.

\subsection{Missing Data Imputation} \label{sec:data_imp}
\citet{rezende2015variational} show that their model is able to impute missing data. Images $\mbox{$\myvec{y}$}_n$ are modified by 'removing' a square of pixels. The resulting corrupted images contain observed and missing pixels, $\mbox{$\myvec{\tilde{y}}$} = \sqr{ \mbox{$\myvec{\tilde{y}}$}_M, \mbox{$\myvec{\tilde{y}}$}_O }$, where the missing data is initialized with random noise. We divide each image into quadrants and observe only a subset of them, while the remaining quadrant(s) is considered to be 'missing'. This modification can be denoted as corruption process $C\rnd{\mbox{$\myvec{\tilde{y}}$} | \mbox{$\myvec{y}$}}$ representing a conditional distribution over corrupted samples $\mbox{$\myvec{\tilde{y}}$}$, given data samples $\mbox{$\myvec{y}$}$.

The autoencoder is trained on the (unmodified) training set $\mbox{$\myvec{Y}$}_{Tr}$. Then, the trained model's performance is measured by encoding corrupted test samples $\mbox{$\myvec{\tilde{y}}$}_n^{(s)}$, decoding the resulting latent vector $\tilde{\mbox{$\myvec{z}$}}_{n}^{(s)}$ and comparing the reproduced image $\widehat{\mbox{$\myvec{y}$}}_{n}^{(s)}$ to its original version $\mbox{$\myvec{y}$}_n$. This procedure results in the following sampling scheme:

\begin{align}
	\mbox{$\myvec{\tilde{y}}$}_n^{(s)} &\sim C(\mbox{$\myvec{\tilde{y}}$}_n|\mbox{$\myvec{y}$}_n \in \mbox{$\myvec{Y}$}_{Te})\\ \label{eq:reconstruction}
	\tilde{\mbox{$\myvec{z}$}}_{n}^{(s)} &\sim q_{\phi}(\mbox{$\myvec{z}$}_{n}|\mbox{$\myvec{\tilde{y}}$}_n^{(s)})\\
	\widehat{\mbox{$\myvec{y}$}}_{n}^{(s)} &\sim p_{\theta}(\widehat{\mbox{$\myvec{y}$}}_{n}|\tilde{\mbox{$\myvec{z}$}}_{n}^{(s)}).
\end{align}

The probability of correctly imputing the missing data $\widehat{\mbox{$\myvec{y}$}}_M$ is proportional to correctly imputing the missing data and correctly reconstructing the observed data
\begin{align}
	p\rnd{ \widehat{\mbox{$\myvec{y}$}}_M | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O}
		&\propto p \rnd{ \widehat{\mbox{$\myvec{y}$}}_M, \widehat{\mbox{$\myvec{y}$}}_O | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O}.
\end{align}
This distribution can be written as follows:
\begin{align*}
	\mbox{$p$} \rnd{ \widehat{\mbox{$\myvec{y}$}}_M, \widehat{\mbox{$\myvec{y}$}}_O | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O }
		&= \int \int \mbox{$p$} \rnd{ \widehat{\mbox{$\myvec{y}$}}_M, \widehat{\mbox{$\myvec{y}$}}_O, \mbox{$\myvec{\tilde{y}}$}_M, \mbox{$\myvec{z}$} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O } d\mbox{$\myvec{z}$} d\mbox{$\myvec{\tilde{y}}$}_M\\
		&= \int \int \mbox{$p$} \rnd{ \widehat{\mbox{$\myvec{y}$}}_M, \widehat{\mbox{$\myvec{y}$}}_O |
			\mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O, \mbox{$\myvec{\tilde{y}}$}_M, \mbox{$\myvec{z}$} } \mbox{$p$} \rnd{ \mbox{$\myvec{z}$} | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O, \mbox{$\myvec{\tilde{y}}$}_M } \mbox{$p$} \rnd{ \mbox{$\myvec{\tilde{y}}$}_M | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O } d\mbox{$\myvec{z}$} d\mbox{$\myvec{\tilde{y}}$}_M \\
		&\simeq \int \int \mbox{$p$}_{\theta} \rnd{ \widehat{\mbox{$\myvec{y}$}} |
			\mbox{$\myvec{z}$} } q_{\phi} \rnd{ \mbox{$\myvec{z}$} | \mbox{$\myvec{\tilde{y}}$} } C(\mbox{$\myvec{\tilde{y}}$}_M | \mbox{$\myvec{y}$} \in \mbox{$\myvec{Y}$}_{Te}) d\mbox{$\myvec{z}$} d\mbox{$\myvec{\tilde{y}}$}_M,
\end{align*}
where $\widehat{\mbox{$\myvec{y}$}} = \sqr{ \widehat{\mbox{$\myvec{y}$}}_O, \widehat{\mbox{$\myvec{y}$}}_M }$ and $\mbox{$\myvec{\tilde{y}}$} = \sqr{ \mbox{$\myvec{\tilde{y}}$}_O, \mbox{$\myvec{\tilde{y}}$}_M }$.

\subsubsection{Iterative Imputation and Performance Measures}
As shown by \cite{rezende2015variational}, the imputation procedure described  above can be repeated iteratively. In iteration $t=1$, $\mbox{$\myvec{\tilde{y}}$}_n^{(s, 0)}$ is sampled from the corruption process. Then, in the following iterations $i>1$ we set $\tilde{y}_{nd}^{(s, t)} = \widehat{y}_{nd}^{(s, t-1)}, \forall d \in M$, i.e. we replace the missing pixels of a corrupted image $\mbox{$\myvec{\tilde{y}}$}_n^{(s)}$ with the reconstructed pixels from the previous iteration.

This can be formally written as a Markov chain and it can be shown that if the approximation of the true posterior distribution is sufficiently close, the fixed point of this Markov chain approximates the target marginal $p\rnd{\widehat{\mbox{$\myvec{y}$}}_M|\mbox{$\myvec{\tilde{y}}$}_O}$ up to an acceptable error $\varepsilon$ with respect to the $L_1$ norm (\cite{rezende2015variational}).

The sampling procedures for the performance measure functions are slightly different compared to the image reconstruction task. We cannot simply sample $z^{(s)} \sim q_{\phi}(\mbox{$\myvec{z}$}|\mbox{$\myvec{y}$})$ anymore, as this distribution now depends on the result of the previous iteration. Instead we sample $z^{(s, t)} \sim q_{\phi}(\mbox{$\myvec{z}$}^{(t)}|\mbox{$\myvec{\tilde{y}}$}_M^{(s, t-1)}, \mbox{$\myvec{\tilde{y}}$}_O)$, where in the first iteration $\mbox{$\myvec{\tilde{y}}$}_M^{(s, 0)}$ is initialized with random noise $\mbox{$\myvec{\tilde{y}}$}_M^{(s, 0)} \sim \text{Bernoulli}\left(\mbox{$\myvec{\tilde{y}}$}_M^{(r, 0)} | 0.5\right)$, and in the following iterations $t > 1$, it is sampled from $\mbox{$p$} \rnd{ \widehat{\mbox{$\myvec{y}$}}^{(t-1)}_M | \mbox{$\myvec{Y}$}_{Tr}, \mbox{$\myvec{\tilde{y}}$}_O }$.

\fi

\section{Exercises}}{}
\newenvironment{myexercises}{\section{Exercises}}{}
\newenvironment{mysolutions}{\section{Solutions}}{}
\newcommand{\myexercise}[1]{\vspace{0.5cm} \subsection{#1}}
\newcommand{\mysoln}[1]{\vspace{0.5cm} \subsection{#1}}

\newcommand{\end{equation}}{\end{equation}}
\newcommand{\begin{eqnarray}}{\begin{eqnarray}}
\newcommand{\end{eqnarray}}{\end{eqnarray}}
\newcommand{\begin{eqnarray*}}{\begin{eqnarray*}}
\newcommand{\end{eqnarray*}}{\end{eqnarray*}}

\newcommand{\conv}[1]{\,\,\,\displaystyle{\operatorname*{\longrightarrow}^{\,_{#1}\,}}\,\,\,}
\newcommand{\conv{D}}{\conv{D}}
\newcommand{\conv{P}}{\conv{P}}
\newcommand{\conv{AS}}{\conv{AS}}
\newcommand{\lpconv}[1]{\conv{L^{#1}}}

\section{Introduction}
\label{sec:intro}
\input{intro}

\section{Variational Optimization}
\label{sec:vo}
\input{vo}

\section{Variational Adaptive-Newton Method}
\label{sec:van}
\input{van}
\input{van1}

\section{VAN for Variational Inference}
\label{sec:vi}
\input{vi}

\section{Applications and Experimental Results}
\label{sec:experiments}
\input{results.tex}

\section{Discussion and Conlcusions}
\label{sec:discussion}
\input{discussion}

\bibliographystyle{apalike}

\subsection{Supervised Learning: Lasso Regression}

\input{lasso.tex}

\subsection{Supervised Learning: Logistic Regression}

In logistic regression, we minimize the following:
\begin{align}
f(\boldsymbol{\theta})
&= \sum_{i=1}^N \left[ \log(1+e^{y_i (\boldsymbol{\theta}^\top\boldsymbol{x}_i)}) \right] + \lambda \sum_{d=1}^D \theta_d^2 ,
\end{align}
where $y_i \in\{-1,+1\}$ is the label.

We compare VAN to Newton's method and AdaGrad both of which standard algorithms for batch and stochastic learning, respectively, on convex problems. We use VAN, its stochastic version sVAN, and the diagonal version sVAN-D.
We use three real-world datasets from the libSVM database~\citep{CC01a}: `breast-cancer-scale' ($N=683,D=10,N_{train}=341,\lambda=1.88$), `USPS' ($N=1,540,D=256,N_{train}=770,\lambda=6.21$), and `a1a' ($N=32,561,D=123,N_{train}=1,605,\lambda=67.23$).
We compare the log-loss on the test set computed as follows: $ \sum_i \log (1+ \exp(y_i (\hat{\mbox{$\myvecsym{\theta}$}}^T\mbox{$\myvec{x}$}_i))/N_{test}$ where $\hat{\mbox{$\myvecsym{\theta}$}}$ is the parameter estimate and $N_{test}$ is the number of examples in the test set.
For sVAN and sVAN-D, we use a mini-batch size of 10 for `breast-cancer-scale' dataset and a mini-batch of size 100 for the rest of the datasets.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.33\linewidth]{figures/YearPredictionMSD_lasso_loss.pdf}
	\includegraphics[width=0.33\linewidth]{figures/bank32nh_lasso_loss.pdf}
	\includegraphics[width=0.33\linewidth]{figures/mnist_vae_loss.pdf}
	\includegraphics[width=0.33\linewidth]{figures/batch_many_seeds_te_breast_cancer_s0_1_bsz_10.pdf}
	\includegraphics[width=0.33\linewidth]{figures/batch_many_seeds_te_usps_3vs5_s0_10_bsz_100.pdf}
	\includegraphics[width=0.33\linewidth]{figures/batch_many_seeds_te_a1a_s0_10_bsz_100.pdf}
	\includegraphics[width=0.33\linewidth]{figures/sto_many_seeds_te_breast_cancer_s0_1_bsz_10.pdf}
	\includegraphics[width=0.33\linewidth]{figures/sto_many_seeds_te_usps_3vs5_s0_10_bsz_100.pdf}
	\includegraphics[width=0.33\linewidth]{figures/sto_many_seeds_te_a1a_s0_10_bsz_100.pdf}
	\caption{Experimental results on different learning tasks. (a-b): Lasso regression. (c): VAE. (d-f):  Logistic regression with VAN and Newton's method. (h-j): Logistic regression with sVAN, sVAN-D and AdaGrad. Datasets are specified in the title. $M$ refers to the mini-batch size for stochastic methods.}
	\label{figure:mix}
\end{figure*}

Results are shown in Figure~\ref{figure:mix} (d)-(j). The first row, with plots (d)-(f), shows comparison of Batch methods, where we see that VAN converges at a comparable rate to Newton's method.
The second row, with plots (h)-(j), shows the performance of the stochastic learning.
Since sVAN uses the full Hessian, it converges faster than sVAN-D and AdaGrad which use a diagonal approximation.
sVAN-D shows overall similar performance to AdaGrad.
The main advantage of sVAN over AdaGrad is that sVAN maintains a distribution $q(\boldsymbol{\theta})$ which can be used to evaluate the uncertainty of the learned solution, as shown in the next subsection on active learning.

In conclusion, VAN and sVAN give comparable results to Newton's method while sVAN-D gives comparable results to AdaGrad.

\subsection{Active Learning for using VAN}
An important difference between active and stochastic learning is that an active learning agent can \emph{query} for its training data examples in each iteration.
In active learning for classification, examples in a pool of input data $\{\mbox{$\myvec{x}$}_i\}_{i=1}^{N}$ are ranked using an \emph{acquisition score} which measures how informative an example is for learning.
We pick the top $M$ data examples as the mini-batch.
Active learning is expected to be more data efficient than stochastic learning since the learning agent focuses on the most informative data samples.

In our experiments, we use the entropy score \citep{Schein2007} as the acquisition score to select data example for binary logistic regression:
\begin{align}
   \mathcal{H}(\mbox{$\myvec{x}$}) = - \sum_{c\in\{-1,1\}} \widehat{p}(y=c|\mbox{$\myvec{x}$}) \log \widehat{p}(y=c|\mbox{$\myvec{x}$}),
\end{align}
where $\widehat{p}(y=c|\mbox{$\myvec{x}$})$ is the estimated probability that the label for input $\mbox{$\myvec{x}$}$ takes a value $y=c$. Within our VAN framework, we estimate these probabilities using distributions $q(\mbox{$\myvecsym{\theta}$}) = \mbox{${\cal N}$}(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\mu}$}_t,\mbox{$\myvecsym{\Sigma}$}_t)$ at iteration $t$. We use the following approximation that computes the probability using samples from $q$:
\begin{align}
\widehat{p}_t(y=c|\mbox{$\myvec{x}$})
&\approx \int p(y=c|\mbox{$\myvec{x}$}, \boldsymbol{\theta}) q(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\mu}$}_t,\mbox{$\myvecsym{\Sigma}$}_t) d\mbox{$\myvecsym{\theta}$} \\
&\approx \frac{1}{S} \sum_{s=1}^S p(y=c|\mbox{$\myvec{x}$}, \boldsymbol{\theta}_t^{(s)}),
\end{align}
where $S$ is the number of MC samples, $\mbox{$\myvecsym{\theta}$}_t^{(s)}$ are sample from $q$, and $p(y|\mbox{$\myvec{x}$},\mbox{$\myvecsym{\theta}$})$ is the logistic likelihood.

Figure~\ref{figure:active} compares the performance on on the USPS dataset with active learning by VAN for mini-batch of 10 examples.
The result clearly shows that VAN with active learning is much more data efficient and stable than VAN with stochastic learning.

\begin{figure*}[t]
	\centering
	\begin{minipage}{0.49\linewidth}
	\includegraphics[width=1\linewidth]{figures/active_test_usps_3vs5.pdf}
	\vspace{-0.3cm}
	\caption{Active learning on logistic regression where we see that sVAN with active learning gives good results with fewer number of passes through the data. }
	\label{figure:active}
	\end{minipage}
\hfill
	\begin{minipage}{0.49\linewidth}
	\includegraphics[width=1\linewidth]{figures/HalfCheetah-v1.pdf}
	\vspace{-0.3cm}
	\caption{Parameter-based exploration for DDPG where we see than VAN based methods perform better than exploration using V-SGD or plain SGD with no exploration.}
	\label{figure:rl}
	\end{minipage}
\end{figure*}

\subsection{Unsupervised Learning with Variational Auto-Encoder}
\input{vae.tex}

\subsection{Parameter Based Exploration in Deep Reinforcement Learning}
Exploration is extremely useful in reinforcement learning (RL), especially in environment where obtaining informative feedback is rare. Unlike most learning tasks where a training dataset is readily available, RL agents needs to explore the state-action space to collect data. An explorative agent that always chooses random actions might never obtain good data for learning the optimal policy. On the other hand, an exploitative agent that always chooses the current optimal action(s) may never try suboptimal actions that can lead to learning a better policy. Thus, striking a balance between exploration and exploitation is a must for an effective learning. However, finding such a balance is an open problem in RL.

In this section, we apply VAN to enable efficient exploration by using the parameter-based exploration~\citep{DBLP:journals/paladyn/RuckstiessSSWSS10}.
In standard RL setup, we wish to learn a parameter $\boldsymbol{\theta}$ of a parametric policy $\mbox{$\myvecsym{\pi}$}$ for action $\boldsymbol{a} = \mbox{$\myvecsym{\pi}$}(\boldsymbol{s}; \boldsymbol{\theta})$ given state $\boldsymbol{s}$. We seek an optimal $\boldsymbol{\theta}$ such that a state-action sequence $(\boldsymbol{s}_1, \boldsymbol{a}_1, \boldsymbol{s}_2 \dots)$ maximizes the expected returns where $\boldsymbol{s}_t$ and $\boldsymbol{a}_t$ are state and action at time $t$, respectively.
To facilitate exploration, a common approach is to perturb the action by a random noise $\mbox{$\myvecsym{\epsilon}$}$, e.g., we can simply add it to the action $\boldsymbol{a} =\mbox{$\myvecsym{\pi}$}(\boldsymbol{s};\boldsymbol{\theta}) + \mbox{$\myvecsym{\epsilon}$}$.
In contrast, in parameter-based exploration, exploration is facilitated in the parameter space as the name suggest, i.e., we sample the policy parameter $\mbox{$\myvecsym{\theta}$}$ from a distribution $\mathcal{N}(\boldsymbol{\theta}|\boldsymbol{\mu}, \boldsymbol{\Sigma})$.
Our goal therefore is to learn the distribution parameters $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$.

The parameter-based exploration is better than the action-space exploration because in the former a small perturbation of a parameter result in significant explorative behaviour in the action space. However, to get a similar behaviour through an action-space exploration, the extent of the noise needs to be very large which leads to instability \citep{DBLP:journals/paladyn/RuckstiessSSWSS10}.

The existing methods for parameter-based exploration independently learn $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ (similar to V-SGD) \citep{DBLP:journals/paladyn/RuckstiessSSWSS10, plappert2017parameter,fortunato2017noisy}. However, this method can be increasingly unstable as learning progresses (as discussed in Section \ref{sec:vo}).
Since VAN exploits the geometry of the Gaussian distribution to jointly learn $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$, we expect it to perform better than the existing methods that use V-SGD.

We consider the \emph{deep deterministic policy gradient} (DDPG) method~\citep{DBLP:conf/icml/SilverLHDWR14, DBLP:journals/corr/LillicrapHPHETS15}, where a local optimal $\boldsymbol{\theta}$ is obtained by minimizing
\begin{align}
f(\boldsymbol{\theta}) = -\mathbb{E}_{p(\boldsymbol{s})}\left[ \widehat{Q}(\boldsymbol{s}, \mbox{$\myvecsym{\pi}$}(\boldsymbol{s},\boldsymbol{\theta}))\right],
\end{align}
where $\widehat{Q}(\boldsymbol{s}, \boldsymbol{a})$ is an estimated of the expected return and $p(\boldsymbol{s})$ a state distribution.
Both $\mbox{$\myvecsym{\pi}$}$ and $\widehat{Q}$ are neural networks.
We compare the performance of sVAN and sVAG against two baseline methods denoted by SGD and V-SGD.
SGD refers to DPG where there are no exploration at all, and V-SGD refers to an extension of DPG where the mean and covariance of the parameter distribution are learned using the update \eqref{eq:vsgd_mean} and \eqref{eq:vsgd_cov}.
Due to the large number of neural networks parameters, we use diagonal approximation to the Hessian for all methods.
More details of experiments are given in Appendix~\ref{appendix:rl}.

The result in Figure~\ref{figure:rl} shows the performance on Half-Cheetah from OpenAI Gym~\citep{1606.01540} for 5 runs, where we see that VAN based methods significantly outperform existing methods.
sVAN-D-1 and sVAG-D-1 both perform equally well. This suggests that both Hessian approximations obtained by using the reparameterization trick shown in \eqref{eq:reparam} and the Gauss-Newton approximation shown in \eqref{eq:Vag_prec}, respectively, are equally accurate for this problem.
We can also see that sVAN-D-10 has better data efficiency than sVAN-D-1 especially in the early stages of learning.
V-SGD is able to find a good policy during learning but has unstable performance that degenerates over time, as mentioned previously.
On the other hand, SGD performs very poorly and learns a suboptimal solution.
This strongly suggests that good exploration strategy is crucial to learn good policies of Half-Cheetah.

We also tried these comparisons on the `pendulum' problem in OpenAI Gym where we did not observe significant advantages from explorations. We believe that this is because this problem does not benefit from using exploration and pure-exploitative methods are good enough for these problems. More extensive experiments are required to validate the results presented in this section.

\section{VAN for Large-Scale Problems}
\label{sec:large}

Applying VAN to problems with large number of parameters is not feasible because we cannot compute the exact Hessian matrix. In this section, we describe several variants of VAN that scale to large problems. Our variants are similar to existing adaptive-gradient methods such as AdaGrad~\cite{duchi2011adaptive} and AROW~\citep{crammer2009adaptive}. We derive these variants by using a mean-field approximation for $q$. Our derivation opens up the possibility of a new framework for designing
computationally efficient second-order methods by using structured distributions $q$.

One of the most common way to obtain scalability is to use a diagonal approximation of the Hessian. In our case, this approximation corresponds to a distribution $q$ with diagonal covariance, i.e., $q(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\eta}$}) = \prod_{d=1}^D \mbox{${\cal N}$}(\theta_d|\mu_d,\sigma_d^2)$, where $\sigma_d^2$ is the variance.
 This is a common approximation in variational inference methods and is called the mean-field approximation \citep{bishop2006pattern}.
Let us denote the precision parameters by $s_d = 1/\sigma_d^2$, and a vector containing them by $\mbox{$\myvec{s}$}$. Using this Gaussian distribution in the update \eqref{eq:Van_mean_1} and \eqref{eq:Van_prec_1}, we get the following diagonal version of VAN, which we call VAN-D:
   \begin{align}
      \textrm{ VAN-D: } \mbox{$\myvecsym{\mu}$}_{t+1} &= \mbox{$\myvecsym{\mu}$}_{t} - \beta_t \,\, \mbox{$\mbox{diag}$}(\mbox{$\myvec{s}$}_{t+1})^{-1} \mathbb{E}_{q_t} \sqr{  \nabla_\theta f(\mbox{$\myvecsym{\theta}$})}  \nonumber\\
      \mbox{$\myvec{s}$}_{t+1} &= \mbox{$\myvec{s}$}_t + \,\, \beta_t \,\, \mathbb{E}_{q_t} \sqr{\mbox{$\myvec{h}$}(\mbox{$\myvecsym{\theta}$})}  \label{eq:Vand_prec_0}
   \end{align}
   where $\mbox{$\mbox{diag}$}(\mbox{$\myvec{s}$})$ is a diagonal matrix containing the vector $\mbox{$\myvec{s}$}$ as its diagonal and $\mbox{$\myvec{h}$}(\mbox{$\myvecsym{\theta}$})$ is the diagonal of the Hessian $\nabla^2_{\theta\theta} f(\mbox{$\myvecsym{\theta}$})$.

The VAN-D update requires computation of the expectation of the diagonal of the Hessian, which could still be difficult to compute. Fortunately, we can compute its approximation easily by using the reparameterization trick \citep{kingma2014adam}. This is possible in our framework because we can express the expectation of the Hessian as gradients of an expectation, as shown below:
\begin{align}
\mathbb{E}_q\sqr{\nabla_{\theta_d \theta_d}^2 f(\mbox{$\myvecsym{\theta}$}) } &= 2 \nabla_{\sigma_d^2} \sqr{\mathbb{E}_q[f(\mbox{$\myvecsym{\theta}$})] } , \\
                                                          & = 2 \nabla_{\sigma_d^2} \mathbb{E}_{\mathcal{N}(\epsilon|0,1)} \sqr{ f(\mu_d + \sigma_d \epsilon) } ,\\
& = 2 \mathbb{E}_{\mathcal{N}(\epsilon|0,1)}\sqr{  \nabla_{\sigma_d^2} f(\mu_d + \sigma_d \epsilon) } , \label{eq:reparam}
\end{align}
where the first step is obtained using \eqref{eq:bonnet}.
In general, we can use the stochastic approximation by simultaneous perturbation (SPSS) method \citep{bhatnagar2007adaptive, spall2000adaptive} to compute derivatives. A recent paper by \cite{2017arXiv170303864S} showed that this type of computation can also be parallelized which is extremely useful for large-scale learning.
Note that these tricks cannot be applied directly to standard continuous-optimization methods. The presence of expectation with respect to $q$ in VO enables us to leverage such stochastic approximation methods for large-scale learning.

Finally, when $f(\mbox{$\myvecsym{\theta}$})$ corresponds to a supervised or unsupervised learning problem with large data, we could compute its gradients by using stochastic methods. We use this version of VAN in our large-scale experiments and call it sVAN, which is an abbreviation for stochastic-VAN.

\subsection{Relationship with Adaptive-Gradient Methods}
The VAN-D update given in \eqref{eq:Vand_prec_0} is closely-related to an adaptive-gradient method called AdaGrad \citep{duchi2011adaptive} which uses the following updates:
\begin{align}
 \textrm{AdaGrad: } \mbox{$\myvecsym{\theta}$}_{t+1} &= \mbox{$\myvecsym{\theta}$}_t - \rho_t \mbox{$\mbox{diag}$}(\mbox{$\myvec{s}$}_{t+1})^{-1/2} \mbox{$\myvec{g}$}(\mbox{$\myvecsym{\theta}$}_t) \label{eq:adgrad2} \\
 \mbox{$\myvec{s}$}_{t+1} &= \mbox{$\myvec{s}$}_t + [\mbox{$\myvec{g}$}(\mbox{$\myvecsym{\theta}$}_t)\odot\mbox{$\myvec{g}$}(\mbox{$\myvecsym{\theta}$}_t)]
\end{align}
Comparing these updates to \eqref{eq:Vand_prec_0}, we can see that both AdaGrad and VAN-D compute the scaling vector $\mbox{$\myvec{s}$}_t$ using a moving average.
However, there are some major differences between their updates: 1. VAN-D uses average gradients instead of a gradient at a point, 2. VAN-D does not raise the scaling matrix to the power of 1/2, 3. The update of $\mbox{$\myvec{s}$}_t$ in VAN-D uses the diagonal of the Hessian instead of the squared gradient values used in AdaGrad. It is possible to use the squared gradient in VAN-D but since we can compute the Hessian using
the reparameterization trick, we do not have to make further approximations.

VAN can be seen as a generalization of an adaptive method called AROW~\citep{crammer2009adaptive}. AROW uses a mirror descent algorithm that is very similar to ours, but has been applied only to problems which use the Hinge Loss. Our approach not only generalizes AROW but also brings new insights connecting ideas from many different fields.

\begin{table}[!t]
\centering
\caption{Variants of VAN used in our experiments.}
\label{tab:listofmethods}
\vspace{.1in}
\resizebox{0.95\columnwidth}{!} {
\begin{tabular}{ |l| l| }
  \hline
  \textbf{Method} & \textbf{Description} \\
  \hline
  VAN & Variational Adaptive-Newton Method using \eqref{eq:Van_mean_1} and \eqref{eq:Van_prec_1}.\\
  sVAN & Stochastic VAN (gradient estimated using mini-batches). \\
  sVAN-Exact & Stochastic VAN with no MC sampling. \\
  sVAN-D & Stochastic VAN with diagonal covariance $\Sigma$. \\
  sVAN-Active & Stochastic VAN using active learning. \\
  sVAG & Stochastic Variational Adaptive Gauss-Newton Method.\\
  sVAG-D & Stochastic VAG with diagonal covariance $\Sigma$. \\
  \hline
\end{tabular}
}
\end{table}

\subsection{VAN for Continuous Optimization}
The above updates corresponds to a second-order method which is very similar to Newton's method.
We can show this using the following identities
\citep{Opper:09}:
\begin{align}
\nabla_{\mu} \mathbb{E}_q \sqr{f(\mbox{$\myvecsym{\theta}$}) } &= \mathbb{E}_q \sqr{ \nabla_\theta f(\mbox{$\myvecsym{\theta}$})}
, \label{eq:bonnet}\\
\nabla_\Sigma \mathbb{E}_q \sqr{f(\mbox{$\myvecsym{\theta}$}) } &= \mbox{$\frac{1}{2}$} \mathbb{E}_q \sqr{ \nabla^2_{\theta\theta} f(\mbox{$\myvecsym{\theta}$}) } .
\end{align}
By substituting these into \eqref{eq:Van_mean_0} and \eqref{eq:Van_prec_0}, we get the following updates which we call the VAN method:
\begin{align}
   \textrm{VAN: } \quad \mbox{$\myvecsym{\mu}$}_{t+1} &= \mbox{$\myvecsym{\mu}$}_{t} - \beta_t \,\, \mbox{$\myvec{P}$}_{t+1}^{-1}  \mathbb{E}_{q_t} \sqr{  \widehat{\nabla}_\theta f(\mbox{$\myvecsym{\theta}$})} \label{eq:Van_mean_1}\\
   \mbox{$\myvec{P}$}_{t+1} &= \mbox{$\myvec{P}$}_t + \,\, \beta_t \,\, \mathbb{E}_{q_t} \sqr{ \widehat{\nabla}^2_{\theta\theta} f(\mbox{$\myvecsym{\theta}$})} \label{eq:Van_prec_1}
\end{align}
where $\mbox{$\myvec{P}$}_t := \mbox{$\myvecsym{\Sigma}$}_t^{-1}$ is the precision matrix and $q_t := \mbox{${\cal N}$}(\mbox{$\myvecsym{\theta}$}|\mbox{$\myvecsym{\mu}$}_t, \mbox{$\myvecsym{\Sigma}$}_t)$.
The precision matrix $\mbox{$\myvec{P}$}_t$ contains a running-sum of the past \emph{averaged Hessians}, and the search-direction for the mean is obtained by scaling the \emph{averaged gradients} by the inverse of $\mbox{$\myvec{P}$}_t$.
If we compare Eq. \ref{eq:Van_mean_0} to the following update of Newton's method,
\begin{align}
\mbox{$\myvecsym{\theta}$}_{t+1} &= \mbox{$\myvecsym{\theta}$}_t - \rho_t \sqr{\nabla^2_{\theta\theta} f(\mbox{$\myvecsym{\theta}$}_t)}^{-1} \sqr{\nabla_\theta f(\mbox{$\myvecsym{\theta}$}_t)},
\end{align}
we can see that the Hessian matrix is replaced by the Precision matrix in the VAN update.
Due to this connection to Newton's method and the use of an adaptive scaling matrix $\mbox{$\myvec{P}$}_t$, we call our method the \emph{Variational Adaptive-Newton} (VAN) method.

The averaged gradient and running sum of averaged Hessian allow VAN to avoid some types of local optima.
Figure~\ref{fig:van_illustration} shows such a result when minimizing\footnote{This example is discussed in a blog by \cite{ferencBlogES}} $f(\theta) = \mathrm{sinc}(\theta)$ with initial solution at $\theta = -3.2$.
Clearly, both the gradient and Hessian at $\theta_0$ suggest updating $\theta$ towards a local optimum at $\theta = -3.5$.
However, VAN computes an averaged gradient over samples from $q(\theta)$ which yields steeper descent directions pointing towards the global optimum. The adaptive scaling of the steps further ensures a smooth convergence.

The averaging property of VAN is strikingly different from other second-order optimization methods. We expect VAN to be more robust due to averaging of gradients and Hessians.
Averaging is particularly useful for optimization of stochastic objectives. For such objectives, application of Newton's method is difficult because reliably selecting a step-size is difficult. Several variants of Newton's method have been proposed to solve this difficulty, e.g., based on quasi-Newton methods \citep{byrd2016stochastic} or incremental approaches \citep{gurbuzbalaban2015globally}, or by simply adapting mini-batch size \citep{mokhtari2016adaptive}.
VAN is most similar to the incremental approach of \citep{gurbuzbalaban2015globally} where a running sum of past Hessians is used instead of just a single Hessian. In VAN however the Hessian is replaced by the average of Hessians with respect to $q$.
For stochastic objectives, VAN differs substantially from existing approaches and it has the potential to be a viable alternative to them.

An issue with using the Hessian is that it is not always positive semi-definite, for example, for non-convex problems. For such cases, we can use a Gauss-Newton variant shown below \cite{bertsekas1999nonlinear} which we call the Variational Adaptive Gauss-Newton (VAG) Method:
\begin{align}
   \textrm{VAG: } \mbox{$\myvec{P}$}_{t+1} &= \mbox{$\myvec{P}$}_t + \beta_t \mathbb{E}_{q_t} \sqr{ \sqr{\nabla_{\theta} f(\mbox{$\myvecsym{\theta}$}) } \sqr{\nabla_{\theta} f(\mbox{$\myvecsym{\theta}$})}^T} .  \label{eq:Vag_prec}
   \end{align}

\section{VAN for Evolution Strategies}
\label{sec:nes}
VAN performs natural-gradient update in the parameter-space of the distribution $q$ (as discussed earlier in Section \ref{sec:van}). The connection to natural-gradients is based on a recent result by \citet{raskutti2015information} that shows that the mirror descent using a KL divergence for exponential-family distributions is equivalent to a natural-gradient descent. The natural-gradient corresponds to the one obtained using the Fisher information of the exponential family distribution. In our case, the mirror-descent algorithm \eqref{eq:md} uses the Bregman divergence that corresponds to the KL divergence between two Gaussians. Since the Gaussian is an exponential-family distribution, mirror descent
\eqref{eq:md} is equivalent to natural-gradient descent in the dual Riemannian manifold of a Gaussian. Therefore, VAN takes a natural-gradient step by using a mirror-descent step.

Natural Evolution Strategies (NES) \citep{wierstra2008natural} is also a natural-gradient algorithm to solve the VO problem in the context of evolution strategies. NES directly applies natural-gradient descent to optimize for $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ and this yields an infeasible algorithm since the  Fisher information matrix has $O(D^4)$ parameters.
To overcome this issue, \cite{wierstra2008natural} proposed a sophisticated re-parameterization that reduces the number of parameters to $O(D^2)$.
VAN, like NES, also has $O(D^2)$ parameters, but with much simpler updates rules due to the use of mirror descent in the mean-parameter space.

redpajama_set_name: RedPajamaArXiv