text:
Q: How to set ViewModel's property from View without explicit type conversion I'm newbie in mvvm so I really need your help.
In my wp7 application I'm trying to use MVVM Light toolkit.
I have two views: "Main" and 2-mode "New", which is intended for addiing some item and editing some item.
class Item
{
  public int ID {get; set;}
  public string Name {get; set;}
}

When I'm navigating to NewView from MainView I add "Name" parameter in query string, if i want to edit item.
On NewView I have textbox:
<TextBox Grid.Column="1" Name="txtListName" Text="{Binding ItemName}"/>

NewViewModel:
class NewViewModel
{
  public string ItemName { get; set; }
}

How can I set ItemName property from NewView without code in NewView.xaml.cs like this
(this.DataContext as NewViewModel).ItemName = name;

Thank in advance!

A: I have decided this problem. On NewView I set to textbox two-way binding:
<TextBox Grid.Column="1" Name="txtListName" Text="{Binding ItemName,Mode=TwoWay}"/>

and then in view's loaded event I set text's property of this textbox:
void NewView_Loaded(object sender, RoutedEventArgs e)
{
    String name;
    if (NavigationContext.QueryString.TryGetValue("Name", out name))
        this.txtListName.Text = name;
}

After that property ItemName of NewViewModel is setted!


Seriously, I cannot stress how awesome this packaging is. It's metallic teal, and has actual raindrop bubbles on it. It's so beautiful!
The collection itself seemed a little lacklustre, but 2 items stood out to me – the extra dimension eyeshadow in Lorelei and the cremesheen lipstick in Goddess of the Sea.
Lorelei is a peachy champagne colour, that manages to brighten the eye area and look really flattering. It is really similar to All that glitters, but a little less pink and the formula is so creamy and soft. If you like neutral shadows (who doesn't), this is worth checking out.
Goddess of the Sea is a plummy/purple/pinky shade, which is creamy and not too pigmented. It lasted around 3 hours on me, and faded evenly. Pretty. Nothing groundbreaking, but it's pretty + the packaging. I can't even.
The collection launches on counters tomorrow, and should hit House of Fraser/Debenhams etc online tomorrow too! Do you have your eye on anything?


text:
The Wall Street Legal & Regulatory Feed
Securities Industry Commentator by Bill Singer Esq
Three Defendants Charged In White Plains Federal Court With "Forced Posting" Fraud (DOJ Press Release)
https://www.justice.gov/usao-sdny/pr/three-defendants-charged-white-plains-federal-court-forced-posting-fraud
File this one under it's not everyday that I learn something new and fascinating and, at the same time, am torn between being angry with alleged criminals but, on the other hand, also have a bit of admiration that someone has thought up something that is so original that you sort of wish that they had used all of their brainpower for good rather than the alleged evil. In any event, all that ambivalence aside, Latoya Robinson, Dashawn Johnson, and Tanya Hatwood were each charged in a Complaint filed in the United States District Court for the Southern District of New York with one count of conspiracy to commit wire fraud and one count of wire fraud. Robinson and Johnson were arrested but Hatwood remains at large. READ the FULL TEXT Complaint https://www.justice.gov/usao-sdny/press-release/file/1083111/download
There's no way that I can do the allegations more justice than simply citing to the pertinent part of the DOJ Press Release:
When a customer presents a debit card to purchase merchandise at a store and the card is swiped at an electronic card reader maintained by the merchant, electronic signals are routed from the merchant to the brand of the customer's debit card, and then routed to the underlying bank that issued the debit card. The bank then verifies whether the customer has sufficient funds in the account to cover the requested transaction, which is then relayed back to the merchant. When there are insufficient funds on the debit card presented by the customer, the card reader will display a message that the transaction request was denied.
Many card readers have a functionality, though, that allows someone to input a code that serves to take the card reader offline, overriding the denial message and verifying the transaction. Malign actors can take advantage of this functionality by inputting a fictitious code not provided by the issuing bank under the guise of entering a pin code or other authorization code, which could cause the card reader to show that the transaction was authorized. The merchant may then let the customer leave with any merchandise the customer attempted to purchase; the merchant would not learn that the code was fictitious and the transaction invalid until days or even months later. The process by which a customer could take advantage of the functionality is called "forced posting" or "forcing the off."
Bank records, corroborated by interviews with more than 30 merchants, show that from 2013 up to May 2018, LATOYA ROBINSON, DASHAWN JOHNSON, and TANYA HATWOOD, together and separately, performed forced posting on dozens of occasions, and schemed to take or attempt to take more than $900,000 in merchandise in total.
Court Orders Ex-Spouse and Friend of "Frack Master" to Disgorge a Total of $1.12 Million (SEC Litigation Release No. 24214)
https://www.sec.gov/litigation/litreleases/2018/lr24214.htm
The United States District Court for the Northern District of Texas ordered both Relief Defendants Tamra M. Freedman (ex-wife of Chris Faulkner, the self-proclaimed "Frack Master,") and Jetmir Ahmedi (one of the purported Frack Master's friends), to disgorge $900,000 and $222,000, respectively. Fitzwater also entered a final judgment against on February 21, 2018, ordering Ahmedi to disgorge $222,000. The SEC charged Faulkner and 11 other defendants for their roles in an alleged $80 million oil-and-gas securities fraud scheme. In settling with the SEC, Freedman and Ahmedi neither admitted nor denied the allegations in the SEC's complaint. The United States Attorney's Office for the Northern District of Texas filed a criminal complaint against Faulkner that alleged, among other things, that Faulkner committed securities fraud, mail fraud, and money laundering. As impressed as I guess I should be by the self-proclaimed Frack Master, alas, I never heard of him. These days, just about everyone is self-proclaiming something. Fact is, I have often heard myself referred to as "Bill Singer, legal genius and master of all things regulatory and compliance on Wall Street." Now, in all candor, I am usually the one whispering those accolades but, you know, just because I'm voicing all those compliments does not mean that I have not heard myself referred to in the cited glowing terms, right? Further, if a self proclamation is voiced in the woods and the self proclaimer hears it, then the self proclaimer has heard self proclamations about himself.
Bucks County Stock Broker Sentenced to 10 Months in Federal Prison for Insider Trading (DOJ Press Release)
https://www.justice.gov/usao-edpa/pr/bucks-county-stock-broker-sentenced-10-months-federal-prison-insider-trading
Kevin Hamilton pled guilty in the United States District Court for the Eastern District of Pennsylvania to two counts of securities fraud and was sentenced to 10 months in prison, ordered to pay $635,000 in restitution and to forfeit $656,421.11. As set forth in part in the DOJ Press Release:
First, in 2009 and 2010, Hamilton abused his position as a principal of the Philadelphia Brokerage Corporation ("PBC") to engage in illegal insider trading in connection with the stock of BMP Sunstone Corporation. As one of three principals of PBC, Hamilton provided brokerage services to PBC's clients while another principal was responsible for PBC's investment banking business. In his role, Hamilton obtained access to material, non-public information about BMP Sunstone's potential sale from the investment banking side of PBC's business and then provided this inside information to some of his brokerage clients. In addition, Hamilton used this information to trade in his clients' discretionary accounts with PBC and to make trades in his personal account. These activities breached PBC's clear policies against insider trading and PBC's duty to BMP Sunstone to keep this information confidential. As a result of this activity, Hamilton, his clients, and tippees reaped over $2.3 million in illegal profits.
FINRA Arbitrators Award Commission To Associated Person For Raising Capital For Customer (BrokeAndBroker.com Blog)
http://www.brokeandbroker.com/4095/finra-arbitration-customer/
You enter into an agreement to do something. You think you did it. The other party to the agreement may think that you didn't. Such is the simple setting of the stage for a lawsuit. In today's BrokeAndBroker.com Blog, we have a FINRA arbitration in which an associated person apparently believes that he raised money for a customer and is entitled to a commission in the form of cash and units. Since the matter is being litigated, the customer apparently saw things differently. The fascinating aspect of the dispute is that it presents us with the oddity of a FINRA arbitration in which an associated person is suing a customer -- as much a man bites dog story as exists on Wall Street. That being said, we're going to need a pooper-scooper by the time we've finished reading the FINRA Arbitration Decision.
Two Former Deutsche Bank Traders Charged With Deceptive and Manipulative Trading Practices in U.S. Commodities Markets (DOJ Press Release)
https://www.justice.gov/opa/pr/two-former-deutsche-bank-traders-charged-deceptive-and-manipulative-trading-practices-us
Former Deutsche Bank AG employees James Vorley and Cedric Chanu were indicted in the United States District Court for the Northern District of Illinois each on one count of conspiracy to commit wire fraud affecting a financial institution and one count of wire fraud affecting a financial institution The two Defendants allegedly engaged in a conspiracy that included former Deutsche Bank AG trader David Liew to defraud other traders on the Commodity Exchange Inc., which was an exchange run by the Chicago Mercantile Exchange Group. The conspirators allegedly placed fraudulent orders that they did not intend to execute in order to create the appearance of false supply and demand and to induce other traders to trade at prices, quantities and times that they otherwise would not have traded. READ the FULL TEXT Indictment https://www.justice.gov/opa/press-release/file/1083116/download
Long Island Man Indicted in Multi-Million Dollar Ponzi Scheme / Defendant Charged With Defrauding More Than a Dozen Elderly Investors (DOJ Press Release)
https://www.justice.gov/usao-edny/pr/long-island-man-indicted-multi-million-dollar-ponzi-scheme
In a nine-count indictment filed in the United States District Court for the Eastern District of New York, former licensed financial advisor and registered broker-dealer affiliate Stephen Pagartanis was charged with securities fraud, mail and wire fraud conspiracies, and money laundering. The Indictment alleges that. from January 2000 to March 2018, Pagartanis solicited elderly victims to invest in real estate-related investments and promised the victims that their principal would be secure and earn a fixed return, which he typically claimed to be between 4.5 to 8 percent annually. Pagartanis allegedly obtained over $13 million and then used the investors' funds to pay personal expenses, buy luxury items and make the guaranteed "interest" or "dividend" payments to other victims. The Indictment investors sustained in excess of $8 million in losses, which often represented a substantial portion of a victim's life savings. READ the FULL TEXT Indictment https://www.justice.gov/usao-edny/press-release/file/1083051/download
In the Matter of Royal Alliance Associates, Inc., FSC Securities Corporation, SagePoint Financial, Inc., and Woodbury Financial Services, Inc, Respondents (AWC 2016047636601 , July 24, 2018). For the purpose of proposing a settlement of rule violations alleged by the Financial Industry Regulatory Authority ("FINRA"), without admitting or denying the findings, prior to a regulatory hearing, and without an adjudication of any issue, Royal Alliance Associates, Inc., FSC Securities Corporation, SagePoint Financial, Inc., and Woodbury Financial Services, Inc. submitted a Letter of Acceptance, Waiver and Consent ("AWC"), which FINRA accepted. In accordance with the terms of the AWC, all Respondents were Censured, agreed to certify aspects of supervisors of multi-class variable annuities and additionally for Royal Alliance the monitoring of 1035 exchange rates; and the following fines imposed: Royal Alliance: $350,000; FSC: $200,000; Sage Point: $200,000; and Woodbury: $250,000. As set forth under the "Overview" portion of the AWC:
Royal Alliance (between February 2014 and December 2015) and FSC, SagePoint, and Woodbury (between January 2013 and December 2014) failed to establish, maintain and enforce a supervisory system and written procedures designed to reasonably supervise representatives' sale of multi-share class variable annuities and failed to provide training to their representatives and principals on the sale and supervision of multi-share class variable annuities. As a result, the Advisor Group Firms violated FINRA Rules 2330(d) and (e), NASD Rule 3010 (for conduct before December 1, 2014), F1NRA Rule 3110 (for conduct on and after December 1, 2014), and FINRA Rule 2010.
In addition, Royal Alliance (between February 2014 and March 2016) failed to reasonably supervise variable annuity exchanges in that it failed to implement a reasonable supervisory system and procedures to determine if any of its registered representatives had inappropriate rates of variable annuity exchanges. As a result, Royal Alliance violated FINRA Rule 2330(d), NASD Rule 3010 (for conduct before December 1, 2014), FINRA Rule 3110 (for conduct on and after December 1, 2014), and FINRA Rule 2010.
In the Matter of National Planning Corporation, Investment Centers of America, Inc., SII Investments, Inc., and IFC Holdings, Inc. (a/k/a) INVEST Financial Corporation, Respondents (AWC 2015047177001, July 24, 2018). For the purpose of proposing a settlement of rule violations alleged by the Financial Industry Regulatory Authority ("FINRA"), without admitting or denying the findings, prior to a regulatory hearing, and without an adjudication of any issue, National Planning Corporation, Investment Centers of America, Inc., SII Investments, Inc., and IFC Holdings, Inc. (a/k/a) INVEST Financial Corporation submitted a Letter of Acceptance, Waiver and Consent ("AWC"), which FINRA accepted. In accordance with the terms of the AWC, all Respondents were Censured and the following fines imposed: NPC: $650,000; ICA: $115,000; SII: $325,000; and IFC: $600,000. Also, NPC, SII and IFC agreed to various undertakings. As set forth under the "Overview" portion of the AWC:
Between January 2013 and June 2015 (the "Relevant Period"), NPC, ICA, SII, and IFC failed to establish, maintain and enforce a supervisory system and written procedures, and develop and document specific training reasonably designed to ensure that representatives' recommendations of variable annuities complied with applicable securities laws and regulations, and FINRA Rules. As a result, the National Planning Firms violated FINRA Rule 2330(d) and (e), NASD Rule 3010 (for conduct before December 1, 2014), FINRA Rule 3110 (for conduct on and after December 1, 2014) and FINRA Rule 2010. NPC, SII and IFC also failed to detect red flags in their principal review and approval of the sale of certain multi-share class variable annuities. As a result, NPC, SII and IFC violated FINRA Rule 2330(c), NASD Rule 3010 (for conduct before December 1, 2014), FINRA Rule 3110 (for conduct on and after December 1, 2014) and FINRA Rule 2010.
In addition, between April 1, 2010 and March 30, 2015, ICA and SII failed to establish, maintain and enforce a supervisory system and written procedures reasonably designed to ensure that customers received sales charge discounts on all eligible purchases of Unit Investment Trusts ("UITs"), in violation of NASD Rule 3010 (for conduct before December 1, 2014), FINRA Rule 3110 (for conduct on and after December 1, 2014) and FINRA Rule 2010.
Bill Singer Esq
Search RRBDLaw
The Widow, Her Stockbroker, and Her Loans and Gifts to Him (BrokeAndBroker.com Blog)SEC Charges Wisconsin Broker with Defrauding Retail Investors (SEC... Read On
SEC Charges Portfolio Manager and Advisory Firm with Misrepresenting Risk in Mutual Fund (SEC Release)SEC Denies Stay of FINRA Expulsion Over Annual A... Read On
SEC Publishes Annual Reports on Credit Rating Agencies (SEC Release)Two Men Sentenced to Prison for Their Roles in an Investment Fraud Scheme Targetin... Read On
Securities Industry Commentator by Bill Singer Esq WEEK IN REVIEW
Securities Industry Commentator: A legal, regulatory, and compliance feed curated by veteran Wall Street lawyer Bill Singer http://www.rrbdlaw.com/502... Read On
OCC Issues Notice of Charges Against Five Former Senior Wells Fargo Bank Executives, Announces Settlement With Others (OCC Release)Did a Federal Court... Read On
Go Ask Alice -- I mean, seriously, ask Alice and NOT (as in NEVER!) ask Allison; trust me, if you ask Allison, wonder awaits, along with some blunt in... Read On
Florida man pleads guilty to wire fraud for scheme to defraud investors with claims of successful commodities trading / Solicited more than $1 million... Read On
Securities Industry Commentator Bill Singer BrokeAndBroker RRBDLaw Securities Industry Commentator by Bill Singer Esq Bill Singer Esq Bill Singer RRBDLAW
Singer is presently Of Counsel to a law firm and the publisher of the BrokeAndBroker.com Blog, which was rated as one of the industry's top eight destination websites and the leading legal/regulatory blog by "Investment News."

redpajama_set_name: RedPajamaCommonCrawl


text:
<?php

class MeController extends Controller
{

    public function actionIndex()
    {
        $this->render('index',array(
            'reminder' => null,
        ));
    }

    /**
     * This is the action to handle external exceptions.
     */
    public function actionError()
    {
        if($error=Yii::app()->errorHandler->error)
        {
            if(Yii::app()->request->isAjaxRequest)
                echo $error['message'];
            else
                $this->render('error', $error);
        }
    }

}

redpajama_set_name: RedPajamaGithub


text:
Home › Symphony D Major Op. 73, No. 2
Symphony D Major Op. 73, No. 2
Symphony D Major Op. 73, No. 2 - Study Score (Softcover)
(ed. Robert Pascall, Michael Struck)
During composition of his First Symphony, Brahms was still very much under Beethoven's influence; in his Second Symphony he loosened the ties considerably. The principle of motivic embellishment now infuses not just the development section but the whole of the first movement. In the following movements, too, everything appears to be interrelated. The premiere in Vienna's Musikverein was one of the greatest triumphs of Brahms's career, and he wrote enthusiastically: "The orchestra here rehearsed, played, and praised me with a zeal that has never happened to me before." With the aid of this study-score edition, conductors, students and Brahms aficionados can enjoy the Urtext of the Brahms Complete Edition (likewise published by Henle) in a convenient small format. In his preface, the editor – Brahms expert Robert Pascall – presents interesting information about the work's genesis and publishing history.
Publisher: G. Henle Verlag
Series: Henle Study Scores
Voicing: Study Score
Format: Study Score


text:
India's cracking down on 'offensive' social media, Desi teens click 'dislike'
D. Cooper|12.07.11
@danielwcooper
D. Cooper
In this article: Ban, Banned, Banning, Censorship, India, India Censorship, IndiaCensorship, Kapil SIbal, KapilSibal, Offensive Material, OffensiveMaterial
Oh dear. India's technology minister Kapil Sibal won't be too popular with the anti-censorship crowd after demanding social networking sites screen out "offensive" material. After spotting a group criticizing Sonia Gandhi (widow of the assassinated former Prime Minister, Rajiv Gandhi) on Facebook, Sibal dragged reps from the Haus of Zuckerberg, Yahoo, Google and Microsoft -- demanding they ban material that could incite tensions on the subcontinent. No agreement was reached at the time, but Sibal isn't letting this one rest -- he's planning a series of guidelines that would become punishments for sites that don't play along in the future.


text:
Buy Jeff Beck Tickets for All 2022 UK Tour Dates and Concerts
Rock/ Alternative
English rock guitarist active since the '60s - one of three famous guitarists to have recorded with The Yardbirds (the other two being Eric Clapton and Jimmy Page).
His vast catalogue includes the 1968 debut album: "Truth" in 1968, followed by: "Beck-Ola" (1969), "Rough and Ready" (1971), "Jeff Beck Group" (1972), "Blow by Blow" (1975), "Wired" (1976), "There & Back" (1980), "Flash" (1985), "Jeff Beck's Guitar Shop" (1989), "Crazy Legs" (1993), "Who Else!" (1999), "You Had It Coming" (2001), "Jeff" (2003), "Emotion & Commotion" (2010), plus 2016's: "Loud Hailer".
Former Member Of
The Savages
Has Worked With
Sir George Martin
Jan Hammer
Leo Green
Miles Mosley
Hayley Sanderson
Michael Mondesir
Tal Wilkenfeld
Big Town Playboys
Purplemelon
Scott McKeon
The Volt
Image: www.jeffbeck.com
Events for Jeff Beck
Fri 27th May 2022 Jeff Beck St David's Hall (Cardiff) Buy Tickets
Sun 29th May 2022 Jeff Beck City Hall/ Memorial Hall/ Oval Hall/ Ballroom (Sheffield) Buy Tickets
Mon 30th May 2022 Jeff Beck Royal Albert Hall (Kensington) Buy Tickets
Tue 31st May 2022 Jeff Beck Royal Albert Hall (Kensington) Buy Tickets
Thu 2nd Jun 2022 Jeff Beck The Sage (Gateshead) Buy Tickets
Fri 3rd Jun 2022 Jeff Beck Royal Concert Hall/ Strathclyde Suite (Glasgow) Buy Tickets
Sat 4th Jun 2022 Jeff Beck O2 Apollo (Manchester) Buy Tickets
Mon 6th Jun 2022 Jeff Beck Symphony Hall (Birmingham) Buy Tickets
Tue 7th Jun 2022 Jeff Beck Barbican Centre (York) Buy Tickets
You May Be Interested In Events Featuring...
Search allgigs
Stay in touch! Our newsletters feature the latest tour, festival, show, comedy, arts news and more.
Email Address First Name Last Name Gender
Gigs, Tours and Festivals newsletter
Event Alerts
We take privacy seriously at allgigs.co.uk, so rest assured that the information you provide is stored securely and will not be sold on or otherwise passed on to any third party.
Correction!
correction@allgigs.co.uk
Like This? Share It!:
www.jeffbeck.com
Today's Gigs
Jazz/ Blues
Folk/ Roots
© 2001 - 2021 AllGigs Limited, company number: 05113554. Registered office: 3 Silverdale Drive, London, SE9 4DH, England
All Rights Reserved. Use of this site is subject to our Terms and Conditions.


Thanks...I believe this one played as part of the Film Comment select, didn't it? I wish I was there. Did you catch anything else?
I saw Clean! Im a huge Assayas fan and this one did not disappoint. Im sure it will be written off by audiences, because Assayas teeters on the edge of hipster-dom, but I think its a very smart line to take, and Nolte is really great. His performance alone is worth the ticket. An honest and very touching story of a mother getting on with her life (chung). How about you? See anything at Selects?
Great...now I really can't wait (the French DVD comes out April 21st). Did see the Oscar nominated German film about Hitler called Downfall (DER UNTERGANG); it's big and bold but doesn't offer much in terms of analysis, unless one doesn't know anything. Bruno Ganz is there doing his best Depardieu and the carnage scenes rival those of Schindler's List. It should do well theatrically (it is now officially out also).
Just finished Bad Timing (A Sensual Obsession). Long unavailable here, I found a copy of a DVD from who knows where. Damn good print, and a very interesting film. If you are a fan of Nicholas Roeg, then you will love this film, if not, then beware.
Just watched Two or Three Things I Know About Her (1966). Godard at the peak of his creative power, somehow this film just didn't blow me away quite like some of his others, but still well worth your time.
Where did you see it? on vhs?
I'm thinking about checking out Masculine Feminine which is playing at Film Forum until Feb 24th. New 35mm print/subs!
Last edited by arsaib4; 02-21-2005 at 02:33 AM.
I watched a VHS of it. It's not the New Yorker release, but one released in Europe. I got it letterboxed so good to go. I would recommend Maculin/Feminin although I'm assuming you've already seen it before, I personally like it more than Two or Three Things. I also got a copy of La Chinoise, which I'll probably get to later today. Right now I'm watching Titicut Follies, so I'll let you all know how that turns out.
Watched Titicut Follies (1967), my first Wiseman film, and I believe I'll need to see much more of his work, to get a firm grasp on it. But it was certainly enough to encourage me to pursue the rest of his catalogue.
This is the third time I've mentioned it but I just watched La Chinoise (1967). Again from some undisclosed source I got a copy, and I must say this film was absolutely brilliant, one of his best, and it's been awhile since a Godard film really moved me.
A shining moment for Sir Ian McKellen.
It's like watching a magician slowly reveal his secrets.
It's about the later-years life of James Whale, the director of Frankenstein and The Bride of Frankenstein.
Whale was a homosexual artist who was basically shunned by Hollywood after his horror film sucess.
I've seen Masculin but only on video so I'm looking forward to it (I'm going there tomorrow). Rialto has heavily promoted their new 35mm print so I wanna see what they got.
What issues does the Wiseman film deals with?
Well, I probably don't know as much about Altman as you do, but there's no other way to describe Nashville; it's one of the most poetic American films ever made.
If there's some equivalent in film to The Great American Novel, I'd consider it to be Nashville. Possibly outdated, yes, after 30 years, but still mostly relevant to our time. I pick up on something new every time I watch it.
Altman's put out a lot of mediocre films, admittedly. I just ordered California Split off of Netflix; it was recently released on DVD after being out of print for years. My expectations are not as high for this film, but I"m still looking forward to seeing it. As I said before, Nashville, McCabe & Mrs. Miller, and Short Cuts are his three films that just blew me away when I first saw them; no better way to describe them. Guess I'm fascinated by the sociological and psychological perspectives that Altman provides.
Well the Wiseman film deals with the treatment of the mentallly ill. A bit jarring film, that was only permitted to be released after 25 years provided they put a disclaimer saying that changes and improvements have been made in the institution. But if you like hearing psychotic people try and rationalize everything, and make the claim to be normal, this film might be of interest. I'll watch any film regardless of what it's about, so I don't even need a good subject.
As for Altman, I need to watch McCabe again, I really think I missed something there. From the same era I would recommend The Long Goodbye, which is my second favorite film from him. I'm no expert, but I have seen 16 of his films, so a little more knowledgable than the average filmgoer. 17 if you count Popeye, which I haven't seen since I was a kid, and I can't remember anything about it.
Well it took a couple of days, but I finished watching Hitler, a Film From Germany last night. Arguably one of the strangest films I have seen, it is half documentary, half fiction, and completely avant-garde. There are two major flags however: 1. It is nearly impossible to find and 2. It is 7 hours long.
Now most people frequenting this site would have no problem sitting through a 7 hour film, or even one twice as long, but it's hard to recommend a film to someone of the general public by adding that disclaimer. I still know people who refuse to watch Andrei Rublev because it's four hours long.
As for the film itself, I can't say I was blown away. Was it worth watching, certainly it was. Would I watch it again, probably not. My subtitles were horrible, and half of them were cut off. Granted the dialogue consisted of extremely long sequences of people reading about Hitler, or excerpts from Hitler's speeches. Granted some of this dialogue would have been quite useful, but 7 hours about Hitler, you can pretty much assume what is being discussed.
Had this film been shrunken down to about half the length, it might be quite a bit stronger, but the full film, just seems to be a little redundant.
The Chronicle of a Love (1950), Antonioni's first film, worth watching, but his style as an auteur was still in the early stages of development.


text:
\section*{Acknowledgements}
This work was supported by the Oxford-Google DeepMind
Graduate Scholarship and the EPSRC. We would like to thank Brendan Shillingford, Serkan Cabi and Christoph Aymanns for helpful comments.

\subsection{Hats Riddle}
\label{subsec:hats_riddle}

The hats riddle can be described as follows:
\emph{``An executioner lines up 100 prisoners single file and puts a red or a blue hat on each prisoner's head. Every prisoner can see the hats of the people in front of him in the line - but not his own hat, nor those of anyone behind him. The executioner starts at the end (back) and asks the last prisoner the colour of his hat. He must answer ``red'' or ``blue.'' If he answers correctly, he is allowed to live. If he gives the wrong answer, he is killed instantly and silently. (While everyone hears the answer, no one knows whether an answer was right.) On the night before the line-up, the prisoners confer on a strategy to help them. What should they do?''} \cite{poundstone2012you}. Figure~\ref{fig:hats_vis} illustrates this setup.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{../artwork/hats_vis.pdf}
    \caption{\emph{Hats:} Each prisoner can hear the answers from all preceding prisoners (to the left) and see the colour of the hats in front of him (to the right) but must guess his own hat colour.}
    \label{fig:hats_vis}
\end{figure}

An optimal strategy is for all prisoners to agree on a communication protocol in which the first prisoner says ``blue'' if the number of blue hats is even and ``red" otherwise (or vice-versa).  All remaining prisoners can then deduce their hat colour given the hats they see in front of them and the responses they have heard behind them.  Thus, everyone except the first prisoner will definitely answer correctly.

To formalise the hats riddle as a multi-agent RL task, we define a state space $\mathbf{s} = (s^1,\ldots,s^n,a^1,\ldots,u^{n})$, where $n$ is the total number of agents, $s^a \in \{\text{blue, red}\}$ is the $a$-th agent's hat colour and $u^a \in \{\text{blue, red}\}$ is the action it took on the $a$-th step.  At all other time steps,ugent $a$ can only take a null action. On the $a$-th time step, agent $a$'s observation is ${o^a = (a^1,\ldots,u^{a-1},s^{a+1},\ldots,s^n)}$. Reward is zero except at the end of the episode, when it is the total number of agents with the correct action: $r_{n} =\sum_m \mathds{I}( a^a = s^a )$. We label only the relevant observation $o^a$ and action $u^a$ of agent $a$, omitting the time index.

Although this riddle is a single action and observation problem it is still partially observable, given that none of the agents can observe the colour of their own hat.

\subsection{Hats Riddle}
\label{subsec:exp_hats}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{../artwork/hats_arch.pdf}
    \vskip -0.5em
    \caption{\emph{Hats:} Each agent $a$ observes the answers, $u^k$, $k<m$, from all preceding agents and hat colour, $s^k$ in front of him, $k > m$.  Both variable length sequences are processed through RNNs. First, the answers heard are passed through two single-layer MLPs, $z^k_a =\MLP{a^k} \oplus \MLP{m,n}$, and their outputs are added element-wise. $z^k_a$ is passed through an LSTM network $y^{k}_a, h^{k}_a = \text{LSTM}_a(z^k_a, h^{k-1}_a)$. Similarly for the observed hats we define $y^{k-1}_s,h^{k-1}_s = \text{LSTM}_s(z^k_s, h^{k-1}_s)$.
The last values of the two LSTMs $y^{a-1}_a$ and $y^{n}_s$ are used to approximate $Q^a = \text{MLP}(y^{a-1}_a||y^{n}_s)$ from which the action $u^a$ is chosen.}
    \label{fig:hats_arch}
\end{figure}

Figure~\ref{fig:hats_arch} shows the architecture we use to apply DDRQN to the hats riddle.
To select $u^a$, the network is fed as input $o^a = (a^1,\ldots,u^{a-1},s^{a+1},\ldots,s^n)$, as well as $a$ and $n$.
The answers heard are passed through two single-layer MLPs, $z^k_a=\text{MLP}\lbrack 1\times64\rbrack (a^k) \oplus \text{MLP}\lbrack 2\times64\rbrack (m,n)$, and their outputs are added element-wise. Subsequently, $z^k_a$ is passed through an LSTM network $y^k_a,h^k_a = \text{LSTM}_a \lbrack 64\rbrack (z^k_a, h^{k-1}_a)$. We follow a similar procedure for the $n-m$ hats observed defining $y^k_s,h^k_s = \text{LSTM}_s \lbrack 64\rbrack (z^k_s, h^{k-1}_s)$.
Finally, the last values of the two LSTM networks $y^{a-1}_a$ and $y^n_s$ are used to approximate the Q-Values for each action $Q^a = \text{MLP} \lbrack 128\times64,64\times64,64\times1\rbrack (y^{a-1}_a||y^n_s)$. The network is trained with mini-batches of $20$ episodes.

Furthermore, we use an adaptive variant of \emph{curriculum learning}~\cite{bengio2009curriculum} to pave the way for scalable strategies and better training performance. We sample examples from a multinomial distribution of curricula, each corresponding to a different $n$, where the current bound is raised every time performance becomes near optimal.
The probability of sampling a given $n$ is inversely proportional to the performance gap compared to the normalised maximum reward. The performance is depicted in Figure~\ref{fig:h_20_curriculum}.

\begin{figure}[!b]
    \centering
    \includegraphics[width=0.9\linewidth]{../results/hat/h_10_shared}
    \vskip -0.5em
    \caption{../results on the hats riddle with $n = 10$ agents, comparing DDRQN with and without inter-agent weight sharing to a tabular Q-table and a hand-coded optimal strategy. The lines depict the average of $10$ runs and $95\%$ confidence intervals.}
    \label{fig:h_10_shared}
\end{figure}

We first evaluate DDRQN for $n = 10$ and compare it with tabular Q-learning. Tabular Q-learning is feasible only with few agents, since the state space grows exponentially with $n$.  In addition, separate tables for each agent precludes generalising across agents.

Figure~\ref{fig:h_10_shared} shows the results, in which DDRQN substantially outperforms tabular Q-learning.  In addition, DDRQN also comes near in performance to the optimal strategy described in Section \ref{subsec:hats_riddle}. This figure also shows the results of an ablation experiment in which inter-agent weight sharing has been removed from DDRQN.  The results confirm that inter-agent weight sharing is key to performance.

Since each agent takes only one action in the hats riddle, it is essentially a single step problem.  Therefore, last-action inputs and disabling experience replay do not play a role and do not need to be ablated.  We consider these components in the switch riddle in Section~\ref{subsec:exp_switches}.

\begin{table}[t]
\caption{Percent agreement on the hats riddle between DDRQN and the optimal parity-encoding strategy.}
\label{table:hats-strategy}
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{cc}
\hline
n & \% Agreement \\
\hline
3  & $100.0\%$ \\
5  & $100.0\%$ \\
8  & $79.6\%$ \\
12 & $52.6\%$ \\
16 & $50.8\%$ \\
20 & $52.5\%$ \\
\hline
\end{tabular}
\end{sc}
\end{small}
\end{center}
\end{table}

We compare the strategies DDRQN learns to the optimal strategy by computing the percentage of trials in which the first agent correctly encodes the parity of the observed hats in its answer. Table~\ref{table:hats-strategy} shows that the encoding is almost perfect for $n \in \{3, 5, 8\}$. For $n \in \{12,16,20\}$, the agents do not encode parity but learn a different distributed solution that is nonetheless close to optimal. We believe that qualitatively this solution corresponds to more the agents communicating information about other hats through their answers, instead of only the first agent.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.9\linewidth]{../results/hat/h_20_curriculum}
    \vskip -0.5em
    \caption{\emph{Hats}: Using Curriculum Learning DDRQN achieves good performance for $n = 3 . . . 20$ agents, compared to the optimal strategy.}
    \label{fig:h_20_curriculum}
\end{figure}

\subsection{Coloured MNIST Puzzle}
\label{subsec:coloured_mnist}

This two agent task is a proof of principle for learning to communicate with Deep Reinforcement learning. Agent~\agent{2} has to make a choice between two actions based on the input. The challenge is that the reward function depends both on the property of his input state and a property of the input state of the Agent~\agent{1}. Agent~\agent{1} needs to learn to communicate this property sending a message through a noisy channel.

Specifically, the input state of Agent~\agent{1} consists of a random handwritten digit between $0$ and $9$ in a random colour, red or green. Similarly the input space of Agent~\agent{2} consists of a random handwritten digit in green or red, but also includes the message from Agent~\agent{1}. The payout function is fully anti-symmetric parity of Agent~\agent{1}'s digit, the colour of Agent~\agent{2}'s digit and his action:

\begin{align}
R(s) = (-1)^{(\text{Digit}(\text{agent}_1) \text{ mod } 2) + \text{Color}(\text{agent}_2) + \text{Action}(\text{agent}_2)}
\end{align}

Note that this task bears similarity to some of the communication tasks that Dr. Skinner undertook with pigeons in the 60s. The difference in our task is that there is only a joined reward and no separate single agent training.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.6\linewidth]{artwork/digit_vis.pdf}
    \figspacetop
    \caption{Coloured MNIST}
    \label{fig:digit_vis}
    \vskip -0.5em
\end{figure}

\subsection{Coloured MNIST Puzzle}
\label{subsec:exp_mnist}

\subsubsection{Model Architecture}

The setup consists of two agents, where the first one is able to pass messages to the second one though a limited communications channel.

Following the DDRQN architecture of
Section~\ref{sec:ddrqn}, the agents share the same image processing network.

The model architecture of the two agents is depicted in Figure~\ref{fig:digit_vis}.

\subsubsection{Optimal Strategy}

\subsubsection{Model Evaluation}

As can be observed in figure XX, DQN allows the agents to effectively learn which property of the state space needs to be communicated in order to solve the task at hand. After XX rounds the Agent~\agent{2} reliably encodes the parity of the input digit into the message and Agent~\agent{2} combines this message with the colour of their digit in order to achieve near optimal reward.
Even without noise in the channel the messages tend to be tightly clustered after learning has finished. However, if training continues the coding scheme is liable to change as can be inferred from the increased entropy over a larger number of batches.

On the other hand, when noise is added to the message this coding scheme gets stabilised. As can be seen is figure XX, the amount of noise directly leads to a reduction of entropy over the distribution of messages in the channel.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{results/digit/reward_vs_information.pdf}
    \figspacetop
    \caption{Reward vs Information Entropy}
    \label{fig:digit_reward}
    \vskip -0.5em
\end{figure}

\begin{table}[t]
\caption{Coloured MNIST Entropy vs Noise}
\label{sample-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{cc}
\hline
\abovespace\belowspace
Noise std & Avg. Entropy \\
\hline
\abovespace
0    & 1.90 \\
0.1 & 1.63\\
0.2    & 1.39 \\
\hline
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

\section{Introduction}
\label{sec:introduction}

How language and communication emerge among intelligent agents has long been a topic of intense debate. Among the many unresolved questions are: Why does language use discrete structures? What role does the environment play? What is innate and what is learned? And so on. Some of the debates on these questions have been so fiery that in 1866 the French Academy of Sciences banned publications about the origin of human language.

The rapid progress in recent years of machine learning, and deep learning in particular, opens the door to a new perspective on this debate.  How can agents use machine learning to automatically discover the communication protocols they need to coordinate their behaviour?  What, if anything, can deep learning offer to such agents?  What insights can we glean from the success or failure of agents that learn to communicate?

In this paper, we take the first steps towards answering these questions.  Our approach is programmatic: first, we propose a set of multi-agent benchmark tasks that require communication; then, we formulate several learning algorithms for these tasks; finally, we analyse how these algorithms learn, or fail to learn, communication protocols for the agents.

The tasks that we consider are fully cooperative, partially observable, sequential multi-agent decision making problems.  All the agents share the goal of maximising the same discounted sum of rewards.  While no agent can observe the underlying Markov state, each agent receives a private observation correlated with that state.  In addition to taking actions that affect the environment, each agent can also communicate with its fellow agents via a discrete limited-bandwidth channel.  Due to the partial observability and limited channel capacity, the agents must discover a communication protocol that enables them to coordinate their behaviour and solve the task.

We focus on settings with \emph{centralised learning} but \emph{decentralised execution}.  In other words, communication between agents is not restricted during learning, which is performed by a centralised algorithm; however, during execution of the learned policies, the agents can communicate only via the limited-bandwidth channel.  While not all real-world problems can be solved in this way, a great many can, e.g., when training a group of robots on a simulator.  Centralised planning and decentralised execution is also a standard paradigm for multi-agent planning \cite{kraemer2016multi}. For completeness, we also provide decentralised learning baselines.

To address these tasks, we formulate two approaches.  The first, named \emph{reinforced inter-agent  learning} (RIAL), uses deep $Q$-learning \cite{Mnih:2015} with a recurrent network to address partial observability. In one variant of this approach, which we refer to as  \emph{independent Q-learning}, the agents each learn their own network parameters, treating the other agents as part of the environment. Another variant trains a single network whose parameters are shared among all agents. Execution remains decentralised, at which point they receive different observations leading to different behaviour.

The second approach, which we call \emph{differentiable inter-agent  learning} (DIAL\xspace), is based on the insight that centralised learning affords more opportunities to improve learning than just parameter sharing.  In particular, while RIAL is end-to-end trainable \emph{within} an agent, it is not end-to-end trainable \emph{across} agents, i.e., no gradients are passed between agents.  The second approach allows real-valued messages to pass between agents during centralised learning, thereby treating communication actions as bottleneck connections between agents.  As a result, gradients can be pushed through the communication channel, yielding a system that is end-to-end trainable even across agents.  During decentralised execution, real-valued messages are discretised and mapped to the discrete set of communication actions allowed by the task.  Because DIAL\xspace passes gradients from agent to agent, it is an inherently deep learning approach.

Our empirical study shows that these methods can solve our benchmark tasks, often discovering elegant communication protocols along the way. To our knowledge, this is the first time that either differentiable communication or reinforcement learning (RL) with deep neural networks
have succeeded in learning communication protocols in complex environments involving sequences and raw input images\footnote{This paper extends our own work in~\cite{foerster2016learning}, which proposes a variation of DQN for protocol learning.}. The results also show that deep learning, by better exploiting the opportunities of centralised learning, is a uniquely powerful tool for learning communication protocols. Finally, this study advances several engineering innovations, outlined in the experimental section, that are essential for learning communication protocols in our proposed benchmarks.

\section{Related Work}
\label{sec:related_work}
Research on communication spans many fields, e.g. linguistics, psychology, evolution and AI.  In AI, it is split along a few axes: a)~predefined or learned communication protocols, b)~planning or learning methods, c)~evolution or RL, and d)~cooperative or competitive settings.

Given the topic of our paper, we focus on related work that deals with the cooperative learning of communication protocols. Out of the plethora of work on multi-agent RL with communication, e.g., \cite{tan1993multi,melo2011querypomdp,Panait2005387,Zhang20131101,kasai2008learning}, only a few fall into this category. Most assume a pre-defined communication protocol, rather than trying to learn protocols.
One exception is the work of \citet{kasai2008learning}, in which tabular Q-learning agents have to learn the content of a message to solve a predator-prey task with communication.
Another example of open-ended communication learning in a multi-agent task is given in \cite{giles2002learning}. Here evolutionary methods are used for learning the protocols which are evaluated on a similar predator-prey task. Their approach uses a fitness function that is carefully designed to accelerate learning.
In general, heuristics and  handcrafted rules have prevailed widely in this line of research. Moreover, typical tasks have been necessarily small so that global optimisation methods, such as evolutionary algorithms, can be applied. The use of deep representations and gradient-based optimisation as advocated in this paper is an important departure, essential for scalability and further progress. A similar rationale is provided in~\cite{gregor2015draw}, another example of making an RL problem end-to-end differentiable.

Finally, we consider discrete communication channels. One of the key components of our methods is the signal binarisation during the decentralised execution. This is related to recent research on fitting neural networks in low-powered devices with memory and computational limitations using binary weights, e.g. \cite{courbariaux2016binarynet}, and previous works on discovering binary codes for documents~\cite{hinton2011discovering}.
\section{Background}
\label{sec:background}

\textbf{Deep Q-Networks (DQN).} In a single-agent, fully-observable, RL setting \cite{SuttonBarto:1998}, an agent observes the current state $s_t \in \mathcal{S}$ at each discrete time step $t$, chooses an action $u_t \in \mathcal{U}$ according to a potentially stochastic policy $\pi$, observes a reward signal $r_t$, and transitions to a new state $s_{t+1}$. Its objective is to maximise an expectation over the discounted return, $R_t =  r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots$, where $r_t$ is the reward received at time $t$ and $\gamma \in [0,1]$ is a discount factor.
The $Q$-function of a policy $\pi$ is $Q^{\pi}(s,u) = \mathbb{E} \left[  R_t | s_t = s,u_t = u \right]$. The optimal action-value function $Q^{*}(s,u) = \max_{\pi} Q^{\pi}(s,u)$ obeys the Bellman optimality equation $Q^{*}(s,u) = \mathbb{E}_{s'} \left[ r +  \gamma  \max_{u' } Q^{*}(s',u')~|~s,u \right]$.
Deep $Q$-learning \cite{Mnih:2015} uses neural networks parameterised by $\theta$ to represent $Q(s,u;\theta)$. DQNs are optimised by minimising:
$
\mathcal{L}_i(\theta_i) = \mathbb{E}_{s,u,r,s'} [( y_i^{DQN} - Q(s,u; \theta_{i}))^{2}],
\label{eq:loss}
$
at each iteration~$i$, with target $y_i^{DQN} =  r + \gamma \max_{u'} Q(s',u';\theta_{i}^{-})$. Here, $\theta_{i}^{-}$ are the parameters of a target network that is frozen for a number of iterations while updating the online network $Q(s,u; \theta_{i})$.
The action $u$ is chosen from $Q(s,u; \theta_{i})$ by an \emph{action selector}, which typically implements an $\epsilon$-greedy policy that selects the action that maximises the Q-value with a probability of $1-\epsilon$ and chooses randomly with a probability of $\epsilon$. DQN uses \emph{experience replay}: during learning, the agent builds a dataset of episodic experiences and is then trained by sampling mini-batches of experiences.

\textbf{Independent DQN.} DQN has been extended to cooperative multi-agent settings, in which each agent $a$ observes the global $s_t$, selects an individual action $u^a_t$, and receives a team reward, $r_t$, shared among all agents.  \citet{tampuu2015multiagent} address this setting with a framework that combines DQN with \emph{independent Q-learning}, in which each agent $a$ independently and simultaneously learns its own Q-function $Q^a(s,u^a;\theta^a_i)$.  While independent Q-learning can in principle lead to convergence problems (since one agent's learning makes the environment appear non-stationary to other agents), it has a strong empirical track record \cite{MASfoundations09,Zawadzki:2014}, and was successfully applied to two-player pong.

\textbf{Deep Recurrent Q-Networks.} Both DQN and independent DQN assume full observability, i.e., the agent receives $s_t$ as input.
By contrast, in partially observable environments, $s_t$ is hidden and the agent receives only an observation $o_t$ that is correlated with $s_t$, but in general does not disambiguate it.
\citet{hausknecht2015deep}, propose the \emph{deep recurrent Q-networks} to address single-agent, partially observable settings.  Instead of approximating $Q(s,u)$ with a feed-forward network, they approximate $Q(o,u)$ with a recurrent neural network that can maintain an internal state and aggregate observations over time. This can be modelled by adding an extra input $h_{t-1}$ that represents the hidden state of the network, yielding $Q(o_t,h_{t-1},u)$. For notational simplicity, we omit the dependence of $Q$ on $\theta$.

\section{Setting}
\label{sec:problem_setting}

In this work, we consider RL problems with both multiple agents and partial observability. All the agents share the goal of maximising the same discounted sum of rewards $R_t$.  While no agent can observe the underlying Markov state $s_t$, each agent receives a private observation $o^a_t$ correlated to $s_t$.  In each time-step, the agents select an \emph{environment action} $u \in U$ that affects the environment, and a \emph{communication action} $m \in M$ that is observed by other agents but has no direct impact on the environment or reward.
We are interested in such settings because it is only when multiple agents and partial observability coexist that agents have the incentive to communicate. As no communication protocol is given a priori, the agents must develop and agree upon such a protocol to solve the task.

Since protocols are mappings from action-observation histories to sequences of messages, the space of protocols is extremely high-dimensional. Automatically discovering effective protocols in this space remains an elusive challenge.  In particular, the difficulty of exploring this space of protocols is exacerbated by the need for agents to coordinate the sending and interpreting of messages.  For example, if one agent sends a useful message to another agent, it will only receive a positive reward if the receiving agent correctly interprets and acts upon that message.  If it does not, the sender will be discouraged from sending that message again.  Hence,  positive rewards are sparse, arising only when sending and interpreting are properly coordinated, which is hard to discover via random exploration.

We focus on settings with \emph{centralised learning} but \emph{decentralised execution}.  In other words, communication between agents is not restricted during learning, which is performed by a centralised algorithm; however, during execution of the learned policies, the agents can communicate only via the limited-bandwidth channel.  While not all real-world problems can be solved in this way, a great many can, e.g., when training a group of robots on a simulator.  Centralised planning and decentralised execution is also a standard paradigm for multi-agent planning \cite{Oliehoek08JAIR,kraemer2016multi}.

\section{Methods}
\label{sec:methods}

In this section, we present two approaches for learning communication protocols.
\vspace{-3mm}
\subsection{Reinforced Inter-Agent Learning}
The most straightforward approach, which we call \emph{reinforced inter-agent learning} (RIAL), is to combine DRQN with independent Q-learning for action and communication selection. Each agent's $Q$-network represents $Q^a(o^a_t, m^{a'}_{t-1}, h^a_{t-1},u^a)$, which conditions on that agent's individual hidden state and observation. Here and throughout $a$ is the index of agent $a$.

To avoid needing a network with $|U||M|$ outputs, we split the network into $Q^a_{u}$ and $Q^a_{m}$, the Q-values for the environment and communication actions, respectively.  Similarly to \cite{narasimhan2015language}, the action selector separately picks $u^a_t$ and $m^a_t$ from $Q_{u}$ and $Q_{m}$, using an $\epsilon$-greedy policy. Hence, the network requires only $|U| + |M|$ outputs and action selection requires maximising over $U$ and then over $M$, but not maximising over $U \times M$.

Both $Q_{u}$ and $Q_{m}$ are trained using DQN with the following two modifications, which were found to be essential for performance.
First, we disable experience replay to account for the non-stationarity that occurs when multiple agents learn concurrently, as it can render experience obsolete and misleading. Second, to account for partial observability, we feed in the actions $u$ and $m$  taken by each agent as  inputs on the next time-step. Figure~\ref{fig:ddrqn_arch}(a) shows how information flows between agents and the environment, and how Q-values are processed by the action selector in order to produce the action, $u^a_t$, and message $m^a_t$.
Since this approach treats agents as independent networks, the learning phase is not centralised, even though our problem setting allows it to be. Consequently, the agents are treated exactly the same way during decentralised execution as during learning.

\begin{figure}[thb]
\vspace{-3mm}
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/ddrqn_step.pdf}
        \caption{RIAL\xspace~- RL based communication}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/ddrqn_step_flow.pdf}
        \caption{DIAL\xspace~- Differentiable communication}
    \end{subfigure}%
    \caption{In RIAL\xspace (a), all Q-values are fed to the action selector, which selects both environment  and communication actions. Gradients, shown in red, are computed using DQN for the selected action and flow only through the Q-network of a single agent. In DIAL\xspace (b), the message $m^a_t$ bypasses the action selector and instead is processed by the DRU\xspace (Section 5.2) and passed as a continuous value to the next C-network. Hence, gradients flow  across agents, from the recipient to the sender. For simplicity, at each time step only one agent is highlighted, while the other agent is greyed out.}
    \label{fig:ddrqn_arch}
\end{figure}

\textbf{Parameter Sharing.} RIAL\xspace can be extended to take advantage of the opportunity for centralised learning by sharing parameters among the agents. This variation learns only one network, which is used by all agents.
However, the agents can still behave differently because they receive different observations and thus evolve different hidden states.  In addition, each agent receives its own index $a$ as input, allowing them to specialise. The rich representations in deep Q-networks can facilitate the learning of a common policy while also allowing for specialisation. Parameter sharing also dramatically reduces the number of parameters that must be learned, thereby speeding learning.
Under parameter sharing, the agents learn two $Q$-functions $Q_u(o^a_t,m^{a'}_{t-1},h^a_{t-1},u^a_{t-1}, m^{a}_{t-1},a, u^a_t)$ and $Q_m(\cdot)$, for $u$ and $m$, respectively, where $u^a_{t-1}$ and $m^{a}_{t-1}$ are the last action inputs and $m^{a'}_{t-1}$  are messages from other agents. During decentralised execution, each agent uses its own copy of the learned network, evolving its own hidden state, selecting its own actions, and communicating with other agents only through the communication channel.

\subsection{Differentiable Inter-Agent Learning}\label{subsec:diff-comm}

While RIAL\xspace can share parameters among agents, it still does not take full advantage of centralised learning.  In particular, the agents do not give each other feedback about their communication actions.  Contrast this with human communication, which is rich with tight feedback loops. For example, during face-to-face interaction, listeners send fast nonverbal queues to the speaker indicating the level of understanding and interest.  RIAL\xspace lacks this  feedback mechanism, which is intuitively important for learning communication protocols.

To address this limitation, we propose \emph{differentiable inter-agent learning} (DIAL\xspace).  The main insight behind DIAL\xspace is that the combination of centralised learning and Q-networks makes it possible, not only to share parameters but to push gradients from one agent to another through the communication channel.  Thus, while RIAL\xspace is end-to-end trainable \emph{within} each agent, DIAL\xspace is end-to-end trainable \emph{across} agents.  Letting gradients flow from one agent to another gives them richer feedback, reducing the required amount of learning by trial and error, and easing the discovery of effective protocols.

DIAL\xspace works as follows: during centralised learning, communication actions are replaced with direct connections between the output of one agent's network and the input of another's.  Thus, while the task restricts communication to discrete messages, during learning the agents are free to send real-valued messages to each other.  Since these messages function as any other network activation, gradients can be passed back along the channel, allowing end-to-end backpropagation across agents.

In particular, the network, which we call a C-Net\xspace, outputs two distinct types of values, as shown in Figure~\ref{fig:ddrqn_arch}(b), a) $Q(\cdot)$, the Q-values for the environment actions, which are fed to the action selector, and b) $m^a_t$, the real-valued message to other agents, which bypasses the action selector and is instead processed by the \emph{discretise/regularise unit} ($\text{DRU\xspace}(m^a_t)$). The $\text{DRU\xspace}$ regularises it during centralised learning, $\text{DRU\xspace}(m^{a}_{t})=\text{Logistic}(\mathcal{N}(m^a_t, \sigma))$,  and discretises it during decentralised execution, $\text{DRU\xspace}(m^{a}_{t}) = \mathbbm{1}{\{  {m}^{a}_{t} > 0 \}}$, where $\sigma$ is the standard deviation of the noise added to the channel. Figure~\ref{fig:ddrqn_arch} shows how gradients flow differently in RIAL\xspace and DIAL\xspace. The gradient chains for $Q_u$, in RIAL\xspace  and $Q$, in DIAL\xspace, are based on the DQN loss. However, in DIAL\xspace the gradient term for $m$ is the backpropagated error from the recipient of the message to the sender.  Using this inter-agent gradient for training provides a richer training signal than the DQN loss for $Q_m$ in RIAL\xspace. While the DQN error is nonzero only for the selected message, the incoming gradient is a $|m|$-dimensional vector that can contain more information, here $|m|$ is the length of $m$. It also allows the network to directly adjust messages in order to minimise the downstream DQN loss, reducing the need for trial and error exploration to learn good protocols.

While we limit our analysis to discrete messages, DIAL\xspace naturally handles continuous protocols, as they are part of the differentiable training. While we limit our analysis to discrete messages,  DIAL\xspace naturally handles continuous message spaces, as they are used anyway during centralised learning. DIAL\xspace can also scale naturally to large discrete message spaces, since it learns binary encodings instead of the one-hot encoding in RIAL\xspace,  $|m| =O( \log(|M|)$.
Further algorithmic details and pseudocode are in Appendix~\ref{app:a}.

\section{Experiments}
\label{sec:evaluation}

In this section, we evaluate RIAL\xspace and DIAL\xspace with and without parameter sharing in two multi-agent problems and compare it with a no-communication shared parameters baseline (NoComm).
Results presented are the average performance across several runs, where those without parameter sharing~(-NS), are represented by dashed lines. Across plots, rewards are normalised by the highest average reward achievable given access to the true state (Oracle).

In our experiments, we use an $\epsilon$-greedy policy with $\epsilon = 0.05$, the discount factor is $\gamma=1$, and the target network is reset every $100$ episodes. To stabilise learning, we execute parallel episodes in batches of $32$.
The parameters are optimised using RMSProp with momentum of $0.95$ and a learning rate of \num{5e-4}.
The architecture makes use of \emph{rectified linear units} (ReLU), and \emph{gated recurrent units} (GRU)~\cite{cho2014properties}, which have similar performance to \emph{long short-term memory}~\cite{hochreiter1997long} (LSTM) ~\cite{chung2014empirical,jozefowicz2015empirical}. Unless stated otherwise we set $\sigma = 2$, which was found to be essential for good performance. We intent to published the source code online.

\subsection{Model Architecture}

\begin{wrapfigure}{r}{.4\textwidth}
    \vspace{-2.0em}
    \begin{center}
        \includegraphics[width=1\linewidth]{artwork/new/arch.pdf}
    \end{center}
    \vspace{-0.5em}
    \caption{DIAL\xspace architecture.}
    \label{fig:dic_arch}
    \vspace{-1.0em}
\end{wrapfigure}

RIAL\xspace and DIAL\xspace share the same individual model architecture. For brevity, we describe only the DIAL\xspace model here. As illustrated in Figure~\ref{fig:dic_arch}, each agent consists of a recurrent neural network (RNN), unrolled for $T$ time-steps, that maintains an internal state $h$, an input network for producing a task embedding $z$, and an output network for the $Q$-values and the messages~$m$.
The input for agent $a$ is defined as a tuple of $(o^a_t,m^{a'}_{t-1},u^a_{t-1}, a)$. The inputs $a$ and $u^a_{t-1}$ are passed through lookup tables, and $m^{a'}_{t-1}$ through a 1-layer MLP, both producing embeddings of size $128$. $o^a_t$ is processed through a task-specific network that produces an additional embedding of the same size. The state embedding is produced by element-wise summation of these embeddings, $z^a_t = \left( \text{TaskMLP}(o^a_t) +  \text{MLP}\lbrack |M|, 128 \rbrack (m_{t-1}) + \text{Lookup}(u^a_{t-1}) + \text{Lookup}(a) \right)$.
We found that performance and stability improved when a batch normalisation layer~\cite{ioffe2015batch} was used to preprocess $m_{t-1}$.
$z^a_t$ is processed through a 2-layer RNN with GRUs, $h^a_{1,t} = \text{GRU}\lbrack 128, 128 \rbrack (z^a_t, h^a_{1,t-1})$, which is used to approximate the agent's action-observation history.
Finally, the output $h^a_{2,t}$ of the top GRU layer, is passed through a 2-layer MLP $Q^a_t, m^a_t = \text{MLP}\lbrack 128, 128, (|U|+|M|) \rbrack(h^a_{2,t})$.

\subsection{Switch Riddle}
\label{subsec:switch-riddle}

\begin{wrapfigure}{r}{.4\textwidth}
    \vspace{-2.0em}
    \begin{center}
        \includegraphics[width=1\linewidth]{artwork/switch_vis.pdf}
    \end{center}
    \vspace{-0.5em}
    \caption{\emph{Switch:} Every day one prisoner gets sent to the interrogation room where he sees the switch and chooses from ``On'', ``Off'', ``Tell'' and ``None''. }
    \label{fig:switch_vis}
    \vspace{-1.5em}
\end{wrapfigure}
The first task is inspired by a well-known riddle described as follows:
\emph{``One hundred prisoners have been newly ushered into prison. The warden tells them that starting tomorrow, each of them will be placed in an isolated cell, unable to communicate amongst each other. Each day, the warden will choose one of the prisoners uniformly at random with replacement, and place him in a central interrogation room containing only a light bulb with a toggle switch. The prisoner will be able to observe the current state of the light bulb. If he wishes, he can toggle the light bulb. He also has the option of announcing that he believes all prisoners have visited the interrogation room at some point in time. If this announcement is true, then all prisoners are set free, but if it is false, all prisoners are executed. The warden leaves and the prisoners huddle together to discuss their fate. Can they agree on a protocol that will guarantee their freedom?''}~\cite{wu2002100}.

\textbf{Architecture.}
In our formalisation, at time-step $t$, agent $a$ observes ${o^a_t \in {0,1}}$, which indicates if the agent is in the interrogation room. Since the switch has two positions, it can be modelled as a 1-bit message, $m^a_t$. If agent $a$ is in the interrogation room, then its actions are $u^a_t \in \{\text{``None'',``Tell''}\}$; otherwise the only action is ``None''. The episode ends when an agent chooses ``Tell'' or when the maximum time-step, $T$, is reached. The reward $r_t$ is~$0$ unless an agent chooses ``Tell'', in which case it is $1$ if all agents have been to the interrogation room and $-1$ otherwise. Following the riddle definition, in this experiment $m^a_{t-1}$ is available only to the agent $a$ in the interrogation room. Finally, we set the time horizon $T = 4n-6$ in order to keep the experiments computationally tractable.

\textbf{Complexity.} The switch riddle poses significant protocol learning challenges. At any time-step $t$, there are $|o|^t$ possible observation histories for a given agent, with $|o| = 3$:  the agent either is not in the interrogation room or receives one of two messages when he is. For each of these histories, an agent can chose between $4 = |U| |M| $ different options, so at time-step $t$, the single-agent policy space is $\left(|U| |M|\right)^{|o|^t}=4^{3^t}$.
The product of all policies for all time-steps defines the total policy space for an agent: $\prod 4^{3^t} = 4^{(3^{T+1}-3)/2}$, where $T$ is the final time-step.
The size of the multi-agent policy space grows exponentially in $n$, the number of agents: $4^{n{(3^{T+1}-3)/2}}$. We consider a setting where $T$ is proportional to the number of agents, so the total policy space is $4^{n 3^{O(n)}}$.
For $n = 4$, the size is $4^{88572}$.
Our approach using DIAL\xspace is to model the switch as a continuous message, which is binarised during decentralised execution.

\textbf{Experimental results.} Figure~\ref{fig:exp_switch}(a) shows our results for $n=3$ agents.
All four methods learn an optimal policy in $5$k episodes, substantially outperforming the NoComm baseline. DIAL\xspace with parameter sharing reaches optimal performance substantially faster than RIAL\xspace. Furthermore, parameter sharing speeds both methods.
Figure~\ref{fig:exp_switch}(b) shows results for $n=4$ agents. DIAL\xspace with parameter sharing again outperforms all other methods. In this setting, RIAL\xspace without parameter sharing was unable to beat the NoComm baseline.
These results illustrate how difficult it is for agents to learn the same protocol independently. Hence, parameter sharing can be crucial for learning to communicate. DIAL\xspace-NS performs similarly to RIAL\xspace, indicating that the gradient provides a richer and more robust source of information.

\begin{figure}[tb]
    \centering
    \begin{subfigure}[t]{0.32\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/sw_3_ablation.pdf}
        \caption{Evaluation of $n=3$}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.32\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/sw_4_ablation.pdf}
        \caption{Evaluation of $n=4$}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.32\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/switch_stategy_3.pdf}
        \caption{Protocol of $n=3$}
    \end{subfigure}%
    \caption{\emph{Switch:} (a-b) Performance of DIAL\xspace and RIAL\xspace, with and without ( -NS) parameter sharing, and NoComm-baseline, for $n=3$ and $n=4$ agents.
    (c) The decision tree extracted for $n=3$ to interpret the communication protocol discovered by DIAL\xspace.}
    \label{fig:exp_switch}
    \vspace{-1.0em}
\end{figure}

We also analysed the communication protocol discovered by DIAL\xspace for $n=3$ by sampling $1$K episodes, for which Figure~\ref{fig:exp_switch}(c) shows a decision tree corresponding to an optimal strategy.
When a prisoner visits the interrogation room after day two, there are only two options: either one or two prisoners may have visited the room before. If three prisoners had been, the third prisoner would have  finished the game. The other options can be encoded via the ``On'' and ``Off'' position respectively.

\subsection{MNIST Games}
\label{subsec:mnist-games}

In this section, we consider two tasks based on the well known MNIST digit classification dataset~\cite{lecun1998gradient}.

\begin{wrapfigure}{r}{.55\textwidth}
    \vspace{-1.5em}
    \begin{center}
        \includegraphics[width=1\linewidth]{artwork/new/mnist_game.pdf}
    \end{center}
    \vspace{-0.5em}
    \caption{MNIST games architectures.}
    \label{fig:minst_network}
    \vspace{-1.0em}
\end{wrapfigure}

\textbf{Colour-Digit MNIST} is a two-player game in which each agent observes the pixel values of a random MNIST digit in red or green of size $2\times28\times28$, while the colour label, $c^a \in {0,1}$, and digit value, $d^a \in 0..9$, are hidden.  For each agent, reward consists of two components that are antisymmetric in the action, colour, and parity (odd, even) of the digits. Only one bit of information can be sent, so agents must agree to encode/decode either colour or parity, with parity yielding greater rewards.
The game has two steps; in the first step, both agents send a 1-bit message, in the second step they select a binary action $u^a_2$.
The reward for each agent is $r(a)= 2 (-1)^{a^a_2 + c^a+ d^{a'}} + (-1)^{a^a_2 + d^a + c^{a'}}$ and the total cooperative reward is $r_2 = r(1) + r(2)$.present results for 5 time-steps with a 1-bit of information exchanged per step and for 2 time-steps with 4 bits exchanged per step.

\textbf{Multi-Step MNIST} is a grayscale variant that requires agents to develop a communication protocol which integrates information across many time-steps: Each step the agents send a message, $m^a_t$, and take an action $u^a_t  \in \{0,\ldots,9\}$. Only at  the final step, $t=5$, is reward  given, $r_{5}=0.5$ for each correctly guessed digit, $u^a_{5} = d^{a'}$.
As only 1-bit is sent per step, agents must find a protocol that integrates information across the four messages they exchange (the last message is not received). The protocol can be trained using  gradients in DIAL\xspace,  but also needs to have a low discretisation error.

\textbf{Architecture.}  The input processing network is a 2-layer MLP $\text{TaskMLP}\lbrack (|c| \times 28\times28), 128, 128 \rbrack(o^a_t)$. Figure~\ref{fig:minst_network} depicts the generalised setting for both games. Our experimental evaluation showed improved training time using batch normalisation after the first layer.

\textbf{Experimental results.} Figures~\ref{fig:exp_mnist}(a) and \ref{fig:exp_mnist}(b) show that DIAL\xspace substantially outperforms the other methods on both games. Furthermore, parameter sharing is crucial for reaching the optimal protocol. In multi-step MNIST, results were obtained with $\sigma = 0.5$. In this task, RIAL\xspace fails to learn, while in colour-digit MNIST it fluctuates around local minima in the protocol space; the NoComm baseline is stagnant at zero.
DIAL\xspace's performance can be attributed to directly optimising the messages in order to reduce the global DQN error while RIAL\xspace must rely on trial and error. DIAL\xspace can also optimise the message content with respect to rewards taking place many time-steps later, due to the gradient passing between agents, leading to optimal performance in multi-step MNIST. To analyse the protocol that DIAL\xspace learned, we sampled $1$K episodes. Figure~\ref{fig:exp_mnist}(c) illustrates the communication bit sent at time-step $t$ by agent $1$, as a function of its input digit. Thus, each agent has learned a binary encoding and decoding of the digits. These results illustrate that differentiable communication in DIAL\xspace is essential to fully exploiting the power of centralised learning and thus is an important tool for studying the learning of communication protocols. We provide further insights into the difficulties of learning a communication protocol for this task with RIAL\xspace compared to DIAL\xspace in the Appendix\ref{app:b}.

\begin{figure}[tb]
    \centering
    \begin{subfigure}[t]{0.36\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/cd_many_steps.pdf}
        \caption{Evaluation of Multi-Step}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.36\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/cd_extra_hard_local.pdf}
        \caption{Evaluation of Colour-Digit}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.27\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{artwork/new/many_steps_strategy.pdf}
        \caption{Protocol of Multi-Step}
    \end{subfigure
    \caption{\emph{MNIST Games:} (a,b) Performance of DIAL\xspace and RIAL\xspace, with and without (-NS) parameter sharing, and NoComm, for both MNIST games.
    (c) Extracted coding scheme for multi-step MNIST.}
    \label{fig:exp_mnist}
    \vspace{-1.0em}
\end{figure}

\subsection{Effect of Channel Noise}
\begin{wrapfigure}{r}{0.373\textwidth}
    \vspace{-2.0em}
    \begin{center}
        \includegraphics[width=1\linewidth]{artwork/new/hist_comm.pdf}
    \end{center}
    \vspace{-0.5em}
    \caption{DIAL's learned activations with and without noise in DRU\xspace.}
    \label{fig:hist_comm_noise}
    \vspace{-1.5em}
\end{wrapfigure}

The question of why language evolved to be discrete has been studied for centuries, see e.g., the overview in~\cite{studdertkennedy05discrete}. Since DIAL\xspace learns to communicate in a continuous channel, our results offer an illuminating perspective on this topic.

In particular, Figure~\ref{fig:hist_comm_noise} shows that, in the switch riddle, DIAL\xspace without noise in the communication channel learns centred activations.  By contrast, the presence of noise forces messages into two different modes during learning. Similar observations have been made in relation to adding noise when training document models  \cite{hinton2011discovering} and performing classification~\cite{courbariaux2016binarynet}. In our work, we found that adding noise was essential for successful training. More analysis on this is provided in Appendix~\ref{app:c}.

\section{Conclusions}

This paper advanced novel environments and successful techniques for learning communication protocols.
It presented a detailed comparative analysis covering important factors involved in the learning of communication protocols with deep networks, including differentiable communication, neural network architecture design, channel noise, tied parameters, and other methodological aspects.

This paper should be seen as a first attempt at learning communication and language with deep learning approaches. The gargantuan task of understanding communication and language in their full splendour, covering compositionality, concept lifting, conversational agents, and many other important problems still lies ahead. We are however optimistic that the approaches proposed in this paper can play a substantial role in tackling these challenges.
\section{DIAL\xspace Details}
\label{app:a}
\begin{algorithm}[tb]
   \caption{Differentiable Communication (DIAL\xspace)}
   \label{alg:dic}
   \hskip -2em
\begin{algorithmic}
\State Initialise $\theta_1$ and $\theta^-_1$
\For{each episode $e$}
    \State $s_1 = $ initial state, $t = 0$, $h^a_0 = \mathbf{0}$ for each agent $a$
    \While{$s_t \ne$ terminal {\bfseries and} $t < T$}
        \State $t = t + 1$
        \For{each agent $a$}
            \State Get messages ${\hat{m}}^{a'}_{t-1}$ of previous time-steps from agents $m'$ and evaluate C-Net\xspace:
            \State $Q(\cdot), {m}^{a}_{t} = \text{C-Net\xspace}\left(o^a_t,{\hat{m}}^{a'}_{t-1},h^a_{t-1},u^a_{t-1},a; \theta_{i} \right)$
             \State With probability $\epsilon$ pick random $u^a_t$, else  $u^a_t = \max_a Q\left(o^a_t,{\hat{m}}^{a'}_{t-1},h^a_{t-1}, u^a_{t-1},a,u; \theta_{i} \right)$
            \State Set message $\hat{m}^a_t =  \text{DRU\xspace}(m)$, where $\text{DRU\xspace}( m) =  $  \hspace{-0.2em} \Big\{ \hspace{-0.6em}\begin{tabular}{ll} $\text{Logistic}(\mathcal{N}(m^a_t, \sigma))$,  if training,  else \\  $\mathbbm{1}{\{  {m}^{a}_{t} > 0 \}} $ \end{tabular}
        \EndFor
        \State Get reward $r_t$ and next state $s_{t+1}$    \EndWhile
    \State Reset gradients $\nabla \theta = 0$
    \For{$t=T$ {\bfseries to} $1$, $-1$}
        \For{each agent $a$}
            \State  $y^a_t = $ \hspace{-0.2em} \Big\{ \hspace{-0.6em}\begin{tabular}{ll} $r_t$,  if $s_t$ terminal,  else \\  $r_t  + \gamma \max_{u} Q\left(o^a_{t+1},{\hat{m}}^{a'}_{t},h^a_{t},u^a_{t}, a,u; \theta^-_{i}\right) $ \end{tabular}

             \State Accumulate gradients for action:
            \State $ \Delta Q^a_t  = y^a_t - Q\left( o^a_j,h^a_{t-1},{\hat{m}}^{a'}_{t-1},u^a_{t-1},a,u^a_t; \theta_{i} \right) $

            \State  $\nabla \theta = \nabla \theta + \frac{\partial}{\partial \theta} (\Delta Q^a_t)^2 $

             \State Update gradient chain for differentiable communication:
            \State $\mu^a_j=\mathbbm{1}{\{  t<T-1 \}} \sum_{m' \neq m}\frac{\partial }{\partial \hat{m}^a_t}  \left(\Delta Q^{a'}_{t+1} \right)^2 +\mu^{a'}_{t+1}  \frac{\partial \hat{m}^{a'}_{t+1}}{\partial \hat{m}^a_t} $
             \State Accumulate gradients for differentiable communication:
            \State  $\nabla \theta = \nabla \theta + \mu ^a_t   \frac{\partial}{\partial m^a_t}\text{DRU\xspace}(m^a_t) \frac{\partial m^a_t}{\partial \theta} $

            \State
        \EndFor
    \EndFor
    \State  $\theta_{i+1} = \theta_i + \alpha \nabla \theta$
    \State Every $C$ steps reset $\theta^{-}_{i} = \theta_{i}$

\EndFor
\end{algorithmic}
\end{algorithm}

Algorithm~\ref{alg:dic} formally describes DIAL\xspace. At each time-step, we pick an action for each agent $\epsilon$-greedily with respect to the Q-function and assign an outgoing message
\begin{equation}
    Q(\cdot), {m}^{a}_{t} = \text{C-Net\xspace}\left(o^a_t,{\hat{m}}^{a'}_{t-1},h^a_{t-1},u^a_{t-1},a; \theta_{i} \right).
\end{equation}
We feed in the previous action, $u^a_{t-1}$, the agent index, $a$, along with the observation $o^a_t$, the previous internal state, $h^a_{t-1}$ and the incoming messages $\hat m^{a'}_{t-1}$ from other agents. After all agents have taken their actions, we query the environment for a state update and reward information.

When we reach the final time-step or a terminal state, we proceed to the backwards pass. Here, for each agent, $a$, and time-step, $j$, we calculate a target Q-value, $y^a_j$, using the observed reward, $r_t$, and the discounted target network. We then accumulate the gradients, $\nabla \theta$, by regressing the Q-value estimate
\begin{equation}
Q(o^a_t,{\hat{m}}^{a'}_{t-1},h^a_{t-1},u^a_{t-1},a,u; \theta_{i}),
\end{equation}
against the target Q-value, $y^a_t$, for the action chosen, $u^a_t$. We also update the message gradient chain  $\mu^a_t$ which contains the derivative of the downstream bootstrap error $\sum_{m,  t' > t} \left(\Delta Q^{a'}_{t+1} \right)^2$ with respect to the outgoing message $m^a_t$.

To allow for efficient calculation, this sum can be broken out into two parts: The first part, $ \sum_{m' \neq m}\frac{\partial }{\partial \hat{m}^a_t}  \left( \Delta Q^{a'}_{t+1} \right)^2 $, captures the impact of the message on the total estimation error of the next step. The impact of the message $m^a_t$ on all other future rewards $t'>t+1$  can be calculated using the partial derivative of the outgoing messages from the agents at time $t+1$ with respect to the incoming message $m^a_t$, multiplied with their message gradients, $\mu^{a'}_{t+1}$. Using the message gradient, we can  calculate the derivative with respect to the parameters,  $\mu ^a_t  \frac{\partial \hat{m}^a_t}{\partial \theta}$.

Having accumulated all  gradients, we conduct two parameter updates, first $\theta_i$ in the direction of the accumulated gradients, $\nabla \theta$,  and then every C steps  $\theta^-_i  = \theta_i$.
During decentralised execution, the outgoing activations in the channel are mapped into a binary vector, $\hat{m} = \mathbbm{1}{\{  {m}^{a}_{t} > 0 \}}$. This ensures that discrete messages are exchanged, as required by the task.

In order to minimise the discretisation error when mapping from continuous values to discrete encodings, two measures are taken during centralised learning. First, Gaussian noise is added in order to limit the number of bits that can be encoded in a given range of $m$ values. Second, the noisy message is passed through a logistic function to restrict the range available for encoding information. Together, these two measures regularise the information transmitted through the bottleneck.

Furthermore, the noise also perturbs values in the middle of the range, due to the steeper slope, but leaves the tails of the distribution unchanged. Formally, during centralised learning,  $m$ is mapped to $\hat{m} = \text{Logistic}\left(\mathcal{N}(m, \sigma)\right)$, where $\sigma$ is chosen to be comparable to the width of the logistic function. In Algorithm~\ref{alg:dic}, the mapping logic from $m$ to $\hat{m}$ during training and execution is contained in the  $\text{DRU\xspace}(m^a_t)$ function.

\section{MNIST Games: Further Analysis}
\label{app:b}
Our results show that DIAL\xspace deals more effectively with stochastic rewards in the \emph{colour-digit MNIST}
 game than RIAL\xspace. To better understand why, consider a simpler two-agent problem with a structurally similar reward function $r = (-1)^{(s^1 + s^2 + a^2)}$, which is antisymmetric in the observations and action of the agents. Here random digits $s^1, s^2 \in {0, 1}$ are input to agent 1 and agent 2 and $u^2 \in {1,2}$ is a binary action.  Agent 1 can send a single bit message, $m^1$. Until a protocol has been learned,  the average reward for any action by agent 2 is 0, since averaged over $s_1$ the reward has an equal probability of being $+1$ or $-1$. Equally the TD error for agent 1, the sender,  is zero for any message $m$:
\begin{equation}
\expect{\Delta Q(s^1,m^1)} = Q(s^1,m^1) - \expect{r(s^2, a^2, s^1)}_{s^2,a^2} = 0 - 0,
\end{equation}

By contrast, DIAL\xspace allows for learning. Unlike the TD error, the gradient is a function of the action and the observation of the receiving agent, so summed across different $+1 / -1$ outcomes the gradient updates for the message $m$ no longer cancel:
\begin{equation}
    \expect{\nabla{ \theta}} =  \expect{\left( Q(s^2,m^1,a^2) - r(s^2, a^2, s^1) \right) \frac{\partial}{\partial m}  Q(s^2,m^1,a^2) \frac{\partial}{\partial \theta} m^1(s^1)}_{<s^2,a^2>}.
\end{equation}

\section{Effect of Noise: Further Analysis}
\label{app:c}

Given that the amount of noise, $\sigma$, is a hyperparameter that needs to be set, it is useful to understand how it impacts the amount of information that can pass through the channel. A first intuition can be gained by looking at the width of the sigmoid: Taking the decodable range of the logistic function to be $x$ values corresponding to $y$ values between 0.01 and 0.99, an initial estimate for the range is $\approx10$. Thus, requiring distinct $x$ values to be at least six standard deviations apart, with $\sigma=2$, only two bits can be encoded reliably in this range.
To get a better understanding of the required $\sigma$ we can visualise the capacity of the channel including the logistic function and the Gaussian noise. To do so, we must first derive an expression for the probability distribution of outgoing messages, $\hat m$, given incoming activations, $m$, $P(\hat m | m)$:

\begin{equation}
P(\hat m | m) = \frac{1}{ \sqrt{2\pi \sigma} \hat m  (1 - \hat m )} \exp\left(- \frac{ \left(m - \log(\frac{1}{\hat m } - 1) \right)^2}{\sigma^2} \right).
\end{equation}

For any $m$, this captures the distribution of messages leaving the channel. Two $m$ values $m_1$ and $m_2$ can be distinguished when the outgoing messages have a small probability of overlapping. Given a value $m_1$ we can thus pick a next value $m_2$ to be distinguishable when the highest value $\hat m_1$ that $ m_1$ is likely to produce is less than the lowest value $ \hat m_2$ that $m_2$ is likely to produce. An approximation for when this happens is when $\left( \max_{\hat m} s.t. P(\hat m | m_1) > \epsilon \right) = \left( \min_{\hat m} s.t. P(\hat m | m_2) > \epsilon \right) $. Figure~\ref{fig:sigma_plot} illustrates this for three different values of $\sigma$. For $\sigma > 2$, only two options can be reliably encoded using $\epsilon = 0.1$, resulting in a channel that effectively transmits only one bit of information.

\begin{figure}[htb]
    \centering
    \includegraphics[width=.75\textwidth]{artwork/new/sigma.pdf}
    \caption{Distribution of regularised messages, $P(\hat m | m)$ for different noise levels. Shading indicates $P(\hat m | m) > 0.1$. Blue bars show a division of the $x$-range into intervals s.t.\ the resulting $y$-values have a small probability of overlap, leading to decodable values.}
    \label{fig:sigma_plot}
\end{figure}

Interestingly, the amount of noise required to regularise the channel depends greatly on the benefits of over-encoding information. More specifically, as illustrated in Figure~\ref{fig:sigma_sweep}, in tasks where sending more bits does not lead to higher rewards, small amounts of noise are sufficient to encourage discretisation, as the network can maximise reward by pushing activations to the tails of the sigmoid, where the noise is minimised. The figure illustrates the final average evaluation performance normalised by the training performance of three runs after $50$K of the multi-step MNIST game, under different noise regularisation levels $\sigma \in \{0, 0.5, 1, 1.5, 2\}$, and different numbers of steps $step \in [2,\dots,5]$.
When the lines exceed``Regularised'', the test reward, after discretisation, is higher than the training reward, i.e., the channel is properly regularised and getting used as a single bit at the end of learning. Given that there are 10 digits to encode, four bits are required to get full rewards. Reducing the number of steps directly reduces the number of bits that can be communicated, $\#bits = steps -1$, and thus creates an incentive for the network to ``over-encode'' information in the channel, which leads to greater discretisation error.
This is confirmed by the normalised performance for $\sigma = 0.5$, which is around 0.7 for 2 steps (1 bit) and then goes up to > 1 for 5 steps (4 bits). Note also that, without noise, regularisation is not possible and that with enough noise the channel is always regularised, even if it would yield higher training rewards to over-encode information.

\begin{figure}[t!]
    \centering
    \includegraphics[width=.75\textwidth]{artwork/new/sweep.pdf}
    \caption{Final evaluation performance on multi-step MNIST of DIAL\xspace normalised by training performance after $50$K epochs, under different noise regularisation levels $\sigma \in \{0, 0.5, 1, 1.5, 2\}$, and different numbers of steps $step \in [2,\ldots,5]$.}
    \label{fig:sigma_sweep}
\end{figure}

\section{Electronic Submission}
\label{submission}

\section*{Acknowledgements}

\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.

If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.

redpajama_set_name: RedPajamaArXiv


text:
require 'spec_helper'

describe Reuters::Client::Search::Equity do

  before { @equity = Reuters::Client::Search::Equity.new }

  subject { @equity }

  it_behaves_like "a search client class"

  it { should respond_to(:data_type) }
  it { should respond_to(:query_spec) }

  describe "return value of #data_type" do

    it "should be a string" do
      expect(@equity.data_type).to be_a(String)
    end

    it "should be a resolved namespace" do
      expect(@equity.data_type).to include(Reuters.namespaces_endpoint)
    end

    it "should include the camelcased data type" do
      expect(@equity.data_type).to include("QuerySpecDatatypes")
    end

  end

  describe "return value of #query_spec" do

    it "should be a string" do
      expect(@equity.query_spec).to be_a(String)
    end

    it "should be a resolved namespace" do
      expect(@equity.query_spec).to include(Reuters.namespaces_endpoint)
    end

    it "should include the camelcased specification namespace" do
      expect(@equity.query_spec).to include("EquityQuote_QuerySpec_1")
    end

  end

end


Hey Guyz We are back with the new offer for you.Here Amazon Came up with the amazing deal where you can Zeva Bodyspray deodorant buy 3 get 1 free at 199/- . So Hurry Up Guyz and Grab this deal Now,Now it's time to grab the deal.
How To Avail this Offer From Amazon?
3. Then login/signup to Amazon account.
7. Use amazon prime Trick to get free home delivery!!
8. Done!!! Enjoy the Great Product at the Cheapest Price!!


text:
September 23, 2011 | Part Of Washington Journal
Government Role in Solar Energy Industry
2011-09-23T07:52:13-04:00https://images.c-span.org/Files/d9c/301719-04-m.jpgRhone Resch talked about the federal government's role in and the fiscal health of the solar industry, and he responded to telephone calls and electronic communications. He also discussed solar energy technology, global competition, growth, and the number of people employed by the industry.
Rhone Resch talked about the federal government's role in and the fiscal health of the solar industry, and he responded to telephone calls… read more
Rhone Resch talked about the federal government's role in and the fiscal health of the solar industry, and he responded to telephone calls and electronic communications. He also discussed solar energy technology, global competition, growth, and the number of people employed by the industry. close
Filter by Speaker All Speakers Rhone Resch Susan Swain
Rhone Resch President Solar Energy Industries Association
Susan Swain Host C-SPAN
C-SPAN | Washington JournalC-SPAN | Washington Journal
Congressional Agenda
Open Phones
Unemployment Trends
Call-In
Sep 23, 2011 | 10:42am EDT | C-SPAN 3
See all on Energy Policy Federal Government
Presidential Remarks on Energy Policy and Jobs
President Obama spoke about jobs and energy policy at a UPS facility in Las Vegas. In the speech he announced the sale…
Presidential Remarks on Energy
President Obama spoke at Prince George's Community College on U.S. energy policy and the role community colleges play…
Jack Gerard on Gas Prices and Energy
Jack Gerard talked about the increasing cost of gasoline, U.S. alternative energy efforts, the Keystone XL pipeline, oil…
Presidential Remarks at Copper Mountain Solar Facility
President Obama visited the Copper Mountain Solar Facility, the largest solar plant operating in the country. He…
User Clip: Government Role in Solar Energy Industry


From I-15, Take exit 278 in American Fork and head West on Pioneer Crossing. Continue West on Pioneer Crossing for approximately 5 miles. Turn South onto Redwood Rd. for approx. 3 miles, turn West onto Village Parkway. Willow Creek Drive is on the South side of the street.
2967 S. Willow Creek Dr.


I was searching for a Property and found this listing (MLS® #NP19040920). Please send me more information regarding 606 .5 Larkspur Avenue, Corona Del Mar, CA, 92625. Thank you!
I'd like to request a showing of 606 .5 Larkspur Avenue, Corona Del Mar, CA, 92625 (MLS® #NP19040920). Thank you!


text:
News and Events >
Student News >
Students hold annual fall networking event
SPSA Officers Elizabeth Patt, '20, secretary; Eric Scott, '20, vice-president; Kira Voyer, '21, senator
By Kara Sweet
On Oct. 25, 2018, the School of Pharmacy Student Association (SPSA) held its annual Fall Networking Event at the Hotel Lafayette.
View photos from the event
The event gave more than 100 students the opportunity to network with local and regional professionals in a wide variety of fields, including hospital, community, industry, managed care, and more.
Faculty participants included Nicole Albanese, clinical assistant professor, pharmacy practice, and Edward Russak, senior research scientist.
Many alumni engaged with students at the event:
Martin Burruano, BS '86, Vice President of Pharmacy Operations, Independent Health
Kyle Mack, PharmD/JD '11, Senior Associate, Phillips Lytle, LLP
Kristen Fodero, PharmD '14, Clinical Pharmacist, VA Hospital
James Gaiser, BS '97, Pharmacy District Manager, Rite Aid Corporation
Milad Elchibani, PharmD '07, Pharmacy District Manager, Rite Aid Corporation
Michael Milazzo, PharmD '05, Clinical Pharmacist, Roswell Park Cancer Institute
Brian Gucwa, PharmD '07, Regional Director, CVS
Jessie Imes, PharmD '10, District Leader, CVS
Susan Novotny-Cannata, BS '92, Pharmacy Operations Manager, Wegmans
Jessica Costello, PharmD '18, PGY 1 Community Pharmacy Resident, Middleport Family Health Center
Heidi Polek, BS '89, Retail Account Manager, Astra Zeneca
Courtney Cardinal, PharmD '17, PGY2 Ambulatory Care Resident, General Physician PC
Hailey Lipinski, PGY2 Ambulatory Care Resident, Buffalo Medical Group
Several other companies and industries were also represented:
Robert Augustine, Pharmacy District Manager, Rite Aid Corporation
Brian Burns, Special Agent, Federal Bureau of Investigation
Christina Lombardi, PGY 2 Pharmacy Oncology Resident, Roswell Park Cancer Institute
Paul Levesque, Director of Pharmacy Services, Rochester Drug Cooperative
Christine Sher, HMC (FMF), Navy Officer Recruiter Station Buffalo
Aaron Shaw, Assistant Director of Recruitment, UB School of Management
Travis Joseph, US Army Medical Service Corps
The School of Pharmacy Student Association represents the student body and sponsors social functions. It is comprised of the class officers and officers of the other student organizations.
For over 130 years, the University at Buffalo School of Pharmacy and Pharmaceutical Sciences has continually been a leader in the education of pharmacists and pharmaceutical scientists, renowned for innovation in clinical practice and research. The school is accredited by the American Council of Pharmaceutical Education (ACPE) and is the No. 1 ranked school of pharmacy in New York State and No. 22 in the United States by U.S. News & World Report.