text:
Q: Ember.js After refresh page data disappear Ive got little basic problem with ember.
Here is app: http://emberjs.jsbin.com/qivamuzu/5 (click test - working like a charm, because model is in memory - loaded in index page)
But when you try page #test directly http://emberjs.jsbin.com/qivamuzu/5#/test all the data disappear (and that's bad - does not fired index route and load model). I follow this question Why isn't my ember.js route model being called? but doesn't help me.
I need to use template with model in other templates - I use {{render index}} but I'm not sure what to use and how. Please help me I am stuck.

A: I'm a little unclear on exactly what you're trying to do.
If you're just trying to use the same model data with a different route (and template) then you can explicitly set the model data to be the same in the route definition:
App = Ember.Application.create();

App.Router.map(function() {
   this.route('test');
});

var myModelData = ['red', 'yellow', 'blue'];

App.IndexRoute = Ember.Route.extend({
  model : function(){
    return myModelData;
  }
});

App.TestRoute = Ember.Route.extend({
  model : function(){
    return myModelData;
  }
});

Here's a working JSBin example:

http://emberjs.jsbin.com/qivamuzu/8/edit

EDIT: Additional Information That May Help
Okay,  one more stab at it =)  When you navigate directly to the test page there is no data because the TestRoute is using the model data from the IndexRoute which hasn't been loaded yet.  What you can do is force the creation of the IndexController and model by initializing it from the ApplicationRoute which will always be invoked when you first go to any route in your application.
First you have to generate the controller since it doesn't exist yet.
this.generateController('index');

Then you can get the controller and set its model data:
this.controllerFor('index').set('model', ['red','green','blue']);

Here's a working fiddle and I actually tested it this time to make sure it works when you go straight to #/test.  I removed the extra routes and things that aren't actually needed from my previous example.

http://emberjs.jsbin.com/qivamuzu/11#/test


text:
When is Mo Salah back for Liverpool? Remaining Egypt fixtures at AFCON fixtures and potential return date
When is the Carabao Cup final? 2022 EFL Cup finale date, UK kick off time and Wembley Stadium ticket detail
Over 1,200 players to test mouth guards for new RFL concussion research project
The Rugby Football League is to launch a pilot scheme involving more than 1,200 players for an extensive game-wide research project into the cause and effects of concussion.
By Ian Laybourn, PA
Wednesday, 14th April 2021, 9:09 am
A trial of a new mouthguard will take place in rugby league.
The 12 Betfred Super League clubs are to work with the RFL on the Instrumented Mouth-Guard Project, with research led by Leeds Beckett University designed to quantify the risk of head impact in rugby league.
The scheme aims to cover around 50 teams at all levels of the game, including academy, the Betfred Women's Super League and community clubs.
Sign up to our World of Sport newsletter
The pilots will run from May to August and will involve the researchers testing different instrumented mouth guards in both training and matches.
Leeds Rhinos have been using them since 2020 and Salford Red Devils started to use them this season, while several rugby union Premiership clubs have also trialled them.
The findings will determine which instrumented mouth guards are selected for the project, which is to begin in November and run for three years.
Researchers will aim to shed more light on head impact exposures and discover how tackle technique and tackle height influence head impacts.
Professor Ben Jones, the lead researcher from Leeds Beckett University and the head of performance at the RFL's England Performance Unit, said: "In rugby league, as in other sports, there is widespread recognition of the need to maximise our understanding of the impact of head collisions.
"Mouth-guard technology has recently developed rapidly, allowing valid measures of head impacts and movement.
"Instrumented mouth guards are already being used by some clubs in rugby league, however a game-wide project will enable a better understanding across different levels of the sport with a bigger data set."
Karen Moorhouse, the RFL's chief regulatory officer, said: "The well-being of players is a top priority of the RFL and clubs.
"The RFL has protocols across the game in relation to concussion (covering recognition, removal and rehabilitation) with the aim of protecting the welfare and health of players.
"These protocols have evolved as a result of increased knowledge of concussions.
"The RFL sees this project as an important next step in the understanding of head impacts and has committed to it for the benefit of current and future players."
RFLLeeds RhinosSalford Red DevilsBen JonesResearchers
When is the Carabao Cup final? 2022 EFL Cup finale date, UK ... Sport
Is Roger Federer playing in Australian Open 2022? Why Swiss ... Sport
Eurosport tennis presenters 2022: Alize Lim and Barbara Sche... Sport
Why aren't there any Premier League games on 29th January? R... Sport

redpajama_set_name: RedPajamaCommonCrawl


text:
'''
Created on Mar 1, 2014

@author: nkavaldj
'''
class sfifo_beh:
    ''' Speculative FIFO: behavioral model '''

    COMMIT = 0
    DISCARD = 1
    DISCARD_COMMIT = 2

    def __init__(self, depth, afull_th=None, aempty_th=None):
        self.depth = depth
        self.data = [] # Data (written, committed, not read )
        self.data_sw = [] # Speculatively written data
        self.data_sr = [] # Speculatively read data
        self.afull_th = afull_th if (afull_th != None) else depth//2
        self.aempty_th = aempty_th if (aempty_th != None) else depth//2
        self.ovf = False
        self.udf = False
        self.count_max = 0

    def reset(self):
        self.__init__(self.depth, self.afull_th, self.aempty_th)

    def isFull(self):
        return self.getCount() == self.depth

    def isEmpty(self):
        return len(self.data) == 0

    def isAFull(self):
        return (self.depth - self.getCount()) <= self.afull_th

    def isAEmpty(self):
        return len(self.data) <= self.aempty_th

    def isOvf(self):
        return self.ovf

    def isUdf(self):
        return self.udf

    def getCount(self):
        return len(self.data_sr) + len(self.data) + len(self.data_sw)

    def getCountMax(self):
        return self.count_max

    def getDout(self):
        return self.data[0]

    def _rcmd(self, cmd):
        if (cmd == self.DISCARD or cmd == self.DISCARD_COMMIT):
            self.data = self.data_sr + self.data
            self.data_sr = []
        elif (cmd == self.COMMIT):
            self.data_sr = []

    def _wcmd(self, cmd):
        if (cmd == self.DISCARD or cmd == self.DISCARD_COMMIT):
            self.data_sw = []
        elif (cmd == self.COMMIT):
            self.data.extend(self.data_sw)
            self.data_sw = []

    def command(self, wcmd=None, rcmd=None):
        self._wcmd(wcmd)
        self._rcmd(rcmd)

    def write(self, val, wcmd=None, rcmd=None):
        if (not self.isFull()):
            self.data_sw.append(val)
        else:
            self.ovf = True

        self._wcmd(wcmd)
        self._rcmd(rcmd)

        if (self.getCount() > self.count_max):
            self.count_max = self.getCount()

    def read(self, wcmd=None, rcmd=None):
        x = None
        if (not self.isEmpty()):
            x = self.data.pop(0)
            self.data_sr.append(x)
        else:
            self.udf = True

        self._wcmd(wcmd)
        self._rcmd(rcmd)

        return x

    def write_read(self, val, wcmd=None, rcmd=None):
        full = self.isFull()
        empty = self.isEmpty()

        if (not full):
            self.data_sw.append(val)
        else:
            self.ovf = True

        self._wcmd(wcmd)

        x = None
        if (not empty):
            x = self.data.pop(0)
            self.data_sr.append(x)
        else:
            self.udf = True

        self._rcmd(rcmd)

        if (self.getCount() > self.count_max):
            self.count_max = self.getCount()
        return x

    def status(self):
        return "FULL = {:}, EMPTY = {:}, COUNT = {:3}, AFULL = {:}, AEMPTY = {:}, OVF = {:}, UDF = {:}, COUNT_MAX = {:}, DOUT = {:}".format(int(self.isFull()), int(self.isEmpty()), int(self.getCount()), int(self.isAFull()), int(self.isAEmpty()), int(self.isOvf()), int(self.isUdf()), self.getCountMax(), self.data[0] if len(self.data)>0 else None)

        return "FULL = {:}, EMPTY = {:}, COUNT = {:3}, AFULL = {:}, AEMPTY = {:}, OVF = {:}, UDF = {:}, COUNT_MAX = {:}, DOUT = {:}".format(int(self.isFull()), int(self.isEmpty()), int(self.count()), int(self.isAFull()), int(self.isAEmpty()), int(self.isOvf()), int(self.isUdf()), self.maxCount(), self.data[0] if len(self.data)>0 else None)

redpajama_set_name: RedPajamaGithub


text:
Accelerate the Waterfowl Production Area Program in Minnesota
Pheasants Forever, Ducks Unlimited, and the U.S. Fish and Wildlife Service partnered to permanently protect, restore and enhance 1,630 acres of grassland and 294 acres of wetland as Waterfowl Production Areas in Western and Southern Minnesota. All lands acquired will be owned and managed in perpetuity by the USFWS as part of the National Wildlife Refuge System and open for public recreation.
Accelerated Shallow Lake Restorations and Enhancements
Ducks Unlimited enhanced 6,882 wetland acres through the bio-engineering and installation of water control structures on managed shallow lake outlets for Minnesota DNR, and protected 76 wetland and 103 upland acres on a shallow lake through a purchased conservation easement.
Acquire 1.27 acres at Bald-Eagle Otter Lakes Regional Park
At Bald Eagle-Otter Lakes Regional Park, acquire 1.27 acre inholding at 1700 North County Line Rd.
Freeborn County Historical Society
To add 277 rolls of microfilmed newspapers to broaden public accessibility to primary records
To add 119 rolls of microfilmed newspapers to broaden public accessibility to primary records.
Aladdin Street BioInfiltration Basin Retrofit Installation
Grass Lake Watershed Management Organization
At 410 acres, Lake Owasso is the largest lake in the Grass Lake Water Management Organization (GLWMO) and one of the most pristine. Maintaining the lakes water quality is a priority for the GLWMO. A long urbanized area along Aladdin Street in Roseville currently lacks stormwater features to remove pollutants and reduce water volume. The rainwater from this area drains directly to a wetland which is hydrologically connected to Lake Owasso. Adjacent to the residential area is a 0.5 acre parking lot which drains into a ditch which eventually enters the same wetland.
A direct appropriation of $400,000 in FY 2010 for the Anoka Conservation District (ACD) is for the metropolitan landscape restoration program for water quality and improvement projects in the seven-county metro area (the law also provides $600,000 for this purpose in FY2011).
Archaeological Survey of the Gibbs Museum
In 2006, Ramsey County Historical Society (RCHS) purchased an additional 1.5 acres of the original Gibbs farmstead located adjacent to the existing Gibbs Museum property in Falcon Heights, Minnesota. This purchase provided the impetus for updating and expanding the interpretive programs at Gibbs Museum of Pioneer and Dakotah Life and creating a new master plan and landscape plan for the Museum.
Artist/Mentor Grant
Name of minor withheld due to MN Statutes 2018, section 3.303.10
Mitchel Hoffmann will work one-on-one with Guthrie metal artist Paula Jensen.
Paula J. Jensen
Gabriela Karam will study jazz and improvisational saxophone under Dr Steve Konecne.
Joseph P. Midthun
For an animated short in a rotoscope style exploring the coming of age in east central Minnesota in the early nineties.
Benjamin A. Doty
To complete six or more short stories toward finishing a short story collection.
Brian T. Miller
To rehearse and record a CD of songs with Irish roots collected in Minnesota between 1870 and 1930.
Sarah A. Rudek
To be mentored by poet Connie Wanek and to attend a writing festival.
Gary E. Peter
For time to complete his novel.
Caly M. McMorrow
To study and create a new body of installation work focusing on audience interaction sound and light.
Anne E. Elias
To produce and shoot a 60 minute non-narrative film for use as a central element in the performance piece A Good Distance.
Ahmad M. Alam
To make and promote a recording of songs focusing on interfaith dialogue.
Katrina M. Vandenberg
To complete the research and writing of a series of poems about the Irish monks and saints and the alphabet for the manuscript--To The Man Who Steals My Electricity.
Kathleen A. Glasgow
For the revision and preparation of a novel draft for submission to agents.
Joshua L. Grubbs
To visually represent what it is like when everyday activities lead to internal moments of clarity and how these moments of clarity transform our perception.
Mike Hazard
For creative time and mentoring to complete The Joy of Roy--a video portrait of the poet Roy McBride.
Peter L. Johnson AKA Peter L Johnson
To create a new presentation form for his work that presents the devastating beauty seen in our relationship to the environment.
Teresa M. Whitman
To revise and refine a collection of stories-Nightlight-and make an intense effort to have these stories published as individual pieces and as a collection.
Tony M. Schaffer
A collection of linked stories titled The Last Days of Moon Lake set over the history of a small Midwest town from the initial removal of the Ojibwe tribe to present day.
Gina L. Dabrowski
To print and frame photographs for an exhibition at Normandale Community College and attend FotoFest to broaden national visibility.
William G. Henry
To edit and revise a completed draft of Seven Acts of Mercy--a historical novel about the baroque painter Caravaggio--before sending it to a literary agent who has requested first look at the manuscript.
Patricia S. Kirkpatrick
To complete a poetry manuscript--Odessa--for travel to the Minnesota prairie and to attend the 2010 Meningioma Awareneness Day conference sponsored by the Brain Science Foundation.
Shelley M. Tougas
For writing time to complete a revision of a young adult novel with a specific focus on character development.
Miriam B. Gerberg
For the rehearsal and performance of a new experimental music/drama-- Desert Dreams: Naomi and Ruth--at the Sabes Jewish Community Center in Minneapolis.
Kenyon C. Williams
To commission four original compositions for solo percussion leading to public performances across Minnesota and for the initial recording of several tracks for his first solo CD.
Tyler J. Stefanich
To support time for ideation process and production of three new projects thus enabling him to apply for programs and exhibitions.


Beautifully crafted dinosaur figures come in a wide variety of colors, all hand painted and bursting with imagination.
Realistic dinosaur figures will bring kids' imaginative play and learn.
Eco-friendly, non-toxic, safe to play.


text:
Agent Advantage
Driver Advantage Program
Driver Application
Founded in 1939 by Albert L. Evans, Sr. the Company started with two trucks handling LTL freight in the local Pottsville area. The company continued to grow in the regional LTL market & gradually expanded operations to Pennsylvania, New Jersey, New York & Maryland. In 1973, the Company established Intermodal services to serve Steamship lines in the port of Philadelphia. Today, our Intermodal drayage offering has expanded to serve the majority of all major ports and rails in the country east of the Rocky Mountains. In 1989, the Company sold its LTL operations and continued expansion in the Intermodal and Truckload markets.
Today, the Company has more than 600 terminal locations throughout the United States, with a fleet of over 7,000 trucks, providing transportation services in the port drayage/intermodal, flatbed, van truckload and freight brokerage markets.
Evans Delivery, Inc. is a member of The Evans Network of Companies.
Evans Delivery Company has locations in all major East Coast ports and major inland port locations. We have interchange agreements with all major steamship lines and railroads as well as direct contact with importers, exporters, freight forwarders, NVOCC's and customs brokers. These solid working relationships assure that all of your international transportation needs will be handled in an efficient, professional manner. Evans Delivery Company services all major east coast port marine terminals and rail facilities and major inland ports and rail ramps.
Intermodal Transportation Services Include:
Ocean Container Drayage – Local & Long Distance
Temperature Controlled Equipment Handled
Rail Trailer Drayage – Local & Long Distance
Flatbed Transportation
Intermodal Transportation
Containers Yard Operations
Container Freight Station Operations
HazMat Certified
The Evans Network of Companies strives to be more than just trucks: we provide small business relationships with big business benefits to our Customers, Network Employees, Agent Partners and Independent Contractors:
We are customer-centric
We are dedicated to safety
We are a good Company for whom to work
We are consistent in our service
We are a stable and supportive network
We treat our Employees with respect and professionalism
The Focus is The Customer…
We find that through implementation of strategic company goals, which focus on service, safety and security, we can do the best job of meeting our Customer's requirements in the most efficient and cost-effective manner. Our corporate mission is to continue to grow as a vital force in the transportation and logistics industry, and to provide superior logistical services and support to our customers. We are a winning team of capable leaders, associates and agent partners who respond quickly and efficiently to change and are able to adapt aggressively to the Customer's needs. Forging alliances with our customers to provide consistent, efficient and economical transportation services will accomplish our goals and objectives. It is our pledge to accomplish these goals by operating safely with uncompromising integrity, availing benefits to our Customers, Employees, Agents and the community at large.
Our Customers are Our Life's Blood
They must be constantly nourished and provided a service of the highest quality at a cost that will allow both we, and they, to make a fair profit. To survive, we must endeavor to understand and meet the expectations and demands of each of our customers. We must be ever mindful of the concerns and needs of our associates and agent partners; for it is through them that our customer's needs are served. Everyone must be considered an individual whose suggestions and concerns are considered and acted upon. Those who are qualified must be given the encouragement and opportunity for advancement and provided an atmosphere in which they can develop and hone their skills. Each associate and agent partner must be treated with respect and recognized for their contributions. Compensation must be fair and adequate and working conditions clean, orderly and safe.
Evans Delivery operates under the following brands:
Evans Gives Back
The Evans Network of Companies gives back to the community in many ways. Multiple employees are involved in local organizations such as Penn State University Schuylkill, The Schuylkill Chamber of Commerce, NEPA Logistics/ Transportation Partnership and many nonprofit groups including, The Walk In Art Center, Schuylkill YMCA, Vision, The United Way, Boy Scouts of America, local fire companies and many other nonprofit organizations.
The Company is on Facebook, YouTube and LinkedIn under Evans Delivery.
If you have any questions about the company, please contact Kim Lorimer klorimer@evansdelivery.com.
About Evans Network of Companies
The network has more than 600 terminal locations throughout the United States, with a fleet of over 7,000 trucks, providing transportation services in the port drayage/intermodal, flatbed, van truckload and freight brokerage markets.
Evans Network of Companies
100-110 West Columbia Street
Schuylkill Haven, PA 17972
Copyright © 2021 by Evans Delivery Company, Inc. All rights reserved.
Website & Hosting by RedX Web Design


text:
Koulibaly's goal puts Senegal into round…
Koulibaly's goal puts Senegal into round of 16 at World Cup
Senegal's Kalidou Koulibaly, right, celebrates with teammates scoring his side's second goal during the World Cup group A soccer match between Ecuador and Senegal, at the Khalifa International Stadium in Doha, Qatar, Tuesday, Nov. 29, 2022. (AP Photo/Francisco Seco)
By The Associated Press |
By GERALD IMRAY
AL RAYYAN, Qatar (AP) — Kalidou Koulibaly literally drew inspiration on his captain's armband, writing the No. 19 on it in memory of one of Senegal's most loved players.
Then Koulibaly became one himself.
The Senegal captain scored the winning goal in the 2-1 victory over Ecuador on Tuesday, putting his team into the round of 16 at the World Cup despite the absence of the injured Sadio Mane, the team's best player and one of the best strikers in the world.
The No. 19 was to honor Papa Bouba Diop, the Senegal midfielder who scored the winning goal against defending champion France in the opening game of the 2002 World Cup. It was one of Senegal's greatest soccer moments and a victory that started that team's run to the quarterfinals — still Senegal's best performance.
"We wanted to make him proud. We wanted to give homage to him," Koulibaly said of Diop, who died at the age of 42 exactly two years ago Tuesday. "He was the one who made me dream when I was a boy. This was something we weren't going to let slip.
"Thank God I was there to send that ball into the back of the net."
Only a win against Ecuador would have taken Senegal through to the next round from Group A, and Koulibaly scored the goal that made it happen at Khalifa International Stadium.
A free kick from Idrissa Gueye bounced off an opponent and came his way. Koulibaly threw out his right foot and hit it past Ecuador goalkeeper Hernan Galindez in the 70th minute. Koulibaly then sprinted toward the corner flag and slid on his knees to celebrate.
The African champions took the lead after a first-half penalty from Ismaila Sarr. But Moises Caicedo evened the score when he was left unmarked after a corner kick in the 67th minute.
At 1-1, Ecuador was going to advance and Senegal was out.
That result would have been as agonizing as Senegal's early elimination four years ago, when the team finished the group stage with a loss to Colombia and even on points with Japan. Senegal was eliminated on a fair play rule because it had picked up two more yellow cards than the Japanese.
This year, Senegal coach Aliou Cisse was forced to reinvent his forward line following the injury to Mane, who sent the team a message before the match. Koulibaly said the Bayern Munich striker has also been an inspiration for the team despite not being in Qatar.
"When it came to his (Mane's) injury, two thirds of the world probably doubted that we would be able to qualify," Koulibaly said.
Senegal lost its opening match against the Netherlands 2-0 but gave itself a chance with a 3-1 win over host Qatar. Still, the team needed a victory over Ecuador.
"We were up against a brick wall," Cisse said.
The 24-year-old Sarr won the penalty when he was knocked over by a clumsy challenge from Ecuador defender Piero Hincapie just before halftime. He side-footed his penalty into the bottom right corner in the 44th minute and almost the entire Senegal bench and coaching staff ran to him.
Sarr also headed over to the corner flag to celebrate while putting his hands over his eyes.
He can look now, because Senegal has at least one more match to play.
SENEGAL'S QUEST
Senegal now has a chance match, or even better, the country's scintillating run to the quarterfinals in 2002, when Cisse was the captain and the team equaled Africa's best performance at a World Cup. No African team has made the semifinals.
ECUADOR EMOTION
It was the end of a strong campaign for Ecuador and striker Enner Valencia, who had scored his team's first three goals at this year's World Cup and played against Senegal despite struggling with a bad knee strain. Valencia was left consoling Galindez, who was tearing up near the end of the game and couldn't hold back after the final whistle as he cried on Valencia's shoulder.
Senegal will play England in the round of 16 on Sunday. The Netherlands, which won Group A, will face the United States on Saturday.
Colorado's trying a public option, drug importation and more. What's actually lowering health care costs?
Colorado rural residents facing hurdles to health care get assistance from a program


If you are able to take current digital marketing trends and apply them specifically to your pay-per-click (PPC) efforts, you will give yourself a boost over your competitors.
Features and enhancements in pay-per-click (PPC) occur regularly, but taking overall marketing trends and applying them to PPC gives advertisers an advantage over the competition.
While they still represent a smaller percentage of the coupons redeemed in stores, the digital coupon saved via mobile devices is sure to catch up with users' increasing mobile behavior.
First, ensure there is a landing page ready that reiterated the offer in the ad.
Use ad copy with offer and call to action (CTA) in headline or body copy to bring the sale home.
Use Sitelink extensions to communicate additional selling text, then link to several offers by categories, promotion type, or deals of the day.
New callout extensions allow an additional up to 25 characters of descriptive text in the standard text ad. These are not clickable, but rather can showcase some additional unique offers such as "free shipping" or "price matching" that can entice searchers to click on the ad.
Through apps and search engines, consumers can now easily compare prices, features, and availability even while shopping in the store.
Be sure to use location targeting in Google and Bing Ads to target areas closest to your stores with ads and then provide special promotions and offers customized to that specific location.
When buyers are comparison-shopping, especially for big-ticket items or during the holiday season, they will be using high-tech gadgets' features to locate where to buy the items.
For PPC, be present in Google Maps through Google My Business, which connects maps with Google+ and search. Ensure all profiles are complete so that maps entries have as much information as possible for the searcher.
Online reviews have evolved into must-haves for shoppers.
1. PPC advertisers in AdWords can use review extensions to surface positive reviews. The review extensions attach a review line to the ad. If you have a review or award from a third-party publication or organization, you can use it.
2. Seller ratings let people know which advertisers are highly rated for quality service. Seller ratings are automatically generated from Google from reputable sources such as Google Shopping and Google Trusted Stores and 29 third-party sources.
With the increase in shopping this holiday season, we will be seeing many shoppers posting "likes" and opinions to sites like Facebook, Twitter, Instagram, Pinterest, and more. They will likely post about the store or online shopping experience, deals, products, and customer service. Using these social platforms for ads is a great opportunity to get ahead of the game by becoming familiar to the audience.
For these social ads, consider the market you are targeting and customize the message to them – don't use the same ad copy for all channels.
Ask for a "share" in the ad so that highly engaged users are reminded to pass it along.
Are there any marketing trends you've been following that are perfect to apply to PPC? Tell us in the comments or tweet it out to @ClickZ or @LisaRocksSEM to discuss.


Our Guest Center is available to assist you 24 hours a day, 7 days a week.
Located in our lobby, or just a button-press away on your in-room telephone, our dedicated team offers genuinely friendly and efficient service.
We're standing by, ready to arrange all your requests as well as answer your questions and queries during your stay.


text:
\section*{Abstract}
Neural network frameworks such as PyTorch and TensorFlow are the
workhorses of numerous machine learning applications ranging from
object recognition to machine translation. While these frameworks are
versatile and straightforward to use, the training of and inference in
deep neural networks is resource (energy, compute, and memory) intensive.

In contrast to recent works focusing on algorithmic enhancements, we
introduce \textsc{BrainSlug}\xspace, a framework that \emph{transparently}
accelerates neural network workloads by changing the default
layer-by-layer processing to a depth-first approach, reducing the
amount of data required by the computations and thus improving the
performance of the available hardware caches.
\textsc{BrainSlug}\xspace achieves performance improvements of up
to 41.1\% on CPUs and 35.7\% on GPUs. These optimizations come at zero cost
to the user as they do not require hardware changes and only need tiny
adjustments to the software.
\section{Background on Neural Networks}
\label{sec:background}
On a basic level, a neural network corresponds to a sequence of
operations which act on numerical input data of a certain predefined size,
so-called tensors.  In computer vision, for instance, the input
tensors are typically three-dimensional data structures, with two
dimensions defining the two-dimensional picture (of $w \times h$
pixels), and the third dimension $d$ containing the information of
each color channel.  The naming of those three dimensions as width,
height, and channels has been adopted as a common general naming
scheme beyond the field of computer vision.

The transformations applied to the input data are grouped into layers,
with each layer executing a certain type of operation (an example of a
deep neural network is given in \autoref{fig-cnn}). The most common
layers are:\\
\noindent(1)~\textit{Element-wise layers} apply a function to each of the input
tensor's values independently. Typical examples of element-wise
layers are normalization and non-linearity layers. The former
normalize the output of a previous layer to conform to a specific
desired distribution, improving convergence behavior and accuracy of
the network. A \textit{non-linear or activation} layer applies an
activation function element-wise to the input. A commonly
used activation function is the rectified linear unit (ReLU), computing $f(x) = \max(0, x)$ on each input~\cite{nair2010rectified}.\\
(2)~\textit{Pooling} layers operate on predefined and fixed regions of
the input tensor. These predefined regions are often non-overlapping and square-shaped.
An example of two pooling operations (average and maximum) is given in
Figure~\ref{fig-pool}.\\
(3)~\textit{Convolutional} layers apply a convolution
operation.  In the image example, a convolutional
layer comprises $k$ groups of $d$ filters each, where in each group
one of the filters gets applied to each channel.  Since each filter
works on a 2-D channel, each of the $k$ groups is also called a 3-D
filter (\autoref{fig-conv}). The output dimension of a convolutional layer has depth $k$, while keeping the
channel dimensions $w \times h$.\\
(4)~\textit{Fully-connected} or \textit{dense} layers: Each value of
the output vector is a weighted sum of all input values, that is, all
output values are connected to all input values. In deep CNNs, there
are usually only a few dense layers and these
are located at the end of the network (see \autoref{fig-cnn}).

\begin{figure}[t]
	\centering
	\includegraphics[width=0.7\columnwidth]{pool.pdf}
	\vspace{1em}
	\caption{\label{fig-pool} Example of a pooling operation on
		one input channel. The aggregation is performed over
		non-overlapping regions of the input space. The figure is
		based on an example from the Caffe
		tutorial~\cite{jia2014caffe}.}
\end{figure}

Each neural network can be mapped to a Directed Acyclic Graph (DAG)
whose nodes correspond to (groups of) element-wise operations and
whose edges represent the input-output relationships between the
computations.  Most deep learning frameworks map a given neural
network specification to such a computation graph and execute the
graph on one or multiple devices. Since the DAG has a unique root
(corresponding to the input data), every node has a unique depth. A
level of a computation DAG is made up of all operations at a specific depth. All
nodes at the same level compute their operations independently of each
other, but nodes at deeper levels might depend on results of nodes in
previous levels.  The computation DAG, therefore, also represents the
dependency structure of the computations. Figure~\ref{fig:trio-bf} illustrates the connection
between the layers of a neural network (right), the computation graph
(left), and the corresponding code snippet (center).

\begin{figure}[t]
	\centering
	\includegraphics[width=0.75\columnwidth]{conv.pdf}
	\vspace{1em}
	\caption{\label{fig-conv} Each convolutional layer moves a set
          of 2-D filters (each being part of one 3-D filter) over the
          input channels from left to right and top to bottom and
          multiplies and accumulates the corresponding values. }
\end{figure}

\section{Discussion and Future Work}
\label{sec:discussion}

We have shown that \textsc{BrainSlug}\xspace (\href{http://brainslug.info}{brainslug.info}) accelerates commonly used deep neural networks by
as much as 41.1\% on CPUs and 35.7\% on GPUs while requiring
minimal code changes. These improvements are significant considering that training
such networks on big data can take up to several weeks. {\textsc{BrainSlug}\xspace}'s speed-up is most
pronounced for the more commonly used batch sizes of 8 and up. For instance, the \textsc{DenseNet} architectures are usually trained with a per-GPU batch size of 32~\cite{densenet}, a batch size where \textsc{BrainSlug}\xspace achieves the best performance improvement. Due to recent results insights into the benefits of increasing the batch size during training~\cite{smith2017don} and the generally growing size of main memory on GPUs, we expect training batch sizes to further increase in the future.

\noindent\textbf{Extending \textsc{BrainSlug}\xspace:} \textsc{BrainSlug}\xspace is designed to make
it easy to extend and, as such, provides APIs that need to be
implemented when developing front and back-ends. Adding a new
front-end requires the biggest effort, since it has to parse through
the NN graph and identify optimizable layers; this cannot be
implemented generically as every NN framework uses a different
representation. Further, it is necessary to connect \textsc{BrainSlug}\xspace's
runtime system with the NN framework, so that \textsc{BrainSlug}\xspace can interact
with framework-specific data structures. For PyTorch, our front-end
implementation consists of 270 lines of Python code and 438 lines of
C++. To add a new back-end requires much less work, as only methods to
load and execute the device code are required. In our implementation,
we have 299 lines of C++ code (code + header files) for NVIDIA GPUs
(including integration of the NVIDIA profiling library) and 165 lines of code for CPUs.

\noindent\textbf{Limitations:} Although in theory our stacking method
can be applied to several different kinds of layers, we figured out
that in certain cases it is not beneficial. While we were able to
achieve significant speed ups for element-wise and pooling layers, we
have not been able to improve linear and convolutional layers. For
convolutional layers the problem is that the operation itself uses
overlapping data areas. Because of this overlap, \textsc{BrainSlug}\xspace would
force neighboring data paths to have to do redundant calculations. As
convolution is already a compute-bound operation, and since \textsc{BrainSlug}\xspace
optimizes memory accesses and not the actual computation, these
redundant calculations reduce overall performance. For linear layers
the problem is more conceptual. A linear layer can be represented by a
matrix--matrix multiplication. Instead of executing this as a
matrix--matrix multiplication, \textsc{BrainSlug}\xspace would strip it down to
multiple vector--matrix multiplications. The problem of this method is
that the processor needs to load the entire weight matrix for each
output vector. In contrast, in a matrix--matrix multiplication the
weight matrix can be significantly better reused, resulting in less
memory transactions compared to multiple vector--matrix multiplications.

\noindent\textbf{Future Work:} We plan to enhance \textsc{BrainSlug}\xspace by
adding more front-ends to support a larger variety of frameworks. Further, we
plan to expand our optimizations to training, as this is the most time consuming
operation for neural networks; we expect \textsc{BrainSlug}\xspace to be able to achieve
equivalent speed-ups for it. Finally, we are also targeting additional types of
hardware including vector or neural network processors.
\section{Evaluation}
\label{sec:eval}
To evaluate \textsc{BrainSlug}\xspace we chose the
TorchVision\cite{torchvision} package.
TorchVision contains a series of broadly used neural network architectures for
computer vision applications. We use the entire set of available networks
ranging from AlexNet (A)~\cite{alexnet}, Densenet-(121, 161, 169, and 201) (D)
\cite{densenet}, Inception v3 (I)~\cite{inception}, Resnet-(18, 34, 50, 101, and
152) (R)~\cite{resnet}, Squeezenet-(1.0 and 1.1) (S)~\cite{squeezenet}, and
VGG-(11, 13, 16, and 19, with and without Batch Normalization) (V)~\cite{vgg}, a
total of 21 different architecture and parameter combinations.

We run all tests on a server with an Intel Xeon E5-2690v4, an NVIDIA GeForce GTX
1080 Ti, Debian 9, NVIDIA GPU driver v384.81, CUDA v9.0, ISPC v1.9.2, Python
v3.5.3 and PyTorch v0.3.0 (using cuDNN). We perform the test times ten times for
the GPU and five times for the CPU and we take the minimum execution time for
both PyTorch and \textsc{BrainSlug}\xspace results.

\begin{figure}[t]
	\centering
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{scaling.pdf}
	\caption{Performance of \textsc{BrainSlug}\xspace's stacked layer mechanism versus PyTorch
		for increasing numbers of $<$Max-Pooling,Batch Normalization,ReLU$>$
		blocks.}
	\label{fig:eval:synthetic_scaling}
\end{figure}

\subsection{Stacked Layers Acceleration}
For the first experiment we want to evaluate the advantage of our proposed layer
stacking mechanism. To do so, we build artificial neural networks consisting only of
layers that can be optimized using \textsc{BrainSlug}\xspace. In particular, we define a block
consisting of a Max-Pooling (Kernel: 3x3, Stride: 1x1 and Padding: 1x1), a Batch
Normalization and a ReLU layer, and create neural networks that comprise between
1 and 40 of these blocks. We execute these networks on a CPU and GPU (see
\autoref{fig:eval:synthetic_scaling}, notice the log scale) and evaluated three
different strategies: only 1 step per sequence, max 5 steps per sequence and
unrestricted.

On the CPU, the PyTorch implementation is always 10-20x slower than
\textsc{BrainSlug}\xspace. This massive increase is partially due to the fact that
the current PyTorch implementation is not particularly optimized for
CPUs: most significantly, it does not use any explicit vector
processing instructions. In contrast, we use ISPC for vector
operations, so in theory we have 8x more computational power (AVX2
with 8x 32Bit float operations). Further, the PyTorch CPU code relies
on OpenMP \texttt{parallel for} constructs, but does not define a
specific execution schedule, yielding sub-optimal
performance. On the GPU, \textsc{BrainSlug}\xspace yields a speed-up of 1.4-2.2x.

\begin{figure*}[t]
	\begin{minipage}[t]{0.49\linewidth}%
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{cpu_time.pdf}%
	\caption{\label{fig:eval_total_cpu_time}Comparison of calculation time between PyTorch and \textsc{BrainSlug}\xspace for TorchVision's neural networks (CPU, batch size 128).}%
	\end{minipage}%
	\hspace{0.02\linewidth}%
	\begin{minipage}[t]{0.49\linewidth}%
	\centering
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{gpu_time.pdf}%
	\caption{\label{fig:eval_total_gpu_time}Comparison of calculation time between PyTorch and \textsc{BrainSlug}\xspace for TorchVision's neural networks (GPU, batch size 128).}%
	\end{minipage}%

	\begin{minipage}[t]{0.49\linewidth}%
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{cpu_speedup.pdf}%
	\caption{\label{fig:eval_total_cpu_speedup}\textsc{BrainSlug}\xspace's speed-up over PyTorch of TorchVision neural networks (CPU, batch size 128).}%
	\end{minipage}%
	\hspace{0.02\linewidth}%
	\begin{minipage}[t]{0.49\linewidth}%
	\centering
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{gpu_speedup.pdf}%
	\caption{\label{fig:eval_total_gpu_speedup}\textsc{BrainSlug}\xspace's speed-up over PyTorch of TorchVision neural networks (GPU, batch size 128).}%
	\end{minipage}%
\end{figure*}

For both devices the performance improves even if we only allow one step per
sequence. It further improves when we stack multiple steps in a sequence, up to
61\% for the GPU and 58\% for the CPU. In the unrestricted case, we can see that
for the lower block counts it is equal or even slightly better than the 5 step
scenario. However, the performance significantly decreases for larger values
until it reaches an artifact (indicated by circles) that happens for the GPU at
16 and 32, and for the CPU at 24 and 38. These artifacts occur whenever the
cache size limit is reached and an additional sequence is required. The cause
for this increase in required cache size is the padding value of the Max-Pooling
layer. This causes an overlap of data and, as previously discussed for the
convolutional layers, results in redundant operations. As each block adds new
padding, the value increases with each additional block. The performance
improves at these points since the new sequence does not suffer from the
redundant operations in the first place, but only if too many blocks are added
to it.

\subsection{Full Network Acceleration}
Next, we evaluate the acceleration that \textsc{BrainSlug}\xspace provides when
executing more realistic neural
networks. Figures~\ref{fig:eval_total_cpu_time}
and~\ref{fig:eval_total_gpu_time} show the total execution time when
running the networks with a batch size of 128 for CPUs and GPUs
respectively, while Figures~\ref{fig:eval_total_cpu_speedup} and
\ref{fig:eval_total_gpu_speedup} show the relative speedup with
respect to PyTorch.

While the networks have significantly varying execution times ranging
from very short (AlexNet) to quite long (Densenet-161 and Resnet-152),
\textsc{BrainSlug}\xspace provides a speed up in all cases, with the most pronounced
improvements for Densenets on both CPU and GPU, VGGs with Batch
Normalization on GPU, and Squeezenets on CPU.  Note that adding the
Batch Normalization layer to the VGG networks has a significant impact
on PyTorch's computation time, while there is virtually no change in
\textsc{BrainSlug}\xspace's case: an effect directly attributable to \textsc{BrainSlug}\xspace
collapsing the normalization into the previous step.

\autoref{tbl:eval_total_table} shows \textsc{BrainSlug}\xspace's full speed-up
results for all networks on CPU and GPU for batch sizes from 1 to
256.
The results clearly indicate that \textsc{BrainSlug}\xspace outperforms PyTorch on the GPU with
batch sizes bigger than 8 (except for Resnet-101 and \mbox{-152}), and
for all cases for the CPU.

\begin{lstlisting}[caption={PyTorch's Max-Pooling implementation.}, label=lst:maxerror, float=t]
#omp parallel for
foreach(batch):
	#omp parallel for
	foreach(channel):
		...
\end{lstlisting}

The results show large performance gains for small batch sizes when
running on the CPU. This is related to a programming error in
PyTorch's Max-Pooling implementation (see \autoref{lst:maxerror}). The
code uses two nested OpenMP \texttt{parallel for} loops, which means
that only the outer loop is parallelized over the CPU cores. In the
extreme case of batch size $= 1$, the entire function utilizes only a
single core. In \textsc{BrainSlug}\xspace we use only one \texttt{parallel for} loop,
iterating over \texttt{BatchSize $\times$ Channels} elements, so we
can always leverage parallelism.

Finally, note that negative values for the GPU batch sizes 1–-4 look
significant but in absolute terms they are not: in these cases the
execution time is only a few milliseconds, while for larger ones, it
is hundreds of milliseconds. This relatively performance difference is
mainly because our implementation is optimized towards larger batch
sizes.

\begin{table*}[t]
	\centering
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{total-table.pdf}
	\caption{\textsc{BrainSlug}\xspace full speed-up results compared to PyTorch on a CPU and
		GPU for all neural networks.}
	\label{tbl:eval_total_table}
	\vspace{-3pt}
\end{table*}

\subsection{Detailed Performance Analysis}
\textsc{BrainSlug}\xspace's performance gains stem from optimizing some layer types,
while leaving others as they are. Here we provide a more detailed
analysis to answer the following question: for real-world neural
networks, how much does \textsc{BrainSlug}\xspace improve the performance of the
optimizable layers, and what fraction of the overall runtime is this
optimizable part?

\begin{table}[t]
	\centering%
	\input{DetailKernel.tex}%
	\vspace{1.0em}
	\caption{For each NN at batch size 128: the number of layers, how
          many \textsc{BrainSlug}\xspace can optimize and into how many stacks.
          Opt.\ Speed Up is the acceleration for the optimizable
          layers, \% of Total Time is the time the optimized layers
          take within the full execution, and Total Speed Up is the
          speed-up for the entire network.}%
	\label{tbl:eval_detailkernel}%
\end{table}

As shown in \autoref{tbl:eval_detailkernel}, the complexity of the networks
ranges from 27 up to 709 layers, with \textsc{BrainSlug}\xspace able to optimize 44 to 64\% of
them using 8 to 204 stacks. For those layers, we achieve speed-ups of 321.2 to
842.9\% on the CPU and 5.7 to 222.9\% on the GPU (all results are for a batch
size of 128). Again, the speed-up for the CPU is significantly higher than for
the GPU due to PyTorch's less-than-optimal CPU implementation. Overall, the
optimizable layers represent 2.5 to 16.9\% for the CPU and 13.7 to 47.4\% for
the GPU of the total computation time (\% of Total Time columns), with the rest
of the time being spent mostly on convolutional layers. This leads to a total
speed-up of 2.1\% to 13.9\% for the CPU and 1.1\% and 20.9\%.
Note that this speed-up only concerns the pure compute kernel time, and is
hence different from the numbers for batch size 128 in Table~\ref{tbl:eval_total_table};
the total improvement is in fact often higher because, for example, \textsc{BrainSlug}\xspace needs
fewer memory allocations.

\begin{figure}[t]
	\vspace{-9pt}
	\centering
	\includegraphics[clip, trim=0 0 0 0, width=\linewidth]{batchsize.pdf}%
	\caption{\label{fig:eval:batch_scaling}Scaling behavior of \textsc{BrainSlug}\xspace (BS) versus PyTorch (Py) for different batch sizes.}%
\end{figure}

\subsection{Batch Size Scaling Behavior}
One important parameter for neural network performance is the batch size, which
represents the number of independent data parts (e.g., images) that are
processed at the same time. Most operations can operate independently on
individual batches, which can be leveraged for parallelism.
\autoref{fig:eval:batch_scaling} shows how PyTorch and \textsc{BrainSlug}\xspace
scale with respect to increasing batch sizes for three selected
networks. As can be seen, both scale with batch size but \textsc{BrainSlug}\xspace
performs always best, showing increasing gains for larger batch sizes.

\section{\textsc{BrainSlug}\xspace: Architecture and Implementation}
\label{sec:implementation}
One of the explicit goals of \textsc{BrainSlug}\xspace is to transparently accelerate
neural networks (NN) \emph{irrespective} of the framework (e.g., PyTorch, Theano,
Caffe) they are implemented in. Further, we want the acceleration to
apply to a wide range of hardware devices including GPUs, CPUs, FPGAs, and vector
processors, among others; this is possible because even though their
architectures may vary widely, they all rely on a memory hierarchy to speed up
memory accesses, precisely the hardware feature that \textsc{BrainSlug}\xspace targets.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{brainslug-arch.pdf}%
	\vspace{1em}%
	\caption{\textsc{BrainSlug}\xspace architecture consiting of front-ends (FE) that plug to
          existing frameworks and convert their NNs into a common
          abstraction; the optimizer that generates the code to transparently
          accelerate them; and the scheduler that executes such code, relying on
          the back-ends (BE) to run it on different target hardware.}%
	\label{fig:brainslug-arch}%
\end{figure}

To comply with these requirements, the \textsc{BrainSlug}\xspace architecture introduces the
notion of \emph{front-ends} to support different NN frameworks, and
\emph{back-ends} to be able to execute on different kinds of hardware (see
\autoref{fig:brainslug-arch}).

The \textsc{BrainSlug}\xspace front-ends are specific to a particular framework. They are in
charge of parsing the NN in whatever format it is in, and providing an
abstraction for it called a \emph{stack} for \textsc{BrainSlug}\xspace's optimizer component to
use. Further, the front-ends provide glue to invoke \textsc{BrainSlug}\xspace's scheduler
component whenever the framework launches the prediction process.

The back-ends provide the necessary glue to have \textsc{BrainSlug}\xspace-generated code
execute on different kinds of hardware, including providing hardware specs to
the optimizer component to help it in generating the code.

Beyond these front and back-ends, \textsc{BrainSlug}\xspace consists of two main components:
an optimizer, corresponding to a \emph{compile} phase, and a scheduler, mostly
in charge of the \emph{execution} phase. Next, we cover each of these phases in
turn, pointing out how the various components in the \textsc{BrainSlug}\xspace architecture
interact to optimize and execute a NN. We end the section by giving a more
detailed description of the PyTorch frond-end we implemented along with its API;
and a discussion of the CPU and GPU back-ends.

\subsection{Compile Phase}

\autoref{fig:brainslug-compile} shows \textsc{BrainSlug}\xspace's compile phase,
which is primarily carried out by \textsc{BrainSlug}\xspace's optimizer. The process
begins when the NN framework calls the front-end's \texttt{optimize}
function method (Step 1 in the Figure). Next, the Network Analyzer
goes through the neural network and identifies sequences of
optimizable layers; a layer is optimizable if its type is in
\textsc{BrainSlug}\xspace's list of optimizable layers (Step 2).

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{brainslug-compile-phase.pdf}%
	\vspace{1em}%
	\caption{\textsc{BrainSlug}\xspace's compile phase. The system parses a NN to identify
		  collapsible layers (steps 1, 2), collapses those layers to fit in
		  cache (3), inserts hardware-optimized functions into the code (4),
		  compiles the code (5) and transparently injects the code back into the
		  NN framework (6).}%
	\label{fig:brainslug-compile}%
\end{figure}

Third, the Collapser retrieves device specs from the back-end(s)
(e.g., cache sizes) and takes care of reducing the layers so that
their memory usage can be fit into a target cache (Step 3). In the
next step (Step 5), the Code Generator retrieves device-specific
pre-processor templates (Step 4) to speed up particular functions (e.g., the
max function maps to \texttt{fmaxf} on a GPU), generates optimized
code and compiles it (Step 5). Finally, the Code Generator uses the
front-end to inject the code back into the NN framework (Step 6).

\noindent\textbf{Collapse Process.} The collapse process merits
further description (see \autoref{fig:brainslug-collapse} for a diagram
and \autoref{lst:collapse} for corresponding pseudo-code). To begin
with, we identify optimizable layers and group them into a
\emph{stack}. We then map those layers onto basic computational
operations: these can either be element-wise (e.g., Batch
Normalization or ReLU) or non-element-wise (e.g., pooling).

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{steps.pdf}%
	\vspace{1em}%
	\caption{\textsc{BrainSlug}\xspace's collapse process makes the execution
          of layers, and particularly the data they need, amenable to
          the available cache sizes. Convolution and linear
          layers cannot be optimized and are left
          untouched. Operations marked in red are
          non-element-wise.}%
	\label{fig:brainslug-collapse}%
\end{figure}

Third, we assign these operations to \emph{steps}. If an operation is
element-wise, it can always be added to a step. If not, it can still
be added to the step if there is not already another non-element-wise
operation present. If this criterion is not met then we create another
step: this is necessary as a non-element-wise layer's operations depend on the output
of a larger number of previous operations.

\begin{lstlisting}[caption={Pseudo code explaining \textsc{BrainSlug}\xspace's collapse process.}, label={lst:collapse}, float=t]
class Stack:
def collapse():
	#3: group operations in steps
	steps = []
	step  = new Step()
	foreach(operation in optimizable):
		if(not step.onlyElementwise() or
			not operation.isElementwise()):
			step.add(operation)
		else:
			steps.add(step)
			step = new Step()

	#4: group steps in sequences
	sequences = []
	sequence  = new Sequence()
	foreach(step in steps):
		sequence.add(step)
		if(sequence.resourceConsumption() >
				device.resourceLimit()):
			sequence.remove(step)
			sequences.add(sequence)
			sequence = new Sequence()

#1: identify optimizable layers
stack = new Stack()
foreach(layer in graph):
	if(isOptimizable(layer)):
		#5: replace layers with stack
		if(stack.empty()):
			graph.insertBefore(layer, stack)
		graph.remove(layer)

		#2: map layer to operations
		foreach(operation in layer);
			stack.add(operation)
	else:
		stack.collapse()
\end{lstlisting}
\begin{lstlisting}[caption={Example code of a collapsed stack.}, label={lst:finalcode}, language=C++, morekeywords={foreach,in}, float=t]
void step_0(in_data, out_data, ...):
	foreach(o in out_data):
		foreach(i in in_data):
			MaxPooling()
	BatchNorm()
	ReLU()

void step_1(in_data, out_data, ...):
	foreach(o in out_data):
		foreach(i in in_data):
			AvgPooling()
	AvgNormalization()

void sequence_0(in_data, out_data, ...):
	float cached_data[...]
	step_0(in_data, cached_data, ...)
	step_1(cached_data, out_data, ...)
\end{lstlisting}
\begin{lstlisting}[caption={Snippet showing how to use \textsc{BrainSlug}\xspace's PyTorch front-end. Only lines 2 and 8 need to be added by the user.}, label={lst:python}, float=t]
import torchvision.models as models
import brainslug

# load the model
model = models.__dict__['...']()

# optimize with BrainSlug
brainslug.optimize(model)

# execute the model
model(...)
\end{lstlisting}

After this, we group the steps in order to utilize hardware resources
efficiently. As each step requires that it is synchronized after it is
processed, all data needs to be stored in a local data cache to pass data from
one step to another. To accomplish this, we bundle these steps into
\emph{sequences}. We iterate over the steps and evaluate if their resource
consumption fits the limitation of the target hardware (e.g., a L1 cache on a
CPU or shared memory on a GPU). The resource consumption is calculated by the
amount of data that each step requires and the number of active SIMD units of
the processor that share this data. For example: If we have 128 SIMD units, a
non-overlapping pooling layer with kernel size 3x3, and 32 channels, we would
require 128*32\,B for the output and 128*3*3*32\,B for the input. An additional
layer would have the previous input size as output, and a corresponding larger
input size. If the stacked steps do not exceed the hardware resources, we add
the step to the sequence, otherwise we create a new sequence.

Finally, we generate the final code. There are two possible scenarios. First, if
a sequence only contains a single step, we iterate over the entire input data:
in this case data locality is achieved by directly passing the values from one
operation to another. If there are multiple steps, we need to perform the
previously-mentioned synchronization between the steps. In the case that the
cache size limit is not reached, we increase the size of it, so that each SIMD
unit may not calculate a single output value, but multiple ones, to better
utilize the given hardware resources. Finally, we compile the code using a
device-specific compiler and replace the optimized layers in the NN with our
collapsed stack. To illustrate, \autoref{lst:finalcode} shows how the example in
\autoref{fig:brainslug-collapse} is mapped onto the actual final code.

\subsection{Execution Phase}
\textsc{BrainSlug}\xspace's execution phase, embodied by the scheduler, handles the execution
of the compute kernels. When a stack is executed, the front-end gathers all
necessary data and parameter tensors. The scheduler then calculates the output
size and allocates memory using the NN framework. After this the kernel function
object (cubin for GPU and dll for CPU) is loaded, executed and the result buffer
is returned to the NN framework. If there is more than one sequence in a stack
the sequences are executed in a serialized fashion.

\subsection{PyTorch Front-end and API}
We implemented a PyTorch front-end. We chose PyTorch as it was the
first NN framework to support dynamic network graphs that can be reshaped at
runtime. This feature complicates the implementation but allows us to show that
our method can be applied even in such a highly dynamic scenario.

The frontend parses through the neural network, groups all optimizable layers in
stacks and passes these to the \textsc{BrainSlug}\xspace optimizer. These are then removed from
the network and replaced by a special \textsc{BrainSlug}\xspace layer (one per stack) that pass
the control flow to the \textsc{BrainSlug}\xspace scheduler whenever they are triggered. If
there are multiple equivalent stacks, \textsc{BrainSlug}\xspace only generates the code once
and reuses it for all identical stacks.

Finally, it is worth noting that the front-end is extremely easy to
use: the user need only add a few lines of code in order to enable
transparent acceleration of the neural networks (see \autoref{lst:python}).

\subsection{CPU and GPU Backends}
\noindent\textbf{GPU}: GPUs are often used for processing of neural networks,
mainly because of their high compute performance and memory throughput. For the
implementation we use two code building blocks.

For steps that only perform element-wise operations, we start as many thread
blocks as there are channels and each thread blocks applies its calculations on
each batch, for a specific channel.

For pooling layers we distinguish between stacked and non-stacked. In the
non-stacked case, we start \texttt{BatchSize * Channels} thread blocks. The SIMD
units iterate over all data elements, while we process as many element as we
have SIMD units in parallel. In the stacked case, we use \texttt{BatchSize *
Channels * Patches} thread blocks, where each patch represents a depth-first
processing and use the SIMD units the same way. To store the data for the
depth-first processing we use two buffers allocated in the devices shared
memory. When a step is processed, we synchronize the entire thread block and
swap the buffers for the next step.

In general we use a thread count of 128, which is a good trade-off between
overhead when synchronizing a thread block and compute utilization. Further, we
limit the usage of shared memory to 16\,kB (depending on the GPU either 64 or
92\,kB would be available), as this can have a negative impact on the
performance because it reduces the amount of blocks that can be scheduled onto the
GPU multiprocessor, resulting in less opportunities to employ latency hiding.

\noindent\textbf{CPU}: The CPU back-end relies on the \textit{Intel SPMD Program
Compiler}~\cite{intelispc} (ISPC). ISPC can be seen as
``CUDA for CPUs'': it adds syntactic sugar to the application and explicitly
defines which computations should be done by a single SIMD unit. Because of the
similarities between ISPC and CUDA we can share many parts of the implementation
between both architectures. As CPUs do not have a dedicated shared memory, we
allocate the memory on the stack. The other parts of the implementation are
 similar and differ only in the way variables are used -- either by all or only single SIMD units -- and replacing the outer loops with the
ISPC specific \texttt{foreach(...)} instruction. ISPC can target different
instruction sets, e.g. SSE[2,4], AVX[1,2] and AVX512 for Intel's Knight's
Landing. We use the default values from the compiler. ISPC further supports a task
system, similar to CUDA thread blocks. This task system has to use some predefined
variants provided by ISPC or can be implemented by the user. As our task system
does not require any special features, we implemented a simple variant
based on \texttt{parallel for} using Intel's Threading Building
Blocks~\cite{inteltbb} to minimize the framework's
overhead.

\section{Introduction}
\label{sec:intro}
Artificial intelligence, and neural networks in particular, have
gained immense notoriety in the past few years. Their flexibility
means that they can be applied to a wide range of applications, from
recommendation systems for online stores, to autonomous driving,
to financial fraud detection or optimization of production lines.

One of the main issues with neural networks is their high
computational time. While over the years many algorithmic and
hardware optimizations to reduce the cost of these computations have
arisen, the sequential way in which a neural network is
processed, taking input data and computing on it from layer to layer
before moving onto the next input data, has remained largely
fixed. Even though this breadth-first processing is straightforward,
the fact that each pass through the network acts on a relatively large
amount of data causes constant cache trashing (whether on CPUs, GPUs,
or other hardware), reducing their effectiveness and ultimately
increasing computation time. This partly explains why processors with
extremely high memory throughput are used for neural networks so that
the processors are never idle.

In this report we propose the use of a depth-first approach: we take a
subset of the input data (e.g., a part of an image) that can fit in
L1 cache and compute all layers, then repeat the process for the next
subset of the data. At this high level the process sounds simple;
however, there are two main issues. First, only certain operations
(i.e., layer types) are able to function when given only a subset of
the data. Second, processing data in this way requires the user to
write specialized compute kernels for each possible sequence of
layers. This is clearly difficult to do by hand and points to the need
of an automated system to carry this out.

We implemented \textsc{BrainSlug}\xspace (\href{http://brainslug.info}{brainslug.info}), a system that enables depth-first
computation of neural networks, providing \emph{transparent}
acceleration through improvements to data locality. We make the
following specific contributions:

\begin{itemize}[noitemsep,nolistsep]
  \item A novel, depth-first method for neural network computation
    that increases data locality and reduces computation time. The
    method does not change the actual results of the computation, and
    is widely applicable to a large set of neural networks and
    different types of hardware.

  \item An implementation of this method, including a modular
    architecture that allows for easy extensibility to multiple neural
    network frameworks (e.g., PyTorch~\cite{pytorch}, Theano~\cite{theano}, Caffe~\cite{caffe}, TensorFlow~\cite{tensorflow}) and hardware
    targets (e.g., CPUs, GPUs, FPGAs, etc.).

  \item The implementation and evaluation of \textsc{BrainSlug}\xspace using a
    PyTorch front-end and CPU and GPU back-ends. Through extensive
    experimentation we show that \textsc{BrainSlug}\xspace achieves speed-ups of up
    to 41.1\% on CPUs and 35.7\% on GPUs while requiring only tiny adjustments
    to a user's program.

\end{itemize}

In the following, we first give a brief background about neural
networks in Section~\ref{sec:background}. Section~\ref{sec:method}
describes the main idea behind \textsc{BrainSlug}\xspace, followed in
Section~\ref{sec:implementation} by a description of the system's
implementation, and an in-depth evaluation in Section~\ref{sec:eval}.
Finally, we discuss related work in Section~\ref{sec:related} and
conclude in Section~\ref{sec:discussion} with a discussion and future work.

\section{\textsc{BrainSlug}: Method Principles}
\label{sec:method}

Every DNN has to perform numerous passes through the network during
training and prediction. In many cases, millions of floating-point
operations are required for one pass and there are millions of passes
per training task. With \textsc{BrainSlug}\xspace we want to improve the
resource utilization of DNN frameworks, with a focus on accelerating
(groups of) element-wise and pooling layers, all the while ensuring
that this acceleration is transparent to users and that it can be
implemented irrespective of the deep learning framework used (e.g.,
PyTorch, Theano, Caffe) and irrespective of the target hardware.

Towards this goal, we address the shortcoming of existing deep learning
frameworks that always execute neural networks layer by layer. The
dependency structure of the computation graph, however, allows us to
exploit situations in which a different execution pattern is possible.
Our work on \textsc{BrainSlug}\xspace is therefore motivated by two basic questions:

\begin{enumerate}[noitemsep]
\item Are there ways to rearrange the standard execution order of the
  computation graph such that the hardware can operate more
  efficiently while delivering the same numerical results?

\item Is there a method for detecting when such a rearranging is
  possible such that it is both frequently applicable and efficient to
  compute?
\end{enumerate}

With the proposed \textsc{BrainSlug}\xspace approach, we answer both questions in the
affirmative. First, we show that the way in which the computation
graph is executed has an impact on the efficiency of the
computations. In several situations, the same set of operations
can be executed such that the intermediate data fits into the
caches and registers of a device, circumventing the need to read and
write from the device's main memory.  This is possible by executing
\emph{independent paths} (or groups of independent paths) in the
computation graph in parallel, essentially parallelizing the computation
graph not only in a breadth-first but, when beneficial, in a
depth-first manner. Second, we show that for a large class of DNNs,
there is a generic method for detecting independent paths whose
intermediate data fits into the caches and registers.

\begin{figure*}[t]%
\centering
	\begin{minipage}[b]{0.5\textwidth}%
		\includegraphics[width=0.85\linewidth]{network-parsing.pdf}%
	\end{minipage}%
	\hspace{0.02\linewidth}%
	\begin{minipage}[b]{0.42\linewidth}%
		\begin{lstlisting}[frame=none]
stack = []
foreach(node in network):
	if(node is optimizable):
		if(stack is empty):
			network.replace(node, stack)
		else
			network.remove(node)
		stack.add(node)
	else:
		stack.collapse()
		stack = []
	\end{lstlisting}%
	\end{minipage}%
	\vspace{1em}%
	\caption{\textsc{BrainSlug}\xspace's layer aggregation process, shown visually (left) and as pseudo-code (right).}%
	\label{fig:network-parsing}%
\end{figure*}

\subsection{Depth-First Parallelism}
Existing deep learning frameworks parallelize the computation graph in a
breadth-first manner, finishing all computations at one level of computation
before starting the next level's computations. We refer to this as
\emph{breadth-first parallelism} because the operations at one DAG level are
executed in parallel before the computation proceeds to lower DAG levels.

\textsc{BrainSlug}\xspace is able to detect and execute more complex independent
computation paths in parallel. While this does not reduce the overall
number of operations, it often leads to situations where the data
accessed by these independent paths fit into the caches and
registers of the hardware, increasing performance.  We refer to the
parallel processing of independent paths in the computation graph as
\emph{depth-first parallelism}. The challenge is now to transparently
detect and interleave these parallelism types to suit the
characteristics of particular hardware.

\autoref{fig:trio-df} illustrates depth-first parallelism for the
example in \autoref{fig:trio-bf}. The computation graph is the same but the operations
are grouped according to the independent computation paths involving
the normalization and non-linear operations, which are merged in the
pooling layer. Whereas in the typical breath-first, layer-wise
execution of DNNs the data has to be written and read from main memory
for each layer, here the intermediate data are small enough to fit
into the hardware's cache, improving overall performance.
Fortunately, as we show in what follows, detecting the existence of
these parallel computation paths in the DAGs of DNNs is both efficient
and frequently possible.

\subsection{Aggregation Detection}
To detect layers that can be aggregated, \textsc{BrainSlug}\xspace parses through the DAG of a given neural
network layer-by-layer and identifies sequences of layers that support data
\emph{locality}, that is, sequences of layers that operate on a sub-set of the data (e.g., a
portion of an image) reducing the number of input-output dependencies in the computation graph. Examples of such layers
are layers performing element-wise and pooling operations.  We add all
consecutive sequences of such layers to a \emph{stack} (see \autoref{fig:network-parsing}),
which we then \emph{collapse} by analyzing the underlying computations and rewriting
them to utilize depth-first parallelism. A stack, therefore, partitions
independent computation paths in the DAG into paralellizable code blocks such
that each such block's intermediate data fits into the device caches. As each of the SIMD units of a single
device core share the same cached data,
we need to find an input data region (and the corresponding combined paths in the computation DAG)  that (1) is big enough to
keep all SIMD units utilized during the computations and (2) does not exceed the
cache size limit. As one can see in \autoref{fig:trio-df}, if one SIMD
unit is mapped onto one white box, we need at least the same number of
boxes as we have SIMD units to operate efficiently.

During the rewriting of the computations we need to take the
dependencies of the original DAG into account. For example, as can be
seen in \autoref{fig:trio-df}, the pooling layer requires all data
from the element-wise layers to be calculated before it can
perform its computations.

 We provide a
detailed description of all of \textsc{BrainSlug}\xspace's mechanisms in the
following sections.

\section{Related Work}
\label{sec:related}
In this work we focus on accelerating the forward pass in deep
neural networks on both CPUs and GPUs. Due to the frequent occurrence
of convolutional and dense layers in these networks (see
Figure~\ref{fig-cnn}), recent work has focused specifically on
improving the multiply-and-accumulate (MAC) operations prevalent in
these layers.
CPUs and GPUs have libraries that support SIMD or SIMT-based processing such as
Intel's ``single program, multiple data'' (SPMD) compilers. In general, the MAC
operations resulting from convolutional and dense layers can be mapped to
multiplications between two matrices. Libraries such as
cuDNN~\cite{chetlur2014cudnn} and cuBLAS~\cite{cublas} for GPUs and Intel
MKL~\cite{intelmkl} and OpenBLAS~\cite{openblas} for CPUs are optimized for
matrix--matrix operations. Moreover, there are specialized algorithms that can
lead to speed-ups for the multiply operation in convolutional layers. For
instance, performing a fast Fourier transform (FFT) has been shown to be
beneficial for convolutional layers with certain properties~\cite{mathieu:2014}.
There are also several other approaches that reduce the number of expensive
operations required for matrix-matrix multiplications. Examples are the
application of Strassen's~\cite{cong2014minimizing} and Winograd's
algorithms~\cite{lavin2016fast} for accelerating the processing of convolutional
layers. Deep learning libraries such as NVIDIA's cuDNN and
TensorRT~\cite{tensorrt} utilize heuristics for choosing the algorithmic method
expected to work best for a given convolutional layer. TensorRT selects
different implementations according to the used hardware and optimizes memory
allocation. Due to the extensive engineering that goes into the design of these
heuristics and implementations, the processing of convolutional layers is highly
optimized and hard to improve on; consequently, \textsc{BrainSlug}\xspace focuses on
improvements to other commonly-used layer types.

The main disadvantage of TensorRT is that it only works if all used layers are
known to the framework, as it directly translates the entire network
into its own implementation for NVIDIA GPUs. \textsc{BrainSlug}\xspace, in contrast, only replaces
parts of the network that it knows. This enables us to create and explore
user-created layers and still benefit from \textsc{BrainSlug}\xspace's
improvements. Further, \textsc{BrainSlug}\xspace is designed as an extensible platform,
allowing users to apply its optimizations not only to GPUs but to all kinds of
processors and accelerators.

The algorithmic optimizations discussed so far do not change the network
architecture or the result of the computations. There are several algorithmic
tricks one can apply to trade-off accuracy for efficiency. For instance,
TensorRT can reduce the precision from 32\,Bit floating point to 16-Bit floating
point or even 8-Bit integer, which improves performance but might decrease
accuracy. It is even possible to work with binarized neural networks, that is,
networks that perform binary instead of floating point
operations~\cite{hubara2016binarized,courbariaux2016binarized,rastegari2016xnor}.
There are numerous other methods that change the structure and parameters of the
original DNN to improve performance. For instance, it is possible to prune
filters during the learning process which reduces the amount of computation
required for the convolutional layers~\cite{li2016pruning}. In a different line
of work, the network units with low-valued activations are pruned. This was
shown to result in a 11\% speed up~\cite{albericio2016cnvlutin} or a
substantial reduction in power consumption~\cite{reagen2016minerva}.
With \textsc{BrainSlug}\xspace, we focus on
optimizations that do not alter the original DNN: both the original
and optimized DNN perform exactly the same operations on the hardware.

Alwani et al.~\cite{2016:alwani:fused} proposed to fuse layers of
convolutional neural networks for faster processing on FPGAs, merging
the first two convolutional layers of a neural network. Their
method uses a data shifting approach to reduce the recomputation of
overlapping data regions. This is quite efficient for FPGAs but is difficult to implement on CPUs and GPUs, and has the limitation of only being applicable to no more than two convolutional layers. As already mentioned, our method
does not focus on convolutional layers but accelerates the entire
network by aggregating consecutive non-convolutional layers.

Sze et al.~\cite{sze2017efficient} discuss different methods for
energy-efficient dataflows on neural network accelerators, suggesting
the development of a specialized neural network accelerator with a
mesh-based processing architecture. In contrast, our method targets
off-the-shelf, cheap hardware that provides excellent compute power per
dollar.

redpajama_set_name: RedPajamaArXiv


« A new disease is sweeping the nation: inconspicuous nonconsumption.
I was once a joyous shopper.
I loved clothes, make up, nail extensions – the works.
But now I'm lurching between depression and near hysteria.
I can't go into shops.
My friends have tried physically forcing me in.
but still can't get through the door.
and my nails are a mess!
This entry was posted on September 14, 2009 at 8:41 pm and is filed under Testimonies.	You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
Good job, ineffectual husband, one boy, one girl, one Labrador.
I had the nanny, the cleaner, the convenience foods.
Now I'm all old clothes and …… I've got my name down for an allotment and I'm collecting horse manure ….
It started slowly. I wore the same outfit to the office two days in a row, then a whole week. Then I realised that my Casual Friday clothes weren't casual at all.
One Wednesday evening I desperately searched the shops for Taramasalata. Nobody seemed to have any. I was so incensed I bought nothing and stormed out of the shops. I couldn't bring myself to go back after that. I started cooking instead. Then cleaning. Spending time with my kids. Taking time off work. Now I am poor but happy, at the forefront of a brave new world. The disease is big but underground – growing every day. There's been nothing on the news except public spending is down. Retail outlets are suffering.
That's my fault. Shops just can't cope without my custom.
A few days ago I saw a woman in the supermarket – harrassed – two wayward toddlers and screaming infant in trolley. She was absent mindedly picking up stuff from her overstuffed trolley, reading the labels as if she was pondering to buy and replacing everything on the shelves.
I think I may have seen you. There was this group of teenagers, singing, chewing gum and chatting – shopping in Primark. They were way too loud, desperate to be seen to be having a good time. Then suddenly one of them goes pale, stops chatting, stops shopping … stops dead. She looks at this ridiculous frou frou thing she was about to buy and shoves it back on the rail, puts back all the clothes she's been collecting to try on.
I've got tops! she screamed I've got tops I've never worn!
She looked genuinely confused, frightened even.
Teenagers you say? Not me … definitely not me! lol!
There must be a lot of it about.
I'm just an observer but there does seem to be a lot of it about.
I'm starting to wish I couldn't go shopping not as an excuse but because it just seems right. It's weird I think I'm a bit jealous.
I didn't realise how much I rely on everybody else to do all the shopping and hardly do any myself. I don't know if I'm shopaphobic or just can't be bothered or like the idea, or something. I am not doing as much on my own as I used to.


Settle My Debts was new in the market, with a new way of dealing with personal unsecured debt. Unlike others who were busy consolidating debts, entering people into Debt Agreements or Bankruptcy, they developed a way to re-negotiate their customer's actual debt amount, reducing repayments and incurring interest. As a new company they were hungry for more business, which is where we came in!
Who are we trying to talk to?
How can Settle My Debts help?
What is their next step?
The tactics we put in place to drive expedient growth.
We began with a brand workshop to discover two things: the customer's needs, pains and gains, and how Settle My Debts was addressing these better than the competitors. In this process, we built solid personas for the target audience, which informed the creative process.
Based on previous results, Settle My Debts could say they can personal unsecured debts by up to 50%, often more. They were willing to back this with confidence with a money back guarantee – if we don't reduce your debts by at least 50% our service is free. Considering how competitive the market is this gave Settle My Debts a competitive edge.
Based on the data we could see there was a portion of people who a) were not able to be contacted or b) did not go on to sign a contract despite showing significant interest. To nurture these contacts we devised ways to reach them via channel expansion – considering Facebook, content, marketing automation and SMS. We also tested the on page form to see if we could reduce initial contact friction by adding or removing field options.
"I'd been looking for a digital agency to grow my business for 2 years. In less than 90 days I've had to take on 3 new people to handle the incoming leads."


"Swap is a rock band with loads of energy. Some classic and some progressive rock in place here with great songwriting, structure and conviction ... I give them a 10 out of 10 for energy and show. These guys are ready. If you are going to be watching an original rock band in Sydney, this should be the one."
Australian finalist in the Emergenza competition & won best unsigned guitarist in Australia award.
2016 - played and interviewed on radio Skid Row 88.9fm and new self-titled album release.
During this period Swap has performed at The Metro, The Factory Theatre, The Lair, The Hornsby Inn, Fitz Centre, Valve Bar - Agincourt Hotel, Lewisham Hotel, The Roundhouse & The Manly Fishos.
'Not One Of Us' was written about someone's (unfortunately to common) view of boat arrival asylum seekers. He had the view that they were separate from us and therefore not our problem. This song discusses and confronts this view. This song was also featured in the documentary film 'Freedom Stories'.


Since bursting onto the fashion scene in 1990, Steve Madden has become renowned as the brand of choice for budget-savvy fashionistas the world over. Offering up chic, urban-inspired footwear, accessories and apparel, the brand continues to bring a rock and roll edge to affordable, wearable fashion. When the temperature drops, you can step out in style with this covetable range of Steve Madden coats. Featuring stylish belted puffer coats, chic funnel necks, military styles and colour-pop duffels, there's a stylish design here to suit every girl's winter closet.


The statistic presents data on the frequency of free music streaming service usage in the United States as of September 2017. During a survey, 20 percent of respondents stated that they listened to free music streaming services several times a day.
paid music streaming services such as Spotify, Pandora, etc.
What devices did you use to listen to audio yesterday?


From Pennsylvania Turnpike - Take Exit 343 - Willow Grove(North). At the first light take a sharp right onto Mill Street. At the first light take a left onto York Road (#263 north). Follow to Street Road (#132) and take a right. Follow to Jacksonville Road (#332) (the Naval Air Warfare Center will be diagonally on the left corner) turn left and follow for 1.7 miles. Spring Mill Manor is on the left, Spring Mill Country Club is on the right.
From I-95 North - Exit at Street Road (#132). Head west for 10 miles to Jacksonville Road (#332) (the Naval Air Warfare Center will be on the right corner) turn right and proceed 1.7 miles. Spring Mill Manor is on the left. Spring Mill Country Club is on the right.
From Route 476 - Take Route 476 North toward Plymouth Meeting. Stay left and follow signs to Pennsylvania Turnpike East. Take Pennsylvania Turnpike East to Exit 343 (Willow Grove). Follow above directions from Pennsylvania Turnpike.


When Sayid shoots the young Ben in Lost, that was pretty badass and horrific. Couldn't believe I was watching a network show, in a good way.
The Lonely Island's "Cool Guys Don't Look at Explosions" video with J.J. Abrams, Will Ferrell, and Andy Samberg. Andy did an amazing job with the MTV Movie Awards. Hosting can be funny—not just a break between Lady Gaga's outfits!
Bryan Cranston in Breaking Bad. He has an internal-burning thing going on like Chris Cooper. He's also a straight-up handsome guy, like a craggier Robert Redford, but also incredibly funny, like Peter Sellers. He's too many good things. And Vince Gilligan's writing jumps off the page.
Pete and Peggy on Mad Men. So many reasons to tune in to this show, but their relationship is my favorite.
I liked when those two people whose names I don't remember got married on The Office.