text:
Rod Trent MVP, myITforum'er
myITforum.com Polls : ConfigMgr 2007 Version
Which version for System Center Configuration Manager 2007 are you running?
Participate here: myITforum.com : ConfigMgr 2007 Version
The myITforum.com Alpha Site
We're working steadily to releasing a brand new myITforum.com web site which will give you some amazing new features, streamlined interface, and offer quicker methods for getting your daily fix of all things System Center. Keep in mind that we're still working on adding, migrating, converting, etc., etc., but the Alpha site is safe to start playing around with. Right now we're just linking to current pages and technologies, but as we progress new things will start to pop up. We'll definitely let you know when there's something significant to check out, but for now, feel free to drop out to the System Center LAUNCH page of myITforum.com 2.0 and give us feedback. We've setup a special beta@myITforum.com email account for your to make comments, suggestions, etc., which is accessible on the System Center LAUNCH page by clicking on the ALPHA SITE graphic at the top right.
http://www.myitforum.com/def-new.asp
There's just so much that goes on at myITforum.com hourly that the old page setup didn't really relay to anyone just how much activity there was. There is quite a bit of traffic on myITforum.com at any given time. The System Center LAUNCH page's goal is to show you what's happening at any given minute of the day. Stay tuned as we change things around and make the site more intuitive for you.
And, since myITforum.com is a *real* community, we are also interested if anyone is willing to get involved in the development of some backend resources on myITforum.com. If you have time to donate to help build the new site into something monumental, just drop us a line. myITforum.com has always been "Powered by You" and now it can also be "Built by You". We've already received some affirmations for help, but it takes a community to raise a web site worthy of such a large and dedicated community as this one. Drop me a note if you are interested in helping out: rodtrent@myITforum.com
Here's some things we need help with:
Database (SQL Server) data migrations
Common task automation, such as newsletter creation
Forums moderation
Columnists for the new WYSIWYG article system (btw: the new article system is also tied into the blogs!)
Post-MMS 2009 myITforum.com Newsletter Coming This Weekend
We have a LOT to talk about since coming off of MMS 2009 this week. We're decompressing this week and following-up on the huge number of vendors who just love myITforum.com. But, expect a Post MMS 2009 report in your email inboxes this weekend. There are lots of stories to tell and photos to share.
Don't receive the myITforum.com newsletter? Shame on you! You are missing out on the most current information about the System Center community. Sign up now and you'll get this weekend's post-MMS 2009 edition:
http://www.myITforum.com/newsletter
Have photos of your own with no place to put them? Drop us an email and we can set up your very own photo gallery on myITforum.com for uploading your MMS 2009 photos.
Have some stories worthy of System Center community reading? Drop us an email and we can have your very own blog setup in a matter of minutes. myITforum.com continues to be the CENTRAL spot if you want the System Center community to take notice of your information and expertise.
admin@myITforum.com
Quick MMS 2009 FAQ: Did you know that the Microsoft Management Summit was actually started by this community back in 1998? Its true. And, this community continues to be the driving factor for MMS success.
In memory…Steve Pruitt
I'm finishing up the myITforum.com awards for MMS 2009 (given at the annual myITforum.com Party) and ran across this again:
Steve won this community award last year. It really makes me a bit sad, seeing it. But, Steve has left a great legacy in this community, and I think he helped instill his same type of dedication and insistence of sharing in others. Steve had a great testimony. I'll always think about him.
Please be in continued prayer for his family.
We just really never know how things will happen, or what will happen in a year's time. This community has become such a family over the years, glued together by myITforum.com and the Microsoft Management Summit. That's also a funny thing to me, that a silly conference and a silly web site could bring so many people together in such a huge way – it just doesn't make sense sometimes. Those who don't participate in this community miss out on a great deal. People who sit back and think that myITforum.com is just a web site, miss the big picture. Those are the folks who think a pretty web site makes a community. There's more than just articles, blogs, and web pages. Here, there's real people from all over the world who feel like they are a part of something. There's people who really care about the health and wealth of this community.
And this just brings me back to Steve. We lost a great member this past year. Yet, there are others who are and may be in the same predicament. Be good to each other, folks. Have fun at MMS. Enjoy this community.
Two new Twibes to follow for MMS 2009
We've created a couple new Twibes for those Tweeters wanting to get the most out of MMS 2009 through up-to-the-minute news and information.
First, there's the System Center Twibe: http://www.twibes.com/group/systemcenter
And, of course, the best one, the myITforum.com Twibe: http://www.twibes.com/group/myITforum
Updated myITforum.com MMS 2009 Party Information!
myITforum.com: Your "Centralized" System Center Community!
April 27th – May 1st
Exhibitors, Presenters, and Community Servants
Find us at: BOOTH #633
What we're doing this year
· Session SU30: "Community Saves Your Company Money, and Why Invest Time in It" – Tuesday, April 28 2:15 PM – 3:30 PM, San Palo 3401A (grab the conference floor plan)
· Session SU31: "Ask the Experts Panel" – Wednesday, April 29 2:15 PM – 3:30 PM, San Palo 3401A (grab the conference floor plan)
· Hourly Booth Raffles
· The myITforum.com Party – Sponsored by Microsoft and 1E
Giving away everything you can imagine
· 1 XBOX 360 A DAY – donated by Microsoft
· Copies of System Center Operations Manager 2007 Unleashed – signed by the authors.
· Copies of System Center Configuration Manager 2007 Administrator's Companion – donated by MS Press – signed by the authors
· Copies of Mastering System Center Configuration Manager 2007 – signed by the authors
· Quest Software has donated two 12 month term licenses (one each) of their Toad for SQL Server and Capacity Manager for SQL Server products with support
· Secure Vantage has donated a Microsoft Zune.
Annual myITforum.com Party at the Microsoft Management Summit 2009
IMPORTANT DATE CHANGE: The Premiere MMS Party – your official invitation
Steeped in tradition, this year, myITforum.com will celebrate it's 9th annual party! Who can say that? The myITforum.com party ticket is the most sought after piece of SWAG at MMS, and the stampede to get one will be as large as ever – maybe larger! Don't believe me? Watch the stampede from last year to understand what you're up against.
http://www.myITforum.com/absolutevc/?v=891
Thanks to the Dell folks for capturing the mad rush on film!
How to get a pass:
The MMS 2009 Expo opens at 5:30pm on Monday, April 27th, but the line to get into the Expo starts forming about an hour prior. You'll need to beat the mad rush and stop by the myITforum.com booth #633 to grab a party button. This button is your ticket into the party. It MUST be worn to get entrance on Wednesday night.
myITforum.com Party buttons are on a first-come first-served basis!
Date: Tuesday, April 28, 2009
Location: Venetian Ballroom B. (grab a floor plan – this is the MMS 2008 floor plan, but it's the same as MMS 2009)
What's happening at this year's party?
Mix and mingle with your peers. Put faces with names and reconnect with old friends.
myITforum.com Annual Awards Ceremony! (Yes, we noticed all you did this year and want to Thank You!)
WIN COOL STUFF! (Enter the Raffle to win a Zune, or an Xbox 360!)
Eat, drink, and be merry! Plenty of food and drinks for everyone!
This year marks the official announcement of our Community "Partners". For a long time, folks have just assumed that Microsoft sponsors and promotes myITforum.com and that's how we have survived for so long. That's simply not true. While we appreciate the type of support that Microsoft gives us, it doesn't pay the bills. Find out who actually invests in the myITforum.com vision and who is benefitting from our experience and reach by visiting our new Partners page:
http://www.myitforum.com/Partners/
ALSO…a very special MMS-only announcement!
Just for MMS 2009 we are EXPANDING our Bronze sponsorship numbers from 4 to 10! If you were wanting to get in on the Bronze level sponsorship for myITforum.com now is your best chance!
In addition – we are offering a Show Special 15% off on Silver and Gold sponsorships if you sign up for one of these sponsorship within 1 month following MMS 2009. Stop by booth #633 for more information.
Rod Trent
myITforum.com, Inc.
1E Gold Speaker Presentation – : Tuesday 28 April, 2009
Attend this session to gain a technical insight into all the latest features of NightWatchman and 1E WakeUp for power management and proactive computer health. Review new Nomad enhancements which minimize the bandwidth used for software and OS distributions and learn how to achieve lowest cost Windows 7 deployment. In addition, be one of the first to hear about 1E's new solution for server energy management.
Location: Room Veronese 2405, Time: 11.45am – 1.00pm
© 2009 myITforum.com, Inc.
myITforum.com MMS 2009 Newsletter
How to hook-up with myITforum.com at MMS 2009 – and your official myITforum.com Party invitation
Back in Vegas, baby!
Microsoft Management Summit 2009
Party invitation and information is at the bottom of this column
MMS is the annual gathering together of the System Center community. This year marks the 10th year of MMS and there are things planned for this momentous event that you just cannot miss! And — of course, we're back in Vegas (grab a copy of the map of the Vegas Strip)
myITforum.com and MMS have grown up together. It's really hard to say where the conference would be today without this community and how our partnerships melded together.
Read about the history of MMS
But, even more, it's also really hard to say what the systems management landscape at Microsoft would look like without this community. You all should be very proud of yourselves. By coming together and voicing your concerns, desires, solutions, rants, and kudos, through myITforum.com you've enabled Microsoft to develop some highly effective solutions for managing environments.
Did you know that myITforum.com turned 10 years old this year??? That's simply amazing. How many other communities can say they have been together for that long, and maintained their importance and relevance?
As always, I want to thank Microsoft for listening in and supporting this community for so long.
But, this year marks the official announcement of our Community "Partners". For a long time, folks have just assumed that Microsoft sponsors and promotes myITforum.com and that's how we have survived for so long. That's simply not true. While we appreciate the type of support that Microsoft gives us, it doesn't pay the bills. Find out who actually invests in the myITforum.com vision and who is benefitting from our experience and reach by visiting our new Partners page:
Cool things you may want
MMS 2009 Agenda
MMS 2009 grid view of all the Breakout and ILL schedules – 11" x 17"
MMS 2009 grid view of all the Breakout and ILL schedules (Day on a Page)
MMS 2009 Session Details PDF
The Premiere MMS Party – your official invitation
Date: Wednesday, April 30, 2009
Remembering MMS 2008 – Looking toward MMS 2009
Here's a tidbit of a memory captured and compiled on YouTube. Remember the myITforum.com booth stampede?
Look for a special MMS-edition newsletter in the next few days, detailing all the myITforum.com happenings at MMS 2009. From the annual party, to sessions, to book signings, to booth give-aways…keep your eyes peeled for the newsletter.
myITforum.com offers new sponsorship levels and a full media kit for download
As a vendor who wants to get noticed by the System Center community, do you wonder how to interact on a daily basis with customers and potential customers? Wonder no more!
Today, myITforum.com has unleashed new Partnership levels, featuring tiered levels designed to work with marketing budgets and your needs and requirements.
In this day and age, it's tough to know if your message is being heard. myITforum.com makes it easy to understand and simple to know through our completely interactive service. Advertising on the web has become a daunting task. Some web sites offer banner placement, and then you are left wondering (hoping) if someone even sees your message. myITforum.com takes your message and blasts it out to a highly targeted group of customers and potential customers who actually WANT to hear your message. That group, by the way, includes close to 60,000 people in varying company sizes, different management levels, and a huge number of countries. And, not only do we work within the System Center community to get your message out, but one email or one phone call gets your message out to the world through our direct interaction in today's social media outlets.
Stop worrying about how to get the best return for you marketing dollars, jump out here and download the Full Media Kit to get started:
http://www.myitforum.com/sponsors/sponsorship.asp
Let us take care of your marketing and get back to building your business.
If you are attending MMS 2009, drop by booth 633 and say "hello".
All inquiries can be lodged to: marketing@myITforum.com
Steve Pruitt – farewell to a friend
I'm saddened today by last night's passing of a good friend and a long-time myITforum member and contributor. Steve Pruitt's fingerprints are all over myITforum.com in the form of helpful posts, extremely useful articles on patching, entire Wiki sections on OSD and patching, posts to the SMS email list and more.
If there's one good thing about the Internet, it's this: contributions live out longer than all of us. And, because of that, the memory of Steve will be ever-present. Steve fought a serious disease for a long time even though he had it licked for some time. If you go back through the pages and pages of information that Steve freely, selflessly shared, you'll never find one hint of sadness, one iota of ego, or a single instance where he went outside the bounds of professionalism. Steve was a fighter, and from my estimation a good man. His contributions will remain forever, and his legacy is a good one.
I think it's extremely fitting that we remember Steve based on his myITforum.com award at MMS 2008. Steve proved his excellence over and over, and he will continue to be a lasting contributor in our hearts and minds.
Previous page Page 1 Page 2 Page 3 Page 4 … Page 8 Next page
How to Deploy to Deploy Office 2010 (Beta) using System Center Essentials 2010
Microsoft Outlook 2010 : Issue with message sizes in Outlook 2010 Beta and "message clipped" in GMail and BBerry, etc.
Data Protection Manager video info – courtesy of the myITforum.com site search…
Take a tour of the upcoming new myITforum.com Front page


text:
\section{Introduction}
\label{intro}

In recent years, the field of deep learning has seen a marked revival and lead to remarkable progress in machine learning \cite{goodfellow2016deep}. However, there remain many open problems concerning its theoretical foundations.
Recent progress has been driven in large part by the development of neural tangent kernel (NTK) theory \cite{lee2017deep,jacot2018neural,lee2019wide,https://doi.org/10.48550/arxiv.1910.00019,arora2019exact,sohl2020infinite,Roberts:2021fes}.
In particular, it has been proven, under certain conditions on the learning rate and weight initialization, that infinitely wide neural nets reduce to linear models.
To study more realistic models, which can learn new representations of the data, one can go beyond the strict infinite width and study perturbative, $1/\text{width}$ corrections \cite{DBLP:journals/corr/abs-1909-08156,DBLP:journals/corr/abs-1909-11304,yaida2020non}.

In this work we are interested in understanding models that display non-trivial, non-perturbative dynamics when trained with full-batch gradient descent.
Specifically, we will study the \emph{catapult mechanism} of \cite{lewkowycz2020large}, which occurs when we train the model with a learning rate $\eta$ that is larger than the maximal learning rate of the corresponding linearized, or infinite-width, model.
We will refer to learning rates above the na\"ive, linear stability threshold as ``super-critical".

The catapult phase is an intermediate phase between the so-called ``lazy phase" and ``divergent phase". In the lazy phase the learning rate is sufficiently small such that the model weights do not deviate significantly from their values at initialization.
In this regime one can use large width perturbation theory to make predictions.
In the ``divergent phase" the learning rate is taken to be large enough such that the loss diverges.
By contrast, in the catapult phase there is an initial transient period where the loss first increases rapidly, before decreasing and settling down to a small value.
In \cite{lewkowycz2020large} it was demonstrated in a variety of examples that the catapult phase leads to a smaller generalization loss than the lazy phase.

Overall, our understanding of the catapult phase, e.g. why it exists and how the neural net evolves in this phase, pales in comparison to our knowledge about neural nets in the lazy phase.
To gain a better understanding of the catapult phase, we will study two classes of toy models: quadratic models and homogenous MLPs with one hidden layer.
Quadratic models are minimal models for representation learning \cite{Roberts:2021fes}, as they are the simplest deviation from linear models whose NTK evolves under gradient descent.
We will derive sufficient, but not necessary, conditions for quadratic models to converge at large learning rates in terms of the maximal eigenvalue of the meta-feature function, which we will define later.
We will also derive sufficient conditions for homogenous MLPs to have a catapult phase, which will depend on the slope of the activation function in the positive/negative regions.
For both classes of models, the existence of the catapult phase is proven by showing that there exists a range of super-critical learning rates such that the weight norm $\bs{\theta}^2$ decreases whenever the loss becomes large.
We will provide evidence for all analytical results by numerically studying various quadratic models and two-layer MLPs.

More generally, the weight norm $\bs{\theta}^2$ is an interesting quantity to study because its behavior changes qualitatively as we change the learning rate \textit{inside} the catapult phase.
We observe that for sufficiently large learning rates, the weight norm in ReLU nets can increase significantly over the course of training.
We empirically observe that this increase in the weight norm does not hurt the generalization performance of ReLU MLPs.
We conjecture that these models still generalize well because the activation map of ReLU nets becomes increasingly sparse as we increase the learning rate.
We believe this result points to interesting future directions for the study of ReLU nets at large learning rates.

\textbf{Notations: }We use bold letters, e.g. $\bs{\theta}$, $\bs{\psi}$ to refer to vectors, matrices, etc. in weight space. We use the corresponding un-bolded letter when referring to individual indices, e.g. $\theta_{\mu}$ and $\psi_{\mu\nu}$ refer to specific components.
We also use un-bolded letters for one-dimensional quantities in weight space.
We use Greek letters from the middle of the alphabet to refer to indices in weight space, as above, and Greek letters from the beginning of the alphabet to refer to points in sample space, e.g. $(\bs{x}_\alpha,\bs{y}_\alpha)$ refers to the $\alpha^{\text{th}}$ data-point and its corresponding label.
We will keep the indices in sample space explicit, except when we consider toy examples with only one data-point, in which case we drop this index for compactness.
The symbols $\mathcal{N}$ and $\mathcal{U}$ refer to the normal and uniform distribution, respectively.

For vector-norms we use the notation $\bs{\theta}^2\equiv|\!|\bs{\theta}|\!|_{2}^{2}$,
where $|\!|\!\cdot\!|\!|_2$ is the $L_2$ norm. For matrices we use the norm induced by the vector $L_2$ norm, $|\!|\bs{\psi}|\!|\equiv |\!|\bs{\psi}|\!|_2$.
In addition, we use $\lambda_i(\bs{\psi})$ to refer to the $i^{\text{th}}$ eigenvalue. We use the $\lambda_i$ notation for matrices in both weight and sample space.
For the NTK we use $\lambda_{\text{max}}(H_{\alpha\beta})$ and $|\!|H_{\alpha\beta}|\!|$ interchangeably.\footnote{Since we reserve index-free notation for objects in weight space, $\lambda_{\text{max}}(H_{\alpha\beta})=|\!|H_{\alpha\beta}|\!|$ is the top eigenvalue of the full NTK matrix, and does not refer to the norm of individual components.}

Finally, we use the $t$-subscript to refer to quantities at step $t$ of gradient descent.
We will exclusively use the $0$ subscript to denote quantities at initialization, e.g. $\bs{\theta}_{0}$ is the vector $\bs{\theta}$ at initialization and not its $0^{\text{th}}$ component.

\section{Related works}
The catapult phase was first discovered in \cite{lewkowycz2020large}. There they proved analytically that the catapult phase exists in a linear, MLP with a single hidden layer. They also showed numerically that the catapult phase exists in a large class of wide neural nets and that a model trained in the catapult phase typically has a smaller generalization loss than the same model trained in the lazy phase. The catapult phase in models trained with logistic loss was studied in \cite{DBLP:journals/corr/abs-2011-12547}. The effect of large learning rates on matrix factorization was studied in \cite{https://doi.org/10.48550/arxiv.2110.03677}.

The work we present here is similar to the interesting analysis of \cite{Belkin_quadratic}.
There they proved that the catapult phase exists in the quadratic model that approximates a two-layer ReLU MLP.
They also demonstrated that the generalization loss of neural nets and their quadratic approximations are qualitatively similar in the catapult phase.
Our work differs from \cite{Belkin_quadratic} in a few key areas.
The first is, we will show that the catapult phase exists in a large class of quadratic models, while in \cite{Belkin_quadratic} they focused on the quadratic approximation of a two-layer ReLU MLP.
We also show that the catapult phase exists in general two-layer, homogenous MLPs, without making a quadratic approximation.
We believe it would be interesting to understand how their analysis for the quadratic approximation to the ReLU net could be used to study the class of quadratic models considered in this work.

Quadratic models have also been studied more generally in the literature.
In \cite{Roberts:2021fes} the quadratic model was proposed as a minimal model to study feature learning.
In \cite{Pennington_second_order} they proved that in quadratic models the maximal eigenvalue of the NTK displays the edge of stability (EOS) phenomena \cite{https://doi.org/10.48550/arxiv.2103.00065}.\footnote{In \cite{https://doi.org/10.48550/arxiv.2103.00065} the EOS was studied for the top eigenvalue of the Hessian. As explained in Appendix A of \cite{Pennington_second_order}, if the model is trained with MSE then the spectrum of the NTK approaches that of the Hessian as the model converges.}
Taylor expansions of neural nets were also studied in \cite{DBLP:journals/corr/abs-2002-04010}.

Finally, we also empirically study the sparsity of ReLU MLPs in the catapult phase as a function of the learning rate.
For a review on sparsity see \cite{https://doi.org/10.48550/arxiv.2102.00554}.
Two recent works \cite{https://doi.org/10.48550/arxiv.2210.05337} and \cite{https://doi.org/10.48550/arxiv.2210.06313} show that increasing the learning rate promotes sparsity, but to our knowledge they do not consider training the model in the catapult phase.
Large initial learning rates in ReLU nets have also been studied in \cite{https://doi.org/10.48550/arxiv.2212.07295}.

\section{Critical points in the learning rate}
\subsection{Model definitions}
\textbf{Quadratic models}. We start by defining the general quadratic model.
We take the data-points to be $d$-dimensional, $\bs{x}_\alpha\in \mathbb{R}^{d}$, where $\alpha=1,\ldots,D$, and the labels to be one-dimensional $y_\alpha\in\mathbb{R}$.

The general quadratic model (for a $1d$ output) is defined by:
\begin{align}
z(\bs{x}_{\alpha})=\boldsymbol{\theta}^T \boldsymbol{\phi}(\mathbf{x}_{\alpha})+\frac{\zeta}{2}\boldsymbol{\theta}^T\boldsymbol{\psi}(\mathbf{x}_{\alpha})\boldsymbol{\theta},
\label{eq:quad_vector}
 \end{align}
where $z(\bs{x})\in\mathbb{R}$, $\boldsymbol{\theta}\in\mathbb{R}^{n}$, $\bs{\phi}(\mathbf{x})\in\mathbb{R}^{n}$, $\boldsymbol{\psi}(\mathbf{x})\in\mathbb{R}^{n \times n}$, and $\zeta\in\mathbb{R}$.
The weights are drawn from a normal distribution with unit variance, $\bs{\theta}\sim \mathcal{N}(0,\mathbb{I}_{n\times n})$.
The functions $\bs{\phi}$ and $\bs{\psi}$ are part of the definition of the model and are not trainable.
We will refer to $\bs{\phi}$ and $\bs{\psi}$ as the feature and meta-feature functions, respectively.

The parameter $\zeta$ is in principle arbitrary and measures the deviation of \eqref{eq:quad_vector} from linearity.
When $\zeta=0$ the quadratic model \eqref{eq:quad_vector} reduces to a linear model, which has static features.
When $\zeta\neq0$ the above model can exhibit non-trivial representation learning.
In this work we take $0<\zeta\ll1$, but we will still see large deviations from linearity when the learning rate $\eta$ is taken to be sufficiently large.

We train the model to minimize the mean-squared error (MSE):
\begin{align}
L=\frac{1}{2D}\sum\limits_{\alpha=1}^{D}\epsilon_{\alpha}^2=\frac{1}{2D}\sum\limits_{\alpha=1}^{D}(z_{\alpha}-y_{\alpha})^2,
\end{align}
where the error term is $\epsilon_{\alpha}=z_{\alpha}-y_{\alpha}$.
Throughout this work we will assume $y_\alpha=O(1)$.
Under gradient descent the parameters evolve as:
\begin{align}
\bs{\theta}_{t+1}&=\bs{\theta}_{t}-\frac{\eta}{D}\frac{\partial L_t}{\partial \bs{\theta}_t}
\nonumber
\\
& = \bs{\theta}_t-\frac{\eta}{D}\sum\limits_{\alpha=1}^D\epsilon_{\alpha,t}\bigg(\bs{\phi}(\bs{x}_\alpha)+\zeta\bs{\psi}(\bs{x}_\alpha)\bs{\theta}_t\bigg),
\end{align}
where $\eta$ is the learning rate and we explicitly introduced the time-dependence.

One quantity of particular interest is the Neural Tangent Kernel (NTK), $H_{\alpha\beta}$, which is defined by:
\begin{align}
H_{\alpha\beta}\equiv H(\mathbf{x}_{\alpha},\mathbf{x}_{\beta})&=\frac{1}{D}\sum\limits_{\mu=1}^{n}\frac{\partial z(\mathbf{x}_{\alpha})}{\partial \theta_{\mu}}\frac{\partial z(\mathbf{x}_{\beta})}{\partial \theta_{\mu}}.
\end{align}
In vector notation $H_{\alpha\beta}=D^{-1}(\partial_{\bs{\theta}}z(x_{\alpha}))^T\partial_{\bs{\theta}}z(x_{\beta})$.
In the quadratic model the NTK is given by:
\begin{align}
H_{\alpha\beta}&=\frac{1}{D}\sum\limits_{\mu=1}^n\phi^{E}_{\mu}(\mathbf{x}_\alpha,\bs{\theta})\phi^E_{\mu}(\mathbf{x}_\beta,\bs{\theta}),
\\
\phi^E_{\mu}(\mathbf{x}_\alpha,\bs{\theta})&=\phi_{\mu}(\mathbf{x}_\alpha)+\zeta\sum\limits_{\nu=1}^n\psi_{\mu\nu}(\mathbf{x}_{\alpha})\theta_{\nu},
\end{align}
where $\bs{\phi}^E(\mathbf{x}_\alpha,\bs{\theta})$ are the effective feature functions of the quadratic model.
When $\zeta\neq0$ they are $\theta$-dependent and evolve under gradient descent, while when $\zeta=0$ they reduce to the static feature functions $\bs{\phi}$.
Correspondingly, the NTK only evolves when $\zeta\neq0$.

We study two classes of quadratic models, the \textit{pure quadratic model} which is defined to have $\bs{\phi}=0$ and the \textit{quadratic model with bias} which is defined by the condition that $\bs{\psi}\bs{\phi}=0$, i.e. that $\bs{\phi}$ is an eigenvector of $\bs{\psi}$ with eigenvalue 0.

\textbf{Homogenous Nets}. The quadratic models capture some aspects of representation learning in finite-width neural nets. Unfortunately, the quadratic approximation of neural nets, i.e. simply taking the output function $z(\bs{x}_{\alpha},\bs{\theta})$ and Taylor expanding $\bs{\theta}$ around its value at initialization, breaks down when the weights $\bs{\theta}$ evolve significantly. When this happens we need to keep higher-order terms in the Taylor expansion to get a reasonable approximation to the full neural net.\footnote{The breakdown of the quadratic approximation is discussed in \cite{Roberts:2021fes,Pennington_second_order,Belkin_quadratic}.}

For this reason, we will also consider a more realistic set of toy models: neural nets with a single hidden layer and a scale-invariant activation function.
These also scale quadratically in the weights, but will not generically be differentiable.
They can therefore be thought of as \textit{generalized quadratic models}.
We will see how the methods which proved essential in studying quadratic models can be carried over to homogenous nets with one hidden layer.

We study the following MLP:
\begin{align}
z(x)=\frac{1}{\sqrt{n}}\bs{v}^T\sigma\left(\bs{u}x\right)=\frac{1}{\sqrt{n}}\sum\limits_{\mu=1}^{n}v_\mu\sigma(u_\mu x), \label{eq:generic_scale_invariant_z}
\end{align}
where $x\in \mathbb{R}$, $\bs{u}\in \mathbb{R}^{n}$, $\bs{v}\in \mathbb{R}^{n}$.
The activation function is:
\begin{align}
\sigma(x)=a_+x\hspace{.05cm}\mathbbm{1}_{x\geq0}+a_{-}x\hspace{.05cm}\mathbbm{1}_{x\leq0},\label{eq:sigma_act_definition}
\end{align}
where $\mathbbm{1}$ is the indicator function.
We assume that $0\leq a_-\leq a_+$.
The ReLU function corresponds to $(a_{-},a_+)=(0,1)$.

The NTK in this model is:\footnote{The derivative of the activation function is not defined at the origin, but for definiteness we can define:
\begin{align}
\sigma'(0)\equiv\frac{1}{2}(a_++a_-).
\end{align}}
\begin{align}
H&=\frac{1}{n}\left(\sigma(\bs{u}x)^2+\left(\bs{v}\circ \sigma'(\bs{u}x)\right)^2\right)\nonumber\\
&=\frac{1}{n}\sum\limits_{\mu=1}^{n}(\sigma^2(u_\mu x)+(v_\mu\sigma'(u_\mu x))^{2}),\label{eq:NTK_generic_scale}
\end{align}
where $\circ$ is the Hadamard product.
Finally, we need that the weight norm is simply given by $\bs{\theta}^2=\bs{u}^2+\bs{v}^2$.

\begin{table*}[!htbp]
\caption{Rigorous guarantees for the existence of the catapult phase in different models trained on the toy dataset $(x,y)=(1,0)$.
 }
  \centering
  \begin{tabular}{@{}ccccc@{}}
    \toprule
    Model  & $z$  & $H$    & Derived catapult phase range for $\eta$\\
    \midrule
    Pure quadratic & $\frac{\zeta}{2}\bs{\theta}^T\bs{\psi}\bs{\theta}$ & $\zeta^2\bs{\theta}^T\bs{\psi}^2\bs{\theta}$ & $(\frac{2}{H_{0}},\frac{4}{\zeta^2\bs{\theta}^2_{0}\lambda_{\text{max}}(\bs{\psi}^2)})$
    \\[13pt]
    Quadratic with biases & $\bs{\phi}^T\bs{\theta}+\frac{\zeta}{2}\bs{\theta}^T\bs{\psi}\bs{\theta}, \hspace{.2cm} \bs{\psi}\bs{\phi}=0$ & $\bs{\phi}^2+\zeta^2\bs{\theta}^T\bs{\psi}^2\bs{\theta}$ & $(\frac{2}{H_{0}},\frac{4}{2\bs{\phi}^{2}+\zeta^2\lambda_{\text{max}}(\bs{\psi}^2)\left(\bs{\theta}^{2}_{0}+\frac{(\bs{\phi}^T\bs{\theta}_{0})^2}{\bs{\phi}^2}\right)})$
    \\[13pt]
    Homogenous nets & $ \begin{matrix}
   \frac{1}{\sqrt{n}}\bs{v}^T\sigma\left(\bs{u}x\right)
      \\
   \sigma=a_+x\hspace{.05cm}\mathbbm{1}_{x\geq0}+a_{-}x\hspace{.05cm}\mathbbm{1}_{x\leq0}\\
   a_->0  \\
\end{matrix}$ & $\frac{1}{n}\left(\sigma(\bs{u}x)^2+\left(\bs{v}\circ \sigma'(\bs{u}x)\right)^2\right)$ & $(\frac{2}{H_0},\frac{4n}{a_+^2(\bs{u}_0^2+\bs{v}_0^2)})$
\\[20pt]
    ReLu nets & $ \begin{matrix}
   \frac{1}{\sqrt{n}}\bs{v}^T\sigma\left(\bs{u}x\right)   \\
   \sigma=a_+x\hspace{.05cm}\mathbbm{1}_{x\geq0}+a_{-}x\hspace{.05cm}\mathbbm{1}_{x\leq0}   \\

   (a_{-},a_+)=(0,1)   \\
\end{matrix}$ & $\frac{1}{n}\left(\sigma(\bs{u}x)^2+\left(\bs{v}\circ \sigma'(\bs{u}x)\right)^2\right)$ & $(\frac{2}{{{H}_{0}}},\frac{4}{{{H}_{0}}})$
\\
    \bottomrule
  \end{tabular}
\label{catares}
\end{table*}

\subsection{Summary of Results}
Here we will summarize the main results.
Table \ref{catares} gives the primary analytic results of our paper, where we summarize our sufficiency conditions for the existence of the catapult phase using a toy dataset.
The toy dataset is defined to have a single, one-dimensional datapoint, $x=1$, with label $y=0$.
The proofs are given in Appendix \ref{app:single_datapoint}.
Although this dataset is too simple to model realistic datasets, it does give qualitatively good insight into the dynamics of these models at large learning rates.
The generalization of these analytic results to more generic datasets is given in Appendix \ref{app:multiple_datapoints}.
Below, we will briefly comment on the results derived.

\begin{itemize}
\item Our results in Table \ref{catares} are rigorously derived and significantly extend the results of \cite{lewkowycz2020large,Belkin_quadratic}.
These conditions are in general sufficient, but not necessary, for the existence of the catapult phase.
Our conditions are sufficient because they are derived by imposing that the weight norm $\bs{\theta}_t^2$ decays monotonically during training.
In quadratic models and homogenous MLPs, the loss can only diverge if some of the weights $\theta_{\mu}$ diverge. Therefore, if $\bs{\theta}^2_t$ is bounded the loss must be finite.
On the other hand, our conditions are not necessary because imposing that $\bs{\theta}_t^2$ monotonically decreases is stronger than only requiring that $\bs{\theta}_{t}^2$ is finite.
We find numerically that the catapult phase generically does exist beyond the range we have derived.

\item Given that our conditions were derived by requiring that the weight norm decreases monotonically under gradient descent, we can restate our results as follows: large learning rates which satisfy the conditions listed in Table \ref{catares} lead to an implicit $L_2$ regularization of the weight norm $\bs{\theta}_t^2$.
This result, along with the fact that the catapult phase lands on flatter minima \cite{DBLP:journals/corr/KeskarMNST16,lewkowycz2020large},\footnote{In the two-layer linear net with $1d$ input, the NTK is proportional to the weight norm \cite{lewkowycz2020large}, so a decrease in the weight norm implies that the minima is flatter.} may help explain why models in the catapult phase generalize better than models in the lazy phase.\footnote{In general however, flatness is not necessarily correlated with better generalization behavior \cite{https://doi.org/10.48550/arxiv.1703.04933,https://doi.org/10.48550/arxiv.2206.10654}.}

\item The analysis of ReLU nets is more subtle than the analysis of general homogenous nets.
If we take our bounds for homogenous nets and set $a_-=0$ and $a_+=1$, the allowed range for $\eta$ shrinks to zero size.\footnote{To see this from the third row of Table \ref{catares}, we use that in the ReLU net with one datapoint on average only half of the weights at $t=0$ contribute to the NTK. Therefore, at $t=0$ the weight norm is twice the NTK on average and the allowed range disappears.}
However, by using special properties of the ReLU function, we can use a different argument to show that a two-layer ReLU net with one, $1d$ datapoint converges for $\eta H_0\leq 4$. This agrees with the bound found in \cite{Belkin_quadratic} for the quadratic approximation of the same ReLU net with one datapoint of generic dimension.

On the other hand, in practice we know that the catapult phase for ReLU nets exists for larger learning rates, up to $\eta H_0 \lesssim 12$ \cite{lewkowycz2020large}.
This presents an interesting tension and suggests that the two-layer ReLU net may have characteristically different behaviors depending on the sign of $4-\eta H_0$.
We observe empirically that there is a qualitative difference: for $\eta H_0\lesssim 4$ the weight norm $\bs{\theta}^2$ decreases during training, while for $\eta H_0\gtrsim 4$ we find that $\bs{\theta}^2$ receives positive updates during training. In particular, if we take $\eta H_0$ large enough, then the final weight norm can be greater than its value at initialization, $\bs{\theta}^2_{\infty}>\bs{\theta}^2_{0}$.
Despite this increase in $\bs{\theta}^2$, models trained with $\eta H_0\gtrsim 4$ still generalize well \cite{lewkowycz2020large}. We conjecture that this is the case because the activation map of ReLU nets becomes sparse in the catapult phase.
\end{itemize}

\begin{figure*}[!ht]
\centering

\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_quadratic_max_epochs_1000_seed_13_hidden_dim_1000_act_identity_meta_type_linear_eval_dist_uniform_1_2_matrix_dist_normal_0_1/train_losses.pdf}
\caption{}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_quadratic_max_epochs_1000_seed_13_hidden_dim_1000_act_identity_meta_type_linear_eval_dist_uniform_1_2_matrix_dist_normal_0_1/weight_norm.pdf}
\caption{}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_quadratic_max_epochs_1000_seed_13_hidden_dim_1000_act_identity_meta_type_linear_eval_dist_uniform_1_2_matrix_dist_normal_0_1/lrNTK.pdf}
\caption{}
\end{subfigure}
\centering
\begin{subfigure}[b]{0.4\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_quadratic_max_epochs_1000_seed_13_hidden_dim_1000_act_identity_meta_type_linear_eval_dist_uniform_1_2_matrix_dist_normal_0_1/final_lrNTK.pdf}
\caption{}
\end{subfigure}
\hspace{.25in}
\begin{subfigure}[b]{0.4\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_quadratic_max_epochs_1000_seed_13_hidden_dim_1000_act_identity_meta_type_linear_eval_dist_uniform_1_2_matrix_dist_normal_0_1/final_weight_norm.pdf}
\caption{}
\end{subfigure}
\hfill
\caption{Results for the pure quadratic model with a linear meta-feature function trained on the toy dataset $(x,y)=(1,0)$. (a)-(c) give the evolution of the loss $L_t$, the weight norm $\bs{\theta}^2_t$, and $\eta H_t$ as a function of time, respectively. The different colors in (a)-(c) correspond to different choices of $\eta H_0$. In (a) we see that the loss experiences a transient phase of exponential growth before decreasing down to 0. In (b)-(c) we see that in the catapult phase both $\bs{\theta}_t^2$ and $\eta H_t$ decrease significantly. In (d)-(e) we show the final value of $\eta H_{t}$ and $\bs{\theta}_t^2$ as a function of $\eta H_0$.
The dashed vertical lines correspond to the upper bound in the first row of Table \ref{catares}.
We correctly predict that the model does have a catapult phase and empirically observe the model converges up to $\eta H_0=4$.}
\label{fig:linear_meta_toy_dataset_ev1_2}
\end{figure*}

\begin{figure*}[!ht]
\centering

\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_MLP_General_max_epochs_1000_seed_120_width_1024_leaky_relu_num_layers_1_slope_half/train_losses.pdf}
\caption{}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_MLP_General_max_epochs_1000_seed_120_width_1024_leaky_relu_num_layers_1_slope_half/weight_norm.pdf}
\caption{}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_MLP_General_max_epochs_1000_seed_120_width_1024_leaky_relu_num_layers_1_slope_half/lrNTK.pdf}
\caption{}
\end{subfigure}
\centering
\begin{subfigure}[b]{0.4\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_MLP_General_max_epochs_1000_seed_120_width_1024_leaky_relu_num_layers_1_slope_half/final_lrNTK.pdf}
\caption{}
\end{subfigure}
\hspace{.25in}
\begin{subfigure}[b]{0.4\textwidth}
\centering
\includegraphics[scale=.25]{figures/toy_MLP_General_max_epochs_1000_seed_120_width_1024_leaky_relu_num_layers_1_slope_half/final_weight_norm.pdf}
\caption{}
\label{fig:linear_scale_invariant_toy_dataset_half_final_weight}
\end{subfigure}
\hfill
\caption{Results for the two-layer, homogenous net with $a_+=1$ and $a_-=1/2$.
(a)-(c) give the evolution of the loss $L_t$, the weight norm $\bs{\theta}^2_t$, and $\eta H_t$ as a function of time, respectively.
The different colors in (a)-(c) correspond to different choices of $\eta H_0$.
(d) and (e) give the final values for $\eta H_t$ and $\bs{\theta}^2_t$ after the model has converged as a function of $\eta H_0$.
The hidden layer has width 1024. The vertical dashed line corresponds to the upper bound in the third row of Table \ref{catares}.
We again correctly predict that the model converges for super-critical learning rates to the left of this line.}
\label{fig:linear_scale_invariant_toy_dataset_half}
\end{figure*}

\subsection{Sketch of a Proof}
\label{ssec:sketch_proof}
Here we will sketch the proof for the results in Table \ref{catares} and leave the details for Appendix \ref{app:single_datapoint}.
In both two-layer, homogenous nets and the pure quadratic model, the output $z$ is a homogenous function of $\bs{\theta}$ with scaling weight two.\footnote{A slightly modified argument is needed for the quadratic model with bias, see Appendix \ref{app:single_datapoint}.}
Restoring the explicit weight dependence we have:
\begin{align}
z(\bs{x}_{\alpha},\lambda \bs{\theta})=\lambda^2z(\bs{x}_{\alpha},\bs{\theta}).\label{eq:homogeneity_weight_two}
\end{align}
This homogeneity property simplifies the update equation for $\bs{\theta}_{t}^2$ and implies that the NTK is also a homogenous function of $\bs{\theta}_t$ with weight two.
Both of these properties are important in deriving the results of Table \ref{catares}.

If we train a net with the property \eqref{eq:homogeneity_weight_two} on the toy dataset $(x,y)=(1,0)$ we find:
\begin{align}
\bs{\theta}_{t+1}^2=\bs{\theta}_{t}^2+\eta z_{t}^2(\eta H_{t}-4).\label{eq:weight_norm_update_simple}
\end{align}
From \eqref{eq:weight_norm_update_simple} we observe that the value $\eta H_t=4$ is special: if $\eta H_t<4$ then the weight norm decreases, $\bs{\theta}_{t+1}^2<\bs{\theta}_{t}^2$, while for $\eta H_t>4$ it increases.
Therefore, if we can guarantee that $\eta H_t<4$ for all $t$, then \eqref{eq:weight_norm_update_simple} implies that the weight norm monotonically decreases.
This condition is sufficient, but not necessary, to ensure the loss does not diverge.
It is also clear that imposing $\eta H_0<4$ does not guarantee convergence: it ensures that the first step of gradient descent decreases $\bs{\theta}^2_t$, but $H_t$ can increase during training and cause the model to diverge.

Therefore, to ensure convergence, we want to bound the NTK $H_t$ throughout training.
Specifically, we will bound $H_t$ in terms of the weight norm $\bs{\theta}_t^2$:
\begin{align}
H_{t}\leq C \bs{\theta}_{t}^2,\label{eq:bound_H_theta}
\end{align}
where $C$ is some constant, positive number.\footnote{In principle, one can also consider bounding the NTK in terms of a positive, monotonic function of $\bs{\theta}_{t}^2$, but we will find the above bound sufficient for the models we study.}
We will show that the bound \eqref{eq:bound_H_theta} holds in the pure quadratic model and for two-layer, homogenous MLPs in Appendix \ref{app:single_datapoint}.
The bound \eqref{eq:bound_H_theta} allows us to ensure that $H_t$ cannot become too large during training and cause the weight norm to get positive updates.
Here it is important that the homogeneity property \eqref{eq:homogeneity_weight_two} implies $H_t$ also has scaling weight two in $\bs{\theta}_t$ in order that both sides of \eqref{eq:bound_H_theta} have the same scaling in $\bs{\theta}_t$.

Finally, if the inequality \eqref{eq:bound_H_theta} holds, then $\bs{\theta}_{t}^2$ is a monotonically decreasing quantity when:
\begin{align}
\eta < \frac{4}{C \bs{\theta}_{t=0}^2}.\label{eq:general_bound_eta}
\end{align}
The argument is simple, if \eqref{eq:general_bound_eta} holds then the first step of gradient descent decreases the weight norm:
\begin{align}
\bs{\theta}_{t=1}^2-\bs{\theta}_{t=0}^2&=\eta z_{t=0}^2(\eta H_{t=0}-4)
\nonumber
\\
& \leq \eta z_{t=0}^2(\eta C \bs{\theta}_{t=0}^2-4)<0.
\label{eq:first_step}
\end{align}
To get the first inequality we plugged in \eqref{eq:bound_H_theta} and to get the second inequality we used \eqref{eq:general_bound_eta}.

Now after a second step of gradient descent we have:
\begin{align}
\bs{\theta}_{t=2}^2-\bs{\theta}_{t=1}^2&=\eta z_{t=1}^2(\eta H_{t=1}-4)
\nonumber
\\
& \leq \eta z_{t=1}^2(\eta C \bs{\theta}_{t=1}^2-4)
\nonumber
\\
& < \eta z_{t=1}^2(\eta C \bs{\theta}_{t=0}^2-4)<0.
\end{align}
Once again, to get the first inequality we used \eqref{eq:bound_H_theta}. To get the second inequality we used \eqref{eq:first_step}, or that the first step decreased $\bs{\theta}^2_t$.
The final inequality follows from condition \eqref{eq:general_bound_eta}.
It now clear that we can extend this argument to all $t$ using an inductive proof, which is given in Appendix \ref{app:single_datapoint}.

\section{Experiments}\label{exp}
Here we will present experimental results for quadratic models and two-layer, homogenous nets.
Further details and experiments, including extensions to more generic quadratic models and datasets, can be found in Appendix \ref{app:more_experiments}.
\subsection{Linear Meta-Feature Function}
In general, the meta-feature function $\bs{\psi}$ is an arbitrary function of $\bs{x}$. In this section we study the pure quadratic model with a linear, meta-feature function:\begin{eqnarray}\psi^{\text{lin}}_{\mu\nu}(\bs{x}_\alpha)=\sum\limits_{i=1}^{d}x_{\alpha,i}W^{i}_{\mu\nu}.
\end{eqnarray}
This is arguably the simplest class of meta-feature functions to consider.
The tensor $W^{i}_{\mu\nu}$ is symmetric under $\mu\leftrightarrow \nu$ and can be diagonalized for each $i$:
\begin{align}
W^{i}_{\mu\nu}=\sum\limits_{\sigma=1}^n\lambda^{i}_{\sigma}q_{\sigma \mu}^{i}q_{\sigma \nu}^{i}.
\label{eq:W_definition}
\end{align}
Here for each, fixed $i$ the matrix $q^i_{\sigma\mu}$ is an orthogonal matrix.

We will train the pure quadratic model on the toy dataset $(x,y)=(1,0)$.\footnote{In Appendix \ref{app:exp_quad} we will consider experiments for both the pure quadratic model and the quadratic model with bias trained on random datasets and teacher-student set-ups.}
We take the number of weights to be $n=1000$ and set $\zeta^2=2/n$. For each $i$ we take $q^i_{\mu\nu}$ to be a random orthogonal matrix. In addition, for each $i$ we split the eigenvalues $\lambda^i_\sigma$ into two sets of identical size, corresponding to the positive and negative eigenvalues. We draw the positive eigenvalues from $\mathcal{U}([1,2])$ and take the negative eigenvalues to be exactly $-1$ times the positive eigenvalues.
We impose that the eigenvalues come in positive/negative pairs so that $\mathbb{E}[z_{0}]=0$.
We also choose the range $[1,2]$ because the two-layer, linear MLP with one datapoint, $x=1$, corresponds to a quadratic model with eigenvalues $\pm 1$ \cite{Belkin_quadratic,Pennington_second_order}. We can then think of this quadratic model as a simple deformation of the linear MLP.
The results for this set-up are shown in Figure \ref{fig:linear_meta_toy_dataset_ev1_2}.
We see that this quadratic model undergoes catapult dynamics and our bounds correctly predict the model does converge for a finite window above $\eta H_0=2$.

\begin{figure*}[!ht]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/train_losses.pdf}
\caption{}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/weight_norm.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_weight_norm}

\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/lrNTK.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_NTK}

\end{subfigure}
\centering
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/final_lrNTK.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_final_NTK}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/final_weight_norm.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_final_weight}

\end{subfigure}
\hfill
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/spars.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_sparsity}
\end{subfigure}
\\
\hfill
\begin{subfigure}[t]{\textwidth}
\centering
\includegraphics[scale=.25]{figures/MNIST01_small_MLP_train_batch_128_stop_criterion_10m8_seed_123_input_size_784_width_1024_relu_num_layers_1/gen_loss.pdf}
\caption{}
\label{fig:MNIST_011_ReLU_one_hidden_layer_gen_loss}
\end{subfigure}
\caption{Results for the two-layer ReLU net trained on a two-class version of MNIST. The hidden layer has width 1024.
(a)-(c) give the early-time evolution of the loss, the weight norm, and $\eta\lambda_{\text{max}}(H_{\alpha\beta,t})$, respectively. The different lines in (a)-(c) correspond to different choices of $\eta\lambda_{\text{max}}(H_{\alpha\beta,0})$. (d)-(g) give the final values for $\eta \lambda_{\text{max}}(H_{\alpha\beta,t})$, $\bs{\theta}^2_t$, the sparsity of the activation map, and the generalization gap, respectively, for the converged model as a function of $\eta\lambda_{\text{max}}(H_{\alpha\beta,0})$. From (f) we observe that increasing the learning rate in the catapult phase promotes sparsity.
}
\label{fig:MNIST_011_ReLU_one_hidden_layer}
\end{figure*}

\subsection{Generic Homogenous Net}
We also perform a similar experiment for homogenous nets with one hidden layer of width 1024. To be concrete, we consider a net with $a_+=1$ and $a_-=1/2$.
We train this model on the toy dataset $(x,y)=(1,0)$.
The results are shown in Figure \ref{fig:linear_scale_invariant_toy_dataset_half}. When $\eta H_0\lesssim 4$, the results are qualitatively the same as for the pure quadratic model.
However, one new feature is this model converges for larger learning rates.
In particular, for $\eta H_0=4.5$ we see that $\bs{\theta}_{t}^2$ initially grows during the catapult phase, before decreasing to a small value.
This growth in $\bs{\theta}^2_t$ occurs because the product $\eta H_t>4$ when $z_{t}=O( \sqrt{n})$.
From the update equation for $\bs{\theta}^2_t$ \eqref{eq:weight_norm_update_simple}, this means that $\bs{\theta}^2_t$ receives a large positive correction during training. However, eventually $\eta H_t$ is pushed below 4 and the model converges.

This result is consistent with our analytic results summarized in Table \ref{catares}.
When $\eta$ obeys the bounds summarized there, the weight norm $\bs{\theta}_{t}^2$ must decrease monotonically.
Since $\eta H_0=4.5$ exceeds our bounds, there is no guarantee $\bs{\theta}_{t}^2$ must decrease.
We can also observe in Figure \ref{fig:linear_scale_invariant_toy_dataset_half_final_weight} that there is a transition in $\bs{\theta}^2_{\infty}/\bs{\theta}^2_{0}$ when $\eta H_0\approx 4$. When $2\leq \eta H_0 \lesssim4$ the final weight norm is smaller than its value at initialization. As we approach $\eta H_0\rightarrow 4$ this trend reverses and there is a peak at $\eta H_0=4$, before $\bs{\theta}^2_t$ decreases again.
To make this change in the evolution of $\bs{\theta}_t^2$ as a function of $\eta$ more pronounced, we will study ReLU nets.

\subsection{ReLU Net}
In this section we will study the behavior of fully-connected ReLU nets trained on a two-class version of MNIST.
We will train the model to distinguish images of ``0" and ``1", which are assigned labels $-1$ and $1$, respectively. The training set has size 128 and the test set has size 2115.
Our analytic bounds for ReLU nets is restricted to $1d$ input data, so here the results will be purely empirical.\footnote{A comparison between analytics and numerics for one-dimensional data and ReLU nets will be given in Appendix \ref{app:scale_inv_MLP_exp}.}

We train a two-layer net with width 1024 using full-batch gradient descent.
The model is trained for each learning rate until the change in the training loss is $<10^{-8}$. The results are shown in Figure \ref{fig:MNIST_011_ReLU_one_hidden_layer}. The new feature, in comparison to the previous experiments, is that the final weight norm $\bs{\theta}_{\infty}^2$ can be greater than the value at initialization, see figures \ref{fig:MNIST_011_ReLU_one_hidden_layer_weight_norm} and \ref{fig:MNIST_011_ReLU_one_hidden_layer_final_weight}.
Despite the increase in $\bs{\theta}^2_t$ the model still generalizes well, see Figure \ref{fig:MNIST_011_ReLU_one_hidden_layer_gen_loss}.
We conjecture that the model is able to generalize well, despite the increase in $\bs{\theta}^{2}_t$, because the activation map becomes sparse in the catapult phase, see Figure \ref{fig:MNIST_011_ReLU_one_hidden_layer_sparsity}.
The sparsity of a given layer is the fraction of nodes in that layer which are zero after acting with the ReLU function for a single input.
To get Figure \ref{fig:MNIST_011_ReLU_one_hidden_layer_sparsity} we averaged the sparsity of the activation map over all inputs.

Finally, we can note that the final value of $\eta \lambda_{\text{max}}(H_{\alpha\beta,t})$, which measures the sharpness of the final minima, first decreases in the catapult phase before increasing again to the edge of stability, $\eta \lambda_{\text{max}}(H_t)=2$, see figure \ref{fig:MNIST_011_ReLU_one_hidden_layer_final_NTK}.
This is consistent with existing work on the edge of stability phenomena for the top eigenvalue of the NTK \cite{https://doi.org/10.48550/arxiv.2103.00065,Pennington_second_order}.

\section{Conclusions}
In this work we have studied the properties of simple machine learning models, the quadratic model and two-layer, homogenous nets, for super-critical learning rates. We have proven that these models can converge beyond the stability threshold of the corresponding linearized model.
In the process, we have shown that training these models for super-critical learning rates can lead to an implicit $L_2$ regularization where the weight norm $\bs{\theta}^2$ decreases significantly during training.
We have also empirically studied ReLU nets for even larger learning rates, $\eta H_0\gtrsim 4$, and shown that training models in this range leads to an increase in both the sparsity and the weight norm. These results, both analytical and empirical, shed new light on the implicit biases of gradient descent at large learning rates.

There are many interesting open questions to consider. One is to understand how to weaken our bounds, which in general can miss large portions of the catapult regime. The reason our bounds can be too strong is because they are designed to hold for all possible configurations of $\bs{\theta}$.
In particular, they hold when inequalities of the form \eqref{eq:bound_H_theta} are saturated.
To derive weaker bounds we need a better understanding on how the weights evolve on average for super-critical learning rates.
In addition, proving that ReLU nets can converge for $4\leq \eta H_0 \lesssim12$ remains an open question.

Another question is: why do models trained in the catapult phase generalize well?
Our empirical and analytic results align with the expectation that large learning rates can lead to implicit regularization \cite{DBLP:journals/corr/abs-1907-04595} which improves model performance.
It would be interesting to understand what features of the catapult mechanism lead to a small generalization error for more generic, wide neural nets.

\section*{Acknowledgements}
We thank Yasaman Bahri, Daniel Roberts, Yi-Hsien Du, and Han Zheng for discussions.
We additionally thank Daniel Roberts for comments on the draft.
DM is supported by the NSF grant PHY-2014071. JL is supported in part by International Business Machines (IBM) Quantum through the Chicago Quantum Exchange, and the Pritzker School of Molecular Engineering at the University of Chicago through AFOSR MURI (FA9550-21-1-0209).

redpajama_set_name: RedPajamaArXiv


(4) the effect of future climate change scenarios on hydrology.
hydrology of these northern regions.
Duration of Position: 1 year, with possible extension.
variability to post-docs and researchers at UCLA, Scripps and DRI/WRCC.
Please contact Professor Mark Schwartz, mwschwartz at ucdavis.edu. 2.
of climate databases to meet needs of the SWCSC partners and affiliates.
performance-based extensions possible. Posted: 6/21/11.
those interested to submit an abstract.
heterogeneity and hydrologic connectivity are not adequately considered.
Daniel Lee Peters, Ph.D., P.Geo.


HPCA is concerned to see a call for the legalising of euthanasia in South Africa. We are saddened to hear of Dr Sue Walter's diagnosis and offer her every assistance at this difficult time. However, we believe that pursuit of legalising euthanasia in South Africa is irresponsible. International discussion on the issue has resulted in the majority of countries rejecting euthanasia and assisted suicide and proposals to change laws designating these acts as illegal.
The recent debate in New Zealand included a submission from Dr Martin Sullivan, a lecturer in Disability Studies & Social Policy at Massey University, stating that "euthanasia and assisted suicide will place already vulnerable people – disabled people, elderly people, people with chronic conditions, sick people – in an even more vulnerable position when they are sick or near the end of life. We live in society which valorizes the 'able-body' and 'able-mind' to the extent that to be disabled in any way is to be highly devalued to the extent that it is better to be "dead than disabled." The 'right to die could become the 'duty to die' for these people.
In South Africa, Prof Daniel Ncayiyana, wrote an editorial in the South African Medical Journal in 2012, describing a "pervasive lack of an ethos of respect for human life" in South Africa. Together with severe constraints on health care facilities, this means that there is "a real risk of euthanasia becoming a substitute for proper care" in the South African situation. He also notes that "euthanasia seems to be a concern among a small, largely white elite in South Africa" and that legalizing "euthanasia in South Africa will potentially reflect the inequalities of the country 'benefitting' only the well-off while opening up the potential for perverse application among the poor and other vulnerable groups".
Calls for legalising euthanasia make the statement that there will be checks so that there is no abuse of a new law. However, crossing the line to allow euthanasia or assisted suicide violates the societal injunction against killing another person. Once the mindset has changed to accept this, additional reasons to allow euthanasia follow.
We recognise that significant emotional adjustment is required when facing a serious diagnosis. We have been concerned for some time by the isolated practice of some palliative care specialists and the loneliness of this position, without the support of a multi-disciplinary palliative care team. HPCA supports personal autonomy but we recognise that an individual's choice is not in isolation of other people, and there is a need to consider relational autonomy and the impact of individual autonomy on others.
It is always hard to make an adjustment from clinician to patient and to accept assistance but hospice in Johannesburg would be willing to assist both Dr Walter & Mr Harckwith with holistic palliative care at this painful and challenging time.
The Supreme Court of Appeal judgement overturning the Fabricius ruling in November 2016, made the point amongst many others that Adv Stranstham-Ford who applied for a ruling to allow euthanasia had benefitted from palliative care and towards the end of his life had expressed ambivalence about euthanasia for himself.


Rumor has it that HOA boards and managers can do anything they want because homeowners have, by buying an association controlled house, ceded all their rights.
The very existence of an HOA creates problems in neighborhoods that would never otherwise occur. The respect one once had for their neighbor, and the sense of community and pride people had for each other have been replaced by hate, discontent and a very adversarial relationship. So, what happens when the power vested in the few is stripped?
Joining us On The Commons this week are Maria and Sam Farran . Maria is an attorney and Sam is a businessman and between them they have a good understanding of both the law and how to run a successful business. But as homeowners in an HOA controlled development they knew they still had rights and they also understood that in order to have a successful business, customers must be treated with respect. Join us as we learn about their 4 year odyssey through the courts, their wins every time, and when the dust settled, what happened in their neighborhood.
Something really sinister seems to happen to people when they are put in a position of perceived power, especially if they are not emotionally and mentally mature enough to handle the responsibilities that go along with the position. We've all heard and read about the horror stories in mandatory membership residential associations but can that really be happening? I mean, this is America, right? People do have rights, don't they?
Joining us On The Commons is Ward Lucas. Ward is an award winning investigative journalist who came face to face with the beast that resides deep in the heart of every HOA. Shocked at first, he decided to use his talents as an investigative journalist to see if what he experienced was "an isolated incident". What he discovered prompted him to write Neighbors at War; The Creepy Case Against Your Homeowners Association. Join us as we take an all too quick peek into what is a must read for every American.
Ward also has a website NeighborsAtWar.com and you can email him at ward@neighborsAtWar.com and his book is available at Amazon.


When e-commerce and online shopping begins to take off in a certain country, transportation networks and logistics facilities benefit from new investment, ideas and products to meet the demands of a changing shopping environment. The supply chain adapts, new warehouses appear to store and dispatch the goods, and logistics service providers – both 3PL and specialist – look for new technologies to adapt to the changing behaviour of their customers.
The pattern is the same around the world. Take India for example – its 40% annual growth rate in online purchases means more and more companies are establishing dedicated e-commerce logistics divisions. Hong Kong is another market to prove the rule, with its excellent air links and connections to China attracting brands to base their B2C distribution there. Local logistics firms that can adapt to this demand are thriving – to take one example, logistics app firm Aftership is seeing a 20% monthly increase in online merchants using its delivery tracking services.
As Hong Kong's example shows, even small countries are rewiring their logistics sectors to meet the new demands of e-commerce. But what about the world's largest country, Russia?
First things first, Russia certainly is an exciting market for e-commerce growth. The total number of Russian internet users, already the highest in Europe, passed the 76 million mark in the first half of 2015, helping boost the country's e-commerce market to $14.5 billion. This figure is growing 15% year on year.
What's more, these already impressive numbers have potential to grow even faster. While Russia's 76 million internet users is a big enough target market, this is only 65% of the adult population. Compare this figure to other countries – Norway's internet penetration rate, for example, is 95% http://www.internetworldstats.com/top25.htm - and it becomes clear that Russia's e-commerce boom shows no signs of running out of steam. Finally, Russians are starting to trust online merchants to handle their transactions, even ones based abroad. 12 million Russians now shop in foreign online stores, according to Bloomberg News. This has led to a flurry of new launches in the country, with H&M becoming the latest retail giant to capitalise on Russia's online potential by opening an online store this October.
So what effect is this having on Russia's transport and logistics market? Russia's huge land mass, with large distances between population centres even in the European part of the country, has always posed a unique challenge for cargo owners and distributors. Now, as more and more Russians feel safe enough to buy online, transport and logistics in the country is looking forward to a huge amount of investment and new ideas.
One area seeing bigger spending is warehousing. Adapting to e-commerce and increased home deliveries means companies need the latest technologies to stay ahead of demand. Christopher van Riet, founder of Radius Group, estimated this year that Russian warehousing logistics will need $30 billion of investment in the long term to keep up. Van Riet is well qualified to make this prediction. Radius signed a deal this year to build a 100,000 square metre national distribution centre for Leroy Merlin near Moscow's Domodevodo Airport, the largest retail warehouse contract in the world in 2015.


Wahl Palm Shampoo Brush 4 out of 5 based on 20 ratings.
We wash our hair with it, brush.
their palm oil is coming from, said Mongabay.com's Butler. "Look at the ingredient lists.
Palm Shampoo Brush This tool ensures that your pet is completely pampered at bath time and simultaneously gives their coat a really deep clean, leaving them dirt free and fresh. Shampoo is stored in the brush and evenly dispensed through the rubber pump.
Wahl Professional Peanut Clipper/Trimmer #8655-200, Black – Great On-the-Go Trimmer for Barbers and Stylists – Powerful Rotary Motor.
Portable Head Scratchers – Hair Brushes Shampoo Beard Pocket Comb Scalp Massager Brush for Travel. by G.B.S. $9.98 $ 9 98 ($0.83/Count) FREE Shipping on eligible orders. 4.2.
Barkshire Fleece Corner Pet Sofa Xlarge Fabric Folding Dog Crate By Petplanet – Medium W46 X D60 X H53cm Details about Easy Folding Pet Crate Travel Carrier PetPlanet Portable Oxford polyester Fabric Danish Design Bobble Deluxe Slumber Bed Pewter 18in Give your dog the good life with this Danish Design bobble pewter deluxe slumber bed, perfect for naps by the fire.
Barbersalon.com – Professional barber & salon supplies super store. Choose from Thousands of Style & Color Combinations. Flawless Customer Service. One Stop Shopping for Professional Barber Supplies, Salon Supplies, Professional Products.
like Japonesque Solid Brush Cleaner, are basically a cross between a bar of soap and baby shampoo. They're generally formulated with a mix of oils (or, in Japonesque's case, goat's milk, olive oil, an.
How to Clean Your Makeup Brushes, According to the Experts – Here, a dermatologist and two makeup artists — including Rihanna's makeup artist Mylah Morales — weigh in on the best practices for cleaning your makeup brushes and sponges, followed by the best produ.
Today Wahl offers a complete line of facial hair trimmers that feature the latest in power through Lithium Ion and convenience, continuing the company's legacy of superior grooming for real guys.
Brushes & Combs. Sprays & Shampoos. Wipes. Nail Clippers. Parts & Accessories. How-To. Dog Care. Cat Care.
All Products ©2018 Wahl Clipper.
Wahl's Four In One is formulated with lavender and chamomile to clean, condition, detangle, and moisturize your dog. This all in one formula makes bath time easy for you and your dog while leaving your dog's coat clean, silky and smooth.
We didn't agree not to buy anything at all, of course; obviously we need things like food and toilet paper and shampoo. But w.
Jon's 24-Hour Product Diary: A Man Who Is Trying – When it came to products, skincare.
Before showering, I trim my face stubble with a Wahl All-in-One Rechargeable Trimmer.
Wahl Dog Grooming Brushes; Skip to page navigation. Filter (2) Wahl Dog Grooming Brushes. Shop by Price.
A Palm-sized brush with an ergonomic grip for removing loose hair, debris and tangles. Nylon ball pin bristles glide through the coat and are suitable for brush drying your pet.
Wahl Double Sided Brush Gentle, rounded tip bristles to.
Here's how she does it: Wet the brush with a little bit of water so the shampoo lathers up. Squeeze a generous amount of sham.
Ingredients Wahl's Oatmeal Shampoo contains oatmeal, coconut, lime, aloe and lemon verbena to soothe dry, itching, irritated skin. Healthy skin means a healthy coat and less shedding.


Real Deal Steel LLC, well known for their high quality USA-built 1955-57 Chevy and 1967-69 Camaro coupe bodies is proud to offer something special to early Henry enthusiasts - their 1940 Ford coupe steel body. These bodies are 100% assembled in their factory in Sanford, Florida USA. Real Deal begins with the highest quality steel structural components and assembles them with their unique (designed in-house) assembly fixture for a factory fit every time. All structural and skin assembly is done using water cooled 220V resistance spot welders – just like you would see in a much larger automotive assembly plant. Real Deal uses 25% more spot welds than original to give their bodies added strength. Complete bodies include top skin and drip rails, quarter panels, rear body panel, doors and deck lid – all fully assembled and fitted. Real Deal Steel pays particular attention to door and deck lid fit which ultimately reduces finish time for the end user. Bodies are available with stock or 5-1/2" recessed firewall configurations. The recessed firewall is ideal for Coyote and other late model V8 installs that require extra clearance. Real Deal works closely with Dennis Carpenter Restoration Parts who offers fenders as well as other body and trim items needed to complete a '40 coupe build. Real Deal Steel's 1940 bodies are officially licensed by Ford Motor Company.


text:
This is a past event. Registration is closed. View other AmCham Dubai events.
SUPPORTED EVENT.
State of American Business 2023
11am ET
Get ready for the State of American Business, the U.S. Chamber of Commerce's premier annual event this Thursday, January 12, 2023. U.S. Chamber President and CEO Suzanne P. Clark will discuss the State of American Business in 2023 and convene business leaders across industries to explore the strength and leadership of business during this complex time and its role in working with lawmakers to pass practical, meaningful, and lasting policies.
During this high-level virtual dialogue, business leaders will discuss the role of government in advancing smart policy and highlight the role of business-led innovation to chart a bright path forward for our nation and its people.
Please join Suzanne, fellow CEOs, and insightful speakers to explore the actionable policies that the government can and must get done to move our nation and the world forward.
CLICK HERE TO REGISTER.
11:00 AM - 12:00 PM (GMT-5)
AmCham Dubai

redpajama_set_name: RedPajamaCommonCrawl


Newborn Baby Boy MUSTACHE Little Man Set.
Includes Onesie Top " Little Man " Bodysuit, Long Pant, Matching Beanie Hat. Cap & Pant are covered with mustache designs.
Available in sized O/4 mo and 9/12mo.
Price includes all 3 pieces..


Yes, you can find other "tomato pie" recipes, and yes, they have some of the same ingredients, and yes, they're all savory with no sugar.
But none match Angeleita's tomato pie, seen above in its fully baked glory.
I know this because I'm married to her, and have seen her bake dozens of them over the years, changing the recipe to fit her unerring instinct for what tastes and textures work.
She has perfected the tomato pie but only makes it when fresh tomatoes, locally grown, are available, usually our own. For us it's a late summer, early fall dish, and compliments other harvest meals nicely.
If you can't find fresh, meaty, ripe tomatoes, don't bother.
I've also seen plenty of friends swoon when they finally taste the exotic warm mix of basil, fresh sliced tomatoes, Vidalia onions, and three cheeses mixed with grapeseed mayonnaise.
It takes a bit of time, since it involves two layers of four ingredients per layer. But it's more than worth it.
1. A Pillsbury pie nine-inch crust, the kind that comes in a box that you unfold to fit into a quiche dish. If you prefer to bake your own crust, great. Angeleita says that using the store-bought crust saves time and the Pillsbury version works well. Note: the "pie" needs to be deep enough to accommodate all these ingredients, so we use a deeper ceramic dish rather than a standard pie plate—technically a quiche dish, 9-10" across.
2. Fresh basil leaves, enough to cover the crust twice, in two separate layers.
3. Thinly sliced Vidalia (sweet) onions, enough to cover crust twice.
4. Thinly sliced fresh tomatoes, enough, etc. NOTE: Make sure that you drain the tomato slices for several minutes onto paper towels; you do not want a soggy tomato pie. Also, salt and pepper each slice.
5. Grapeseed mayonnaise. Do not use sweet versions of mayo; this will ruin the savory character of the pie. "Vegenaise" makes the kind we prefer.
6. Three kinds of cheese, all freshly grated. We prefer (1) sharp white cheddar, (2) regular mozzarella (not the kind that comes in a moist ball) and (3) smoky gouda—about eight ounces each. Experiment with jalapeno cheese, or whatever you prefer, except do not use cheeses that don't melt well, such as feta (goat) or parmesan. For freshness, grate immediately before using.
We decided to make two, since once you have all the ingredients, it isn't much trouble to double it. However, all the instructions and ingredients are for one pie.
First, bake the pie shell and let it cool completely. Do not put the basil leaves on a warm crust; they will turn black.
Second, place the first layer of basil leaves (washed and dried) in the bottom of the cool shell.
Third, place a layer of dried, thinly sliced fresh tomato slices over the basil leaves.
Fourth, place a layer of thinly sliced Vidalia onion slices over the tomato slices.
Fifth, place half of the grated cheeses mixed together over the onion slices. Note: NO mayo with this first layer.
Sixth, repeat the basil, tomato, and onion layers above, then for the final layer, dollop in about two tablespoons of the grapeseed mayonnaise with the second half of the three-cheese mix. Spread the final cheese mix in chunks over the now-completed pie.
Pre-heat oven to 350 degrees and bake until brown and bubbly—75 (or more) minutes, depending on your oven.
If you choose to store the pie, let it cool completely, cover well with foil and/or parchment paper to refrigerate, and when ready to eat, heat it again until bubbly.
This may take up to 45 minutes. Cover lightly with foil when reheating so the cheese doesn't burn; remove foil for ten minutes toward the end to make sure the reheated pie is bubbling hot.
If you follow this recipe carefully, I wouldn't be surprised if you too swoon at having created the perfect tomato pie. Let me know how it works, especially if you try variations with bacon, different cheeses, etc. And of course do comment with any suggestions or corrections to this recipe.
Believe me, this is a dish that causes paroxysms of pleasure from everyone who tries it--especially hot, right out of the oven, with fresh ingredients.


text:
About Civil War Monitor
Posted 7/15/2013 By Tom Huntington
On July 7 Major General George Gordon Meade left Gettysburg and traveled to Frederick, Maryland. He found the streets crowded with people eager to get a glimpse of him. The citizens treated him "like a lion," but he did not allow it to go to his head. After the botched opportunities of Antietam, George McClellan had written to his wife that he had fought a "masterpiece of war." Meade was cut from a different cloth. The papers, he said, were making too much of him. "I claim no extraordinary merit for this last battle, and would prefer waiting a little while to see what my career is to be before making any pretensions," he said. "I did and shall continue to do my duty to the best of my abilities, but knowing as I do that battles are often decided by accidents, and that no man of sense will say in advance what their result will be, I wish to be careful in not bragging before the right time."1
Meade had every intention of completing his work before Robert E. Lee and his army could cross the Potomac River to safety at Williamsport. As the Union cavalry harassed the retreating Confederates, the infantry made its tortuous way though driving rain and cloying mud on a roundabout flank march. The torrential rains transformed roads into quagmires that sucked the shoes off the feet of the soldiers still lucky enough to wear them. Even traveling on the National Road proved difficult. "Napoleon crossing the Alps will no longer be mentioned as the climax of heroic achievements. Sedgwick marching over the Catoctin Mountains has entirely eclipsed that," one soldier noted.2
On July 8 Meade made his headquarters in Middletown. With both Mother Nature and his supply situation taking a toll on his army, Meade began to worry about his ability to catch Lee before the rebels crossed the river. He wired Union General -in-Chief Henry Halleck. "Be assured I most earnestly desire to try the fortunes of war with the enemy on this side of the river, hoping through Providence and the bravery of my men to settle the question, but I should do wrong not to frankly tell you of the difficulties encountered," he warned. "I expect to find the enemy in a strong position, well covered with artillery, and I do not desire to imitate his example at Gettysburg, and assault a position where the chances were so greatly against success. I wish in advance to moderate the expectations of those who, in ignorance of the difficulties to be encountered, may expect too much. All that I can do under the circumstances I pledge this army to do."3
In Washington this no doubt conjured up memories of McClellan and his constant excuses and requests for resupply and reinforcements. Yet the comparison is not fair to Meade. McClellan waited for more than a month after the Battle of Antietam before he initiated a full pursuit of the enemy. Meade allowed only a day to elapse before he began his pursuit, after a three-day battle that had left his army seriously bloodied and his officer corps sadly diminished. Still, the telegraph wires between Halleck and Meade hummed with messages from Washington. "There is reliable information that the enemy is crossing at Williamsport," Halleck wired on July 8. "The opportunity to attack his divided forces should not be lost. The President is urgent and anxious that your army should move against him by forced marches."
Meade began to bristle. "My information as to the crossing of the enemy does not agree with that just received in your dispatch," he replied. "My army is and has been making forced marches, short of rations, and barefooted. One corps marched yesterday and last night over 30 miles. I take occasion to repeat that I will use my utmost efforts to push forward this army."
"Do not understand me as expressing any dissatisfaction," Halleck replied; "on the contrary, your army has done most nobly. I only wish to give you opinions formed from information received here."
On July 9 Meade made his headquarters at the Mountain House at Turner's Gap. The little town of Boonsboro sat at the base of South Mountain on the west side. If the Confederates held Boonsboro they could keep the Federals from streaming down South Mountain. Meade ordered Howard and the XI Corps and Sedgwick and the VI to hurry to Boonsboro, where John Buford's cavalry was opposed by Jeb Stuart's horsemen. On July 8 Buford's cavalry managed to force their opponents back toward the little town of Funkstown, but Stuart had accomplished his goal of standing between the Union and Lee's retreating army.
Halleck continued to telegraph jittery messages to Meade. "Do not be influenced by any dispatch from here against your own judgment," he wired on July 9. "Regard them as suggestions only." The next day Halleck seemed to be sharing Meade's caution. "I think it will be best for you to postpone a general battle till you can concentrate all your forces and get up your reserves and re-enforcements," he wired. "Beware of partial combats. Bring up and hurl upon the enemy all your forces, good and bad." The arriving troops, though, were not all that Meade could have desired. Back in Harrisburg, Darius Couch sent a force under cantankerous Brig. Gen. William "Baldy" Smith south to reinforce Meade. But these men were untested militia, and Meade had no intention of using them in situations where they could do more harm than good.
On July 12 Meade established his headquarters on Antietam Creek at a spot called the Devil's Backbone. He sent another wire to Halleck. He planned to attack the next day "unless something intervenes to prevent it." That night another council took place in Meade's small and crowded tent at Devil's Backbone. The XI Corps' Oliver O. Howard and Maj. Gen. James Wadsworth, who had taken over the I Corps from an ill John Newton, voted to attack. The rest of the corps commanders voted to hold off until the army could better investigate Lee's defenses. Perhaps remembering what had happened at Chancellorsville when Hooker overruled his corps commanders, Meade decided to defer to his generals' advice. He postponed his attack for a day.
The next day Meade wired Halleck about his plans. He received a terse message in reply. "You are strong enough to attack and defeat the enemy before he can effect a crossing," said the general in chief. "Act upon your own judgment and make your generals execute your orders. Call no council of war. It is proverbial that councils of war never fight. Re-enforcements are pushed on as rapidly as possible. Do not let the enemy escape."
On the morning of July 13 war correspondent Charles Carleton Coffin of the Boston Journal rode over to Meade's headquarters at Devil's Backbone. As Coffin talked with Seth Williams, Meade entered the tent, dripping wet from the rain. "We shall have a great battle to-morrow," he told Coffin. "The reinforcements are coming up, and as soon as they come we shall pitch in.'"4
What Meade didn't know was that the Confederates had been busily constructing a bridge across the Potomac. Engineering teams had been tearing apart Williamsport warehouses and other structures for raw materials and using borrowed teapots to heat the tar they needed to seal the sixteen boats they built to support the bridge. Starting late at night on July 13 Longstreet's and A. P. Hill's corps crossed the Potomac here, while Ewell's men forded the river just above Williamsport. Once the last of the Southern soldiers who could make it across the bridge reached the Virginia side, the commander of the final regiment cut the ropes behind him, and the current pushed the bridge over to the far side. Meade had lost his chance to bag the Army of Northern Virginia.
Halleck sent a wire to Meade as soon as he received the bad news. "I need hardly say to you that the escape of Lee's army without another battle has created great dissatisfaction in the mind of the President," he said, "and it will require an active and energetic pursuit on your part to remove the impression that it has not been sufficiently active heretofore."
Quartermaster Rufus Ingalls entered the tent as Meade was reading the telegram. "Ingalls, don't you want to take command of this army?" Meade asked.
"No, I think it's too big an elephant for me," replied Ingalls.
"Well, it's too big for me, too," said Meade. "Read that." He handed him the telegram.5
Meade then sent a reply to Halleck. "Having performed my duty conscientiously and to the best of my ability," he replied, "the censure of the President conveyed in your dispatch of 1 p.m. this day, is, in my judgment, so undeserved that I feel compelled most respectfully to ask to be immediately relieved from the command of this army."
Tom Huntington is the former editor of Historic Traveler and American History magazines and the author of Ben Franklin's Philadelphia (2005), Pennsylvania Civil War Trails (2007) and Guide to Gettysburg Battlefield Monuments (2013). His magazine articles have appeared in American Heritage, Smithsonian, Preservation, Air & Space, America in WWII, Civil War Times, America's Civil War, and many other publications. He lives in Camp Hill, Pennsylvania, with his wife, Beth Ann, and his children, Katie and Sam.
1George Meade, The Life and Letters of George Gordon Meade, Major-General United States Army, 2 vols (New York: Charles Scribner's Sons, 1913), II: 132.
2Eric J. Wittenberg, David Petruzzi, and Michael F. Nugent, One Continuous Fight: The Retreat from Gettysburg and the Pursuit of Lee's Army of Northern Virginia, July 4-14, 1863 (New York: Savas Beatie, 2008) reprint, (Philadelphia: Casemate Publishers, 2011) 148.
3The communications between Meade and Halleck are all from Official Records, Series I, Volume 27, Part One, but they are compiled on the web at www.civilwarhome.com/aftergettysburgor.htm.
4Charles Carleton Coffin, Four Years of Fighting: A Volume of Personal Observation with the Army and Navy, from the First Battle of Bull Run to the Fall of Richmond (Boston: Ticknor and Fields, 1866) 302-305.
5Account from Evelyn Page, "After Gettysburg: Frederick Law Olmsted on the Escape of Lee," Pennsylvania Magazine of History and Biography 75, no. 4 (October 1951): 436-446.
Photo Courtesy Library of Congress (loc.gov).
"The Pursuit" is adapted from the author's book Searching for George Gordon Meade: The Forgotten Victor of Gettysburg courtesy of Stackpole Books.
News from Gettysburg
By: Waters Whipple Braman
During the Battle of Gettysburg, First Lieutenant Waters Whipple Braman, 23, and his regiment, the 93rd New York Infantry, served as Army of the Potomac commander George G. Meade's headquarters guard. Two days after battle's end, Braman, who had sold his lumber business in 1861 to join the army, wrote a letter (copied below) to his uncle in which he provided his intital thoughts about the epic...
‹ Previous: JOHNSON: River of Dark Dreams (2013)
Next: DANIEL: Battle of Stones River (2012) ›
© 2020 The Civil War Monitor
Website Design and Development by Blind Acre Media


text:
package edu.illinois.my.wiki.plugin.servlet.action.response;

import java.io.IOException;

import javax.servlet.http.HttpServletResponse;

import edu.illinois.my.wiki.services.streaming.ObjectStreamer;
import edu.illinois.my.wiki.services.streaming.data.ExceptionOption;
import edu.illinois.my.wiki.services.wrappers.OutputStreamInterface;
import edu.illinois.my.wiki.services.wrappers.OutputStreamWrapper;

final class SendExceptionOptionResponse implements ServletActionResponse {
    private final ExceptionOption<?> objectToSend;
    private final ObjectStreamer streamer;

    private static final String SERIALIZED_CONTENT_TYPE =
            "application/x-java-serialized-object";

    SendExceptionOptionResponse(ExceptionOption<?> objectResponse,
            ObjectStreamer streamer) {
        this.objectToSend = objectResponse;
        this.streamer = streamer;
    }

    @Override
    public void send(HttpServletResponse httpResponse) throws IOException {
        httpResponse.setContentType(SERIALIZED_CONTENT_TYPE);
        OutputStreamInterface responseStream = null;
        try {
            responseStream = getOutputStream(httpResponse);
            streamer.sendObject(objectToSend, responseStream);
            responseStream.flush();
        } finally {
            if (responseStream != null)
                responseStream.close();
        }
    }

    private OutputStreamInterface getOutputStream(
            HttpServletResponse httpResponse) throws IOException {
        return new OutputStreamWrapper(httpResponse.getOutputStream());
    }
}

redpajama_set_name: RedPajamaGithub


Any time you're seeking industrial or domestic Natural Gas Furnaces in West Virginia, look no further than Natural Gas Furnace Guys. We've got a very capable crew of workers who deliver Natural Gas Furnaces in West Virginia at an affordable price. We fully understand all of your expectations and challenges, and we are available to deliver the very best and least expensive services around. Contact us now! 888-398-7770.
When you require superior resources, qualified professional advice, and rates that suit your budget, work with Natural Gas Furnace Guys. When you call us by dialing 888-398-7770, we'll answer all your inquiries related to Natural Gas Furnaces in West Virginia and give you a quote to the expense and time frame for your job. We know that you have a financial budget to stick to and significant goals and objectives to meet, and we do our best to carry out your project quickly, skillfully, and affordably. We will outshine the opposition in customer service and customer happiness each and every time. We pride ourselves in remaining your go to source to get Natural Gas Furnaces in West Virginia, and so we complete the project the correct way to start with. You'll need a company which will save you costs while offering 5 star Natural Gas Furnaces services. Natural Gas Furnace Guys is going to be that provider.
Our skilled and experienced experts excel at supplying the very best services while saving your money. We can carry out your work promptly and stay clear of wasting supplies on usual mistakes to help you stay within your spending budget. We all work quickly with a thorough plan that's guaranteed to save time and cash, but your job will undoubtedly be of the highest quality when completed. You can keep on save your money after we finish given that our superior quality products can last for years without the need of expensive repairs and maintenance. To learn just how long your project materials might last, call up 888-398-7770.
Our objective is always to make certain that you're totally happy when you deal with our company for top level Natural Gas Furnaces in West Virginia. Our client care representatives are trained to offer friendly and knowledgeable help and advice and services. Feel free to ask any sort of concerns you might have, and we are pleased to answer them. Your Natural Gas Furnaces mission is our specialization, and we are going to be sure that you are happy. When you'd like specialized and timely products and services, you're able to have confidence in our staff to be here for your needs. We stay away from unforeseen expenditures and supply a precise estimate prior to getting started out. We will carry out the project properly on the first try, and you'll spend less money. Whenever you contact us, we will respond to all your questions and provide you with a free quote.


HomeCare & Hospice of the Valley provides peace, comfort, and compassionate support to those facing advanced illness. HomeCare & Hospice of the Valley serves Eagle, Garfield and Pitkin Counties, as well as portions of Gunnison and Summit Counties.
Click on photo images to enlarge.


I decided today that it was high time to finish up the leftover Easter ham. That, and my post-gym legs were craving some serious protein. These mini spinach pies are more like quiche than the Greek phyllo-dough treats of the same name (spanakopita translates to "spinach pie") and are a quick and easy way to spice up a solo breakfast or serve up a weekend brunch for friends.
With a pastry blender, mix dry ingredients until coarse and crumbly. Slowly add water, kneading the dough by hand until it is soft and gummy. Form into a disc, wrap with parchment paper and refrigerate at least 30 minutes.
About 5-6 mini pie tins, muffin cups, tart cups, etc.
In a large pan, saute the onions, spinach and ham in the olive oil until nicely browned, adding salt and pepper to taste. Add the cream and stir until creamy, but take care to not burn the cream. Remove from heat.
While the vegetables are cooling, roll out the cheddar dough on a floured surface. Using a stencil (I use a small glass prep bowl) cut out enough rounds to fill each pie tin. Spritz the non-stick cooking spray onto the bottoms and sides of each tin if necessary, then gently press each dough round into the tins.
In a separate bowl, beat the eggs. Fold the vegetable mixture into the eggs, then spoon the batter into each tin, leaving about 1/4" of space at the top, as the egg mixture will slightly rise. Brush edges of crust with egg wash, or with remaining batter if possible. Slice cherry tomatos and place two halves in the center of each pie.
Bake at 350 degrees for 18-20 minutes, or until edges are browned and egg is fully cooked through; insert and remove a skewer to check for doneness - if it comes out clean, the eggs are done. Let cool on a wire rack, then gently remove the pies from their tins.


text:
Q: php declaration in loop? $image im resizing an image using a $image. how can I make this:
    scaleImageFile(
        "gif/SB-99JCX-000021.gif",
        500,
        500,
        "gifanim/output/SB-99JCX-000021-500x500.gif",
        4
    );

into this? and make it work..
$image = is in a loop.. counting each image. so SB-99JCX-000021.gif, 22.gif, 23.gif... and so on
        scaleImageFile(
            "gif/$image",
            500,
            500,
            "gifanim/output/$image-500x500.gif",
            4
        );

more of the code to give you an idea:
foreach ($sortedImages as $image) {

        scaleImageFile(
            "gif/SB-99JCX-000021.gif",
            500,
            500,
            "gifanim/output/SB-99JCX-000021-500x500.gif",
            4
        );

A: Just substitute the variable into the filenames:
foreach ($sortedImages as $image) {

    # Get the name of the image, stripped from image folder path and file type extension
    $name = substr($image,strlen($imagefolder),strpos($image, '.')-strlen($imagefolder));

    scaleImageFile(
        "gif/$name.gif",
        500,
        500,
        "gifanim/output/$name-500x500.gif",
        4
    );
}

You might also find the basename function to be useful, in place of your substr code. It can remove the directory prefix and a known extension. So you could write:
$name = basename($image, '.gif');

A: foreach ($sortedImages as $image) {

$name2 = basename($image, '.gif');
$location_vars = 'gif/'.$name2.'.gif';
$location_vars2 = 'gifanim/output/'.$name2.'-400x400.gif';

        scaleImageFile(
            $location_vars, //"gif/$name2.gif",
            400,
            400,
            $location_vars2, //"gifanim/output/$name2-500x500.gif",
            4
        );

}

redpajama_set_name: RedPajamaStackExchange


It�s the March birthstone, but it�s so much more. As the ancient philosopher Pliny said, aquamarine �has charms not to be denied.� Apples of Gold�s collection of aquamarine rings showcases this alluring stone in styles from demure to dramatic.
The meaning of the word �aquamarine� provides a clue to the stunning appearance of this mesmerizing gem: �aqua� means �water� and �mare� means �the sea.� These two words are a fitting description for a stone that glimmers like the ocean on a sunny day. Aquamarine�s light blue color is both soothing and refreshing, offering a gentle burst of vibrancy that makes it hard to look away. Read more.
As a member of the beryl family of gemstones, aquamarine is related to the emerald. However, unlike its green cousin, aquamarine is almost completely without inclusions, or flaws. It�s also a hard stone, able to resist scratches and continue to sparkle for years to come.
Simple aquamarine rings such as solitaires and three-stone styles allow the natural beauty of the gem they contain to steal the show. These unfussy looks have a timeless appeal that will never go out of style.
Vintage styles are particularly lovely when set with aquamarine. The soft hue of aquamarine is a perfect complement for designs inspired by Victorian Era, a time when feminine looks were in vogue. When accented by this light indigo gem, these ladylike styles take on a sparkle that both honors the past and renders them perfect for the present. With elements such as filigree, paisley patterns and floral designs, Apples of Gold�s vintage aquamarine rings harken to a simpler time, but are perfect for the present.
Of course, styles from the past don�t corner the market on truly stunning aquamarine rings. Modern looks seem even fresher, taking on the relaxing yet invigorating vibe of a glimmering blue gem. Whether you choose a minimalistic look or one with a bold, sizeable stone, you�re sure to get a piece that makes a sophisticated statement.
For a ring that�s soothing as the sea and refreshing as the splash of an ocean wave, aquamarine is the answer. Slide on a ring that features this dazzling blue gem and escape to your own piece of paradise.


A great idea when teaching students about how Jesus fed the 5,000 is to have a fun little snack for them to eat while they are learning about it! Catholic Icing has come up with a simple and easy snack to use.
This is what you will need to do. Put the chocolate on top of the pretzel and put it in the oven and melt it just slightly(or microwave it). Then once it is slightly melted put the little fish on top of it and let it harden into it.
This snack will be a great yummy treat for your kids and they will also allow you to make your story more hands on. This could be great for sunday school, vacation Bible school, a bible study, or just a fun family time moment.
Start with reading Matthew 14:13-21 and then when it gets to the part about Jesus handing out the food, pull out the tasty treats and hand one to each kid.
Thanks Catholic Icing for this fun idea!
If you like what you read, check out our children's book, Hooked on the Book, on this website. It provides the Big Story behind all 66 books in the Bible including colorful illustrations, rhymes and a surprise on every page.
This entry was posted in Bible Experiences, Bible Miracles, Free Resources, Parents, Teachers and tagged Feeding the 5000, Snack, Sunday School. Bookmark the permalink.