text:
What can we learn from the UCL ransomware attack?
Themen: Advanced Threat Protection, Backup, Data Protection, Email Protection, Malware Protection, Network and Application Security, Ransomware Protection, State and Local Government, Virus Protection
15. Juni 2017 15. September 2020 Christine Barry
Just in time for our Thursday ransomware post, news breaks that University College London has been hit by a ransomware attack that brought down the UCL student management system as well as shared network drives. Some of the closely related University College London Hospitals have also shut down their email systems to quarantine themselves from attack. There are no reports of the hospital group being affected by the ransomware other than this preventative downtime.
What do we know?
The particular ransomware strain is unknown, as is the infection method. UCL initially believed that the ransomware attack was launched through a phishing attack on Wednesday afternoon (BST). Phishing is one of the most common methods of ransomware deployment. According to a recent report by PhishMe, the number of phishing emails hit 6.3 million in the first quarter of this year, and 93% of them included ransomware.
Put simply; a phishing attack is an email designed to trick the recipient into taking a particular action. The action may be to enter credentials into a form, visit a compromised website, or open a malicious attachment. Attackers research their targets and craft their email attack based on what they learn. A more sophisticated version of this is a spear phishing attack. Spear phishing is more personal and has a higher success rate. There aren't enough details about the UCL attack to know what kind of phishing may have been used to deploy the ransomware.
The second and most recent unconfirmed hypothesis is that the ransomware was deployed through a compromised website. This type of attack is made possible by vulnerable code in a website and visitors with vulnerable clients. Even high-profile sites like cracked.com and stanford.edu have been used to deploy drive-by downloads, phishing kit attacks, and other exploits. Drive-by downloads deliver and execute malware attacks on client machines, often without user interaction. This is not an uncommon threat: Google currently block-lists approximately 20,000 websites a week for malware and approximately 50,000 a week for phishing.
While UCL staff is still not able to confirm details on the attack, here is what is being reported so far:
The ransomware attack started early afternoon on Wednesday, June 14.
The university network drives and shared drives were locked down with read-only access to protect them from encryption.
The drives already encrypted will be recovered by data backup. There should be minimal data loss.
UCL believes that this was a zero-day attack because university antivirus did not stop the malware.
Some NHS hospital trusts have shut down their email systems as a precaution to prevent the spread of malware into their systems.
Additionally, NHS Digital has been fighting off speculation that NHS itself has suffered a cyber attack today. The speculation is likely due to the confusion surrounding the hospital trusts that took systems offline.
What can we learn?
Despite how little is known about the attack, there's still quite a bit to consider.
The dangers of phishing and spear phishing cannot be overstated. A single user can bring down a university, a hospital, a government, with just one careless click. User fatigue, distraction, or foolish confidence, can only be dealt with by user training and awareness.
Visiting a website can be very dangerous. A compromised website can deliver a set of exploits, or direct a visitor to a separate site that hosts a larger set of attacks. The more sophisticated sites can evaluate a client computer for any number of vulnerabilities, among thousands of possible combinations. Operating systems, applications, missing or incomplete antivirus … all of these things are checked by these exploit kits.
Consider the comment that the university antivirus did not stop the threat, which led to the speculation that it was a zero-day threat. The university system probably has other security in place, but for our purposes let's assume that antivirus was the only defense. Antivirus protection is necessary but not sufficient protection against modern threats. The best security will include Advanced Threat Protection (ATP) that will detect and detonate threats in the cloud before they reach the target's infrastructure. Machine learning and other advanced techniques allow ATP to adapt quickly and identify zero-day attacks as they are getting started. This type of protection could have stopped that phishing email or the drive-by download before it arrived in the university system.
The university changed their drives to 'read only' access to prevent encryption. Some ransomware strains have delayed deployment techniques, and other non-ransomware malware may have been deployed in this attack without the staff noticing. The example of locking down drives shouldn't be taken as anything more than a stop-gap while cleaning and data recovery take place.
Affiliated organizations shut down email servers to stop the ransomware from getting into their networks. This was probably in response to cooperation and shared information between the organizations. It's important to remember that supply chains can also be attacked, and those victims might not communicate the attack as well as a peer or partner organization does.
A common attack technique is to distract the victim with one attack while conducting a different stealth attack. This is usually done with a DDoS attack as the distraction and a stealth data exfiltration as the real attack. With a phishing – ransomware attack keeping everyone busy, it may have been a good time for an attacker to try to get into the network to pull out copies of university research.
To be clear, there is no indication that UCL suffered any other attacks, or that the UCL IT staff do not have advanced protection in place. It's obvious from reading the reports on this attack that the staff was intentionally disclosing as little as possible. However, in this climate of non-stop cybercrime, it just makes sense to review these attacks and look for things that we might not have known.
For information on how Barracuda can help you deploy multiple layers of security and data protection, visit our corporate site here. For specific information on how we protect you from ransomware attacks, visit our corporate ransomware site here and our ransomware blog here.
Christine Barry
Christine Barry ist Senior Chief Blogger und Social Media Manager bei Barracuda. In dieser Rolle hilft sie, Barracuda-Geschichten zum Leben zu erwecken und die Kommunikation zwischen der Öffentlichkeit und den internen Barracuda-Teams zu erleichtern. Bevor sie zu Barracuda kam, war Christine über 15 Jahre lang als Außendiensttechnikerin und Projektmanagerin für K12- und KMU-Kunden tätig. Sie hat mehrere Abschlüsse in Technologie, einen Bachelor of Arts und einen Master of Business Administration. Sie ist Absolventin der University of Michigan.
Vernetzen Sie sich hier auf LinkedIn mit Christine.
Sicherung von mit dem Internet verbundenen Geräten im Gesundheitswesen
Threat-Spotlight: Phishing im Zusammenhang mit dem Coronavirus
Mein Blick zurück in die Zukunft
So unterstützt Barracuda CloudGen WAN ein globales Unternehmen …
Advanced Threat ProtectionBackupData ProtectionE-Mail ProtectionMalware ProtectionNETWORK AND APPLICATION SECURITYSchutz vor RansomwareLandes- und KommunalverwaltungVirenschutz
Are You Cloud Ready? Q & A on providing flexible consumption models
Top Security Technologies for 2017 Put Too Much Strain on the IT Budget


text:
xLua's build engine supports secondary development, and you can use it to generate some text files (for example, code and configuration files). The xLua's link.xml file is generated by the build engine plugin. Other application scenarios (such as generating a Lua IDE auto configuration file) can also be accomplished using this feature.

## Overview

The plugin needs to provide two things: 1. a template for generated files, 2. a callback function that accepts the user configuration and returns the data that needs to be injected into the template and the output stream of the files.

## Template syntax

The template syntax is simple, with only three elements:

* eval: The syntax is <%=exp%>. Exp is an arbitrary expression that will calculate and output the value of exp as a string.
* Code: The syntax is <% if true then end%>. The blue part is any Lua code that will be executed.
* Literal: This contains the parts other than eval and code. Literal means output as it is.

Example:

~~~xml
<%
require "TemplateCommon"
%>

<linker>
<%ForEachCsList(assembly_infos, function(assembly_info)%>
	<assembly fullname="<%=assembly_info.FullName%>">
	    <%ForEachCsList(assembly_info.Types, function(type)
		%><type fullname="<%=type:ToString()%>" preserve="all"/>
		<%end)%>
	</assembly>
<%end)%>
</linker>
~~~

TemplateCommon has some predefined functions that can be used (for example ForEachCsList). You can search in TemplateCommon.lua.txt of the project to see which functions are available. For ordinary Lua, you can write one.

## API

~~~csharp
public static void CSObjectWrapEditor.Generator.CustomGen(string template_src, GetTasks get_tasks)
~~~

* template_src: template source code
* get_tasks: This is a callback function. The type is GetTasks. This function is used to accept the user configuration and return the data that needs to be injected into the template and the output stream of the files.

~~~csharp
public delegate IEnumerable<CustomGenTask> GetTasks(LuaEnv lua_env, UserConfig user_cfg);
~~~

* lua_env: This is a LuaEnv object. Because the returned template data needs to be placed in LuaTable, LuaEnv.NewTable is required.
* user_cfg: user configuration
* return: Among the returned values, CustomGenTask represents a generated file, and IEnumerable type indicates that the same template can generate multiple files.

~~~csharp
public struct UserConfig
{
    public IEnumerable<Type> LuaCallCSharp;
    public IEnumerable<Type> CSharpCallLua;
    public IEnumerable<Type> ReflectionUse;
}
~~~

~~~csharp
public struct CustomGenTask
{
    public LuaTable Data;
    public TextWriter Output;
}
~~~

Example:

~~~csharp
public static IEnumerable<CustomGenTask> GetTasks(LuaEnv lua_env, UserConfig user_cfg)
{
    LuaTable data = lua_env.NewTable();
    var assembly_infos = (from type in user_cfg.ReflectionUse
                          group type by type.Assembly.GetName().Name into assembly_info
                          select new { FullName = assembly_info.Key, Types = assembly_info.ToList()}).ToList();
    data.Set("assembly_infos", assembly_infos);

    yield return new CustomGenTask
    {
        Data = data,
        Output = new StreamWriter(GeneratorConfig.common_path + "/link.xml",
        false, Encoding.UTF8)
    };
}
~~~

* Only one file is generated here, so only one CustomGenTask is returned.
* data is the data to be used in the template. There is an assembly_infos field included. See the template section for how to use this field.

## Tag

Generally speaking, you can use MenuItem to create a menu to trigger a custom generate operation. However, sometimes you may want the generate operation to be triggered directly by the xLua "Generate Code" menu. In this situation, you will need to use CSObjectWrapEditor.GenCodeMenu

Example:

~~~csharp
[GenCodeMenu]//加到Generate Code菜单里头
public static void GenLinkXml()
{
    Generator.CustomGen(ScriptableObject.CreateInstance<LinkXmlGen>().Template.text, GetTasks);
}
~~~

PS: All the code related to the content above is in the XLua\Src\Editor\LinkXmlGen directory, which is also the implementation of the link.xml generation function that was explained at the beginning.

redpajama_set_name: RedPajamaGithub


text:
American Airlines Group Inc. - FORM 8-K - August 11, 2015
EX-99.1 - EX-99.1 - American Airlines Group Inc. d17282dex991.htm
Pursuant to Section 13 or 15(d)
of the Securities Exchange Act of 1934
Date of Report (Date of earliest event reported): August 11, 2015
AMERICAN AIRLINES, INC.
Delaware 1-8400 75-1825172
4333 Amon Carter Blvd., Fort Worth, Texas 76155
4333 Amon Carter Blvd., Fort Worth, Texas
Registrant's telephone number, including area code:
ITEM 7.01 REGULATION FD DISCLOSURE.
On August 11, 2015, American Airlines Group Inc. ("American") announced via press release certain traffic statistics for July 2015. A copy of American's press release is furnished as Exhibit 99.1 to this Current Report on Form 8-K.
The information in this Item 7.01, including Exhibit 99.1, is being furnished and shall not be deemed to be "filed" for purposes of Section 18 of the Securities Exchange Act of 1934, as amended, or otherwise subject to the liabilities of that Section and shall not be deemed incorporated by reference into any registration statement or other document filed pursuant to the Securities Act of 1933, as amended, except as shall be expressly set forth by specific reference in such filing.
(d) Exhibits.
99.1 Press Release, dated August 11, 2015.
Pursuant to the requirements of the Securities Exchange Act of 1934, American Airlines Group Inc. has duly caused this report to be signed on its behalf by the undersigned hereunto duly authorized.
Date: August 11, 2015 By:
/s/ Derek J. Kerr
Derek J. Kerr
Executive Vice President and
Pursuant to the requirements of the Securities Exchange Act of 1934, American Airlines, Inc. has duly caused this report to be signed on its behalf by the undersigned hereunto duly authorized.
99.1 Press Release, August 11, 2015.
American Airlines Group Inc. Reports

redpajama_set_name: RedPajamaCommonCrawl


East Timor Action Network The West Papua Advocacy Team (WPAT) is pleased to announce that it is awarding the 2011 "John Rumbiak Human Rights Defenders Award" to the Honorable Eni F.H. Faleomavaega (D-AS), a member (non-voting delegate) of the U.S. House of Representatives.
Congressmember Faleomavaega has been an articulate and effective advocate for the defense of human rights in West Papua, and has long worked for a peaceful resolution of the serious problems confronting Papuans.
His extensive knowledge regarding West Papua and his manifest sincerity and good will have enabled him to draw on the respect accorded him by his Congressional colleagues and members of successive Administrations to alert them and the U.S. public more broadly to justice, good governance and development concerns in West Papua.
Congressmember Faleomavaega continues to do all he can to hold the Indonesian government accountable so that a better way forward may be found for and on behalf of the people of West Papua.
On September 22, 2010, in his capacity as the Chairman of the Foreign Affairs Subcommittee on Asia, the Pacific and the Global Environment, Congressmember Faleomavaega convened the first hearing in the history of the U.S. Congress to include testimony from West Papua's traditional and religious leaders. The hearing, Crimes Against Humanity: When Will Indonesia's Military Be Held Accountable for Deliberate and Systematic Abuses in West Papua, also included testimony from scholars and administration officials from the U.S. Departments of State and Defense.
Driven by a sense of personal responsibility to carry forward the work of his Samoan relatives who are buried in West Papua and in honor of all those who have lived the struggle, Congressmember Faleomavaega continues to do all he can to hold the Indonesian government accountable so that a better way forward may be found for and on behalf of the people of West Papua.
Past recipients of the award include Carmel Budiardjo (UK) and TAPOL (2008); John M. Miller (U.S.) and the East Timor and Indonesia Action Network (ETAN) (2009), and Andreas Harsono (Indonesia) of Human Rights Watch (2010).
The award includes a plaque and a financial prize which Congressmember Faleomavaega has directed be donated to a charity selected by him. The award is named in honor of Papuan John Rumbiak, a renowned champion of human rights and founder of WPAT.


Today, the EFTA Surveillance Authority has fined Color Line EUR 18.811 million for an infringement of the competition rules of the EEA Agreement.
Color Line is a Norwegian ferry company which operates routes from Norway to Denmark, Germany and Sweden. With its route between Sandefjord in Norway and Strömstad in Sweden, Color Line remained for many years the only provider of short haul passenger ferry services with tax-free sales between these two countries.
The Authority's case concerns an agreement that was concluded in 1991 with the public harbour of Strömstad in Sweden. Through that agreement Color Line secured long-term exclusive access to harbour facilities in Strömstad harbour.
"Color Line has breached the competition rules in the EEA Agreement. The long-term exclusivity in its agreement prevented free competition on the route between Sandefjord and Strömstad and harmed consumers," Mrs Oda Helen Sletnes, President of the EFTA Surveillance Authority, said.
The Authority has concluded today that Color Line's long-term exclusivity in Strömstad harbour in Sweden restricted competition and constituted an abuse of Color Line's dominant market position. An infringement is found from the entry into force of the EEA Agreement in 1994 until December 2005, when a competitor of Color Line was granted access to the harbour.
Following the receipt of a complaint, the Authority opened the case in 2006 and has carried out an extensive market investigation. It presented its preliminary objections to Color Line in 2009 (see PR(09)98) and examined carefully Color Line's reply to those objections before adopting its decision.
For background information, read the Authority's Q&A.


Meet our Team We're always up to something good!
Brendan Toomey is a Certified Financial Planner, Registered Investment Adviser, Portfolio Manager and Tax Preparer. He has been working with the public since 1986. He became a CFP professional in 1993 and has at least 20 years experience in investments, insurance and tax preparation.
Austin is a licensed financial adviser and candidate for the ChFC® assisting the community for multiple years. His belief in ethics and professionalism has given him the motivation and tools necessary to help construct and implement our innovative planning system: the loop©. Austin enjoys the challenge every new client presents, and it shows.
Tammy has been the core of all administrative activities at TIMI for nearly 20 years. She is the catalyst for office operations, client service, communications and tax processing.
The TIMI business model is designed to treat all of our clients equally and fairly. We realized long ago that the financial industry dedicated many resources to capture money from prospective clients but much less on service and accountability for existing clients. At TIMI, we listen carefully, keep in touch and return your calls/communications as quickly as we can. We insist that you count on us. We believe that if you always rely on us, we can work effectively to optimize your financial world and solve your problems. We want to KNOW YOU; We want to help you and your family!
The staff of TIMI is available to speak at meetings and functions. Please contact TIMI to inquire.
TIMI has a long history of providing seminar events for the community. We will send notifications and advertise the events when the schedule is available.


This is our preliminary assessment of the new proposal prepared for purposes of a public meeting.
It has not been possible in such a short timeframe to fully evaluate the proposal. However we feel this preliminary assessment gives a clear but unflattering view of the proposal and highlights issues which support the conclusion that the matter needs a full and comprehensive review by the Minister for the Environment.
This will be achieved if everyone possible can demand that the Minister conducts a "controlled action", which requires a formal assessment.
The developer would prefer a "not a controlled action" decision, known as a tick and flick.
We hope that this information will assist in letters and e-mails to the Minister.
How is the new proposal different?
The proposed development is now on the south side of Middle Head Road and does not include the re-development of the 3 barracks including the underground parking.
The Sydney Harbour Federation Trust (SHFT) has not confirmed that the barracks will be removed. Removal of the barracks is a separate issue.
The buildings on the south side of the road (the 10 Terminal Complex) have been enlarged in scale and bulk.
There is SIGNIFICATLY MORE DEMOLITION of the heritage listed 10 Terminal and we are advised that virtually nothing will remain.
There is also a new parking lot shown on the new proposal, which is presently green space.
There is also a new 100+ metre continuous steel heat-shield firewall between 2-3 metres high along the edge of the bushland/ridge which would block views of the Harbour from the proposal walking track.
HPG disagrees with these statements posted on the Sydney Harbour Federation Trust (SHFT) Website about the new proposal and feels they are misleading.
The overall footprint* seems reduced but the facility is still about the same size as four suburban hotels and roughly as large as all the other buildings on the precinct combined. The number of BEDS has been reduced from only 93 to 89 (86 rooms).
*footprint must include buildings, enclosed spaces, and car parking requirements, whether reserved or public. Because the previous underground parking (36 spaces) is now located above ground, this also takes public space and must be included.
We have not had time to measure the new footprint* but we do not think it is substantially smaller if parking is included.
We are advised that it is quite likely that these buildings would not survive demolition. The interiors are all to be demolished.
It is not an extension it is a massive new 2-storey building.
This is a completely new building completely replacing the existing garage and larger than previously.
The highest existing building would be one of the garages, and this is high (not one storey) and it appears that the new building height is not much lower than the previous proposal but there is no height statistic.
10 Terminal is listed on the Commonwealth Heritage List.
However they intend to demolish most of the buildings and conserve the heritage by "Archival Recording and Interpretation Strategy". Such strategy is yet to be prepared.
"The Trust strongly disagrees with this assessment. The Heritage Impact Statement relies on Sydney Harbour Federation Trust, Comprehensive Plan 2003 which adopts the heritage significance assessment undertaken by Godden Mackay Logan for Georges Heights and Middle Head Defence Site in 1998, while the site was still owned and controlled by the Defence Department. That heritage significance assessment states that Ten Terminal is rated as having some significance, except Building 6, which is rated as having low significance.
"form an exceptionally rare collection of such buildings at a National level. Within NSW only the almost contemporary School of Artillery at North Head and the former 12th Lines of Communication complex at Boronia Park (Gladesville) are known to survive outside military establishments.
Although used variously for training and administration, substantial documentary and physical evidence of the original configuration and detail of the building survives, including window joinery, doors and partitions.
"The significance of the site and the building complexes is such that, with very few exceptions, all the buildings should be retained, adaptively re-used with minimal interference in the fabric of the buildings and landscape and interpreted with appropriate signage. Original World War 2 buildings should not be demolished but be adaptively re-used for uses compatible with their original and community uses.
The proponent has amended its original Heritage Impact Statement to now include many references, including a heritage report by Godden MacKay Logan, which was undertaken in 1998 on behalf Defence Department Sydney Property Disposal Unit.
This was undertaken when Defence was trying to sell the sites and before the SHFT was established. It is assumed that the tone of this report is a reflection of the fact that a lower rating would assist the sale. A similar position applies today, unfortunately.
The Godden MacKay report gave a low heritage value to the site.
Conclusion – the proposal would involve a total loss of a nationally significant rare complex.
Because the development is in a Category 1 dangerous bushfire-prone area, the new plan includes a very long unsightly barrier on the southeast edge of the bushland/ridgeline that will prevent views of the Harbour.
The legal advice is that the new proposal is not legal under SHFT's Middle Head Management Plan 2007, which is the statutory document for the land.
There are 27 parking bays proposed for the west side and this was not on the previous proposal.
This area is presently beautiful trees and open green-space.
The other is a narrow access between 2 large buildings.
We do not consider this usable public space.
Conclusion – there is a significant impact and alienation of public parkland due to footprints, enclosed gardens, additional parking lot to be built, appropriated public car parking, and unusable open space. As the facility is by far the most dominant building in the precinct and a case of overbuilding, it will compete for the use of all adjacent public land, to a degree not examined by either the proponent or the SHFT.
Adaptive reuse is the adaptation of historic buildings from one use to another while preserving their heritage value.
Sometimes this may be nothing more than a change in activities, however in many situations buildings require modifications to suit their new purpose. As a result, reuse allows historic buildings to be properly cared for, and their origins to be understood and remembered, even where their purpose and function have changed.
Adaptive reuse is an important element in retaining historic buildings and other places for future generations. What's more, there are considerable benefits for environmental sustainability.
HPG does not believe this proposal is adaptive reuse.
The ENTIRE INTERIOR OF ALL BUILDINGS WILL BE DEMOLISHED. THE INTERIORS ARE STILL INTACT HISTORICALLY AND NOT MUCH ALTERED.
The number of beds is only reduced from 93 to 89.
To accommodate this almost same number of beds but on only side of the road requires larger buildings and significantly more demolition.
We are advised that the Demolition Plan is not a viable plan and would result in at least 75% demolition and quite possibly all.
The demolition as documented is misleading.
By the time the roof, ceiling & the internal walls are demolished there will be nothing bracing the external walls. They will become unstable and for safety will have to be demolished.
Even if, at a great cost, existing walls are braced & retained as shown on the plans, when new windows, doors, floor to ceiling heights etc are put in, there will not be anything left of the existing walls.
When the internal walls are demolished, existing floors will likewise have to be demolished to accommodate new layout and consequent new services. This is not even mentioned on the Demolition Plans which in my opinion are misleading to say the least.
Above comments apply to the demolition of Buildings 1,3,6 & 7. I believe it is proposed Building 2 to be demolished in its entirety.
Michal K Tomaszewski B. Arch., FRAIA.
There has been no public consultation on this new plan, and the proponent has chosen to seek approval first from the Minister rather than the SHFT. Apparently this is allowed.
The proponent will still need approval from the SHFT in addition to the Minister, not only for the proposal but the SHFT would need to amend its Management Plan which would require the further approval of the Minister.
The proponent states that is has consulted with the SHFT to develop the new plan (around 2 months at least), however the community was not made aware of the new plan until it went on exhibition last week (30 April 2014) at the Department of the Environment.
The fact that the SHFT has been consulted on this new plan which requires almost total demolition of buildings that the SHFT itself rates the 10 Terminal Complex of high heritage significance AND that is not legal under its own Management Plan is not understandable to us.
"Public consultation on the draft amendment to the Middle Head Management Plan indicated strong community desire for the removal of the two-storey timber barracks buildings on the northern side of Middle Head Road.
There is no relation between the new proposal and the ability of the SHFT to remove the 3 Barracks.
Under the SHFT's Middle Head Management Plan they could always be removed.
We have not had time to obtain exact measurements but believe that the overall floor area is similar to the original proposal which was 2 sides of the road.
While the SHFT and proponent maintain that the new proposal is more acceptable than the previous one which they supported and then, apparently abandoned, any analysis does not support their joint views. The plan seems to reduce the footprint somewhat but adds a car park and a much larger and bulkier buildings.
In the process the 10 Terminal Complex, high heritage buildings and a critical part of our military heritage and history, will be lost completely. The operation of the facility cannot fail to push other activities to the side and take away the community's use of public land as granted by the SHFT, in addition to the land already lost to the facility.
Once built it would be irreversible. It has been undertaken without sufficient effort to develop alternatives which are less intrusive and more in keeping with the environment of Middle Head.


Adorable sterling silver children's ring with twisted band and accented with white sapphire stones. This ring is rhodium plated to prevent tarnish for everlasting quality. Jewelry comes in our Timeless Collection gift box and is available in sizes 1-4.


text:
Q: Cython: How to declare default values in pxd files I have a function I'd like to abstract into an importable module:
cdef list generate_random_vectors(int num_vectors, int length, float scale=1):
    cdef list return_list
    np.random.seed()
    return_list = []
    for n in range(num_vectors):
    return_list.append((1 - 2 * np.random.rand(length)) / scale)
    return return_list

To put this in a module, I have to define it in the pxd file.
When I try this:
cdef list generate_random_vectors(int num_vectors, int length, float scale)

I get the error: Function signature does not match previous declaration.
When I try this:
cdef list generate_random_vectors(int num_vectors, int length, float scale=1)

I get the error: Expected ')', found 'INT'

A: Found it. The syntax is different in pxd files.
I need to use:
cdef list generate_random_vectors(int num_vectors, int length, float scale=*)

redpajama_set_name: RedPajamaStackExchange


Getaway Magazine Special Issue 2: American "Zen"
Rand McNally's special-issue Getaway magazine inspires vacations that are both realistic and extraordinary. Time-strapped travelers will love the 50+ pages of getaways that work for a weekend or a week. Budget-savvy travelers will love the value-for-money advice that allows for the occasional splurge. All travelers will love the rich destination experiences—as opposed to just trips—both on and off the beaten path.
Experience American "Zen": Rejuvenate at wellness spas, yoga retreats, motivational workshops, sacred spaces, and meditation meccas. Rekindle the flame with romantic resort adventures. Reconnect with your inner child---hunting for beach treasures or playing in the snow.
Discover the Latest, Coolest, and Best: Goggles, gloves, and footwear touched by tech; skis, sleds, and snow togs with a touch of the artisanal; soothing apps and travel accessories for touches of zen.
Take In 8 Great Places: Find Old Florida on the Gulf Coast and barbecue---old-style and new---in Dallas. Customize your Palm Springs desert adventure or your Utah ski escape. Explore regionalist art in Cedar Rapids or liberal arts in Cambridge. Beach-hop on Kaua'i; island-hop in the U.S. Virgins.


text:
\section{Error Rate of Convex Problems}
\label{sec:convex}

For the convex and Lipschitz functions, we consider the
low-rank structure of the gradient space, i.e,  the population gradient second momment
$\Sigma_t$ is of rank-$k$, which is a special case of the principal gradient dominate assumption when $|[\nabla \hat L_n ({\mathbf{w}}_t)]^\bot \|_2 = 0$.

\theolipconvex*

\emph{Proof:}
By the convexity of $\hat L_n({\mathbf{w}})$, we have
\begin{equation}
\hat L_n({\mathbf{w}}_t) - \hat L_n({\mathbf{w}}^\star) \leq \langle {\mathbf{w}}_t - {\mathbf{w}}^\star, \nabla \hat L_n({\mathbf{w}}_t) \rangle.
\end{equation}

At iteration $t$, we have ${\mathbf{w}}_{t+1} = {\mathbf{w}}_{t} -\eta_t \tilde \mathbf{g}_t,$
where $\tilde \mathbf{g}_t = \hat V_k \hat V_k ^\intercal \mathbf{g}_t + \hat V_k \hat V_k ^\intercal \b_t$.

Let $\bar \mathbf{g}_t = \mathbf{g}_t +\hat V_k \hat V_k ^\intercal \b_t,$ and $\Delta_t =  \hat V_k \hat V_k ^\intercal \mathbf{g}_t - \mathbf{g}_t.$
Then we have
\begin{equation}
  \tilde \mathbf{g}_t = \bar \mathbf{g}_t +  \Delta_t.
\end{equation}

Recall that $\mathbf{g}_t = \frac{1}{|B_t|}\sum_{z_i \in B_t}\nabla \ell({\mathbf{w}}_t,z_i)$. With $B_t$ uniformly sampled from $S$, we have
\begin{equation}
\mathbb{E}_t[ \mathbf{g}_t ]= \nabla \hat L_n({\mathbf{w}}_t).
\end{equation}

Since $\b_t$ is a zero mean Gaussian vector, we have
$\mathbb{E}_t[ \bar \mathbf{g}_t ] = \mathbf{g}_t$.

By convexity, conditioned at ${\mathbf{w}}_t$, we have
\begin{align}
& \nonumber \mathbb{E}_t [\hat L_n({\mathbf{w}}_t) - \hat L_n({\mathbf{w}}^\star)]  \\
& \nonumber \leq \mathbb{E}_t \langle {\mathbf{w}}_t - {\mathbf{w}}^\star, \mathbf{g}_t \rangle \\
\nonumber & = \mathbb{E}_t \langle {\mathbf{w}}_t - {\mathbf{w}}^\star, \bar \mathbf{g}_t \rangle \\
\nonumber & = \frac{1}{\eta_t} \mathbb{E}_t \langle {\mathbf{w}}_t - {\mathbf{w}}^\star, {\mathbf{w}}_t -{\mathbf{w}}_{t+1} - \eta_t \Delta_t \rangle \\
\nonumber & = \frac{1}{2\eta_t}\mathbb{E}_t \left( \|{\mathbf{w}}_t -{\mathbf{w}}^\star \|_2^2 + \eta_t^2 \| \bar \mathbf{g}_t  \|_2^2 - \|{\mathbf{w}}_{t+1} -{\mathbf{w}}^\star -\eta_t \Delta_t\|_2^2
\right)\\
\nonumber & = \frac{1}{2\eta_t}\mathbb{E}_t \left( \|{\mathbf{w}}_t -{\mathbf{w}}^\star \|_2^2 + \eta_t^2 \| \bar \mathbf{g}_t  \|_2^2 - \|{\mathbf{w}}_{t+1} -{\mathbf{w}}^\star\|_2^2 -\eta_t^2 \| \Delta_t \|_2^2 + 2\langle {\mathbf{w}}_{t+1} -{\mathbf{w}}^\star, \eta_t \Delta_t \rangle
\right) \\
\nonumber & = \frac{1}{2\eta_t}\mathbb{E}_t \left(\|{\mathbf{w}}_t -{\mathbf{w}}^\star \|_2^2 - \|{\mathbf{w}}_{t+1} -{\mathbf{w}}^\star\|_2^2 \right) + \frac{\eta_t}{2} \mathbb{E}_t [\| \bar \mathbf{g}_t  \|_2^2] -  \frac{\eta_t}{2}\mathbb{E}_t [\|  \Delta_t  \|_2^2] \\
\nonumber & \ \ \ +  \mathbb{E}_t \langle {\mathbf{w}}_{t+1} -{\mathbf{w}}^\star,   \Delta_t \rangle \\
 & \overset{(a)}{\leq} \frac{1}{2\eta_t}\mathbb{E}_t \left(\|{\mathbf{w}}_t -{\mathbf{w}}^\star \|_2^2 - \|{\mathbf{w}}_{t+1} -{\mathbf{w}}^\star\|_2^2 \right) +  \frac{\eta_t}{2}\left(G^2 + k\sigma^2 \right) +  B \mathbb{E}_t \|  \Delta_t\|_2,
\end{align}
where $(a)$ is true since
\begin{align}
\nonumber    \mathbb{E}_t \| \bar \mathbf{g}_t\|_2^2&  \leq  \mathbb{E}_t [ \| \mathbf{g}_t\|_2^2 + \|\hat V_k \hat V_k^\intercal \b_t \|_2^2 ] \\
    & \leq  G^2 + k\sigma^2,
\end{align}
and $ \| {\mathbf{w}}_{t+1} -{\mathbf{w}}^\star\|_2 \leq B$ \footnote{For convex problem, we consider ${\mathbf{w}} \in {\cal H}$, where ${\cal H} = \{{\mathbf{w}}: \| {\mathbf{w}}\|\leq B\}$.}.

Let $\eta_t =\frac{1}{\sqrt{T}}$, taking the expectation over all iterations and sum over $t =1, .., T$, we have
\begin{align}\label{eq:lip_con_sum}
\frac{1}{T}\mathbb{E}\left[ \sum_{t = 1}^\intercal \hat L_n({\mathbf{w}}_t) - \hat L_n({\mathbf{w}}^\star) \right] \leq \frac{\| {\mathbf{w}}_0 -{\mathbf{w}}^\star\|_2^2 + G^2 + k\sigma^2 }{2\sqrt{T}} + \frac{B \sum_{t=1}^\intercal \mathbb{E}_t \|  \Delta_t\|_2}{T}.
\end{align}

From Theorem \ref{theo:sc}, we have

\begin{align}
\nonumber \mathbb{E}_t \left[\| \Delta_t\|_2\right] &
=\mathbb{E}\left[\left\|\hat{V}_{k}(t) \hat{V}_{k}(t)^\intercal \mathbf{g}_{t}-V(t) V(t)^\intercal \mathbf{g}_{t}\right\|_2\right] \\ \nonumber
& \leq \mathbb{E}\left[\left\|\hat{V}_{k}(t) \hat{V}_{k}(t)^\intercal-V(t) V(t)^\intercal\right\|_{2} G\right] \\
& \leq G \mathbb{E}\left[\left\|\hat{V}_{k}(t) \hat{V}_{k}(t)^\intercal-V_k(t) V_k(t)^\intercal\right\|_{2}
+  \left\|V_k(t) V_k(t)^\intercal - V(t)V(t)^\intercal\right\|_{2}
\right] \nonumber
\\
& \leq
O\left(\frac{G \kappa \ln p \gamma_{2}(\mathcal{W}, d)}{\alpha_t \sqrt{m}}\right),
\end{align}
where the last inequality holds because the $\Sigma_t$ is of rank $k$ and $V(t) = V_k(t)$.

Bring this to \eqref{eq:lip_con_sum}, use the fact that $\|{\mathbf{w}}_0 -{\mathbf{w}}^\star\| < B$,  with Jensen's inequality we have
\begin{equation}
\mathbb{E}\left[\hat L_n(\bar {\mathbf{w}})\right]  - \hat L_n({\mathbf{w}}^\star)  \leq \frac{B^2 + G^2 + k\sigma^2 }{2\sqrt{T}} + O\left(\frac{\Lambda G\kappa \ln p \gamma_{2}(\mathcal{W}, d)}{\sqrt{m}}\right).
\end{equation}
where $\Lambda = \sum_{t=1}^\intercal \frac{1}{\alpha_t}$.

With $\sigma^2 = \frac{G^2T}{n^2\epsilon^2}$,
let $T = n^2\epsilon^2$, we have
\begin{equation}
\mathbb{E}\left[\hat L_n(\bar {\mathbf{w}})\right]  - \hat L_n({\mathbf{w}}^\star)  \leq O\left(\frac{kG^2B^2}{n\epsilon}\right) + O\left(\frac{\Lambda G \kappa \ln p \gamma_{2}(\mathcal{W}, d)}{\sqrt{m}}\right).
\end{equation}
That completes the proof. \qed

\subsection{Uniform Convergence Bound}
\label{sec:app_ucproof}

Our proofs of Theorem \ref{theo:smc} heavily rely on the advanced probability tool, Generic Chaining (GC) \citep{tala14}. Typically the results in generic chaining are characterized by the so-called $\gamma_2$ function (see Definition \ref{def:gamma_2}). \cite{tala14} shows that for a process $
\left(X_{t}\right)_{t \in T}
$ and a given metric space $(T,d)$, if  $
\left(X_{t}\right)_{t \in T}
$
satisfies the increment condition
\begin{equation}  \label{eq:ic_x}
\forall u>0, \mathbb{P}\left(\left|X_{s}-X_{t}\right| \geq u\right) \leq 2 \exp \left(-\frac{u^{2}}{2 d(s, t)^{2}}\right),
\end{equation}
then the size of the process can be bounded as
\begin{equation}
\mathbb{E} \sup _{t \in T} X_{t} \leq c \gamma_2(T,d),
\end{equation}
with $c$ to be an absolute constant.

To apply the GC result to establish the Theorem \ref{theo:smc}, we treat  $\left\|M_{t}-\Sigma_{t}\right\|_{2}
$ as the process $X_t$ over the iterations. In detail, since $
M_{t}=\frac{1}{m} \sum_{i=1}^{m} \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right) \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right)^\intercal,
$
and
$
\Sigma_{t}=\mathbb{E}_{z \sim \mathcal{P}}\left[\nabla \ell\left(\mathbf{w}_{t}, z\right) \nabla \ell\left(\mathbf{w}_{t}, z\right)^\intercal \right]
$, the $\left\|M_{t}-\Sigma_{t}\right\|_{2}
$ is a random process indexed by ${\mathbf{w}}_t \in {\cal W}$, with ${\cal W}$ to be the set of all possible iterates obtained by the algorithm.

We first show that the variable $\left\|M_{t}-\Sigma_{t}\right\|_{2}
$ satisfies the increment condition as stated in \eqref{eq:ic_x} in Lemma \ref{lemm:ic_b}. Before we present the proof of Lemma \ref{lemm:ic_b}, we introduce the  Ahlswede-Winter Inequality \citep{horn2012matrix,wainwright2019high}, which will be used in the proof of Lemma \ref{lemm:ic_b}. Ahlswede-Winter Inequality shows that positive semi-definite random matrix with bounded spectral norm concentrates to its expectation with high probability.

\begin{theo} \label{theo:a_w_eq}
({\bf Ahlswede-Winter Inequality}) Let $Y$ be a random, symmetric, positive semi-definite $p \times p$ matrix. such that  such that $\|\mathbb{E}[Y]\| \leq 1$. Suppose $\|Y\| \leq R$ for some fixed scalar $R \geq 1$. Let $\{Y_1, \ldots, Y_m\}$ be independent copies of $Y$ (i.e., independently sampled matrices with the same distribution as $Y$). For any $u \in [0,1]$, we have
\begin{equation}
\mathbb{P}\left(\left\|\frac{1}{m} \sum_{i=1}^{m} Y_{i}-\mathbb{E}\left[Y_{i}\right]\right\|_2 >u  \right) \leq 2 p \cdot \exp \left(-m u^{2} / 4 R\right).
\end{equation}
\end{theo}

To make the argument clear, we use a more informative notation for $M_t$ and $\Sigma_t$.
Recall the notation of $M_t$ and $\Sigma_t$ such that
\begin{equation}
M_{t}=\frac{1}{m} \sum_{i=1}^{m} \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right) \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right)^\intercal,
\end{equation}
and
\begin{equation}
\Sigma_{t}=\mathbb{E}_{z \sim \mathcal{P}}\left[\nabla \ell\left(\mathbf{w}_{t}, z\right) \nabla \ell\left(\mathbf{w}_{t}, z\right)^\intercal \right],
\end{equation}

given the dataset $S_h =\{\tilde z_1,..,\tilde z_m\}$ and distribution ${\cal P}$ where $\tilde z_i \sim {\cal P}$ for $i\in [m]$, the $M_t$ and $\Sigma_t$
are functions of parameter ${\mathbf{w}}_t$, so we use $M({\mathbf{w}}_t)$ and $\Sigma({\mathbf{w}}_t)$ for $M_t$ and $\Sigma_t$ interchangeably in the rest of this section, i..e,
\begin{equation}
M({\mathbf{w}}) = \frac{1}{m} \sum_{i=1}^{m} \nabla \ell\left(\mathbf{w}, \tilde{z}_{i}\right) \nabla \ell\left(\mathbf{w}, \tilde{z}_{i}\right)^\intercal
\end{equation}
and
\begin{equation}
\Sigma({\mathbf{w}}) = \mathbb{E}_{z \sim \mathcal{P}}\left[\nabla \ell\left(\mathbf{w}, z\right) \nabla \ell\left(\mathbf{w}, z\right)^\intercal \right].
\end{equation}

\begin{lemma} \label{lemm:ic_b}
With Assumption \ref{asmp: zi_grad} and \ref{asmp: bounded_gradient} hold,
for any ${\mathbf{w}}, {\mathbf{w}}^\prime \in {\cal W}$ and $\forall u>0$, we have
\begin{align}
 \label{eq:ic_mg}
 \mathbb{P} \left(  \left\|M({\mathbf{w}}) -\Sigma({\mathbf{w}})\right\|_2- \left\|M({\mathbf{w}}^\prime) -\Sigma({\mathbf{w}}^\prime)\right\|_2 \geq \frac{u}{\sqrt{m}}\cdot 4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) \right) \leq  2 p \cdot \exp \left(- u^{2} / 4 \right),
\end{align}
where $ d: \mathcal{W} \times \mathcal{W} \mapsto \mathbb{R}$
 is the pseudo-metric in Assumption \ref{asmp: zi_grad}.
\end{lemma}

\noindent {\bf Proof:\ } We consider random variable
\begin{equation}
 X_i = \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p,
\end{equation}
where $\mathbb{I}_p \in \mathbb{R}^{p \times  p}$ is the identity matrix.

Note that $2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)$ is deterministic and the randomness of $X_i$ comes from $\nabla \ell({\mathbf{w}}, \tilde z_i)$ and $\nabla \ell({\mathbf{w}}^\prime, \tilde z_i)$.

By  triangle inequality and the construction of $X_i$, we have
\begin{align} \label{eq:bound_by_x}
 & \nonumber \left\|M({\mathbf{w}}) -\Sigma({\mathbf{w}})\right\|_2- \left\|M({\mathbf{w}}^\prime) -\Sigma({\mathbf{w}}^\prime)\right\|_2 \\
\nonumber & = \left\|M({\mathbf{w}}) -\mathbb{E}[M({\mathbf{w}})]\right\|_2- \left\|M({\mathbf{w}}^\prime) -\mathbb{E}[M({\mathbf{w}}^\prime)] \right\|_2 \\
\nonumber & \leq \left\| M({\mathbf{w}}) -\mathbb{E}[M({\mathbf{w}})] - \left( M({\mathbf{w}}^\prime) -\mathbb{E}[M({\mathbf{w}}^\prime)]\right) \right\|_2 \\
\nonumber & = \left\| M({\mathbf{w}}) - M({\mathbf{w}}^\prime) - \mathbb{E}[\left(M({\mathbf{w}})- M({\mathbf{w}}^\prime)\right)]
 \right\|_2 \\
\nonumber & = \left\| \frac{1}{m} \sum_{i=1}^m \left( \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal\right)  - \mathbb{E}\left[  \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \right]\right\|_2\\
\nonumber & = \bigg\| \frac{1}{m} \sum_{i=1}^m \left( \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p \right) \\  \nonumber
& \ \ \ \ \ \  - \mathbb{E}\left[  \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p \right] \bigg\|_2 \\
& = \left\| \frac{1}{m}\sum X_i - \mathbb{E}[X_i]  \right\|_2
\end{align}

To apply Theorem \ref{theo:a_w_eq} for $\left\| \frac{1}{m}\sum X_i - \mathbb{E}[X_i]  \right\|_2 $, we first show that the random symmetric matrix $X_i$ is positive semi-definite.

By Assumption \ref{asmp: bounded_gradient} and Assumption \ref{asmp: zi_grad}
and definition
\begin{align}
& \nonumber\left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal-\nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal\right\|_{2} \\
\nonumber & = \sup_{
\mathbf{x}:\|\mathbf{x}\|=1} \mathbf{x}^\intercal \left(\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal-\nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal
\right)\mathbf{x} \\ \nonumber
& = \sup_{
\mathbf{x}:\|\mathbf{x}\|=1}  \mathbf{x}^\intercal \nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal \mathbf{x} - \mathbf{x}^\intercal \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \mathbf{x} \\ \nonumber
 & = \sup_{
\mathbf{x}:\|\mathbf{x}\|=1} \langle \mathbf{x}, \nabla \ell({\mathbf{w}}, \tilde z_i) \rangle^2 - \langle \mathbf{x}, \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \rangle^2 \\ \nonumber
& = \sup_{
\mathbf{x}:\|\mathbf{x}\|=1} \left(
\langle \mathbf{x}, \nabla \ell({\mathbf{w}}, \tilde z_i) \rangle + \langle \mathbf{x}, \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \rangle \right)\left(
\langle \mathbf{x}, \nabla \ell({\mathbf{w}}, \tilde z_i) \rangle - \langle \mathbf{x}, \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \rangle \right) \\ \nonumber
& \leq \sup_{
\mathbf{x}:\|\mathbf{x}\|=1} 2G \left(
\langle \mathbf{x}, \nabla \ell({\mathbf{w}}, \tilde z_i) \rangle - \langle \mathbf{x}, \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \rangle \right)  \\ \nonumber
& = 2G \| \nabla \ell({\mathbf{w}}, \tilde z_i) - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)\|_2 \\
& \leq 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)
\label{eq:sp_to_l2}
\end{align}

For any non-zero vector  $\mathbf{x} \in \mathbb{R}^{p}$, we have
\begin{align}
\nonumber \mathbf{x}^\intercal X_i \mathbf{x} & = \mathbf{x}^\intercal \left(\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal +2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p \right) \mathbf{x} \\
\nonumber & = \mathbf{x}^\intercal \left(\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \right) \mathbf{x} + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) \|\mathbf{x} \|_2^2 \\
\nonumber & \geq - \left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \right\|_2\|\mathbf{x} \|_2^2 + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) \|\mathbf{x} \|_2^2 \\
\nonumber & = \left(2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) - \left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \right\|_2\right)\|\mathbf{x} \|_2^2 \\
& \overset{(a)}{\geq} 0,
\end{align}
where (a) is true because $ \left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal \right\|_2 \leq 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)$ as shown in \eqref{eq:sp_to_l2}.

Let $Y_i = \frac{X_i}{4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)}$, with \eqref{eq:sp_to_l2}, we have
\begin{align}
  \nonumber  \| Y_i \|_2 & = \frac{ \left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p\right\|_2 }{4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) }\\
\nonumber    & \leq \frac{\left\|\nabla \ell({\mathbf{w}}, \tilde z_i) \nabla \ell({\mathbf{w}}, \tilde z_i)^\intercal - \nabla \ell({\mathbf{w}}^\prime, \tilde z_i) \nabla \ell({\mathbf{w}}^\prime, \tilde z_i)^\intercal\right\|_2 + \left\|2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)\mathbb{I}_p\right\|_2}{4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)} \\
\nonumber    & \leq \frac{2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) + 2G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)}{4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)}\\
    & = 1.
\end{align}
So that $\| Y_i \|_2 \leq 1$ and $\|\mathbb{E}[Y_i]\|_2 \leq 1$. Then, from Theorem \ref{theo:a_w_eq}, with $R = 1$, we have for any $u \in [0,1]$
\begin{align}
\mathbb{P}\left(\left\|\frac{1}{m} \sum_{i=1}^{m} Y_{i}-\mathrm{E}\left[Y_{i}\right]\right\|> u \right) \leq 2 p \cdot \exp \left(- m u^{2} / 4 \right).
\end{align}
Note that $\left\|\frac{1}{m} \sum_{i=1}^{m} Y_{i}-\mathrm{E}\left[Y_{i}\right]\right\|$ is always bounded by $1$ since $\| Y_i \|_2 \leq 1$ and $\|\mathbb{E}[Y_i]\|_2 \leq 1$. So the above inequality holds for any $u >1$ with probability 0 which is bounded by $ 2 p \cdot \exp \left(- m u^{2} / 4 \right)$. So that we have for any $u > 0$,
\begin{align}
\mathbb{P}\left(\left\|\frac{1}{m} \sum_{i=1}^{m} Y_{i}-\mathrm{E}\left[Y_{i}\right]\right\|> u \right) \leq 2 p \cdot \exp \left(- m u^{2} / 4 \right).
\end{align}
So that for any $u > 0$,
\begin{align} \label{eq:aw2}
\mathbb{P}\left(\left\|\frac{1}{m} \sum_{i=1}^{m} X_{i}-\mathrm{E}\left[X_{i}\right]\right\|> \frac{u}{\sqrt{m}}\cdot 4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)  \right) \leq 2 p \cdot \exp \left(- u^{2} / 4 \right).
\end{align}

Combine \eqref{eq:aw2} and \eqref{eq:bound_by_x}, we have
\begin{align}
\nonumber \mathbb{P} & \left(  \left\|M({\mathbf{w}}) -\Sigma({\mathbf{w}})\right\|_2- \left\|M({\mathbf{w}}^\prime) -\Sigma({\mathbf{w}}^\prime)\right\|_2 \geq \frac{u}{\sqrt{m}}\cdot 4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) \right) \\
\nonumber & = \mathbb{P}  \left(   \left\|M({\mathbf{w}}) -\mathbb{E}[M({\mathbf{w}})]\right\|_2- \left\|M({\mathbf{w}}^\prime) -\mathbb{E}[M({\mathbf{w}}^\prime)] \right\|_2 > \frac{u}{\sqrt{m}}\cdot 4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime) \right) \\
\nonumber  & \leq \mathbb{P}\left(\left\|\frac{1}{m} \sum_{i=1}^{m} X_{i}-\mathrm{E}\left[X_{i}\right]\right\|> \frac{u}{\sqrt{m}}\cdot 4G \kappa d({\mathbf{w}}, {\mathbf{w}}^\prime)  \right) \\
 & \leq 2 p \cdot \exp \left(- u^{2} / 4 \right)
\end{align}

That completes the proof. \nolinebreak[1]~~~\hspace*{\fill} \rule{5pt}{5pt}\vspace*{\parskip}\vspace*{1ex}

Based on the above result, now we come to the proof of Theorem \ref{theo:smc}. The proof follows the Generic Chaining argument, i.e., Chapter 2 of \cite{tala14}.

\theosmc*

\noindent {\bf Proof:\ } Note that equation \eqref{eq:exp_uc_sm} is a uniform bound over iteration $t \in [T]$. To bound $
\sup _{t \in[T]}\left\|M_{t}-\Sigma_{t}\right\|_{2}
$, it is sufficient to bound
\begin{align}
     \sup_{{\mathbf{w}} \in {\cal W}} \left\| M({\mathbf{w}}) - \Sigma({\mathbf{w}}) \right\|_2,
\end{align}
where ${\cal W}$ contains all the possible trajectorys of  ${\mathbf{w}}_1, ..., {\mathbf{w}}_T$.

We consider a sequence of subsets ${\cal W}_n$ of ${\cal W}$, and
\begin{equation}
\operatorname{card} {\cal W}_{n} \leq N_{n},
\end{equation}
where $
N_{0}=1 ; N_{n}=2^{2^{n}} \text { if } n \geq 1.
$

Let $\pi_{n}({\mathbf{w}}) \in {\cal W}_n$ be the approximation of any ${\mathbf{w}} \in {\cal W}$.
We decompose the $\|M({\mathbf{w}}) -\Sigma({\mathbf{w}})\|_2$ as
\begin{align} \label{eq:decom}
 \nonumber &\| M({\mathbf{w}}) -\Sigma({\mathbf{w}})\|_2 - \|M(\pi_{0}({\mathbf{w}})) -\Sigma(\pi_{0}({\mathbf{w}}))\|_2 \\ &=\sum_{n \geq 1}\left(\left\|M(\pi_{n}({\mathbf{w}})) -\Sigma(\pi_{n}({\mathbf{w}}))\right\|_2 - \|M(\pi_{n-1}({\mathbf{w}})) -\Sigma(\pi_{n-1}({\mathbf{w}}))\|_2 \right),
\end{align}
which holds since $
\pi_{n}({\mathbf{w}})= {\mathbf{w}}$ for $n$ large enough.

Based on Lemma \ref{lemm:ic_b}, for any $u > 0$, we have
\begin{align} \label{eq:mark}
 \left\|M(\pi_{n}({\mathbf{w}})) -\Sigma(\pi_{n}({\mathbf{w}}))\right\|_2 - \|M(\pi_{n-1}({\mathbf{w}})) -\Sigma(\pi_{n-1}({\mathbf{w}}))\|_2 \geq
\frac{u}{\sqrt{m}}\cdot 4G \kappa d(\pi_{n}({\mathbf{w}}),\pi_{n-1}({\mathbf{w}})),
\end{align}
with probability at most $2 p\exp(-\frac{u^2}{4})$.

For any $n > 0$ and ${\mathbf{w}} \in {\cal W}$, the number of possible pairs $
\left(\pi_{n}({\mathbf{w}}), \pi_{n-1}({\mathbf{w}})\right)$ is
\begin{equation}
\operatorname{card} {\cal W}_{n} \cdot \operatorname{card} {\cal W}_{n-1} \leq N_{n} N_{n-1} \leq N_{n+1}=2^{2^{n+1}}.
\end{equation}

Apply union bound over all the possible pairs of $(\pi_{n}({\mathbf{w}}), \pi_{n-1}({\mathbf{w}}))$, following  \cite{tala14} (Chapter 2.2),
for any $n > 0$, $u >0$, and ${\mathbf{w}} \in {\cal W}$, we have
\begin{equation}
     \left\|M(\pi_{n}({\mathbf{w}})) -\Sigma(\pi_{n}({\mathbf{w}}))\right\|_2 - \|M(\pi_{n-1}({\mathbf{w}})) -\Sigma(\pi_{n-1}({\mathbf{w}}))\|_2  \geq u 2^{n/2}d(\pi_{n}({\mathbf{w}}),\pi_{n-1}({\mathbf{w}})) \cdot  \frac{4G \kappa}{\sqrt{m}}
\end{equation}
with probability
\begin{equation}
 \sum_{n \geq 1} 2p \cdot 2^{2^{n+1}} \exp \left(-u^{2}2^{n-2} \right) \leq
c^\prime p \exp \left(-\frac{u^{2}}{4}\right),
\end{equation}
where $c^\prime$ is a universal constant.

Then we have
\begin{align} \label{eq:decom_hprob}
\sum_{n \geq 1}&\left(\left\|M(\pi_{n}({\mathbf{w}})) -\Sigma(\pi_{n}({\mathbf{w}}))\right\|_2 - \|M(\pi_{n-1}({\mathbf{w}})) -\Sigma(\pi_{n-1}({\mathbf{w}}))\|_2 \right) \nonumber \\
\nonumber &\geq
\sum_{n \geq 1} u 2^{n/2}d(\pi_{n}({\mathbf{w}}),\pi_{n-1}({\mathbf{w}})) \cdot  \frac{4G \kappa}{\sqrt{m}}\\
& \geq \sum_{n \geq 0} u 2^{n/2}d({\mathbf{w}}, {\cal W}_n) \cdot  \frac{4G \kappa}{\sqrt{m}}
\end{align}
with probability at most $c^\prime p \exp \left(-\frac{u^{2}}{4}\right)$.

From Theorem \ref{theo:a_w_eq}, let $Y_i = \frac{\nabla \ell(\pi_{0}({\mathbf{w}}),\tilde z_i)\nabla \ell(\pi_{0}({\mathbf{w}}),\tilde z_i)^\intercal}{G}$, so that $\| Y_i\| \leq 1$ and $\| \mathbb{E}[Y_i]\| \leq 1$. Then we have
\begin{align} \label{ep:pi_0}
    \|M(\pi_{0}({\mathbf{w}})) -\Sigma(\pi_{0}({\mathbf{w}}))\|_2 \geq u\frac{G}{\sqrt{m}},
\end{align}
with ptobability at most $2p \exp \left(-\frac{u^{2}}{4}\right)$.

Combine \eqref{eq:decom}, \eqref{eq:decom_hprob} and \eqref{ep:pi_0}, we have
\begin{align} \label{eq:smc_cons}
 \sup_{{\mathbf{w}} \in {\cal W}}\| M({\mathbf{w}}) -\Sigma({\mathbf{w}})\|_2  & \geq  \sup_{{\mathbf{w}} \in {\cal W}} \sum_{n \geq 0} u 2^{n/2}d({\mathbf{w}}, {\cal W}_n) \cdot  \frac{4G \kappa}{\sqrt{m}} + u \frac{G}{\sqrt{m}}\nonumber \\
 & = u\left(\frac{G\left(4\kappa\gamma_2({\cal W}, d) +1\right)}{\sqrt{m}}

 \right)
\end{align}
with probability at most $(c^\prime+2) p \exp \left(-\frac{u^{2}}{4}\right)$.

That completes the proof. \nolinebreak[1]~~~\hspace*{\fill} \rule{5pt}{5pt}\vspace*{\parskip}\vspace*{1ex}

Now we provide the proof of Theorem \ref{theo:sc}.

\theosubc*

\noindent {\bf Proof:\ }
Recall that $\hat V_k(t)$ is the top-$k$ eigenspace of $M_t$. Let $V_k(t)$ be the top-$k$ eigenspace of $\Sigma_t$.
\begin{align}
    \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal & = \Pi_{M_t}^{(k)}(\I - \Pi_{\Sigma_t}^{(k)}) + (\I - \Pi_{M_t}^{(k)}) \Pi_{\Sigma_t}^{(k)}~ \\
\Rightarrow \qquad \| \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal \|_2 & \leq \|\Pi_{M_t}^{(k)}(\I - \Pi_{\Sigma_t}^{(k)}) \|_2 + \| (\I - \Pi_{M_t}^{(k)}) \Pi_{\Sigma_t}^{(k)} \|_2~.
\end{align}
where $\Pi_{M_t}^{(k)} = \hat{V}_k(t) \hat{V}_k(t)^\intercal$ denotes the projection to the top-$k$ subspace of the symmetric PSD $M_t$ and $\Pi_{\Sigma_t}^{(k)} = V_k(t)V_k(t)^\intercal$ denotes the projection to the top-$k$ subspace of the symmetric PSD $\Sigma_t$. Then, from Davis-Kahan (Corollary 8 in \cite{mcsherry2004spectral}) and using the fact for symmetric PSD matrices eigen-values and singular values are the same, we have
\begin{align}
    \|\Pi_{M_t}^{(k)}(\I - \Pi_{\Sigma_t}^{(k)}) \|_2 & \leq \frac{\| M_t - \Sigma_t \|_2}{\lambda_k(M_t) - \lambda_{k+1}(\Sigma_t)} \label{eq:dkab1} \\
    \| (\I - \Pi_{M_t}^{(k)}) \Pi_{\Sigma_t}^{(k)} \|_2 & \leq \frac{\| M_t - \Sigma_t \|_2}{\lambda_k(\Sigma_t) - \lambda_{k+1}(M_t)}~. \label{eq:dkba1}
\end{align}
Recall, from \cite{horn2012matrix} (Section 4.3) and \cite{GoluVanl96} (Section 8.1.2), e.g., Corollary 8.1.6, we have
\begin{equation}
    |\lambda_k(M_t) - \lambda_k(\Sigma_t)| \leq \| M_t - \Sigma_t \|_2~.
    \label{eq:hjdk}
\end{equation}
From Theorem \ref{theo:smc} and Lemma \ref{lemm:hb_exp}, with $Y = \| M_t - \Sigma_t \|_2$, $A = c$, and $B = \frac{4 G \kappa \sqrt{\ln p} \gamma_{2}(\mathcal{W}, d)}{\sqrt{m}}$, we have
\begin{equation}
    \mathbb{E}\left[ \| M_t - \Sigma_t \|_2\right] \leq O\left(\frac{G \kappa \sqrt{\ln p} \gamma_{2}(\mathcal{W}, d)}{\sqrt{m}}\right).
    \label{eq:exp_smc}
\end{equation}

Let $c_0 =  O\left(G \kappa \sqrt{\ln p} \gamma_{2}(\mathcal{W}, d)\right)$. For
$m \geq\frac{c_0^2}{\alpha_t^2}$, we have
\begin{equation}
\mathbb{E}\left[ \| M_t - \Sigma_t \|_2\right] \leq \frac{c_0}{\sqrt{m}} \leq \frac{\alpha_t}{2}~.
    \label{eq:specdiff}
\end{equation}

Then, for \eqref{eq:dkab1}, we have
\begin{align}
\nonumber \|\Pi_{M_t}^{(k)}(\I - \Pi_{\Sigma_t}^{(k)}) \|_2 & \leq \frac{\| M_t - \Sigma_t \|_2}{\lambda_k(M_t) - \lambda_{k+1}(\Sigma_t)} \\
\nonumber  & = \frac{\| M_t - \Sigma_t \|_2}{(\lambda_k(\Sigma_t) - \lambda_{k+1}(\Sigma_t)) - (\lambda_k(\Sigma_t) - \lambda_k(M_t))} \\
 & \leq \frac{\| M_t - \Sigma_t \|_2}{\alpha_t - \| M_t - \Sigma_t \|_2}.

\end{align}

Then, for \eqref{eq:dkba1}, we have
\begin{align}
\nonumber \| (\I - \Pi_{M_t}^{(k)}) \Pi_{\Sigma_t}^{(k)} \|_2 & \leq \frac{\| M_t - \Sigma_t \|_2}{\lambda_k(\Sigma_t) - \lambda_{k+1}(M_t)} \\
\nonumber & = \frac{\| M_t - \Sigma_t \|_2}{(\lambda_k(\Sigma_t) - \lambda_{k+1}(\Sigma_t)) + (\lambda_{k+1}(\Sigma_t) - \lambda_{k+1}(M_t))} \\
 & \leq \frac{\| M_t - \Sigma_t \|_2}{\alpha_t - \| M_t - \Sigma_t \|_2}.
 \end{align}

Combining these two bounds and \eqref{eq:exp_smc}, with $c_0 =  O\left(G \kappa \sqrt{\ln p} \gamma_{2}(\mathcal{W}, d)\right)$, we have
\begin{align}
\mathbb{E}\left[\| \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal \|_2 \right]
&\leq \frac{2 \alpha_t}{\alpha_t - \mathbb{E}\left[\| M_t - \Sigma_t \|_2\right] } -2 \nonumber \\
& \leq \frac{\frac{2c_0}{\sqrt{m}}}{\alpha_t -\frac{c_0}{\sqrt{m}}}
\end{align}
Using \eqref{eq:specdiff} such that $\frac{c_0}{\sqrt{m}} \leq \frac{\alpha_t}{2}$, we have

\begin{equation}
\mathbb{E}\left[\| \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal \|_2 \right] \leq O\left(\frac{G \kappa \sqrt{\ln p} \gamma_{2}(\mathcal{W}, d)}{\alpha_t \sqrt{m}}\right).
\end{equation}

That completes the proof. \nolinebreak[1]~~~\hspace*{\fill} \rule{5pt}{5pt}\vspace*{\parskip}\vspace*{1ex}

\begin{lemma}
[Lemma 2.2.3 in \cite{tala14}]\label{lemm:hb_exp}
Consider a r.v. $Y \geq 0$ which satisfies
\begin{equation}
\forall u>0, \mathbb{P}(Y \geq u) \leq A \exp \left(-\frac{u^{2}}{B^{2}}\right)
\end{equation}
for certain numbers $A \geq 2$ and $B > 0$. Then
\begin{equation}
\mathbb{E} Y \leq C B \sqrt{\log A},
\end{equation}
where $C$ denotes a universal constant.
\end{lemma}

\subsection{Geometry of Gradients and $\gamma_2$ functions.}
\label{sec:app_gamma2}

In this section, we provide more intuitions and explanations of $\gamma_2$ functions. We justify our assumptions about the gradient space and provide more examples of the gradient space structure and the corresponding $\gamma_2$ functions.

At a high level, for a metric space $(M,d)$, $\gamma_2(M,d)$ is related to $\sqrt{\log N(M,d,\epsilon)}$ where $N(M,d,\epsilon)$ is the covering number of $M$ with $\epsilon$ balls with metric $d$, but it is considerably sharper. Such sharpening has happened in two stages in the literature: first, based on {\em chaining}, which considers an integral over all $\epsilon$ yielding the {\em Dudley bound}, and subsequently, based on {\em generic chaining}, which considers a hierarchical covering, developed by Talagrand and colleagues, and which yields the sharpest bounds of this type. The official perspective of generic chaining is to view $\gamma_2(M,d)$ as an upper (and lower) bound on suprema of Gaussian processes indexed on $M$ and with metric $d$ [Theorem 2.4.1 in \cite{tala14}].

Considering $d$ to be the $\ell_2$ norm distance, $\gamma_2(M,d)$ will be the same order as the Gaussian width of $M$ \citep{vershynin2018}, which is a scaled version of the mean width of $M$. Structured sets (of gradients) have small Gaussian widths, e.g., a $L_1$ unit ball in $\mathbb{R}^p$ has a Gaussian width of $O(\sqrt{\log p}$, wheras a $L_2$ unit ball in $\mathbb{R}^p$ has a Gaussian width of $O(\sqrt{p})$.

To utilize the structure of gradients as example shown in Figure \ref{fig:mnist_grad}, instead of focusing on the $\gamma_2({\cal W},d)$, we can also derive the uniform convergence bound using the measurement $d$ on the set of population gradient $\boldsymbol{\mu} = \mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$. We consider a mapping $f: {\cal W} \mapsto {\cal M}$ from the parameter space ${\cal W}$ to the gradient space ${\cal M}$,  where $f$ can be considered as $f({\mathbf{w}}) =\mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$. With  $\boldsymbol{\mu}_t = \mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$ and ${\cal M}$ to be the space of the population gradient $\boldsymbol{\mu}_t \in {\cal M}$, the pseudo metric $d({\mathbf{w}}, {\mathbf{w}}^\prime)$ can be written as  $d({\mathbf{w}}, {\mathbf{w}}^\prime) = d(f({\mathbf{w}}), f({\mathbf{w}}^\prime)) = d(\boldsymbol{\mu}, \boldsymbol{\mu}^\prime)$ with $\boldsymbol{\mu} = \mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$ and $\boldsymbol{\mu}^\prime = \mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}^\prime, z)\right]$. With such a mapping $f$, the admissible sequence $\Gamma_{{\cal W}} = \{{\cal W}_n: n \geq 0\}$ of ${\cal W}$ in the proof of Theorem \ref{theo:smc} corresponds to the admissible sequence $\Gamma_{{\cal M}} = \{{\cal M}_n: n \geq 0\}$ of ${\cal M}$. The $\gamma_2({\cal W},d) = \inf_{\Gamma_{{\cal W}}} \sup _{{\mathbf{w}} \in {\cal W}} \sum_{n \geq 0} 2^{n / 2} d\left({\mathbf{w}}, {\cal W}_{n}\right) = \inf_{\Gamma_{{\cal M}}} \sup _{\boldsymbol{\mu} \in {\cal M}} \sum_{n \geq 0} 2^{n / 2} d\left(\boldsymbol{\mu}, {\cal M}_{n}\right) = \gamma_2({\cal M}, d)$.
Considering $d(\boldsymbol{\mu}, \boldsymbol{\mu}^\prime) = \|\boldsymbol{\mu} -\boldsymbol{\mu}^\prime \|_2$,  the $\gamma_2({\cal M}, d)$ will be the same order as the Gaussian width of ${\cal M}$, i.e., $w({\cal M}) = \mathbb{E}_{\mathbf{v}}\left[\sup _{\mathbf{m} \in \mathcal{M}}\langle\mathbf{m}, \mathbf{v}\rangle\right]$, where $\mathbf{v} \sim N\left(0, \mathbb{I}_{p}\right)$. Below, we provide more examples of the gradient space structure and the $\gamma_2$ functions.

{\bf Ellipsoid.} The Gaussian width $w({\cal M})$ depends on the structure of the gradient $\boldsymbol{\mu}$. In Figure \ref{fig:mnist_grad}, we observe that, for each coordinates of the gradient is of small value
along the training trajectory and thus ${\cal M}$ includes all gradients living in an ellipsoid, i.e., ${\cal M} = \{ \boldsymbol{\mu}_t \in \R^p \mid \sum_{j=1}^p \boldsymbol{\mu}_t(j)^2/\mathbf{e}(j)^2 \leq 1, \mathbf{e} \in \R^p\}$. Then we have $\gamma_2({\cal M}, d) \leq c_1 w({\cal M}) \leq c_2 \| \mathbf{e} \|_2$~\cite{tala14}, where $c_1$ and $c_2$ are absolute constants. If the elements of $\mathbf{e}$ sorted in decreasing order satisfy $\mathbf{e}(j) \leq c_3/\sqrt{j}$
for all $j \in [p]$, then $\gamma_2({\cal M},d) \leq O\left( \sqrt{\log p} \right)$.

{\bf Composition.} Based on the composition properties of $\gamma_2$ functions~\cite{tala14}, one can construct additional examples of the gradient spaces.  If ${\cal M} = {\cal M}_1 + {\cal M}_2 = \{ \boldsymbol{\mu}_1 + \boldsymbol{\mu}_2, \boldsymbol{\mu}_1 \in {\cal M}_1, \boldsymbol{\mu}_2 \in {\cal M}_2\}$, the Minkowski sum, then $\gamma_2({\cal M},d) \leq c (\gamma_2({\cal M}_1,d) + \gamma_2({\cal M}_2,d))$ (Theorem 2.4.15 in \cite{tala14}), where $c$ is an absolute constant. If ${\cal M}$ is a union of several subset, i.e., ${\cal M} = \cup_{h=1}^D M_{h}$, then by using an union bound on Theorem \ref{theo:smc}, we have $\gamma_2({\cal M},d) \leq \sqrt{\log D} \max_{h} \gamma_2({\cal M}_{h},d)$. Thus, if  ${\cal M}$ is an union of $D = p^s$ ellipsoids, i.e., polynomial in $p$, then $\gamma_2({\cal M},\| \cdot \|_2) \leq O\left( \sqrt{s}\log p\right)$.

\section{Introduction}
\label{sec:intro}
\input{sec1_intro}

\section{Preliminaries}
\label{sec: pre}
\input{sec2_pre}

\section{Projected Private Gradient Descent}
\label{sec: algorithm}
\input{sec3_hold_out_alg}

\section{Experiments}
\label{sec: exp}
\input{exp_v2}

\section{Conclusion and Future Work}
\label{sec: con}
While differentially-private stochastic gradient descent (DP-SGD) algorithms and variants have been well studied for solving differentially private empirical risk minimization (ERM), the error rate of DP-SGD has a dependence on the ambient dimension $p$. In this paper, we aim at bypassing such dependence by leveraging a special structure of gradient space i.e., the stochastic gradients for deep nets usually stay in a low dimensional subspace in the training process.
We propose PDP-SGD which projects the noisy gradient to an approximated subspace evaluated on a public dataset. We show that the subspace reconstruction error is small and PDP-SGD reduces the $p$ factor in the error rate to the projection dimension. We evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of PDP-SGD over DP SGD.

There are several interesting directions for future work. First, it will be interesting to provide an private optimization method that can identify the  gradient subspace at each iteration without access to a public dataset. The existing "analyze-Gauss" techniques in \cite{DworkTT014} will give a bound with reconstruction error scaling with $\sqrt{p}$, which will be propagated to the optimization error.  More recently, \cite{song2020} show that, for a class
of generalized linear problems, DP gradient descent (without any projection) achieves an excess empirical risk bound depending on the rank of the feature matrix instead of the ambient dimension. It will also be interesting to explore whether their rank-dependent bounds hold for a more general class of problems.
Finally, a question that applies both to private and non-private optimization is to characterize the class of optimization problems with low-dimensional gradient subspaces.

\section*{Acknowledgement}
The research was supported by NSF grants IIS-1908104, OAC-1934634, IIS-1563950, a Google Faculty Research Award, a J.P. Morgan Faculty Award, and a Mozilla research grant. We would like to thank the Minnesota Super-computing Institute (MSI) for providing computational resources and support.

\bibliographystyle{abbrvnat}

\subsection{Gradient Subspace Identification}
\label{sec:grad_sub_id}
\vspace{-2mm}

We now analyze the gradient deviation between the  approximated subspace $\hat V_k(t) \hat V_k(t)^\intercal$ and true (population) subspace $ V_k(t) V_k(t)^\intercal$, i.e., $\|\hat V_k(t) \hat V_k(t)^\intercal - V_k(t) V_k(t)^\intercal\|_2$. To bound $\|\hat V_k(t) \hat V_k(t)^\intercal - V_k(t) V_k(t)^\intercal\|_2$, we first bound the deviation between second moment matrix $\| M_t -\Sigma_t \|$~\citep{DworkTT014,mcsherry2004spectral}. Note that, if $M_{t}=\frac{1}{m} \sum_{i=1}^{m} \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right) \nabla \ell\left(\mathbf{w}_{t}, \tilde{z}_{i}\right)^\intercal$
is evaluated on \emph{fresh} public samples, the $\Sigma_t$ is the expectation of $M_t$, and the deviation of $M_t$ from $\Sigma_t$ can be easily analyzed by the Ahlswede-Winter Inequality \citep{horn2012matrix,wainwright2019high},
i.e.,  at any iteration $t$, if we have fresh public sample $\{\tilde z_1(t), ..., \tilde z_m(t)\}$ drawn i.i.d. from the distribution ${\cal P}$, with suitable assumptions we have, $\left\| \frac{1}{m}\sum_{i=1}^m \nabla \ell({\mathbf{w}}_t, \tilde z_i(t))\nabla \ell({\mathbf{w}}_t, \tilde z_i(t))^\intercal- \mathbb{E}\left[ \nabla\ell({\mathbf{w}}_t, \tilde z_i(t))\nabla \ell({\mathbf{w}}_t, \tilde z_i(t))^\intercal\right]\right\|_2 > u, $
$\forall u \in [0,1]$,
with probability at most $p\exp (-m u^2/4G)$ and $G$ is as in Assumption \ref{asmp: bounded_gradient}.

However, this concentration bound does not hold for ${\mathbf{w}}_t, \forall t> 0$ in general, since the public dataset $S_h$ is \emph{reused} over the iterations and the parameter ${\mathbf{w}}_t$ depends on $S_h$. To handle the dependency issue, we bound  $\| M_t -\Sigma_t \|_2$ uniformly over all iterations $t \in [T]$ to bound the worst-case counterparts that consider all possible iterates. Our uniform bound analysis is based on generic chaining (GC)~\citep{tala14}, an advanced tool from probability theory.
Eventually, the error bound is expressed  in terms of a complexity measure called $\gamma_2$ function \citep{tala14}. Note that one may consider the idea of sample splitting to bypass the dependency issue by splitting $m$ public samples into $T$ disjoint subsets for each iteration. Based on Ahlswede-Winter Inequality, the deviation error scales with $O(\frac{\sqrt{T}}{\sqrt{m}})$ leading to a worse trade-off between the subspace construction error and optimization error due to the dependence on $T$.

\begin{figure*}[t]
\centering
  \subfigure[DPSGD, $\sigma = 1.0$]{
 \includegraphics[trim =3mm 1mm 4mm 4mm, clip, width=0.30\textwidth]{top_mt_no1.png}
 }
  \subfigure[DPSGD, $\sigma = 2.0$]{

 \includegraphics[trim = 3mm 1mm 4mm 4mm, clip, width=0.30\textwidth]{mt_top_log_no2.png}
 }
  \subfigure[DP-SGD, $\sigma = 4.0$]{

 \includegraphics[trim = 3mm 1mm 4mm 4mm, clip, width=0.30\textwidth]{mt_top_log_no4.png}
 }  \vspace{-2mm}
\caption[]{Sorted components of population gradients for DP-SGD with $\sigma = 1.0,~ 2.0,~ 4.0$. Dataset: MNIST; model: 2-layer ReLU with 128 nodes each layer. The network has roughly 130,000 parameters. Y-axis is the absolute value of sorted gradient coordinates, i.e., $|\boldsymbol{\mu}_{t}(j)|$, X-axis is the order of sorted gradient component.}
\label{fig:mnist_grad}
\vspace{-2mm}
\end{figure*}

\vspace{-2mm}
\begin{defn}
[$\gamma_2$ function \citep{tala14}] \label{def:gamma_2} For a metric space $({\cal A},d)$, an admissible sequence of ${\cal A}$ is a collection of subsets of ${\cal A}$, $\Gamma = \{{\cal A}_n : n \geq 0\}$, with $|{\cal A}_0| = 1$ and $|{\cal A}_n| \leq 2^{2^n}$ for all $n \geq 1$, the $\gamma_{2}$ functional is defined by  $\gamma_{2}({\cal A},d) = \inf_\Gamma \sup_{A \in {\cal A}} \sum_{n \geq 0} 2^{n/2} d(A,{\cal A}_n)~,$
where the infimum is over all admissible sequences of ${\cal A}$.
\end{defn}
\vspace{-2mm}

In Theorem \ref{theo:smc}, we show that the uniform convergence bound of $\| M_t - \Sigma_t\|_2$ scales with $\gamma_2({\cal W},d)$, where $d$ is the pseudo metric as in Assumption \ref{asmp: zi_grad} and ${\cal W} \in \mathbb{R}^p$ is the set that contains all possible iterates in the algorithm, i.e., ${\mathbf{w}}_t \in {\cal W}$ for all $t \in [T]$.

Based on the majorizing measure theorem (e.g., Theorem 2.4.1 in \cite{tala14}), if the metric $d$ is $\ell_2$-norm,  $\gamma_2({\cal W},d)$ can be expressed as Gaussian width \citep{vershynin2018,wainwright2019high} of the set ${\cal W}$, i.e.,  $w({\cal W}) = \E_{\v}[\sup_{{\mathbf{w}} \in {\cal W}} \langle {\mathbf{w}},\v \rangle]$ where $\v \sim {\cal N}(0,\I_{p})$, which only depends on the size of the ${\cal W}$.
In Appendix \ref{sec:app_gamma2}, we show the complexity measure $\gamma_2({\cal W},d)$ can be expressed as the $\gamma_2$ function measure on the gradient space by mapping the parameter space ${\cal W}$ to the gradient space, i.e., $f: {\cal W} \mapsto {\cal M}$, where $f$ can be considered as $f({\mathbf{w}}) =\mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$. To simplify the notation, we write $\boldsymbol{\mu} = \mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$ as the population gradient at ${\mathbf{w}}$ and ${\cal M}$ as the space of the population gradient. Considering $d(\boldsymbol{\mu}, \boldsymbol{\mu}^\prime) = \|\boldsymbol{\mu} -\boldsymbol{\mu}^\prime \|_2$ for $\boldsymbol{\mu}, \boldsymbol{\mu}^\prime \in {\cal M}$,  $\gamma_2({\cal M}, d)$
will be the same order as the Gaussian width $w({\cal M})$.

To measure the value of $\gamma_2({\cal M}, d)$, we empirically explore the gradient space ${\cal M}$ for deep nets.  Figure \ref{fig:mnist_grad} gives an example of the population gradient along the training trajectory of DP-SGD with $\sigma = \{1, 2, 4\}$ for training a 2-layer ReLU on MNIST dataset. Figure \ref{fig:mnist_grad} shows that each coordinate of the gradient is of small value and gradient components decay very fast \citep{li2021experiments}. Thus, it is fair that the gradient space ${\cal M}$ is a union of ellipsoids, i.e., there exists $\mathbf{e} \in \mathbb{R}^p$ such that ${\cal M} = \{ \boldsymbol{\mu} \in \R^p \mid \sum_{j=1}^p \boldsymbol{\mu}(j)^2/\mathbf{e}(j)^2 \leq 1, \mathbf{e} \in \R^p\}$, where $j$ denotes the $j$-th coordinate. Then we have $\gamma_2({\cal M}, d) \leq c_1 w({\cal M}) \leq c_2 \| \mathbf{e} \|_2$~\citep{tala14}, where $c_1$ and $c_2$ are absolute constants. If the elements of $\mathbf{e}$ are sorted in a decreasing order satisfy $\mathbf{e}(j) \leq c_3/\sqrt{j}$
for all $j \in [p]$, then $\gamma_2({\cal M},d) \leq O\left( \sqrt{\log p} \right)$.\footnote{In the appendix, we provide more examples ${\cal M}$ that are consistent with empirical observations of stochastic gradient distributions and have small $\gamma_2({\cal M},d)$.} Now we give the uniform convergence bound of $\|M_t -\Sigma_t \|_2$.

\vspace{-2mm}
\begin{restatable}[Second Moment Concentration]{theo}{theosmc} \label{theo:smc}
Under Assumption \ref{asmp: bounded_gradient}, \ref{asmp: zi_grad}, the second moment matrix of the  public gradient $M_t = \frac{1}{m}\sum_{i=1}^m \nabla \ell({\mathbf{w}}_t, \tilde z_i)\nabla \ell({\mathbf{w}}_t, \tilde z_i)^\intercal$ approximates the population second  moment matrix $\Sigma_t = \mathbb{E}_{z\sim {\cal P}}[\nabla\ell({\mathbf{w}}_t,z)\nabla \ell({\mathbf{w}}_t,z)^\intercal]$ uniformly over all iterations, i.e., for any $u > 0$,
\vspace{-2mm}
\begin{equation} \label{eq:exp_uc_sm}
\sup_{t \in [T]} \left\| M_t - \Sigma_t \right\|_2 \leq  O\left( \frac{u G \kappa \sqrt{\ln p} \gamma_2({\cal W}, d)}{\sqrt{m}} \right)~,
\vspace{-2mm}
\end{equation}
with probability at least $1- c\exp  \left(-u^{2}/4\right)$, where $c$ is an absolute constant.
\end{restatable}
\vspace{-2mm}

Theorem \ref{theo:smc} shows that $M_t$
approximates the population second moment matrix $\Sigma_t$ uniformly over all iterations. This uniform bound is derived by the technique GC, which develops sharp upper bounds to suprema of stochastic processes indexed by a set with a metric structure in terms of $\gamma_2$ functions. In our case, $\|M_t - \Sigma_t\|$ is treated as the stochastic process indexed by the set ${\cal W} \in \mathbb{R}^p$ such that ${\mathbf{w}}_t \in {\cal W}$, which is the set of all possible iterates. The metric $d$ is the pseudo-metric $d: \mathbb{R}^{p} \times \mathbb{R}^{p} \mapsto \mathbb{R}$ defined in Assumption \ref{asmp: zi_grad}.
To get a more practical bound,
following the above discussion, instead of
working with ${\cal W}$ over parameters, one can
consider working with the set ${\cal M}$ of population gradients by defining the pseudo-metric as $d\left(\mathbf{w}, \mathbf{w}^{\prime}\right)=d\left(f(\mathbf{w}), f\left(\mathbf{w}^{\prime}\right)\right)=d\left(\mathbf{m}, \mathbf{m}^{\prime}\right)$, where $f({\mathbf{w}}) =\mathbb{E}_{z \in {\cal P}}\left[\nabla ({\mathbf{w}}, z)\right]$ that maps the parameter space to gradient space.
Thus, the complexity measure $\gamma_2({\cal W},d)$ can be expressed as the $\gamma_2$ function measure on the population gradient space, i.e., $\gamma_2({\cal M},d)$. As discussed above, using the $\ell_2$-norm as $d$, the $\gamma_2({\cal M},d)$ will be a constant if assuming the gradient space is a union of ellipsoids and uniform bound only depends on logarithmically on $p$.

Using the result in Theorem \ref{theo:smc} and Davis-Kahan sin-$\theta$ theorem \citep{mcsherry2004spectral}, we obtain the subspace construction error $\| \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal \|_2$ in the following theorem.

\vspace{-2mm}
\begin{restatable}[Subspace Closeness]{theo}{theosubc}\label{theo:sc}
 Under Assumption \ref{asmp: bounded_gradient} and \ref{asmp: zi_grad},
with $V_k(t)$ to be the top-$k$ eigenvectors of the population second moment matrix $\Sigma_t$ and $\alpha_t$ be the eigen-gap at $t$-th iterate such that $\lambda_{k}\left(\Sigma_{t}\right)-\lambda_{k+1}\left(\Sigma_{t}\right) \geq \alpha_{t}$, for the $\hat V_k(t)$ in Algorithm \ref{algo: full batch},  if $m \geq  \frac{ O \left(G \kappa \sqrt{\ln p} \gamma_2({\cal W}, d)\right)^2}{\min_{t}{\alpha_t^2}}$,
we have
\begin{equation}
  \mathbb{E} \left[ \| \hat{V}_k(t) \hat{V}_k(t)^\intercal - V_k(t) V_k(t)^\intercal \|_2\right]  \leq O\left(\frac{ G \kappa \sqrt{\ln p} \gamma_2({\cal W}, d)}{\alpha_t \sqrt{m}}\right)~, \forall t \in [T].
  \end{equation}
\end{restatable}
\vspace{-2mm}
Theorem \ref{theo:sc} gives the sample complexity of the public sample size and the reconstruction error, i.e., the difference between $\hat V_k(t)\hat V_k(t)^\intercal$ evaluated on the public dataset $S_h$ and $V_k(t) V_k(t)^\intercal$
given by the population second moment $\Sigma_t$.
The sample complexity and the reconstruction error both depend on the $\gamma_2$ function and eigen-gap $\alpha_t$. A small eigen-gap $\alpha_t$ requires larger public sample $m$.

\subsection{Empirical Risk Convergence Analysis}
\label{sec:erm_rate}
In this section, we present the error rate of PDP-SGD for
non-convex (smooth) functions.
For non-convex case, we first give the error rate of the $\ell_2$-norm of the principal component of the gradient, i.e.,  $ \|V_k(t) V_k(t)^\intercal\nabla \hat L_n({\mathbf{w}}_t)\|_2$. Then we show that the gradient norm also converges if the principal component dominates the residual component of the gradient as suggested by Figure \ref{fig:mnist_eva} and recent observations \citep{papyan2019,LiGZCB20}.
To present our results, we introduce some new notations here.
We write $[\nabla \hat L_n({\mathbf{w}}_t)]^\parallel = V_k(t) V_k(t)^\intercal\nabla \hat L_n({\mathbf{w}}_t)$ as the principal component of the gradient and $[\nabla \hat L_n({\mathbf{w}}_t)]^\bot = \nabla \hat L_n({\mathbf{w}}_t) - V_k(t) V_k(t)^\intercal\nabla \hat L_n({\mathbf{w}}_t)$ as the residual component.

\vspace{-2mm}
\begin{restatable}[Smooth and Non-convex]{theo}{theoi}
\label{thm: full_batch_lsgd}
For $\rho$-smooth function $\hat L_n({\mathbf{w}})$,
under Assumptions \ref{asmp: bounded_gradient} and \ref{asmp: zi_grad}, let $\Lambda = \frac{\sum_{t=1}^T 1/\alpha^2_t}{T}$,  for any $
\epsilon, \delta>0$,  with $T = O(n^2\epsilon^2)$ and $\eta_t = \frac{1}{\sqrt{T}}$, PDP-SGD achieves:
\vspace{-2mm}
\begin{align}
 \frac{1}{T} \sum_{t=1}^T \mathbb{E} \|  V_k(t) V_k(t)^\intercal \nabla \hat L_n ({\mathbf{w}}_t) \|_2^2
    \leq \tilde O\left(\frac{k \rho G^2}{n\varepsilon}\right)   + O\left(\frac{\Lambda G^4  \kappa^2  \gamma_{2}^2(\mathcal{W}, d) \ln p }{m}\right).
\vspace{-2mm}
\end{align}
Additionally, assuming the principal component of the gradient dominates, i.e., there exist $c >0$, such that
$\frac{1}{T} \sum_{t=1}^T\|[\nabla \hat L_n ({\mathbf{w}}_t)]^\bot \|_2^2 \leq c  \frac{1}{T} \sum_{t=1}^T \|[\nabla \hat L_n ({\mathbf{w}}_t)]^\parallel\|_2^2$, we have
\begin{align}
 \mathbb{E}\| \nabla \hat L_n ({\mathbf{w}}_R) \|_2^2
    \leq \tilde O\left(\frac{k \rho G^2}{n\varepsilon}\right)   + O\left(\frac{\Lambda G^4  \kappa^2  \gamma_{2}^2(\mathcal{W}, d) \ln p }{m}\right),
\end{align}
where ${\mathbf{w}}_R$ is uniformly sampled from
$\{{\mathbf{w}}_1, ..., {\mathbf{w}}_T\}$.
\end{restatable}
\vspace{-2mm}

Theorem \ref{thm: full_batch_lsgd} shows that PDP-SGD reduces the error rate of a factor of $p$ to $k$ compared to existing results for non-convex and smooth functions \citep{waxu19}. The error rate also includes a term depending on the $\gamma_2$ function and the eigen-gap $\alpha_t$, i.e., $\Lambda = \sum_{t=1}^T 1/\alpha^2_t/T$. This term comes from the subspace reconstruction error.
As discussed in the previous section, as the gradients stay in a union of ellipsoids, the $\gamma_2$ is a constant. The term $\Lambda$ depends on the eigen-gap $\alpha_t$, i.e, $\lambda_{k}\left(\Sigma_{t}\right)-\lambda_{k+1}\left(\Sigma_{t}\right) \geq \alpha_{t}$. As shown by the Figure \ref{fig:mnist_eva}, along the training trajectory, there are a few dominated eigenvalues and the eigen-gap stays significant (even at the last epoch). Then the term $\Lambda$ will be a  constant and the bound scales logarithmically with $p$. If one considers the eigen-gap $\alpha_t$ decays as training proceed, e.g., $\alpha_t = \frac{1}{t^{1/4}}$ for $t >0$, then we have $\Lambda = O(\sqrt{T})$. In this case, with $T = n^2\epsilon^2$, PDP-SGD requires the public data size $m = O(n\epsilon)$.

For the convex and Lipschitz functions, we consider the
low-rank structure of the gradient space, i.e,  the population gradient second momment $\Sigma_t$ is of rank-$k$, which is a special case of the principal gradient dominate assumption when $|[\nabla \hat L_n ({\mathbf{w}}_t)]^\bot \|_2 = 0$. We present the error rate of PDP-SGD for such case in the following theorem.

\begin{restatable}[Convex and Lipschitz]{theo}{theolipconvex} \label{theo:con}
 For $G$-Lipschitz and convex function $\hat L_n ({\mathbf{w}})$, under Assumptions \ref{asmp: bounded_gradient},\ref{asmp: zi_grad} and assuming $\Sigma_t$ is of rank-$k$, let $\Lambda = \frac{\sum_{t=1}^T 1/\alpha_t}{T}$, for any $
\epsilon, \delta>0$, with $T = O(n^2\epsilon^2)$, step size $\eta_t = \frac{1}{\sqrt{T}}$, the PDP-SGD  achieves
\begin{equation} \label{eq:bound_con}
\mathbb{E}\left[\hat L_n(\bar {\mathbf{w}})\right]  - \hat L_n({\mathbf{w}}^\star)  \leq O\left(\frac{kG^2}{n\epsilon}\right) + O\left(\frac{\Lambda G \kappa \gamma_{2}(\mathcal{W}, d) \ln p }{\sqrt{m}}\right),
\end{equation}
where $\bar {\mathbf{w}} = \frac{\sum_{t=1}^T{\mathbf{w}}_t}{T}$, and ${\mathbf{w}}^\star$ is the minima of $\hat L_n({\mathbf{w}})$.
\end{restatable}

Compared to the error rate of DP-SGD for convex functions \cite{basm14,bafe19}, PDP-SGD also demonstrates an improvement from a factor of $p$ to $k$. PDP-SGD also involves the subspace reconstruction error, i.e.,  $O\left(\frac{\Lambda G \kappa \gamma_{2}(\mathcal{W}, d) \ln p }{\sqrt{m}}\right)$ depending on  the $\gamma_2$ function and eigen-gap term $\Lambda = \frac{\sum_{t=1}^T 1/\alpha_t}{T}$. Based on the discussion in previous section and a more detailed discussion in Appendix \ref{sec:app_gamma2}, with suitable assumptions of the gradient structure, e.g., ellipsoids, the $\gamma_2$ is a constant. For the eigen-gap term, if $\alpha_t$ stays as a constant in the training procedure as shown by Figure \ref{fig:mnist_eva}, $\Lambda$ will be a constant and  the bound scales logarithmically with $p$.
If we assume the eigen-gap $\alpha_t$ decays as training proceed, e.g., $\alpha_t = \frac{1}{t^{1/2}}$ for $t >0$, then we have $\Lambda = O(\sqrt{T})$. In this case, with $T = n^2\epsilon^2$, PDP-SGD requires  public data size $m = O(n\epsilon)$.

Recently, \cite{krrt} propose a noisy version of AdaGrad algorithm for unconstrained convex empirical risk minimization. By assuming the gradients lie in a constant rank subspace and knowing an estimate of the gradient subspace (e.g., from a public dataset), they obtain a dimension-free excess risk bound, i.e., $\tilde O(1/n\epsilon) + \gamma$, where $\gamma$ is the gradient subspace estimation error, which can be interpreted as the subspace construction error in our paper. If the subspace is estimated on the private data with differential privacy, the estimation error $\gamma$ scales with $\sqrt{p}$ \citep{DworkTT014}.
In this case, the bounds are not directly comparable since the assumptions in \cite{krrt} are different from those in this paper, i.e.,  \cite{krrt} assume that the accumulated gradients along the training process lie in a constant rank subspace which requires the rank of the gradient space does not explode when adding more stochastic gradients. Our work does not impose such a constant subspace assumption on $\Sigma_t$ allowing the subspace of $\Sigma_t$ to be different along the training process. In other words, our bounds hold even when the rank of the accumulated stochastic gradients space increases as training proceeds.

redpajama_set_name: RedPajamaArXiv


text:
Impacts of AI on Thailand's Economy and Society
Tue, 25th Feb 2020
Article tags RiskEIU Public Policy
A scenario modelling assessment by The Economist Intelligence Unit
Artificial intelligence (AI) has moved from the margins to the mainstream over the last five years, reflecting remarkable improvements in performance, range and speed. In response, policymakers must think carefully about how AI and AI-assisted automation will affect their country's economy and labour market.
To support decision-makers in Thailand, the Institute of Public Policy and Development (IPPD) commissioned The Economist Intelligence Unit (The EIU) to study the impacts of increased AI adoption on the Thai economy.
Key findings from our analysis show that:
If Thai workers whose jobs are at risk of automation fail to retrain, by 2035 the employment rate could fall by 23 percentage points compared to today.
Without an effective industrial policy that harnesses AI, Thailand risks losing 6% in GDP in 2035 compared to our forecast, due to losses in export competitiveness and declines in foreign direct investment.
By introducing the right interventions, governments and policy stakeholders can harness AI and its associated automation tools to boost skills, strengthen international competitiveness, and improve public services. If Thailand acts on both skills and industrial policy, by 2035 it could keep unemployment levels low and see an increase in GDP to over US$1trn by 2035, up from US$544bn today.
The study assesses three potential scenarios – Ox, Bees and Elephant – that present the risks and opportunities of AI in Thailand out to 2035. Click here for a comprehensive breakdown of each scenario's key potential outcomes.
Tue, 25th Feb 2020 Article tags RiskEIU Public Policy


text:
League Information and Rules
Select list(select one): Division Menu Novice Games Standings Rosters
Novice ball is for 6-7 year olds. The Novice division is a more competitive league, as the concept of three outs per inning is introduced. All players remain in the batting lineup and they will all get to hit each inning.
Games are played at 5:30 or 6:45 on Sundays, Mondays or Wednesdays. Two games per week and they are played on Greenwood Field across from Dominion on Old Placentia Road.
The instructor or coach still acts as the pitcher using a cushioned ball which is thrown underhand. A greater emphasis is placed on making "outs" at this level.
The schedule is usually Mondays and Wednesdays but if we get larger numbers Sunday evening will be added to accommodate the extra teams. These sessions are one hour and fifteen minute sessions. Teams will only play two sessions per week. Greenwood Field across from Dominion is home field.
In the event of rain, an announcement concerning the scheduled games for that evening will be posted to the website at 4:00 pm.
Players are to notify their coach if they are unable to make a scheduled house league game.
Please arrive at least 10 minutes prior to the scheduled start time for a pre-game warm up.
Fair Play Rules For Coaches:
Coaches will be issued a score book. They must set their batting order at the start of the year. Once this batting order is set, the player who is after the last out of the previous game will start the next game. If a player does not show up for the game then he will stay in the same spot in the batting order but crossed out for that particular game. Coaches should use the first three games of the season to finalize the batter order. Batting order may be reset for playoffs. Players should receive equal time on the field. Players should be rotated in positions on the field during games.
Novice Rules:
Official Playing Rules: All games shall be played in accordance with regular baseball rules except where modified by the rules below
Players must wear full uniform, which includes the wearing of white baseball pants, and are advised to wear a protective cup.
All batters must wear helmets when on deck or at bat.
Batters will receive 5 underhand pitches, then a batting tee will be used for 2 attempts.
All teams may use all players in the field.
An inning will be complete after 3 outs are made or 3 runs are scored.
Players must return to the bench when an out is recorded.
Only one base on a ball hit to the infield, two bases on a ball hit to the outfield (the ball cannot roll or bounce to the outfield)
Only one base on an overthrow. One overthrow is allowed per play to encourage players to try to make the play. For example, if the ball is overthrown to 1st base, the runner can advance at his own risk to 2nd. If the ball is overthrown on the same play to 2nd, the runner remains on 2nd.
All plays to the plate are force plays with the catcher just having to touch the plate. This prevents unnecessary collisions.
No lead-offs are permitted.
Throwing the ball back to the pitcher will end the play. If this occurs when a runner is less than half-way to a base, the runner must return to the prior base.
Umpires, at their discretion, will call a foul if the ball doesn't go approximately 10 feet in front of home plate. Score is kept by the coaches and validated with the officials several times during the game.


I should start out by saying that I don't truly believe there is an off-season for endurance athletes. Following the end of your racing season, it's critical that you take a complete break from structure and stress of racing and training, and allow healing, but a break only needs to be a couple of weeks before you can add some structure back into your life and training plan. The key is that this structure should provide a much lower level of physical stress than IRONMAN training, and allow you to make strides toward a great upcoming season while avoiding an accumulation of fatigue. At my coaching company, purplepatch, we refer to this most critical block of work—which much of the next season's success will be built on—as the post season. Let's explore the goals of this phase and your optimal approach to ensuring that it's successful.
To keep things simple, we always discuss the post season as a critical phase of training which should prepare your body for the rigors of heavy training ahead. This frames it as a critical preparatory phase and allows the athlete to emphasize planning, technical improvements, equipment changes, as well as careful improvements to foundational fitness and musculoskeletal resilience that will help them absorb and adapt to harder training ahead. Many triathletes seem to only have an on and off switch when it comes to training. This phase should include lower overall physical strain and a much lower physical load.
1. A commitment to technical improvement.
So often technique and skills acquisition become an afterthought as racing looms. The post season is a great time to refocus on "doing things right."
You don't always need a new bike. After all, we always say "new bike, same old engine." This said, any equipment changes or positional refinements best come at this point in the season. Immerse yourself in finding the right bike seat (for you!), the right fit, or ensuring you are in best running shoes for your comfort and run performance.
A coach can and should guide you to enhanced performance. There is a high degree of athlete responsibility in any coaching relationship, but a high-quality coach will help you. Now is the time. Ensure you choose carefully, and don't just pick the big name coaches. Find the right fit for your needs and style, as this is going to be a real relationship over the coming months and years. Check out IRONMAN U's CoachMatch service that will help you land the right coach for you.
Our running training is aimed at some technical improvements, as well as a careful strengthening of the musculoskeletal system (tendons, ligaments, muscles) to set up the ability to withstand a heavier load in the next block. Much of the running is shorter in duration and applied frequently.
Our sport is swim-bike-run, not swimming, cycling and running. Swim fitness and resilience is critical; don't delay on your post season swimming project. I like athletes to be swim-fit, and we place a great emphasis on swimming development in the post season and beyond. Find a group, focus on some technical work, and fall in love with swimming.
Strength and conditioning is a season-long endeavor for endurance athletes, not a short-term fix. With this, it is not as central as our swim-bike-run sessions. Get race season off to the right start by integrating an efficient but specific focus on strength and conditioning training. Sessions should be short (under 45 minutes), specific, and ready to progress in focus throughout the season. A smart coach should absolutely assist you with this component.
For busy triathletes, keeping all intensity low in this part of the season is a wasted opportunity. Incorporate some very high-intensity intervals to develop neurological conditioning, coordination, and power potential. Short and fast hill repeats with a lot of rest, short fartlek foot speed intervals and other speed work is key. Know that the aim is not to break you, but for you to complete them with great form and real purpose. No session should leave you depleted or destroyed. We call it "technical speed" to place emphasis on effort with great form.
Many athletes fail to fuel properly following workouts, and it impairs recovery and adaptations as well as being highly stressful. Gaining knowledge into the importance of post-workout fueling and how to apply it is the first step, then commit to making it a true training habit. Remember, real food is king.
A platform of healthy daily eating is important for your health and performance. You don't need to become a monk or abstain from your favorite foods forever, but learn about optimal habits, and begin to apply them into life. If your nutrition is falling by the wayside, the post season is also a great time to enroll in IRONMAN U's Sports Nutrition for Endurance Athletes.
When training stress is low, it's easier to establish positive recovery habits. Beyond the nutrition and fueling, other key factors are sleep and going easy enough is the lower stress training sessions. These are your foundations of recovery. Sure, they are modalities and recovery tools that help, but nail the basics first to set up success.
As you can see, there is no reason to get bored during the so-called "off" season. The post season is all about review, planning, and setting the stage for the season ahead. So much of a successful season will be built on what you accomplish now, even though none of your sessions will stand out as the toughest or most challenging of the year.
Don't miss this critical component of performance consistency—embrace and engage your post season now.