How Many Death Certificates Should I Purchase?
How many death certificates you should purchase, the actual number of death certificates, really varies greatly by family. If the person has life insurance, then each life insurance company or policy would require its own certified copy of a death certificate. Any banks or accounts that are solely in the individual's name would require a death certificate. Any pension or 401k plans would require its own copy of a death certificate. These are the types of things and places that would require a death certificate, so it varies greatly depending on the type of estate the deceased person has.
The average is 8-12 originals. However it is very different for every person and dependent on how many assets the deceased had in their name. Assets to consider include insurance policies, credit life insurance, employer benefits, vehicle titles, real estate titles, stocks & bonds, income tax returns, social security, utilities, financial institutions, and attorneys.
It depends on the number of items still in the name of the deceased. The average needed ranges from 6-10 certificates.
We recommend 10 copies. The deceased's assets will dictate how many will be needed. The family will need to review assets and determine the actual amount needed. We also recommend order a couple of extra than you think you will require.
That depends on how much estate planning has been done before the time of death. Ten certificates is the average for most families. More copies can always be purchased at any time directly through the registrar that is in charge of the region where the person passed. Most times this can be done online.
It depends on the individual estate of the deceased. Your funeral director can provide you with a checklist of who might need a death certificate.


At the moment, UK driving licences may be used to drive anywhere in the EEA (this is the EU plus Iceland, Liechtenstein and Norway).
If Theresa May's withdrawal agreement is passed, UK licences will still be valid for visiting EEA countries.
If the UK leaves the EU without a deal, then the situation gets complicated.
The government will try to do deals allowing UK licences to be recognised for visits to Europe. But if it doesn't manage to do so, then drivers will need an International Driving Permit (IDP) for all EEA countries except the Republic of Ireland. IDPs can be bought at Post Offices for £5.50. You may also need to carry your UK driving licence.
There are three different types of IDP, known as the 1926, 1949 and 1968 IDPs – the numbers refer to the dates of the conventions on road traffic that established them.
There will also be changes to the types of IDP you need to be able to drive in countries outside the EEA. For example, if you want to visit Turkey by car before 29 March, you will need a 1949 IDP – but after 29 March, you will need a 1968 IDP. You can find the full list of which IDP you need on this page.
IDPs apply only for visiting other countries. The government has said that if you are a UK licence-holder living in another EU country, then you need to exchange your UK licence for a licence issued by an EU country, before the UK leaves. If you wait until after the UK leaves, then you may need to take another driving test.
EU and EEA licenses will continue to be accepted in the UK for visitors and residents.
If the UK leaves the EU without a deal, you will also need to get a Green Card from your insurer to prove your car is covered.
The Green Card is only proof of a minimum level of third-party cover – it will not necessarily match the level of cover that you pay for in the UK. Check with your insurer to find out what level of cover you will get.
Taking your car to the EEA without a Green Card will be against the law.
The government recommends that you have a GB sticker on your car, even if you also have a GB symbol on your number plate.
You'll need to carry your V5C log book with you if you own the car. If it is a car you have hired or leased then you will need to get a VE103 form to show you have permission to take it out of the UK.
Perhaps most inconveniently, if you are involved in a road traffic accident in an EEA country after a no-deal Brexit, then you may need to make a claim against the responsible driver or their insurer in the country where the accident happened.
And that could involve bringing the claim in the local language.
S-P-M Number Plates is currently seeking experienced external Area Sales Managers due to the expansion of our current field based sales force.
The ideal candidate will be experienced in the Automotive Industry but it's not essential.
If you're interested email your CV to sales@spmltd-uk.com.
When it comes to being ready for the new 'Trailer Plate Laws', we leave the competition trailing !
From 28 March 2019, it will be a requirement for certain trailers travelling abroad to be registered with DVLA. When registered, trailers must display a trailer registration number plate as well as the registration number plate of the towing vehicle.
Registration plates must be supplied by registered number plate suppliers.
Rest assured, we are pleased to announce that S-P-M is ready; the software on our patented Compact number plate system is already capable to produce these new trailer plates.
Please ring 0113 833 0407 to place your stock orders for this new requirement, the demand has already started and becomes law at the end of March 2019.
Do I need to register separately to become a supplier of trailer registration plates?
No, as long as you are a registered number plate supplier you will be able to supply trailer registration plates as well as vehicle registration plates.
I do not have all the necessary components to produce trailer registration plates. Where can I get them?
Your current supplier should be able to provide you with the vehicle registration plate components or equipment needed to produce trailer registration plates.
Is it compulsory to supply trailer registration plates?
No, it is down to each RNPS to decide if they will offer trailer registrations plates. DVLA will advise customers to check with their chosen supplier beforehand to ensure they are available.
Are RNPS required to keep a sales record for trailer registration plates sold?
You must keep records of number plate sales for three years and show them upon request to the police, DVLA, Driver and Vehicle Standards Agency (DVSA) and Trading Standards enforcement officers if they ask to see them. A sample template of a sales record can be found on the INF195/2.
THE owner of the 'world's most expensive number plate' has turned down an offer of £10,000,000 for the plate.
A motorist has turned down an offer for what is referred to the 'world's most expensive' number plate.
The registration plate says 'F1' and is owned by entrepreneur Afzal Khan.
He bought the plate for £440,625 in 2008 from Essex Council and it is currently fit on his Bugatti Veyron Super Sport.
Our friends and colleagues at Parts 2000 proudly exhibited our innovative printer system at the recent Auto Expo in Dublin. It was a great show and on behalf of S-P-M and Parts 2000 we would like to express our thanks to everyone who attended.
Earlier this year the BSI published a new specification regarding Retroreflecting Number Plates with the intention being to replace BS AU 145d. The DVLA was unable to finalise the details / content in time for its pre-arranged date before Parliament. This situation means that currently there is no change to the legislation and the proposed BS AU 145e has not been implemented.
Rest assured, as and when this new legislation comes into force, S-P-M's software, systems and materials will be ready and fully compliant and of course we will update you accordingly.
Once again S-P-M group were exhibiting at the European Automechanika Show in Frankfurt.
This years show was our busiest and most successful to date, with several of our European Partners joining us throughout the week.
Police forces in Thames Valley, Hampshire and the West Midlands are stopping motorists and immediately revoking their drivers licences if they fail a series of simple tests.
Every motorist must pass the test before being allowed to carry on with their journey. It is suspected that 1000's of drivers everyday drive on UK roads with poor eyesight.
If the driver is unable to clearly read a number plate from 20 metres away they will be prevented from proceeding.


text:
School Street School
John Hill School
Boonton High School
District Home Link
BHS Staff Directory
W.A.M.P.U.S. Newspaper
BHS Expressions
Gifted & Talented Opportunities
Academies & Programs
Academic Pathways
The Business Academy
The Criminal Justice Academy
The STEM Academy
The Teaching and Learning Academy
The Gateway Academy
Education Resource Websites
Climate Survey Presentations
Anti-Bullying Information & Contacts
Realtime Portal Links
BHS Parent & Student Handbook
District Policies & Regulations
BHS Media Center
For ELL Families
BHS Alumni Association
BHS HSA
Enriching lives. Empowering tomorrow's leaders.
Message from the Athletic Director
Athletic Forms & Notices
Scholastic Student Athlete Safety Act Information for Parents
NJSIAA - NJ Dept. of Health Update
Welcome to the Boonton High School Athletic Program. The Athletic Department looks forward with much anticipation to competing in the Northwest Jersey Athletic Conference (NJAC). For the 7th year the conference will be made up of forty-one schools from Morris, Sussex, and Warren counties. The Bombers will strive to win NJAC Independence Divisional Championships while competing in cross country, field hockey, football, soccer and tennis. Also, while competing for championships, the Bombers will look to add a NJSIAA Sportsmanship Award as a school.
Come support our coaching staff and athletes as they participate on the courts or on the fields. Our coaches will be challenging our student-athletes skills as well as their character as they compete in interscholastic athletics. The Athletic Department expects its athletes to represent the school and community by demonstrating the importance of sportsmanship, teamwork, and hard work. This is Bomber Pride!
You are encouraged to stay up-to-date with all the athletic happenings by logging onto the Boonton Public Schools Website. The websites provide accurate information regarding all game schedules. For daily athletic information please call the Athletic Hotline at 973-335-9700, option 4. Any additional athletic information can be obtained by calling the Boonton High School Athletic Office at 973-335-9700 ext. 4020.
Hope you all enjoy our sports seasons, and we hope to see you at the games. We wish all our student-athletes a safe and successful fall season. As Knute Rockne, former Notre Dame Head Football Coach, said, "Go out that door to victory!" Here's to a great fall season at Boonton High School. GO BOMBERS!
David Hughen,
david.hughen@boontonschools.org
Boonton, NJ 07005
Phone: 973-335-9700 Option 2
websupport@boontonschools.org
Boonton Public Schools - Enriching lives. Empowering tomorrow's leaders.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: multidex requires build tools 21.0.0 приложение уровня "hello, world"
только, что запускалось
сделал Rebuild Project
вылезла ошибка multidex requires build tools 21.0.0 / Current: 19.1
как отключить обновление?
зачем HelloWorld нужно обновить библиотеки?

A: Об обновлении говорит не ваше приложение а сама студия. Обновите инструменты сборки при помощи Android SDK Manager. На текущий момент актуальной является версия 23.0.3 от марта 2016г. Она и должна поддержать выбранный Вами compileSdkVersion 23.

A: Думаю, дело в том, что в BuildTools 21.0.0 была добавлена поддержка Android 5.0 (API21), а вы используете compileSDKversion 23 (хотя тогда бы, наверное, build tools 23 запросила бы) или возможно какие то классы из этого API.
Либо установите compileSDKversion < 21, либо (что предпочтительнее) обновите Build Tools (и другие инструменты) до актуальных версий через SDK Manager. Хотя может придется ограничить и maxSDKversion = 20, я просто никогда не задавался целью ограничить API сверху, как и пытаться запустить что то на неактуальных инструментах, поэтому с уверенностью сказать не могу, что надо сделать.
Обновления инструментов требует не приложение, а система сборки (IDE), так как ей, по всей видимости, нужны возможности этих обновленных инструментов, чтобы собрать проект с вашими настойками сборки.

redpajama_set_name: RedPajamaStackExchange


types of engineers and scientists, as well as students.
University of Munich, DLR & Lufthansa.
transforms as well as highlighting recent trends.
Transform, Linz, March 27-31, 2017 is open.
speakers by August, 31st to Office RICAM office@ricam.oeaw.ac.at .
the University of Bath, United Kingdom, September 11-15, 2017.
institution has not yet been established by the application deadline.
establishment of the research chair at AIMS-Cameroon is intended.
chairs at AIMS centres in, Ghana, South Africa, Cameroon and Tanzania.
aspects of broadly defined inverse problems.
- Strong expertise in an area of scientific computing.
- Experience in machine learning, data mining and statistics.
- Experience in ab initio simulations.
chemistry, computer engineering, or a similar area.
explaining your planned research topic and its relevance for AICES.
assurance, and they have to be properly accounted for.
and Chalmers but will be employed at CeMEAI-ICMC-USP at São Carlos.
salary is according to the German civil service tariff E13 TV-L.
studies in reactive multicomponent - multiphase transport problems.


text:
\section{Notations and Problem setup}

\textbf{General convex ERM:}  We consider the problem of  $$\min_{\theta\in\Theta} \sum_{i=1}^n \ell_i(\theta) :=  \mathcal{L}(\theta)$$
where $\ell_i: \mathbb{R}^d\rightarrow \mathbb{R}$ is assumed to be convex.   Note that the objective function is the sum, rather than the average of the loss functions of the dataset.

Sometimes we also consider regularized objective function $J_{\lambda,\theta_{\text{ref}}}(\theta) := \mathcal{L}(\theta) + \frac{\lambda}{2} \|\theta-\theta_{\text{ref}}\|^2$ to make it strongly convex.  The subscript $\lambda$ and reference $\theta_{\text{ref}}$ is often abbreviated to avoid clutter.

Besides convexity, we also make the following assumptions: for all $i=1,...,n$,
$\ell_i$ is  convex, $L$-Lipschitz and $\beta$-smooth in its first argument. $J$ is $M$-smooth and $\mu$-strongly convex.  Note that $M\leq  n \beta + \lambda$ and $\mu\geq \lambda$.

Specifically for the interest of  defining differential privacy,  $\ell_1,...,\ell_n$  belongs to a universe of the loss functions induced by the universe of data points. The assumptions stated above for $\ell_i$ are required for all $\ell$ in this universe, thus by adding or removing individual data points, the data point satisfy the same assumptions.

\textbf{General linear models (GLM):}  In some cases, we consider the restriction to a family of generalized linear models when the loss function is $\ell(\theta,(x,y)) := f(\theta^Tx,y)$ for some convex link function $f$. In this cases, $\ell_i(\theta):=\ell(\theta,(x_i,y_i))$.

\textbf{Algorithm:}
The algorithm we consider is the Noisy Gradient Descent (NoisyGD), which updates the parameter vector iteratively using
$$
\theta_{t+1} = \Pi_{\Theta}\left(\theta_t - \eta_t (\nabla J(\theta) + \mathcal{N}(0,\sigma^2 I_d) )\right),
$$
where $\Theta$ is a convex feasible region and $\Pi_{\Theta}$ is the Euclidean projection to this set. When we consider unconstrained problems, then $\Theta=\mathbb{R}^d$.

\section{Convergence guarantees}

\subsection{General convex case}

\begin{theorem}[Convex, smooth and Lipschitz]
	Let the learning rate $\eta_t = \eta \leq \frac{1}{M}$ , assume $\beta <+\infty$
	\begin{align}
		\mathbb{E} J(\bar{\theta}_{1:T})  - J^* \leq  \frac{\|\theta_1 - \theta^*\|^2}{T\eta} +  \eta d\sigma^2.
	\end{align}
In particular, if $\eta_t = \min\{\frac{1}{M}, \frac{\|\theta_1-\theta^*\|}{\sqrt{d\sigma^2 T}}\}$ and that $\sigma$ is chosen such that the algorithm satisfies $\rho$-zCDP, i.e., $\rho:= \frac{TL^2}{2\sigma^2}$, then
	\begin{align}
	\mathbb{E} J(\bar{\theta}_{1:T})  - J^* \leq  \frac{M\|\theta_1-\theta^*\|^2}{T}+  \frac{2 \|\theta_1-\theta^*\|\sqrt{ d\sigma^2}}{\sqrt{T}}\\
	= \frac{M\|\theta_1-\theta^*\|^2}{T}+ \sqrt{\frac{2d}{\rho}}L\|\theta_1-\theta^*\|.
\end{align}
\end{theorem}

\begin{theorem}[Convex and Lipschitz]
		Let the learning rate $\eta_t = \eta$ ,  also assume $\sup_{\theta\in\Theta}\|\theta - \theta_{\text{ref}}\|_2 \leq B$ , then
	\begin{align}
		\mathbb{E} J(\bar{\theta}_{1:T})  - J^* \leq  \frac{\|\theta_1 - \theta^*\|^2}{T\eta} +  \eta \left( (nL+\lambda B)^2 + d\sigma^2\right).
	\end{align}
In particular, if $\eta_t = \frac{\|\theta_1-\theta^*\|}{\sqrt{(d\sigma^2 +(nL + \lambda B)^2)T}}$, and that $\sigma$ is chosen such that the algorithm satisfies $\rho$-zCDP, i.e., $\rho:= \frac{TL^2}{2\sigma^2}$, then
	\begin{align}
	\mathbb{E} J(\bar{\theta}_{1:T})  - J^* &\leq \sqrt{\frac{\|\theta_1 - \theta^*\|^2(nL + \lambda B)^2}{T}} +  \sqrt{\frac{ 2d\sigma^2\|\theta_1 - \theta^*\|^2}{T}} \\
	&= \sqrt{\frac{\|\theta_1 - \theta^*\|^2(nL + \lambda B)^2}{T}}  + \sqrt{\frac{2d}{\rho}}L\|\theta_1 - \theta^*\|.
\end{align}
\end{theorem}

\subsection{Strongly convex  case}
\begin{theorem}
	Let the learning rate be $\eta_t = \frac{2}{\mu (t+1)}$,  and $\|\Theta\|_2 \leq B$ then
		\begin{align}
		\mathbb{E} \left[J\left(\frac{2}{T(T+1)}\sum_{t=1}^T  t \theta_t\right)\right]  - J^*
		\leq& \frac{4(n L + \lambda B)^2 }{\mu T} + \frac{d\sigma^2}{\mu T}\\
		 =& \frac{4(n L + \lambda B)^2  }{\mu T} + \frac{d L^2}{\mu \rho},
	\end{align}
where $\rho:= \frac{TL^2}{2\sigma^2}$ is the zCDP parameter of the algorithm.
\end{theorem}

Note that when $\rho \leq \sqrt{\log (1/\delta)}$, zCDP implies $(\epsilon,\delta)$-DP with $\rho \geq \frac{\epsilon^2}{8\log (1/\delta)}$. This means that as $T$ goes to $\infty$ and $\frac{\sigma^2}{T}$ stays a constant, the first term vanishes and second term matches the information-theoretic limit of differentially private learning under $(\epsilon,\delta)$-DP in the convex case and in the strongly convex case respectively.

\subsection{Utility in the regularized case}

When additional regularization with parameter $\lambda $, the utility we consider should still be considered in terms of $\mathcal{L}(\hat{\theta}) - \mathcal{L}(\theta^*)$. Let $\theta^*$ be any comparator satisfying $B> \|\theta^*\|$
\begin{align*}
	 \mathcal{L}(\hat{\theta}) - \mathcal{L}(\theta^*) &=J(\hat{\theta}) - J_\lambda(\theta^*_\lambda)  +J_\lambda(\theta^*_\lambda) - J(\theta^*) +  \frac{\lambda}{2}\|\theta^*-\theta_{\text{ref}}\|^2 -   \frac{\lambda}{2}\|\hat{\theta}-\theta_{\text{ref}}\|^2\\
	 &\leq J(\hat{\theta})  - J_\lambda(\theta^*_\lambda) + \frac{\lambda}{2}\|\theta^*-\theta_{\text{ref}}\|^2.
\end{align*}
Take expectation on both sides and substitute the bound with an initialization at $o$
\begin{align*}
	\mathbb{E}[\mathcal{L}(\hat{\theta})]- \mathcal{L}(\theta^*) \leq \frac{4(n L + \lambda B)^2  }{\mu T} + \frac{d L^2}{\mu \rho} +  \frac{\lambda}{2}\|\theta^*-\theta_{\text{ref}}\|^2.
\end{align*}
In the case when $\mathcal{L}$ is not strongly convex, $\mu = \lambda$ we get
$$
 \frac{4(n L + \lambda B)^2  }{\lambda T} + \frac{d L^2}{\lambda\rho} +  \frac{\lambda}{2}\|\theta^*\|^2
$$
when we choose $\lambda  = \sqrt{\frac{\rho}{2\|\theta^*-\theta_{\text{ref}}\|d L^2}}$ we get
$$
\mathbb{E}[\mathcal{L}(\hat{\theta})]- \mathcal{L}(\theta^*) \leq   \frac{4(n L + \lambda B)^2  }{\lambda T}  + \frac{\sqrt{d} L\|\theta^*-\theta_{\text{ref}}\|}{\sqrt{2\rho}}.
$$

Again, when $T\rightarrow +\infty$, the first term vanishes and the second term is the information-theoretic limit for general convex problems. when $\theta_{\text{ref}}=0$. However, if we choose $\theta_{\text{ref}}$ according to the public data, then if the public data gives either a good initialization or a good reference point for the strongly convex regularization, then it provably improves the utility under the same privacy parameter $\rho$.

\subsection{Provably good initialization via a public data set}

Assume the public data with $m$ samples are drawn from the same distribution of the private data. Assume that the (population risk)
$$
R(\theta) = \mathbb{E}_{(x,y)\sim \mathcal{D}}[ \ell(\theta^*, (x,y))]
$$
is $c$-strongly convex at $\theta^*$ for some constant $c$.   Note that this does not require strong convexity everywhere and it also does not require strong convexity to hold on the empirical risk objective function finite dataset.

Moreover, assume that the $\mathbb{E}_{(x,y)\sim \mathcal{D}}[\|\nabla \ell(\theta,(x,y))\|^2]\leq  G^2$. Then running one-pass SGD which returns $\theta_{\text{ref}}$ to be the weighted average satisfies that
$$
\mathbb{E}[\|\theta_{\text{ref}} - \theta^*\|^2] \leq \frac{4G^2}{ c^2 m }.
$$
This result  together with the bound above, implies that
$$\mathbb{E}[\mathbb{E}[\mathcal{L}(\hat{\theta})]- \mathcal{L}(\theta^*) ]\leq  \frac{4\sqrt{d} LG^2}{c^2 m \sqrt{2\rho}}$$

The outer expectation is taken over the distribution of the public data, which implies a learning bound of the form (by dividing both sides by $n$ and adjust for the difference between empirical risk and population risks)
$$
\mathbb{E}[R(\hat{\theta})]  -  R^* \leq \frac{4\sqrt{d} LG^2}{c^2 m n\sqrt{2\rho}} +  \text{GeneralizationGap}(n).
$$
where the $\text{GeneralizationGap}$(n) is difference between  the empirical and population risk of $\hat{\theta}$, typically obtained via a uniform convergence argument.  The generalization gap is required even for the non-private learning and is between $O(\sqrt{1/n})$ and $O(1/n)$ when we qualify for a condition for fast rate (e.g., strong convexity, low-noise, etc). Note that the above bound extra factor due to privacy  is on the order of $O(\frac{\sqrt{d}}{mn \sqrt{\rho}})$ comparing to $O(\frac{\sqrt{d}}{n\sqrt{\rho}})$  when we have private data only.

\subsection{Improved dimension dependence for GLMs}
For the generalized linear model case, all dimension $d$ above can be replaced with $\mathrm{rand}(X)$ where $X\in \mathbb{R}^{n\times d}$ is the data matrix and  $ \leq \min\{d, n\}$.  The arguments follow from \citep{song21evading} by leveraging that the gradients are all within the row-space of $X$ and it suffices to consider.

This observation is quite useful to us because the linearization of a deep model is often very high-dimensional and $d \geq n$.  In this case, this is saying that we can chooses the learning rate as a function of $n$ instead.

\subsection{Gradient clipping under GLM}
It was established in \citep{chen2020understanding} that the gradient clipping is known to have the effect of Huberization of the loss function in the GLM cases. The convergence to the optimal solution of a modified objective function is guaranteed if we fix the clipping parameter. Our approach on the adaptive clipping has the effect of selecting the hyperparameter adaptively in an homotopy style algorithm, which shows further empirical benefits.

\subsection{Related work} The theory on the convergence of SGD is well-studied under various regimes. The exposition above is based on the analysis of \citep{ghadimi2013stochastic} (for the convex cases) and \citep{lacoste2012simpler} (for the strongly convex cases).  \citet{song21evading} provided the first analysis of this algorithm in the general convex case with dependence on the $\min{n,d}$, and first illustrated the interpretation of gradient clipping as a Huberization of the link function.  \citet{chen2020understanding} provided an alternative explanation of the gradient clipping in terms of how it effectively modify the data distribution.  To the best of our knowledge, we are the first that states these results for GLM that depends on $\min{n,d}$ in the strongly convex case, as well as showing that the use of public data gives provably smaller sample complexity comparing to the standard private learning setting.

\section{Discussion}

We make a few interesting discussion here.

\bibliographystyle{apa-good}

\section{Introduction}
\label{sec:intro}

When training a deep neural network for visual classification, it is critical to protect privacy by limiting the amount of information that could be gleaned about any individual training sample. Differential Privacy (DP) \cite{dwork2006calibrating} is a theoretical framework to provide strong guarantees on the maximum amount of information that any attacker can extract about an individual training sample. In particular, DP allows users to select the desired trade-off between privacy and accuracy, mediated by a \textit{privacy parameter} $\epsilon$, which typically depends on the application scenario.

Training large machine learning models while guaranteeing strong privacy for each sample is challenging. In practice, however, one often has a pool of data available for which there are no privacy concerns. This could be a synthetic datasets or datasets designed to be available for public use. Such \textit{public} data are distinct from the \textit{private} ones, for which we seek strong privacy guarantees.\footnote{Note that here public data is not the same as data from public sources, as the latter may still require privacy guarantees.}
In particular, using large amounts of generic public data for pre-training has recently enabled the creation of language models that achieve DP on the target task while remaining close to state-of-the-art performance  \cite{li2021large,yu2021differentially}. The same strategy in vision \cite{tramer2020differentially}, however, still yields punishing error increases (\cref{table:mixdp}) ranging from 311\% to 167\% on average across datasets, depending on the desired level of privacy.

\textit{Can we, then, make use of public data that is better suited for vision, so we can retain close-to-paragon performance while ensuring privacy?}

One sure way of preserving privacy is not to use private data altogether, and recent literature has given us multiple ways to do so. For example, using zero-shot learning \cite{larochelle2008zero,lampert2009learning,lei2015predicting,socher2013zero,frome2013devise} one can leverage public data from a different modality (e.g., text) to train a visual model without ever seeing the private data. More generally, one can source or synthesize a few samples of labeled public data from the task distribution,
and train using few-shot learning \cite{wang2019few}, still without using private data. Surprisingly, simple few-shot techniques outperform private training on much larger dataset (Fig. \ref{fig:boost}, left).

Of course, there may be subtle domain shifts between the public and private data, so ignoring the latter is not a desirable strategy to maintain privacy. The question, then, is how to use even small amounts of public data, in addition to the private data, to break the trade-off between privacy and accuracy. To do so, we change the setting from most work on DP to include \textit{labeled} public data sourced with the \textit{same} labels as the target task. We call this setting \textit{mixed differential privacy}, or MixDP.

To address MixDP, we propose to use the public data not just for pre-training the backbone, but for few-shot or zero-shot learning of a classifier on the target tasks, prior to private fine-tuning (Sec.~\ref{sec:adamix}).
In Tab.~\ref{table:mixdp}, we show that indeed in the MixDP setting it is possible to achieve significant gains compared to training with only the private or only the public data, even using a small amount of the latter. To do so, we have to modify existing DP training algorithms to the mixed setting, which leads us to the development of AdaMix, a method for MixDP that uses public data to tune and adapt all major steps of private training, in particular model initialization, clipping of the gradients, and projection onto a lower-dimensional sub space. Some of these ideas to use auxiliary public data were applied in different contexts in isolation, but have suboptimal privacy/performance trade-offs for visual tasks. %

In visual classification tasks, the long-tails of the data
play an important role in achieving high classification performance \cite{feldman2020does,feldman2020neural}. However, DP is a worst-case framework which is highly-influenced by outliers or long-tails. MixDP eases the problem, since it allows to collect public data to ensure that each sub-population is sufficiently covered. This reduced the cost of privacy for the long tails, as we show using a per-instance DP (pDP) analysis of \cite{wang2019per}.%

Unlike most related work, we use NoisyGD rather than DP-SGD as we find that it is faster and more private for our use case. We give a tight analysis of its $(\epsilon, \delta)$-DP properties using the Analytical Gaussian mechanism \cite{balle2018improving,dong2021gaussian} (Sec.~\ref{sec:dp}). In addition, we present a convergence proof for our algorithm together with new stronger bound for the strongly convex case. This also allows us to describe the utility of the algorithm relative to its non-private counterpart.

To summarize, our contributions are: \textbf{(1)} we introduce AdaMix, a state-of-the-art adaptive method for mixed privacy learning; \textbf{(2)} we show significant improvement w.r.t.\ baselines on a novel benchmark for MixDP vision tasks; \textbf{(3)} we show that the zero-shot text information can improve  the performance of private models (Sec.~\ref{sec:experiments}); \textbf{(4)} we analyze the goodness of our method both theoretically (new theorems) and empirically (pDP analysis\textbf{}). %

\section{Related Work}
\label{sec:related}

Most work on Differential Privacy \cite{chaudhuri2008privacy,song2013stochastic,wang2019differentially,wang2019differentially,yu2021differentially,bassily2014private,shokri2015privacy,iyengar2019towards,jayaraman2018distributed,lee2018concentrated} uses public data either for: generic pre-training unrelated to the task \cite{tramer2020differentially}, or to tune parameters \cite{kairouz2020fast,zhou2020bypassing,yu2021not}, or as additional \textit{unlabeled} data \cite{papernot2016semi,papernot2018scalable}. Instead, we use a \textit{small amount of labeled public data related to the task} to improve the accuracy of a private model under a given privacy parameter $\epsilon$.

\paragraph{Unlabeled public data.}
PATE \cite{papernot2016semi,papernot2018scalable,uniyal2021dp} trains teacher models on different partitions of the data to predict labels, and privately vote on how to label public (unlabelled) data. Then, a student model is trained in semi-supervised fashion on public data, now partially labelled using the private data. This strategy does not directly allow to train a model on a mix of labeled public and private data. \cite{zhu2020private} noted that the number of partitions required for meaningful privacy guarantees may be prohibitive for vision datasets. As an alternative, they suggest labeling with a differentially private KNN. In MixDP, where we assume that the public data is already labeled, these algorithms would perform at best similarly to our \textit{Only-Public} baseline. Instead, what we are interested in effectively using labeled private data to improve performance relative to already labeled public data.

\paragraph{DP for Language models.} \cite{li2021large,yu2021differentially} show that a large language model pre-trained on generic public data can be fine-tuned on task-specific private data with only modest loss in accuracy.
In contrast, our focus is on using small amounts of public data for fine-tuning.
We note that in language models strong transfer of information from the pre-training is facilitated by the fact that the output space (a sequence of tokens) is the same for all tasks. Conversely, vision models need to relearn different output spaces based on the task (i.e., the last layer is always trained from scratch) reducing the transfer of information. We show that we can partly recover this advantage by either using an initialization based on public data \textit{of the same task}, or a \textit{multi-modal} vision-language model. We use  CLIP \cite{radford2021learning} for zero-shot learning.

\paragraph{Adaptive clipping.} Similarly to us, \cite{wang2020differentially} studies DP training with access to a small amount of public data, and use adaptive gradient clipping \cite{zhang2021understanding,andrew2019differentially,bagdasaryan2019differential,van2018three} based on the public gradients. Contrary to us, they first train a private model separately and then a mixed public-private model using the private one as a  regularizer with compelling results on tabular data. We found this approach ineffective for computer vision, where usually models are pre-trained on public data. In Sec.~\ref{sec:experiments} we show that AdaMix works better on vision tasks, and is further improved with multi-modal data.%

\paragraph{Adaptive projection.}
\cite{kairouz2020fast,zhou2020bypassing,yu2021not} suggest improving DP-SGD by projecting the gradients into a lower-dimensional space estimated from public data.
However, these methods do not leverage the performance improvements derived from training on public data, which are needed to make vision models viable.
Using public data both to train and to estimate the subspace introduces more challenges, since the gradients of the public data and private data will then behave differently. Moreover, while they perform an SVD on the individual public sample gradients, we do the SVD of the total gradient matrix. Unlike the SVD of the matrix of individual sample gradients, this is cheaper to compute at each step.

\paragraph{Analysis.} \cite{alon2019limits} studies the theoretical limits of mixed privacy learning, but does not provide practical algorithms. We provide both a method for MixDP and a proof of convergence.
Our analysis leverages the results of %
\cite{lacoste2012simpler} which shows that a particular non-uniform averaging scheme of SGD converges at an $O(1/t)$ rate, rather than the classical $O(\log t/t)$ rate. In the context of differentially private learning, \cite{bassily2014private} showed that NoisySGD achieves the optimal rates for (convex) empirical risk minimization (ERM) \cite{chaudhuri2011differentially}. \cite{song21evading} provided the first analysis of NoisyGD in the general convex case.
To the best of our knowledge, we are the first to formally establish that the use of an arbitrarily small public data gives provably smaller sample complexity in learning compared to the standard private learning setting.

\section{Method}

\paragraph{Notation.} Let $z\in \mathcal{Z}$ be a datum, with $z=(x, y)$ being the feature/label pair; $\mathcal{Z}^*=\cup_{n=0}^\infty \mathcal{Z}^n$ is the universe of datasets with arbitrary size.
For learning, we have access to a private dataset $D_{\text{pri}} := \{z_1,...,z_{N_{\text{pri}}}\}$ and a public one $D_{\text{pub}}:=\{\tilde{z}_1,...,\tilde{z}_{N_{\text{pub}}}\}$. The loss function $\ell: \mathcal{Z}\times\mathcal{W}\rightarrow \mathbb{R}$ takes data $z\in\mathcal{Z}$ and ``weights'' $w\in\mathcal{W} \subset \mathbb{R}^d$ to yield the total loss on the private and public dataset $\mathcal{L}_\text{pri}(w):=\sum_{i=1}^{N_\text{pri}}\ell_i(w)$ and $\mathcal{L}_\text{pub}(w):=\sum_{j=1}^{N_\text{pub}}\tilde{\ell}_j(w)$, with $\ell_i$ and $\tilde{\ell}_j$ short-hands for $\ell(w,z_i)$ and $\ell(w,\tilde{z}_j)$ respectively. Notice that $\mathcal{L}_\text{pri}$ and $\mathcal{L}_\text{pub}$ are sums, not averages, of the loss functions.
Their gradients are indicated by $g_i^\text{pri}(w) = \nabla_w \ell_i(w)$, $g_j^\text{pub}(w) = \nabla_w \tilde{\ell}_j(w)$, $G^\text{pri}(w) = \nabla_w \mathcal{L}_\text{pri}(w)$, $G^\text{pub}(w) = \nabla_w \mathcal{L}_\text{pub}(w)$.
In addition, we say $N = N_\text{pri} + N_\text{pub}$ and $\mathcal{L}(w) = \mathcal{L}_\text{pri}(w) + \mathcal{L}_\text{pub}(w)$.

The average \textit{empirical risk} is written as $\hat{\mathcal{R}}(w):=\mathcal{L}(w)/N$. When the data are drawn from some distribution $\mathcal{D}$, the risk, i.e., generalization error, $\mathcal{R}(w):=\mathbb{E}_{z\sim \mathcal{D}}[\ell(z,w)].$ Our goal is to design a differentially private algorithm that returns $\hat{w}$ which minimizes $\mathcal{L}(\hat w)$. Its performance is measured by the \textit{excess empirical risk}, i.e., $\hat{\mathcal{R}}(\hat{w}) - \min_{w\in\mathcal{W}}\hat{\mathcal{R}}(w)$; and by the \textit{excess risk} $\mathcal{R}(\hat{w}) -\min_{w\in\mathcal{W}}\mathcal{R}(w)$. In the experiments, the latter is measured by the error on a test set.

\subsection{Differential privacy}
\label{sec:dp}
Two datasets $D$, $D' \in \mathcal{Z}^*$ are said to be \textit{neighboring}, written as $D \simeq D'$, if we can obtain $D'$ from $D$ by \textit{adding or removing} a single data point.
\begin{definition}[Differential privacy{\cite{dwork2006calibrating,dwork2014algorithmic}}]\label{def:dp} For $\epsilon > 0$ and $\delta \ge 0$, a randomized algorithm $\mathcal{M}$ is $(\epsilon, \delta)$-differentially private if for any neighboring datasets $D \simeq D^\prime$ and any measurable $S \subseteq \mathrm{range}(\mathcal{M})$,
\begin{align*}
\mathsf{Pr}[\mathcal{M}(D) \in S] \le e^\epsilon \cdot \mathsf{Pr}[\mathcal{M}(D^\prime) \in S] + \delta.
\end{align*}
\end{definition}
In our context, $\mathcal{M}$ is the randomized learning algorithm, and $\mathcal{Y}$ is the space of model weights. This condition bounds the maximum amount of information that an attacker, having access to the weights of the model, can extract about one of the training samples. The parameter $\epsilon$ is referred to as the \textit{privacy parameter}:
lower values of $\epsilon$ carry better privacy guarantees, but also make it harder to train an accurate model, as it reduces the amount of information about the training samples that can be represented via the parameters of the model. The goal of a good DP training algorithm $A$ is to train an accurate model while satisfying the $(\epsilon,\delta)$-privacy constraint selected by the user where, ideally, $\epsilon \approx 1$ and $\delta = o(1/n)$.

\paragraph{NoisyGD.} NoisyGD is defined in Algorithm~\ref{alg:noisyGD}. We now show that NoisyGD is differentially private and that, in the convex case, it converges to the optimal solution.

\begin{algorithm}[t]
\caption{(Projected) Noisy Gradient Descent}
\label{alg:noisyGD}
\KwData{Constraint set $\mathcal{W}$, initialization $w_1$, noise level $\sigma$, clipping threshold $\tau$, number of iteration $T$, weight decay parameters $\lambda$ and $w_\text{ref}$, learning rates $\eta_t$.}
\KwResult{$w_1,...,w_{T+1}$}

\For{$t = 1,\ldots,T$}{
$G_t = \sum_{i=1}^{N_\text{pri}} \operatorname{clip}_\tau (g_i(w_t))$\;
$n_t \sim N(0, \sigma^2 \tau^2 I)$\;
$w_{t+1} \gets w_t - \eta_t (G_t + n_t + \lambda (w_t - w_\text{ref}))$\;
$w_{t+1} \gets \Pi_{\mathcal{W}}(w_{t+1})$\;
}
\end{algorithm}

\begin{proposition}[NoisyGD is $(\epsilon,\delta)$-DP, informal]
\label{prop:noisygd-is-private-informal}
Algorithm~\ref{alg:noisyGD}
with parameter $T,\sigma^2$ such that $\rho:= \frac{T^2}{2\sigma^2}$ satisfies the $(\epsilon,\delta(\epsilon))$-DP, where $\delta(\epsilon) := \Phi(\frac{\mu}{2}-\frac{\epsilon}{\mu}) - e^\epsilon \Phi(-\frac{\mu}{2}- \frac{\epsilon}{\mu})$, $\mu = \sqrt{2\rho}$ and $\Phi$ is the CDF of the normal distribution.
\end{proposition}
For any prescribed privacy parameter $(\epsilon,\delta)$, it suffices to choose this $\rho$ parameter, thus from here onward we will use $\rho$ as the privacy parameter %
of interest (note that, asymptotically, $\rho \asymp \min\{\epsilon, \epsilon^2/\log(1/\delta)\}$).

\paragraph{Mixed private learning.} We define the problem of differentially private learning when we have access to an additional public dataset \emph{mixed private learning}. We say the algorithm $\mathcal{M}$ (now taking two arguments $D_{\text{pri}}$ and $D_{\text{pub}}$) satisfies $(\epsilon,\delta)$-DP if $\mathcal{M}(\cdot,D_{\text{pub}})$ satisfies $(\epsilon,\delta)$-DP (Def.~\ref{def:dp}) for every fixed $D_{\text{pub}}$.
\cite{alon2019limits} shows that a small public dataset from the same distribution allows one to privately PAC-learn any classes with bounded VC-dimension, but with an inefficient algorithm. We focus on designing practical DP algorithm
that can effectively use both public and private data under the setting of convex empirical risk minimization \cite{chaudhuri2011differentially} and convex fine-tuning of deep models \cite{achille2021lqf}.

\subsection{AdaMix}
\label{sec:adamix}

\paragraph{Model.} Following \cite{tramer2020differentially}, we use a linear logistic regression model
trained on top of the last layer features of an ImageNet pre-trained network. This reduces the expressivity of the model compared to training a full network. However, the significant reduction in number of parameters makes it a better choice DP algorithms and recent deep networks allow for competitive performance on most tasks even when training only the last layer. We normalize the features before feeding them to the logistic model, which both improves the accuracy and bound the gradients at each step.

\paragraph{Initialization and regularization.}
The choice of initialization is normally of minor concern for logistic regression: since the problem is convex we will converge to a global optimum regardless of the initialization. L2 regularization (weight decay), when employed, is usually centering at $0$ by default. However, centering the regularizer is critical in differential privacy, as we prove in the following result:

\begin{theorem}[Convergence of NoisyGD on strongly convex problems]
\label{thm:excess_empirical_risk_bound}
Assume $\mathcal{W}$ is a convex set satisfying $\sup_{w\in\mathcal{W}}\|w\|\leq B$. Let $w_1,...,w_{T+1}$ be the parameter vectors returned by running Algorithm~\ref{alg:noisyGD}
on the following regularized loss
$J(w) = \mathcal{L}(w) + \frac{\lambda}{2}\|w- w_{\text{ref}}\|^2$
with $w_1 = w_{\text{ref}}\in \mathcal{W}$,
learning rate $\eta_t = \frac{2}{\lambda(t+1)}$ and $\lambda = \sqrt{\frac{\rho}{2\|w^*-w_{\text{ref}}\|d L^2}}$.
Then, the ensemble solution:
\[
\textstyle \bar{w} = \frac{2}{T(T+1)} \sum_{t=1}^T t\, w_t
\] obeys the bound
\[
\textstyle
\mathbb{E}[\hat{\mathcal{R}}(\bar{w})]- \hat{\mathcal{R}}(w^*) \leq \frac{4(N \tau + \lambda B)^2 }{\lambda TN} + \frac{\sqrt{d} \tau\|w^*-w_{\text{ref}}\|}{\sqrt{2\rho}N}
\]
for any $w^*\in\mathcal{W}$, where $\rho$ is the privacy parameter of the algorithm.
\end{theorem}

Note that as $T \to +\infty$ the first term vanishes and the second term matches the information-theoretic limit for privately solving general convex ERM problems when $w_{\text{ref}} = 0$ \cite{bassily2014private}.
However, this can improve if we choose $w_{\text{ref}}$ according to the public data such that $w_{\text{ref}}$ is closer to $w^*$ than $0$. This motivated us to propose choosing $w_{\text{ref}}$ by training on the public data.
Then, we proceed to fine-tune the model on the concatenation of private and public data, using Noisy-GD while preserving $(\epsilon, \delta)$-DP.

\paragraph{Provable benefits of public data access.}
Next we provide a theoretical guarantee for the above approach under standard statistical learning assumptions.

\begin{theorem}[Generalization with access to public data, informal]
\label{thm:provable_benefit_of_public_data_informal}
Assume that private and public data are drawn i.i.d.\ from the same distribution $\mathcal{D}$ and that $R(w) = \mathbb{E}_{(x,y)\sim \mathcal{D}}[ \ell(w, (x,y))]$ is $c$-strongly convex in $\mathcal{W}$. Let $w_{\mathrm{ref}}$ be obtained by training for one epoch on the public dataset and let $\bar{w}$ be as in Thm.~\ref{thm:excess_empirical_risk_bound}, then at the limit when $T$ is sufficiently large, the \emph{excess risk} obeys
\begin{align*}
&\mathbb{E}[\mathcal{R}(\bar{w})] - \mathcal{R}(w^*) \\
\leq& \frac{4\sqrt{d} L^2}{c\sqrt{N_\text{pub}}N\sqrt{2\rho} }+ \text{Gen}(\bar{w},N) + \text{Gen}(w^*,N),\end{align*}
where $w^* = \mathop{\mathrm{argmin}}_{w\in\mathcal{W}} \mathcal{R}(w)$ and
$
\text{Gen}(w,N):= \left|\mathbb{E}[\mathcal{R}(w) - \hat{\mathcal{R}}(w)]\right|
$ is the expected generalization gap of (a potentially data-dependent) $w$.
\end{theorem}

To the best of our knowledge, Theorem~\ref{thm:provable_benefit_of_public_data_informal} is the first demonstrated claim that access to even a very small public dataset can improve private learning in the convex ERM setting.
We note that the generalization bound decomposes in a first term, which describes the generalization cost incurred by using DP, and a second part which is common to non-private learners. Interestingly, in the Appendix we show that in MixDP the cost of privacy asymptotically vanishes even if the percentage of public data becomes neglegible (i.e., when $N_\text{pub}\rightarrow \infty$, $N_{\text{pub}}/N\rightarrow 0$).

Besides providing a good initialization, access to a small public dataset also enables ways to further improve Noisy-GD in practice, as we will now see.

\paragraph{Adaptive threshold.} Since the norm of the gradients changes significantly during training, a fixed clipping threshold $\tau$ is suboptimal throughout training. \cite{wang2020differentially} suggests to tune this threshold adaptively using public data. However, the norm of the average gradient of the public sample, may not yield good clipping thresholds for the \textit{per-sample} gradients. Instead, we find the following adaptive threshold to be more robust:
\[
\tau = \operatorname{quantile}_{90}(\{\|g_i^\text{pub}(w)\|\}_{i=i}^{N_\text{pub}}),
\]
that is, we take the 90-th quantile of the norm of the public gradients, where $g_i^\text{pub}(w) = \nabla_w \tilde{\ell}_i(w)$ is the gradient of the $i$-th public sample.
Then the clipped gradient we use is
\[
\tilde{g}_i^\text{pri}(w) = \frac{g_i^\text{pri}(w)}{\|g_i^\text{pri}(w)\|} \min(\|g_i^\text{pri}(w)\|, \tau).
\]
This quantile rule is also used by \cite{andrew2019differentially}, but we differ in the use of public data to determine $\tau$, rather than allocating part of the privacy-budget for releasing the quantile.
It was established in \cite{chen2020understanding} that gradient clipping has the effect of Huberization of the loss function in the GLM cases. The convergence to the optimal solution of a modified objective function is guaranteed if we fix the clipping parameter. Our approach on the adaptive clipping has the effect of selecting the hyperparameter adaptively in a homotopy style algorithm, which shows further empirical benefits.

\paragraph{Adaptive subspace projection.} From Thm.~\ref{thm:provable_benefit_of_public_data_informal}, the utility of a DP algorithm decreases with the number of parameters $d$ of the model. In view of this, we do not train the whole network but rather use it as a feature extractor and only train a linear model on top. However, for modern vision architectures, even the number of parameters of the final layer is in the order of tens of thousands. Therefore, we use an adaptive mechanism to project private gradients onto a low-dimensional subspace before adding noise. More precisely,
\[
G^\text{pub}(w) = \nabla_w L_{D_\text{pub}}(w) = \sum_{i=1}^{N_\text{pub}} \nabla_w \ell (\tilde{z}_i, w)
\]
be the total gradient of the public dataset $D_\text{pub}$. Since we are using a linear logistic model, $G^\text{pub}(w)$ is a $D \times C$ matrix where $D$ is the number of features and $C$ is the number of classes.
Consider the SVD decomposition:
\[
G^\text{pub}(w) = U S V.
\]
We now project each (clipped) private gradient matrix $\tilde{g}_i^\text{pri}$ on the top $P$ components, and obtain a matrix $\hat{g}_i^\text{pri} = U^t \tilde{g}_i^\text{pri}$ of size $P \times C$.
Other methods projecting the gradient using public data \cite{zhou2020bypassing,yu2021not,kairouz2020fast} do not compute the SVD of the total gradient matrix, as noted in Sec.~\ref{sec:related}. Rather, use the vectorized \textit{per-sample} gradients of the public examples to compute the principal components to project. Since, in vision, the size of matrix of all per-sample gradients has a much larger size $N \times FC$ (rather than our $F \times C$), several approximations would be required to carry out this strategy.

\paragraph{Final algorithm.} The final gradient update step at time $t$ is:
\[
w_{t+1} \gets w_t - \eta_t ( G^\text{pub} + U \hat{G}^\text{pri} + \lambda \cdot w_t)
\]
where
\[
\textstyle \hat{G}^\text{pri} = \sum_{i=1}^{N_\text{pri}} U^T \tilde{g}^\text{pri}_i + n_t.
\]
and $n_t \sim \mathcal{N}(0, \sigma^2 \tau^2 I_C)$. We then train for $T$ steps. The pseudo-code of algorithm is given in Algorithm~\ref{alg:mix-s}.
\begin{proposition}
Algorithm~\ref{alg:mix-s}
with parameter $T,\sigma^2$ such that $\rho:= \frac{T^2}{2\sigma^2}$ satisfies the $(\epsilon,\delta(\epsilon))$-DP of a Gaussian mechanism in Theorem 8 \cite{balle2018improving} with parameter $\mu = \sqrt{2\rho}$.
\end{proposition}
Note that the privacy guarantee is the same as long as the ratio $\sigma/\sqrt{T}$ remains constant, which gives a degree of freedom in choosing $\sigma$ (hence $T$). %
Higher values of $\sigma$ requires more training steps $T$, but, by Thm.~\ref{thm:excess_empirical_risk_bound}, it also ensures better convergence (which we validate this empirically in Sec.~\ref{sec:experiments}). Hence, the user should generally pick the largest $\sigma$ allowed by their computational budget.

\begin{algorithm}[t]
\caption{AdaMix training algorithm.
}\label{alg:mix-s}
\KwData{Public dataset $D_\text{pub}$, private dataset $D_\text{pri}$, privacy parameter $(\epsilon, \delta)$, noise variance $\sigma$, learning rate $\eta$, $L_2$ reg. coefficient $\lambda$}
\KwResult{$w = A(S)$}
$T_\text{max} \gets \operatorname{Calibrate}(\epsilon, \delta, \sigma)$\;
$w \gets \operatorname{Pretrain}(S_u)$\;
\For{$t = 1,\ldots,T_\text{max}$}{
\Comment{Adaptive clipping}
$\tau_t \gets \operatorname{quantile}_{90}\big(\{\|g_i^\text{pub}(w)\|\}_{i=1}^{N_\text{pub}}\big) $\;
\Comment{Adaptive projection}
$U, S, V \gets \operatorname{SVD}(G_\text{public}(w))$\;
\Comment{Clip, project and add noise to private gradients}
$n_t \sim N(0, \sigma^2 \tau_t^2 I)$\;
$\tilde{g}^\text{pri}_i(w) \gets \frac{g_i^\text{pri}(w)}{\|g_i^\text{pri}(w)\|} \min(\tau_t, \|g_i^\text{pri}(w)\|)$\;
$\tilde{G}^\text{pri}(w) = \sum_{i=1}^{N_\text{pri}} U^T \tilde{g}^\text{pri}_i(w) + n_t$\;
\Comment{Final weights update}
$w \gets w - \eta (G^\text{pub}(w) + U \tilde{G}^\text{pri}(w) + \lambda \cdot w)$\;
}
\end{algorithm}

\begin{table*}
\centering
\small
\setlength{\tabcolsep}{2pt}
\begin{tabular}{|lc|c|c|ccc|ccc|}
\hline
& Public Shots & Non-Private & Only-Public & Fully-Private & PPGD\cite{wang2020differentially} & AdaMix & Fully-Private & PPGD\cite{wang2020differentially} & AdaMix\\
& & (paragon) & (baseline) &\multicolumn{3}{c|}{$\epsilon=1$} &\multicolumn{3}{c|}{$\epsilon=3$} \\
\hline
Ox.\ Flowers \cite{Nilsback06}\!\! & 2 & 12.34 & 40.16 & 95.12 & 40.99 & \textbf{39.48} & 81.42 & 39.22 & \textbf{36.11} \\
CUB-200 \cite{WelinderEtal2010} & 2 & 31.97 & 64.01 & 96.79 & 64.20 & \textbf{63.58} & 81.61 & 61.59 & \textbf{59.15}\\
Stanf.\ Dogs \cite{KhoslaYaoJayadevaprakashFeiFei_FGVC2011}\!\! & 2 & 10.08 & 16.58 & 33.87 & 16.42 & \textbf{15.38} & 15.93 & 15.61 & \textbf{12.72}\\
MIT-67 \cite{quattoni2009recognizing} & 5 & 25.17 & 43.58 & 69.55 & 42.02 & \textbf{40.46} & 42.99 & 39.60 & \textbf{33.03}\\
Oxford Pets \cite{parkhi12a} & 5 & 7.17 & 12.83 & 26.02 & 12.85 & \textbf{11.54} & 12.37 & 11.15 & \textbf{9.53} \\
Caltech-256 \cite{griffin2019caltech} & 5 & 14.64 & 24.88 & 60.72 & 24.61 & \textbf{23.74} & 27.15 & 23.71 & \textbf{20.85}\\
\hline
\end{tabular}
\caption{\textbf{Mixed privacy, full privacy and AdaMix.}
We report the result (test errors) of different methods on a diverse set of vision tasks (see text for description). Surprisingly \textbf{Only-Public}, which throws away private data and only trains a model on the small amount of public data outperforms \textbf{Fully-Private}, which trains on all data as if it was private. \textbf{AdaMix} is instead able to effectively use both the private and public data at the same time, and achieves a significant improvement even at relatively small value of $\epsilon$ (e.g., 10\% improvement on MIT-67 w.r.t.\ Only-Public). Instead, the closest method to us, \textbf{PPGD} does not significantly improve over the Only-Public baseline.
}
\label{table:mixdp}
\end{table*}

\subsection{Textual side information}
\label{sec:multi-modal}

CLIP \cite{radford2021learning} is an embedding trained so that the representation of an image is close to the representation of its textual description.
This enables zero-shot learning, i.e., initializing an high-performance image classification model using only the name of the labels or a textual description of the classes, which is generally public.
AdaMix can be easily adapted to work in this multi-modal MixDP setting, where the public data comes from a different modality. First, we initialize a linear logistic model using the names of the classes: Let $c_i$ be the text associated to the $i$-th class, the weight matrix is \cite{radford2021learning}:
\[
W = \beta [\enc(c_1), \ldots, \enc(c_C)],
\]
that is, each row is the normalized encoding of a label scaled by a parameter $\beta$ (we use $\beta=100$).
Starting from this public initialization, we train with AdaMix on image data.

\section{Results}
\label{sec:experiments}
\vspace{-5pt}
\paragraph{Setting.} Unless otherwise specified, we train a linear logistic model on top of the last layer features of a ResNet-50 \cite{he2016deep} pre-trained on ImageNet \cite{deng2009imagenet}. For NoisyGD, we initalize the logistic weights to zero, fix $\sigma=20$ and $\delta=10^{-5}$. We train with $\lambda=1e-2$, weight decay centered at zero, learning rate $\eta \in \{0.0005,0.001,0.0025,0.005\}$, and select the best learning rate for each value of $\epsilon$. For AdaMix, we initialize the weights by training on the public data until convergence. We use the Gaussian mechanism implementation of \texttt{autodp}\footnote{\url{https://github.com/yuxiangw/autodp}} to calibrate the parameters of the training algorithm, in particular $T_{\max}$, in order to achieve the desired $(\epsilon, \delta)$-DP requirement.

\paragraph{Datasets.} We test our algorithm on the following vision classification datasets commonly used in transfer learning (see Tab.~\ref{table:mixdp}).
We split each training dataset into a ``public'' dataset consisting of $N$-samples per class (see Tab.~\ref{table:mixdp}), and consider the remaining training data as private. In all cases, the public data is less than $10\%$ of the private data.

\paragraph{Baselines considered.}
We compare AdaMix with the following baselines:
\textbf{(Fully-Private)} Ignore the MixDP setting and simply train on all data as if it was private, starting form a zero initialization and using NoisyGD;
\textbf{(Only-Public)} Train a standard non-private model using only the few public samples and discarding the private ones;
\textbf{(PPGD)} A version of PPSGD \cite{wang2020differentially}, but trained with NoisyGD rather than DP-SGD for easier comparison. Unlike AdaMix, PPGD has several hyper-parameters: we manually searched for the best combination to ensure the best comparison;
\textbf{(NGD)} An ablated version of AdaMix which uses fixed hyper-parameters rather than the adaptive components.

\subsection{Experiments}
\vspace{-5pt}
\paragraph{Mixed Privacy vs Full Privacy.}
In \Cref{table:mixdp} we see that, perhaps surprisingly, Only-Public is always significantly better than Fully-Private even if it actually throws away most of the available data. This suggests that, in vision, collecting a small amount of public data may be preferred to collecting a much larger amount of private data (Fig.~\ref{fig:boost}). However, we see that AdaMix is able to effectively use both the private and public data at the same time, and achieves a significant improvement even at relatively small value of $\epsilon$ (e.g., 10\% improvement on MIT-67 w.r.t.\ Only-Public at $\epsilon=3$). Conversely, the method closest to us, PPGD, does not significantly improve over the Only-Public baseline even after our changes to adapt it better to vision tasks.

\begin{figure*}
\centering
\includegraphics[width=.99\linewidth]{figures/mit_err_mi_eps.pdf}
\caption{\textbf{Test error vs privacy for several methods.} \textbf{(Left)}
On MIT-67, we show the test error obtained by different methods at different privacy parameters $\epsilon$. The top horizontal red line shows the perfectly private Only-Public baseline (not using the private data at all). The bottom green line shows the non-private paragon (training on all data as if they where public). By changing the privacy parameter $\epsilon$, we can interpolate between the two extrema.
We see that AdaMix
performs uniformly better than other methods, and can use private and public data together to obtain significant boost at all $\epsilon$.
\textbf{(Center)} Same as the before, but we zoom in to show the behavior of the different solutions in the low-$\epsilon$ regime.
\textbf{(Right)} For the same algorithms, we plot the robustness to adversarial attacks (measured by AUC) vs test error obtained using different values of $\epsilon$ (annotated near each point). AdaMix obtains the best trade-off between accuracy and robustness.
}
\label{fig:comparison}
\end{figure*}

\paragraph{Ablation study and finer comparison.} In Fig.~\ref{fig:comparison} (left and center) we plot the average accuracy obtained by various methods and ablations of AdaMix for different privacy parameters $\epsilon$. Both AdaMix and the ablated NGD are able to improve over the Only-Public baseline. Interestingly, this holds true even when considering relatively low privacy parameters $\epsilon \approx 1$ which is usually considered too restrictive to train in a fully private setting (as seen in Tab.~\ref{table:mixdp}). On the other hand, thanks to the MixDP training we can obtain high-accuracy models even at restrictive privacy settings. We note that AdaMix performs uniformly better than other methods at all privacy regimes. From the ablation of AdaMix, we see that initializing the model with the public data has the largest impact, followed by adaptive clipping and adaptive projection.

\paragraph{Membership attacks.} Differential privacy is a very strong requirement, and is it often assumed that using higher value of $\epsilon$ may still provide valuable defense to attacks \cite{shokri2017membership,carlini2019secret,carlini2020extracting,rahman2018membership,bernau2019assessing,zhu2020deep,sablayrolles2019white,lecuyer2019certified,ma2019data} even if the theoretical guarantees are weak. Conversely, \cite{jagielski2020auditing} suggests using attacks to give a lower-bound on the privacy parameter of the algorithm. In Fig.~\ref{fig:comparison} (right) we plot the Area Under Curve of a membership attack executed against models for different values of $\epsilon$.
We use a thresholding based membership attack as proposed in \cite{sablayrolles2019white}. The loss for samples in the private training set are labelled as 0 and test samples are labelled as 1. Then we simply compute the AUC for this. We find that the sorting of the algorithms does not change: algorithms with better theoretical bounds are also more robust to attacks.

\paragraph{Performance Boost and size of public data.}
We hypothesize that, when enough public data is present, the contribution of using private data may be less noticeable. To test this, we define the Performance Boost of a Mixed Privacy method as $\text{PB} = (\text{err}_\text{public} - \text{err}_\text{paragon}) / (\text{err}_\text{DP} - \text{err}_\text{paragon})$, where $\text{err}_\text{paragon}$ is the error obtained by training on non-private model on both public and private data, $\text{err}_\text{public}$ is the error obtained by training only on the public data, and $\text{err}_\text{DP}$ is the error obtained by using training a mixed privacy model. This measures how well we can learn from the private data in the MixDP setting compared to the non-private upper-bound. In Fig.~\ref{fig:boost} we plot the performance boost for different sizes of the public set. We observe that the performance boost derived of using mixed privacy is very high when the public data is scarce, disappears when the amount of public data is comparable or larger than the private data.

\paragraph{Multi-modal initialization.}
In Fig.~\ref{fig:multimodal} we compare CLIP models initialized with (1) a random initialization, (2) a zero-shot initialization based on the public label names (Sec.~\ref{sec:multi-modal}), and (3) using few public image samples. While the latter performs the best at a given privacy level $\epsilon$, the zero-shot initialization still improves significantly over the random initialization, thus showing that even just small amount of textual information (which is often available) can be used to significantly boost accuracy.
Multi-modal models may also learn a better lower-dimensional representation of the data that is more amenable to DP. In Fig.~\ref{fig:multimodal} we compare AdaMix using an ImageNet pretrained ResNet-50 and using CLIP. This is not a direct comparison, but we see a significant performance boost by using CLIP at a given privacy level $\epsilon$, suggesting that the move to multi-modal model may be a logical next step for DP.

\begin{figure}[b]
\centering
\includegraphics[width=.98\linewidth]{figures/mit_priv_non_priv_boost.pdf}
\caption{\textbf{(Left) Trade-off between public and private examples.} We show the accuracy reached on MIT-67 using $N$ samples per class (x-axis) when they are public (green) or private (in red for different values of $\epsilon$). Fully private training requires as much as 10 times more samples. This leaves users with a trade-off between collecting more private data or, if possible, a smaller amount of public data.
\textbf{(Right) Performance boost using a private set for different methods.} We plot the Performance Boost (PB) for different DP methods as the ratio of public to private samples changes (see text for definition). We see that AdaMix achieves the best performance boost. As expected, when the public set is relatively large, the boost from adding private data decreases.%
}
\label{fig:boost}
\end{figure}

\begin{figure*}[t]
\centering
\includegraphics[width=0.9\linewidth]{figures/pDP_domain_shift_final_v2.pdf}
\caption{\textbf{(Left and center) Effect of public data on per-instance DP.} For $\epsilon=3$, we plot the sorted pDP scores for each sample in the dataset with Fully-Private training \textbf{(left)} and with AdaMix \textbf{(center)}. AdaMix makes the the target $\epsilon$ (horizontal line) closer to pDP value used by most samples (blue curve).
\textbf{(Right) MixDP and domain-shifts.}  We plot the performance of AdaMix when (purple curve) the public data come from the same domain as the test (real images), and (blue curve) from different domains (public images are paintings). We see that AdaMix use the private data from the target domain (real images) to adapt the solution to the target task. This significantly improves over using only the public data from the wrong domain (red line) and for large $\epsilon$ approaches the non-private limit (green line).
}
\label{fig:individual}
\end{figure*}

\paragraph{Effect of differential privacy on individual samples.}
DP is a worst-case guarantee which may penalize the accuracy of a model on unusual data, or data on the long-tails of the distribution, which however may play an important role for generalization in vision \cite{feldman2020does,feldman2020neural}. MixDP may ease the problem, since it allows to collect public data to ensure that each sub-population is sufficiently covered, lessening the accuracy cost of making that data private. We show this effect using the per-instance DP (pDP) analysis of \cite{wang2019per}, which measures the privacy loss $\epsilon'$ incurred by each individual $z$ during the private training. This allows us to provide a finer analysis of different methods by comparing the histogram of pDP loss $\epsilon$ for each individual in the dataset $D$ on top of the worst case DP bounds, which we do in Fig.~\ref{fig:individual} (see Appendix for details and theoretical analysis).
We observe that, when using AdaMix, the pDP losses of most samples are the same, and is close to the maximum privacy budget $\epsilon=3$,  indicating that we are using the prescribed privacy budget more effectively and more uniformly across samples, thereby improving utility.

\paragraph{Domain shift.} Often, the private and public data may come from slightly different domains. To test the robustness of AdaMix to domain shifts, we use the DomainNet dataset, which aims to classify objects in different domains. In particular, we train a model to classify objects in painting, using however public data coming from real pictures (while the private data is in the painting domain). In Fig.~\ref{fig:individual} (right), we show that, even if the public comes from a different domain, AdaMix is still able to use the public data to significantly improve performance of private training.

\paragraph{Effect of $\sigma$.} Outside of the learning rate $\eta$, the main other sensitive hyper-parameter of AdaMix is the noise variance $\sigma$. From Prop.~\ref{thm:excess_empirical_risk_bound} we expect that the best accuracy will be obtained using a large $\sigma$ and training for a respectively larger amount of epochs $T$. In the Appendix, we plot the test error as a function of $\sigma$ and we see that indeed, a larger $\sigma$ give better results at the expense of longer training time.

\vspace{-8pt}
\section{Discussion}
\vspace{-1pt}
We study Mixed Differential Privacy learning in computer vision, and show that in this setting differential privacy can be used to provide high accuracy models while respecting a given privacy parameter. To do this, we introduce AdaMix, an adaptive differential privacy mechanism that uses the public data to initialize and compute, at each step, an optimal subspace and clipping threshold. We also show that multi-modal vision and text can significantly improve accuracy under a privacy constrain.

\paragraph{Limitations.} In our analysis we train a linear model on pre-trained features, rather than fine-tuning the whole network. This is common in DP, since fine-tuning multiple layers may not significantly improve the accuracy while the additional parameters negatively affects the privacy bounds. This limitation is partially offset by new models that are trained to have generic and transferable last-layer features without the need of additional fine-tuning. We discuss several theoretical bounds that guide us in understanding the behavior of the algorithm. However, these bounds refer to a slighlty simplified version of AdaMix (see Appendix for details) and use hyper-parameters that are asymptotically optimal, but may not be practical.

\paragraph{Conclusions.} Differentially Private models in computer vision often cannot reach the same level of performance as non-private model for realistic privacy parameters. Using AdaMix in the MixDP learning setting, we have shown that, assuming the presence of a small amount of public data, accurate networks can be produced that reach better accuracies than fully private training without compromising the privacy of the data. We hope that further research in this field may help the adoption of private models in more areas of computer vision.
\vspace{-1cm}
\begin{figure}[b]
\centering
\includegraphics[width=.98\linewidth]{figures/clip_mit_initialization_features_quality.pdf}
\caption{\textbf{(Left) Multi-modal initialization is better than zero initialization.} We compare the test accuracy on MIT-67 of a CLIP model trained with $\epsilon=3$ using different initialization strategies. We observe that initializing the logistic weights to zero leads to the worst results under a given privacy parameter. Using the public label names to initialize the weights (language initialization) performs significantly better. However, using the few available public images to initialize the weights still outperforms the other choices.
\textbf{(Right) Multi-modal models.} We compare the performance of ResNet-50 model, and a CLIP model. We see that CLIP performs significantly better under a given privacy parameter. While the ResNet-50 model and the CLIP model are not directly comparable due to different pre-training, this further reinforces that stronger pre-training can indeed benefit the privacy setting and that the feature space learned by multi-modal models may be better suited for privacy than standard vision networks. }
\label{fig:multimodal}
\end{figure}

\clearpage

{\small
\bibliographystyle{ieee_fullname}

\section{Differential privacy basics}
\label{sec:dp-basics}
In this section, we describes tools we need from differential privacy and use them to prove that Algorithm~\ref{alg:mix-s} and Algorithm~\ref{alg:mix_theory} satisfies a family of differential privacy parameters.

\begin{lemma}[Analytical Gaussian mechanism {\cite{balle2018improving}}]
For a numeric query $f : \mathcal{X}^* \rightarrow \mathbb{R}^d$ over a dataset $\mathcal{D}$, the randomized algorithm that outputs $f(\mathcal{D}) + Z$ where $Z \sim \mathcal{N}(0, \sigma^2 I_d)$  satisfies $(\epsilon,\delta(\epsilon))$-DP for all $\epsilon\geq 0$ and
$\delta(\epsilon) = \Phi(\frac{\mu}{2}-\frac{\epsilon}{\mu}) - e^\epsilon \Phi(-\frac{\mu}{2}- \frac{\epsilon}{\mu})$
with parameter $\mu:=\Delta/\sigma$, where $\Delta:=\Delta_2^{(f)} = \max_{\mathcal{D} \sim \mathcal{D}^\prime} \| f(\mathcal{D}) - f(\mathcal{D}^\prime) \|_2$ is the global L2 sensitivity of $f$ and $\Phi$ is the CDF function of $\mathcal{N}(0,1)$.
\label{lm:ana_gau}
\end{lemma}
The above lemma tightly characterizes the $(\epsilon,\delta)$-DP guarantee of a single invocation of the Gaussian mechanism, and the following lemma shows that we can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.

\begin{definition}[Gaussian Differential privacy {\cite{dong2021gaussian}}]
We say a mechanism $\mathcal{M}$ satisfies $\mu$-Gaussian differential privacy (GDP), if it satisfies $(\epsilon,\delta(\epsilon))$-DP for all $\epsilon\geq 0$ and $\delta(\epsilon)$ being that of a single Gaussian mechanism (in Lemma~\ref{lm:ana_gau}) with parameter $\mu$.
\end{definition}

\begin{lemma}[Composition of Gaussian mechanisms \cite{dong2021gaussian}]
The adaptive composition of a sequence of Gaussian mechanism with noise level $\sigma_1,\sigma_2,\dots$ and global L2 sensitivity $\Delta_1, \Delta_2, \dots$ satisfies $\mu$-GDP with
parameter $\mu = \left(\sum_{i} (\Delta_{i}/\sigma_i)^2\right)^{1/2}$.
\label{lm:composition_gau}
\end{lemma}
Specifically, the noisy gradient descent (NoisyGD) algorithm (Algorithm~\ref{alg:noisyGD}) we use is a composition of $T$ Gaussian mechanisms for releasing the gradients and its privacy guarantee is equivalent to that of a single Gaussian mechanism.
\begin{proposition}
\label{prop:noisygd-is-private}
Let $\nabla f(w)$ be a function of the private dataset with global L2 sensitivity smaller than $L$ for any $w\in \mathcal{W}$, then Algorithm~\ref{alg:noisyGD}
with parameter $T,\sigma^2$ such that $\rho:= \frac{T^2L^2}{2\sigma^2}$ satisfies $\sqrt{2\rho}$-Gaussian differential privacy.
\end{proposition}
\begin{proof}
The proof follows from Lemma~\ref{lm:composition_gau} as Algorithm~\ref{alg:noisyGD} is an adaptive composition of $T$ Gaussian mechanisms with global sensitivity $L$. %
\end{proof}

\section{Per-instance differential privacy}
\label{sec:perinstance-dp}
In this section, we provide details on the per-instance differential privacy \cite{wang2019per} that we used to generate Figure~\ref{fig:individual}.
To cleanly define per-instance differential privacy, we first define indistinguishability.
\begin{definition}[$(\epsilon,\delta)$-indistinguishability]
We say two distributions $P,Q$ are $(\epsilon,\delta)$-indistinguishable if for any measurable set $S$
\begin{align*}
&\mathsf{Pr}_P[S] \le e^{\epsilon} \cdot \mathsf{Pr}_Q[S] + \delta
\text{ and } \mathsf{Pr}_Q[S] \le e^{\epsilon} \cdot \mathsf{Pr}_P[S] + \delta.
\end{align*}
\end{definition}

\begin{definition}[\cite{wang2019per}]
We say a randomized algorithm $\mathcal{M}$ is $(\epsilon(\cdot),\delta)$-per-instance differentially private (pDP) for scalar $\delta\geq 0$ and function $\epsilon: \mathcal{Z}^* \times \mathcal{Z} \rightarrow \mathbb{R}_+$, such that for any dataset $D\in\mathcal{Z}^*$, individual $z\in\mathcal{Z}$,
$\mathcal{M}(D)$ and $\mathcal{M}(D\cup \{z\})$ are $(\epsilon(D,z),\delta)$-indistinguishable.
\end{definition}

pDP loss $\epsilon$ is a function of one particular pair of neighboring datasets. It describes the formal privacy loss incurred to the particular individual $z$ that is added (if $z$ is not part of the input dataset) or removed (if $z$ is part of the input dataset).
pDP is a strict generalization of DP, as we can recover DP from pDP by maximizing $\epsilon(\cdot)$ over $D,z$.
\begin{lemma}
If $\mathcal{M}$ is $(\epsilon(\cdot),\delta)$-pDP, then $\mathcal{M}$ is also $(\sup_{D\in\mathcal{Z}^*,z\in\mathcal{Z}}\epsilon(D,z),\delta)$-DP.
\end{lemma}

We emphasize that the pDP loss $\epsilon(\cdot)$ itself is data-independent, but specific evaluations of the pDP losses (e.g., $\epsilon(D_{-z},z)$ or $\epsilon(D,z)$) depends on the private dataset $D$, thus should not be revealed unless additional care is taken to privately release these numbers.

For our purpose, we are interested in the distribution of pDP losses of individuals in the dataset induced by different DP algorithms. This is used to provide a theoretically-sound alternative to the prevalent practices of using specific attacks (such as membership inference attacks) for evaluating the data-dependent privacy losses.
Before we state the pDP bounds of our algorithms, we extend the standard $(\epsilon(\cdot),\delta)$-pDP definition to a per-instance version of the Gaussian DP.
\begin{definition}
We say a mechanism is $\mu(\cdot)$-per-instance Gaussian Differentially Private (pGDP), if $(D,D\cup z)$ (and $(D\cup z, D)$) are $(\epsilon,\delta)$-indistinguishable for all $\epsilon,\delta$ parameters described by $\mu(D,z)$-GDP.
\end{definition}

This allows us to obtain precise pDP bounds under composition.
\begin{proposition}[pDP analysis of AdaMix]\label{prop:pDP_of_adamix}
Let $z_1,...,z_n$ be the data points of the private dataset. Algorithm~\ref{alg:mix_theory} satisfies $\mu(\cdot)$-pGDP with
$$\mu(D_{-i},z_i) = \sqrt{\sum_{t=1}^{T} \frac{\min\{\|\nabla\ell_i(w_t)\|,\tau\}^2}{\sigma^2}}.$$
Similarly, Algorithm~\ref{alg:mix-s} satisfies $\mu(\cdot)$-pGDP with
$$
\mu(D_{-i},z_i) = \sqrt{\sum_{t=1}^{T} \frac{\|U^T \tilde{g}^\text{pri}_i(w_t)\|_2^2}{\tau_t^2\sigma^2}}.
$$
\end{proposition}
\begin{proof}
    Both algorithms are the composition of $T$-Gaussian mechanisms. Thus the results follow by the composition of pGDP. The composition of pGDP is implied by the composition theorem of GDP (Lemma~\ref{lm:composition_gau}) by choosing the space of datasets to be just $\left\{ D,D\cup\{z\}\right\}$.
\end{proof}

Fixing any $\delta$, we can then compute the corresponding $\epsilon(\cdot)$ for $(\epsilon(\cdot),\delta)$-pDP using the formula of Gaussian mechanism with $\mu$ taken to be $\mu(\cdot)$ pointwise.

\section{Proofs of the technical results}
\label{sec:proofs}

\begin{algorithm}[t]
\caption{(Theoretical version of) AdaMix training algorithm (no clipping, no adaptive projection, slightly different pretraining, theoretically chosen learning rate).
}\label{alg:mix_theory}
\KwData{Public dataset $D_\text{pub}$, private dataset $D_\text{pri}$, privacy parameter $(\epsilon, \delta)$, noise variance $\sigma$, Lipschitz constant $L$\footnotemark, population-level strong convex parameter $c$, regularization parameter $\lambda$ and a constraint set $\mathcal{W}$.}
\KwResult{$\bar{w}$}

\Comment{Public Pretraining Phase (OnePassSGD on $D_\text{pub}$):}
$w_1 = 0$.
\For{$t = 1,\ldots,N_{\text{pub}}$}{
\Comment{In a shuffled order}
$\eta_t = \frac{2}{c(t+1)}$\;
$w_{t+1} \gets \Pi_{\mathcal{W}}\big(w_{t} - \eta_t \nabla \ell(w_t, (\tilde{x}_t,\tilde{y}_t))\big)$\;
}
$w_{\text{ref}} \gets w_{N_{\text{pub}}+1}$.

\Comment{Mix Training Phase (NoisyGD on $D_\text{pub}\cup D_\text{pri}$):}

$T \gets \operatorname{Calibrate}(\epsilon, \delta, \sigma)$ \Comment{(i.e., $\frac{T\tau^2}{2\sigma^2} =: \rho$)}
\Comment{NoisyGD on objective function $\mathcal{L}(w) + \frac{\lambda}{2}\|w-w_{\text{ref}}\|^2$}

$w_1 = w_{\text{ref}}$\;
\For{$t = 1,\ldots,T$}{
    $
    \eta_t \gets \frac{2}{\lambda(t+1)}
    $\;

    $n_t \sim N(0, \sigma^2 I)$\;

    $
    w_{t+1} \gets \Pi_{\mathcal{W}}\big(w_t - \eta_t (\sum_{i=1}^{N_{\text{pri}}} \nabla \ell_i(w_t) + \sum_{j=1}^{N_{\text{pub}}} \nabla \tilde{\ell}_j(w_t) + n_t ) \big)
    $\;
}
\Comment{The following averaging can be implemented incrementally without saving $w_t$}
$\bar{w} \gets \sum_{t=1}^T\frac{2t}{T(T+1)}w_t$

\end{algorithm}
\footnotetext{As we discussed earlier, per-example gradient clipping can be viewed as Huberizing the loss function in GLMs \cite{song21evading}, all our results apply to the updated loss function. The adaptive clipping approach we took, can be viewed as an heuristic that automatically identifies an appropriate level of Huberization.}

We will be using the following $O(1/t)$ convergence bound due to Lacoste-Julien, Schmidt and Bach \cite{lacoste2012simpler}, which uses a decaying learning rate and a non-uniform averaging scheme.
\begin{theorem}[Convergence of SGD for Strongly Convex Objectives {\cite{lacoste2012simpler}}]\label{thm:lacoste-julien}
Let $f$ be a $m$-strongly convex and defined on a convex set $\mathcal{W}$. Assume stochastic gradient oracle $g_t$ satisfies that $\mathbb{E}[g_t | w_t] \in \partial f(w_t)$ and $\mathbb{E}[\|g_t\|^2] \leq G^2$ for all $t=1,...,T$.  Then the (projected) stochastic gradient descent with learning rate $\eta_t=\frac{2}{m(t+1)}$ satisfies
\begin{align}
    &\mathbb{E}[f(\sum_{t=1}^T \frac{2t}{T(T+1)}w_t)] - f(w^*)\leq \frac{2 G^2}{m(T+1)}\\
    \text{and }&\mathbb{E}[\|w_{T+1} - w^*\|^2] \leq \frac{4G^2}{m^2(T+1)}.
\end{align}
\end{theorem}

\begin{corollary}[NoisyGD for Strongly Convex Objectives]\label{cor:noisygd_bound}
Let $J(w) = \mathcal{L}(w) + \frac{\lambda}{2}\|w\|^2$ with individual loss functions $\ell$ being $L$-Lipschitz on $\mathcal{W}$.  Assume $\sup_{w\in\mathcal{W}} \|w\| \leq B$.
Let the learning rate be $\eta_t = \frac{2}{\lambda (t+1)}$, then
		\begin{align}
		\mathbb{E} \left[J\left(\frac{2}{T(T+1)}\sum_{t=1}^T  t w_t\right)\right]  - J^*
		\leq& \frac{2(N L + \lambda B)^2 }{\lambda T} + \frac{2d\sigma^2}{\lambda T}\\
		 =& \frac{2(N L + \lambda B)^2  }{\lambda T} + \frac{d L^2}{\lambda \rho},
	\end{align}
where $\rho:= \frac{TL^2}{2\sigma^2}$ is the privacy parameter of the algorithm ($\sqrt{2\rho}$-GDP).
\end{corollary}
\begin{proof}
First check that $NL+ \lambda B$ upper bounds the Lipschitz constant of $J$ on because the Lipschitz constant of $\frac{\lambda}{2}\|w\|^2$ is smaller than $\lambda B$ due to the bounded domain.
Second, check that the noisy gradient oracle satisfies that it is unbiased, and the added noise has a variance of $\sigma^2$ per coordinate for all $d$ coordinates. Thus
$$
\mathbb{E}[\|g_t\|^2 | w_t] = \mathbb{E}[\|\nabla J(w_t)\|^2 | w_t] + \mathbb{E}[\|n_t\|^2|w_t] \leq (NL + \lambda B)^2 + d\sigma^2.
$$
Thus by taking expectation on both sides we verify that we can take $G^2 = (NL + \lambda B)^2 + d\sigma^2$.

It remains to substitute these quantities and apply the first statement of Theorem~\ref{thm:lacoste-julien}.
\end{proof}

\begin{corollary}[One-Pass SGD on public data]\label{cor:sgd_public_data}
Assume the public data with $N$ samples are drawn from the same distribution of the private data. Assume that the (population risk)
$$
R(\theta) = \mathbb{E}_{(x,y)\sim \mathcal{D}}[ \ell(\theta^*, (x,y))]
$$
is $c$-strongly convex at $\theta^*$ for some constant $c$. Then the one-pass SGD algorithm below
$$
w_{t+1} = w_t - \frac{2}{c(t+1)} \nabla \ell(w_t, (x_t,y_t))
$$
for $t=1,...,N_\text{pub}$
obeys that
	$$
	\mathbb{E}[\|w_{N_\text{pub} +1} - w^*\|^2] \leq \frac{4L^2}{ c^2 N_\text{pub}}
	$$
	where $w^* = \mathop{\mathrm{argmin}}_{w\in \mathcal{W}} \mathcal{R}(w).$
\end{corollary}
\begin{proof}
First note that since the  data is drawn iid, running one-pass SGD by going through the data points in a random order uses a \emph{fresh sample} to update the parameters. This is is equivalent to optimizing the population risk directly. Check that for any fixed $w$ and all $t=1,...,N_\text{pub}$ $\mathbb{E}_{(x_t,y_t)\sim \mathcal{D}}[\nabla_{w} \ell(w,(x_t,y_t))] = \nabla \mathcal{R}(w).$
Moreover, we need this stochastic gradient oracle to satisfy
$\mathbb{E}_{(x,y)\sim \mathcal{D}}[\|\nabla \ell(\theta,(x,y))\|^2]\leq  G^2$. Notice that by our assumption $\|\nabla \ell(\theta,(x,y))\|^2\leq L^2$, thus we can take $G=L$.
By invoking the second statement of Theorem~\ref{thm:lacoste-julien} the result follows.
\end{proof}

With these two corollaries stated, we are now ready to prove Theorem~\ref{thm:excess_empirical_risk_bound} and Theorem~\ref{thm:provable_benefit_of_public_data}.

\begin{proof}[Proof of Theorem~\ref{thm:excess_empirical_risk_bound}]
    The proof relies on Corollary~\ref{cor:noisygd_bound}, and the following argument.
    When additional regularization with parameter $\lambda $, the utility we consider should still be considered in terms of $\mathcal{L}(\hat{w}) - \mathcal{L}(w^*)$. Let $w^*$ be any comparator satisfying $B> \|w^*\|$
\begin{align*}
	 &\mathcal{L}(\bar{w}) - \mathcal{L}(w^*)
	 =J(\bar{w}) - J_\lambda(w^*_\lambda)  \\
	 &+J_\lambda(w^*_\lambda) - J(w^*) +  \frac{\lambda}{2}\|w^*-w_{\text{ref}}\|^2 -   \frac{\lambda}{2}\|\hat{w}-w_{\text{ref}}\|^2\\
	 \leq& J(\hat{w})  - J_\lambda(w^*_\lambda) + \frac{\lambda}{2}\|w^*-w_{\text{ref}}\|^2.
\end{align*}
Take expectation on both sides and apply Corollary~\ref{cor:noisygd_bound}
{\small
\begin{align*}
	\mathbb{E}[\mathcal{L}(\bar{w})]- \mathcal{L}(w^*) \leq \frac{2(N L + \lambda B)^2  }{\lambda T} + \frac{d L^2}{\lambda\rho} +  \frac{\lambda}{2}\|w^*-w_{\text{ref}}\|^2.
\end{align*}
}
Finally, choosing $\lambda  = \sqrt{\frac{\rho}{2\|w^*-w_{\text{ref}}\|d L^2}}$ yields
$$
\mathbb{E}[\mathcal{L}(\hat{w})]- \mathcal{L}(w^*) \leq   \frac{4(n L + \lambda B)^2  }{\lambda T}  + \frac{\sqrt{d} L\|w^*-w_{\text{ref}}\|}{\sqrt{2\rho}}
$$
as claimed (dividing $N$ on both sides to get $\hat{R}$).
\end{proof}

\begin{theorem}\label{thm:provable_benefit_of_public_data}
	Assume the private data and public data are drawn i.i.d.\ from the same distribution $\mathcal{D}$ and that $R(w) = \mathbb{E}_{(x,y)\sim \mathcal{D}}[ \ell(w, (x,y))]$ is $c$-strongly convex in $\mathcal{W}$.  Let $w_{\mathrm{ref}} = w_{N_\text{pub} +1}$ --- the last iterate of a single pass stochastic gradient descent on $\hat{\mathcal{R}}_\text{pub} (w)$ (initializing from $w_0= 0$) that goes over the public dataset exactly once one data-point at a time with learning rate $\eta_t = \frac{2}{c(t+1)}$. Let  $w_{\mathrm{ref}}$ be passed into Theorem~\ref{thm:excess_empirical_risk_bound}'s instantiation of NoisyGD, which returns $\bar{w}$ (The pseudo-code of this algorithm is summarized in Algorithm~\ref{alg:mix_theory} in the appendix), then at the limit when $T$ is sufficiently large, the \emph{excess risk} obeys that
	\begin{align*}
	&\mathbb{E}[\mathcal{R}(\bar{w})]  -  \mathcal{R}(w^*) \\
	\leq& \frac{4\sqrt{d} L^2}{c\sqrt{N_\text{pub}}N\sqrt{2\rho} }+  \text{Gen}(\bar{w},N) + \text{Gen}(w^*,N),\end{align*}
	where $w^* = \mathop{\mathrm{argmin}}_{w\in\mathcal{W}} \mathcal{R}(w)$ and
	$
	\text{Gen}(w,N):= \left|\mathbb{E}[\mathcal{R}(w) - \hat{\mathcal{R}}(w)]\right|
	$ is the expected generalization gap of (a potentially data-dependent) $w$.
\end{theorem}

\begin{proof}[Proof of Theorem~\ref{thm:provable_benefit_of_public_data}]
	Let $w^* = \arg\min_{w\in \mathcal{W}} \mathcal{R}(w)$. By Corollary~\ref{cor:sgd_public_data}, we have
	$$
	\mathbb{E}[\|w_{\text{ref}} - w^*\|^2] \leq \frac{4L^2}{ c^2 N_\text{pub}},
	$$
	which implies (by Jensen's inequality) that
	$\mathbb{E}[\|w_{\text{ref}} - w^*\|] \leq \sqrt{\frac{4L^2}{ c^2 N_\text{pub}}}.$

Now by plugging in the $w^*$ in theorem,  take expectation over the public dataset, and substitute the above bound, we get
$$
	\mathbb{E}[\mathbb{E}[\hat{\mathcal{R}}(\bar{w})]- \hat{\mathcal{R}}(w^*) ]\leq   \frac{4(N L + \lambda B)^2  }{\lambda TN}  + \frac{2\sqrt{d} L^2}{c\sqrt{N_\text{pub}}N\sqrt{2\rho} }.
$$
Take $T$ to be sufficiently large so that the second term dominates, we obtain the stated bound.

Finally, to convert the above bound into that of  the excess risk:
\begin{align*}
	&\mathbb{E}[\mathbb{E}[\hat{\mathcal{R}}(\bar{w})]- \hat{\mathcal{R}}(w^*) ] -  (\mathbb{E}[\mathcal{R}(\bar{w})]- \mathcal{R}(w^*) )\\
	\leq&  |\mathbb{E}[\mathbb{E}[\hat{\mathcal{R}}(\bar{w})] - \mathcal{R}(\bar{w}) ] | +  |\mathbb{E}[\hat{\mathcal{R}}(w^*) ] - \mathcal{R}(w^*)|\\
	 :=& \text{Gen}(\bar{w},N)+  \text{Gen}(w^*,N),
\end{align*}
which completes the proof.
\end{proof}

We make two additional remarks. First, we do not require the \emph{empirical} objective $\mathcal{L}_\text{pub}$ to be strongly convex. In practice, we do not have strong convexity when $N_\text{pub} < d$. The assumption of $c$-strong convexity is on the \emph{population-level} risk function $\mathcal{R}$.
Second, our bound decomposes the excess (population) risk of the private learner into a (local) uniform-convergence bound (which is required by a non-private learner too) and an additional cost due to privacy. Note that $\text{Gen}(N)$ is usually $O(1/\sqrt{N})$ but could be $O(1/N)$ when certain data-dependent ``fast rate'' conditions are met, e.g., realizability, low-noise, or curvature (see, e.g., \cite{koren2015fast}).
Our results suggest that the cost of privacy asymptotically vanishes (fix $\rho$, $N_\text{pub}\rightarrow \infty$, and $N_{\text{pub}}/N\rightarrow 0$) even under these fast rate conditions relative to the non-private rate.

redpajama_set_name: RedPajamaArXiv


text:
Xanax XR (Page 2 of 7)
Pharmacia and Upjohn Company 1 October 2013
CLINICAL EFFICACY TRIALS
The efficacy of XANAX XR Tablets in the treatment of panic disorder was established in two 6-week, placebo-controlled studies of XANAX XR in patients with panic disorder.
In two 6-week, flexible-dose, placebo-controlled studies in patients meeting DSM-III criteria for panic disorder, patients were treated with XANAX XR in a dose range of 1 to 10 mg/day, on a once-a-day basis. The effectiveness of XANAX XR was assessed on the basis of changes in various measures of panic attack frequency, on various measures of the Clinical Global Impression, and on the Overall Phobia Scale. In all, there were seven primary efficacy measures in these studies, and XANAX XR was superior to placebo on all seven outcomes in both studies. The mean dose of XANAX XR at the last treatment visit was 4.2 mg/day in the first study and 4.6 mg/day in the second.
In addition, there were two 8-week, fixed-dose, placebo-controlled studies of XANAX XR in patients with panic disorder, involving fixed XANAX XR doses of 4 and 6 mg/day, on a once-a-day basis, that did not show a benefit for either dose of XANAX XR.
The longer-term efficacy of XANAX XR in panic disorder has not been systematically evaluated.
Analyses of the relationship between treatment outcome and gender did not suggest any differential responsiveness on the basis of gender.
Xanax XR Indications and Usage
XANAX XR Tablets are indicated for the treatment of panic disorder, with or without agoraphobia.
This claim is supported on the basis of two positive studies with XANAX XR conducted in patients whose diagnoses corresponded closely to the DSM-III-R/IV criteria for panic disorder (see CLINICAL EFFICACY TRIALS).
Panic disorder (DSM-IV) is characterized by recurrent unexpected panic attacks, ie, a discrete period of intense fear or discomfort in which four (or more) of the following symptoms develop abruptly and reach a peak within 10 minutes: (1) palpitations, pounding heart, or accelerated heart rate; (2) sweating; (3) trembling or shaking; (4) sensations of shortness of breath or smothering; (5) feeling of choking; (6) chest pain or discomfort; (7) nausea or abdominal distress; (8) feeling dizzy, unsteady, lightheaded, or faint; (9) derealization (feelings of unreality) or depersonalization (being detached from oneself); (10) fear of losing control; (11) fear of dying; (12) paresthesias (numbness or tingling sensations); (13) chills or hot flushes.
The longer-term efficacy of XANAX XR has not been systematically evaluated. Thus, the physician who elects to use this drug for periods longer than 8 weeks should periodically reassess the usefulness of the drug for the individual patient.
XANAX XR Tablets are contraindicated in patients with known sensitivity to this drug or other benzodiazepines. XANAX XR may be used in patients with open angle glaucoma who are receiving appropriate therapy, but is contraindicated in patients with acute narrow angle glaucoma.
XANAX XR is contraindicated with ketoconazole and itraconazole, since these medications significantly impair the oxidative metabolism mediated by cytochrome P450 3A (CYP3A) (see CLINICAL PHARMACOLOGY, WARNINGS and PRECAUTIONS–Drug Interactions).
Dependence and Withdrawal Reactions, Including Seizures
Certain adverse clinical events, some life-threatening, are a direct consequence of physical dependence to alprazolam. These include a spectrum of withdrawal symptoms; the most important is seizure (see DRUG ABUSE AND DEPENDENCE). Even after relatively short-term use at doses of ≤ 4 mg/day, there is some risk of dependence. Spontaneous reporting system data suggest that the risk of dependence and its severity appear to be greater in 6 patients treated with doses greater than 4 mg/day and for long periods (more than 12 weeks). However, in a controlled postmarketing discontinuation study of panic disorder patients who received XANAX Tablets, the duration of treatment (3 months compared to 6 months) had no effect on the ability of patients to taper to zero dose. In contrast, patients treated with doses of XANAX Tablets greater than 4 mg/day had more difficulty tapering to zero dose than those treated with less than 4 mg/day.
Relapse or return of illness was defined as a return of symptoms characteristic of panic disorder (primarily panic attacks) to levels approximately equal to those seen at baseline before active treatment was initiated. Rebound refers to a return of symptoms of panic disorder to a level substantially greater in frequency, or more severe in intensity than seen at baseline. Withdrawal symptoms were identified as those which were generally not characteristic of panic disorder and which occurred for the first time more frequently during discontinuation than at baseline.
The rate of relapse, rebound, and withdrawal in patients with panic disorder who received XANAX XR Tablets has not been systematically studied. Experience in randomized placebo-controlled discontinuation studies of patients with panic disorder who received XANAX Tablets showed a high rate of rebound and withdrawal symptoms compared to placebo treated patients.
In a controlled clinical trial in which 63 patients were randomized to XANAX Tablets and where withdrawal symptoms were specifically sought, the following were identified as symptoms of withdrawal: heightened sensory perception, impaired concentration, dysosmia, clouded sensorium, paresthesias, muscle cramps, muscle twitch, diarrhea, blurred vision, appetite decrease, and weight loss. Other symptoms, such as anxiety and insomnia, were frequently seen during discontinuation, but it could not be determined if they were due to return of illness, rebound, or withdrawal.
In two controlled trials of 6 to 8 weeks duration where the ability of patients to discontinue medication was measured, 71%–93% of patients treated with XANAX Tablets tapered completely off therapy compared to 89%–96% of placebo treated patients. In a controlled postmarketing discontinuation study of panic disorder patients treated with XANAX Tablets, the duration of treatment (3 months compared to 6 months) had no effect on the ability of patients to taper to zero dose.
Seizures were reported for three patients in panic disorder clinical trials with XANAX XR. In two cases, the patients had completed 6 weeks of treatment with XANAX XR 6 mg/day before experiencing a single seizure. In one case, the patient abruptly discontinued XANAX XR, and in both cases, alcohol intake was implicated. The third case involved multiple seizures after the patient completed treatment with XANAX XR 4 mg/day and missed taking the medication on the first day of taper. All three patients recovered without sequelae.
Seizures have also been observed in association with dose reduction or discontinuation of XANAX Tablets, the immediate release form of alprazolam. Seizures attributable to 7 XANAX were seen after drug discontinuance or dose reduction in 8 of 1980 patients with panic disorder or in patients participating in clinical trials where doses of XANAX greater than 4 mg/day for over 3 months were permitted. Five of these cases clearly occurred during abrupt dose reduction, or discontinuation from daily doses of 2 to 10 mg. Three cases occurred in situations where there was not a clear relationship to abrupt dose reduction or discontinuation. In one instance, seizure occurred after discontinuation from a single dose of 1 mg after tapering at a rate of 1 mg every three days from 6 mg daily. In two other instances, the relationship to taper is indeterminate; in both of these cases the patients had been receiving doses of 3 mg daily prior to seizure. The duration of use in the above 8 cases ranged from 4 to 22 weeks. There have been occasional voluntary reports of patients developing seizures while apparently tapering gradually from XANAX. The risk of seizure seems to be greatest 24–72 hours after discontinuation (see DOSAGE AND ADMINISTRATION for recommended tapering and discontinuation schedule).
The medical event voluntary reporting system shows that withdrawal seizures have been reported in association with the discontinuation of XANAX Tablets. In most cases, only a single seizure was reported; however, multiple seizures and status epilepticus were reported as well.
Interdose Symptoms
Early morning anxiety and emergence of anxiety symptoms between doses of XANAX Tablets have been reported in patients with panic disorder taking prescribed maintenance doses. These symptoms may reflect the development of tolerance or a time interval between doses which is longer than the duration of clinical action of the administered dose. In either case, it is presumed that the prescribed dose is not sufficient to maintain plasma levels above those needed to prevent relapse, rebound, or withdrawal symptoms over the entire course of the interdosing interval.
Risk of Dose Reduction
Withdrawal reactions may occur when dosage reduction occurs for any reason. This includes purposeful tapering, but also inadvertent reduction of dose (eg, the patient forgets, the patient is admitted to a hospital). Therefore, the dosage of XANAX XR should be reduced or discontinued gradually (see DOSAGE AND ADMINISTRATION).
CNS Depression and Impaired Performance
Because of its CNS depressant effects, patients receiving XANAX XR should be cautioned against engaging in hazardous occupations or activities requiring complete mental alertness such as operating machinery or driving a motor vehicle. For the same reason, patients should be cautioned about the simultaneous ingestion of alcohol and other CNS depressant drugs during treatment with XANAX XR.
Page 2 of 7 « 1 2 3 4 5 … » Last Page »
Also by this Manufacturer
Additional medications in the 'Prescription Medications' section by Pharmacia and Upjohn Company.
Or get Pharmacia and Upjohn Company product information by RSS.
Within the 'Prescription Medications' section:
Brand names beginning with the letter 'X' (Xanax XR).
Medications with generic name beginning with the letter 'A' (Alprazolam).
Medication information from October 2013, newest first or October 2013, earliest first.
Use Trial Bulletin to search for clinical trials involving this product:
Xanax XR
All MedLibrary.org resources are included in as near-original form as possible, meaning that the information from the original provider has been rendered here with only typographical or stylistic modifications and not with any substantive alterations of content, meaning or intent.
https://medlibrary.org/lib/rx/meds/xanax-xr/page/2/