text:
cd ${INSTALL_DIR}/${PACKAGE_SUB_DIR}

#make config shared=1 prefix=$PREFIX cc=${CK_ENV_LIB_MPI_CC}

echo "Making..."
make -j ${CK_HOST_CPU_NUMBER_OF_PROCESSORS} CXX=${CK_CXX} CUDA_HOME=${CUDA_HOME}
if [ "${?}" != "0" ] ; then
  echo "Error: making failed!"
  exit 1
fi

echo "Installing to '${PREFIX}'..."
export PREFIX=${INSTALL_DIR}/install
make install
if [ "${?}" != "0" ] ; then
  echo "Error: installing failed!"
  exit 1
fi

#make check


text:
Rebuilding Haiti's Hospitality Industry
As part of Haiti's recovery from the 2009 earthquake that devastated the region, the Huffington Post reports that the Yéle Haiti Foundation has a plan to get the country's hospitality industry back on track.
CEO Derek Johnson and Parris Jordan, managing director of the hospitality consulting firm HVS' Caribbean office, have joined forces behind a program designed to educate Haiti residents in the hotel industry. The vocational-training initiative will operate out of Jacmel, a port city that suffered severe damage after the earthquake.
"We're still in the bowel of re-emergence. As relief dollars dissipate, we believe that people desperately need to create and pursue opportunity for themselves," said Johnson. "This series of vocational training initiatives are all aimed at sectors we believe the country must grow in if it is to re-emerge from the catastrophic circumstances."
Yéle Haiti, which was founded in 2005 by Grammy Award-winning musician Wyclef Jean, was set up as a grassroots, nonpolitical, charitable organization focusing on emergency relief, employment, youth development and education.
The first three months of the program will focus solely on customer service and expectations, and the last three months will focus on a specific area of the hotel business, like housekeeping, concierge or front desk.
"What we're after is the investment in human capital — giving people a chance to build a life whereby they can sustain themselves," said Johnson. "If we can work to develop a workforce, then business will come and the government will invest the resources."
It looks as if the investment is already paying off. While the program hasn't even launched yet (it officially starts this September), Yéle Haiti has already secured job placements for 95 of the 120 students.
In other news: 'Rent Too Damn High' Candidate Faces Eviction.

redpajama_set_name: RedPajamaCommonCrawl


text:
//
// Timeout.cs
//
// Author:
//       Carlos Alberto Cortez <calberto.cortez@gmail.com>
//
// Copyright (c) 2011 Carlos Alberto Cortez
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using System.Windows;
using System.Windows.Threading;

namespace Xwt.WPFBackend
{
	static class Timeout
	{
		public static uint Add (Func<bool> action, TimeSpan interval, Dispatcher dispatcher)
		{
			var timer = new DispatcherTimer (DispatcherPriority.Normal, dispatcher);
			timer.Interval = interval;
			timer.Tick += delegate {
				if (!action ())
					timer.IsEnabled = false;
			};

			uint id = RegisterTimer (timer);
			timer.IsEnabled = true;

			return id;
		}

		public static void CancelTimeout (uint id)
		{
			var timer = UnregisterTimer (id);
			if (timer != null)
				timer.IsEnabled = false;
		}

		static uint RegisterTimer (DispatcherTimer timer)
		{
			lock (locker) {
				while (timers.ContainsKey (counter))
					counter++;

				timers [counter] = timer;
				return counter++;
			}
		}

		static DispatcherTimer UnregisterTimer (uint id)
		{
			lock (locker) {
				if (!timers.ContainsKey (id))
					return null;

				var timer = timers [id];
				timers.Remove (id);

				return timer;
			}
		}

		static Dictionary<uint, DispatcherTimer> timers = new Dictionary<uint, DispatcherTimer> ();
		static uint counter;
		static object locker = new object ();
	}
}

redpajama_set_name: RedPajamaGithub


text:
Entertainment & Arts selected
Video duration 1:25
Roxette, Rarotonga and the new Doctor Who Jump to media player Four things you may not have known about new Doctor Who, Jodie Whittaker.
https://www.bbc.com/news/av/entertainment-arts-42411392/roxette-rarotonga-and-the-new-doctor-who
What else has Jodie Whittaker starred in? Jump to media player Before becoming the Doctor, Jodie Whittaker starred in a number of high profile television shows
https://www.bbc.com/news/av/entertainment-arts-40632042/new-doctor-who-star-jodie-whittaker-s-former-roles
What can we expect from the first female Doctor? Jump to media player Former Doctor Who writer says fans shouldn't be upset about a woman assuming the role
https://www.bbc.com/news/av/entertainment-arts-40629225/what-can-we-expect-from-the-first-female-doctor
Jodie Whittaker on becoming the new Doctor Jump to media player Jodie Whittaker reveals how she feels about becoming the 13th Time Lord.
https://www.bbc.com/news/av/entertainment-arts-40851893/jodie-whittaker-reveals-how-she-feels-about-becoming-the-13th-doctor
Daisy Ridley: Four things Jump to media player The Star Wars actor reveals a few things you might not know about her.
https://www.bbc.com/news/av/entertainment-arts-41683467/star-wars-daisy-ridley-four-things-you-might-not-know-about-her
'My drummer tattooed me while eating pizza' Jump to media player Dua Lipa and four things you (maybe) didn't know.
https://www.bbc.com/news/av/entertainment-arts-40866274/my-drummer-tattooed-me-while-eating-pizza
Why Geri Horner named her cockerel Usain Jump to media player Geri Horner reveals why she named her cockerel after an Olympic megastar in Four Things You Don't Know About Me
https://www.bbc.com/news/av/entertainment-arts-40606674/how-usain-bolt-is-in-geri-horner-s-chicken-shed
Roxette, Rarotonga and the new Doctor Who
Four things you may not have known about new Doctor Who, Jodie Whittaker.
Video journalist: Sophie Van Brugen
Go to next video: What else has Jodie Whittaker starred in?
Video Top Stories
More from BBC Reel


text:
Entire Site AcupunctureAnnouncementsBasic ScienceBook & movie reviewsCancerChiropracticChronic FatigueClinical TrialsCommentaryComputers & InternetCritical ThinkingDentistryDiagnostic tests & proceduresEnergy MedicineEpidemiologyEthicsEvolutionFaith Healing & SpiritualityGenetically modified organisms (GMOs)GuidelinesHealth FraudHerbs & SupplementsHistoryHomeopathyHumorInternet & social mediaLegalLymeMedical AcademiaMedical devicesMedical EthicsMedical MythsNaturopathyNeuroscience/Mental HealthNutritionObstetrics & gynecologyOphthalmologyPharmaceuticalsPolitics and RegulationPublic HealthQuality ImprovementReligionScience and MedicineScience and the MediaSurgical ProceduresTraditional Chinese MedicineVaccinesVeterinary medicine
Science and MedicinePolitics and RegulationClinical TrialsScience and the MediaVaccinesHerbs & SupplementsCancerPublic HealthAcupunctureHomeopathyHealth FraudChiropracticNeuroscience/Mental HealthMedical AcademiaNaturopathyMedical EthicsNutritionBasic SciencePharmaceuticalsBook & movie reviews
vaccinesacupuncturehomeopathycancerchiropracticnaturopathyquackerycomplementary and alternative medicineantivaccineNational Center for Complementary and Integrative HealthautismFood and Drug Administrationclinical trialspseudoscienceintegrative medicinealternative medicinepediatricsquackademic medicinebreast cancertraditional Chinese medicine
About SBM
Steven P. Novella, MD – Founder and Executive Editor
David H. Gorski, MD, PhD – Managing Editor
Kimball C. Atwood IV, MD (Emeritus)
Mark A. Crislip, MD (Emeritus)
Harriet Hall, MD
Steven P. Novella, MD
David H. Gorski, MD, PhD
Jann Bellamy, JD
Scott Gavura, BScPhm, MBA, RPh
Help with logging in & commenting
SBM Translations
Vaccines & Autism
Media & CAM
SBM Patreon
Exploring issues and controversies in the relationship between science and medicine
NextWhen homeopaths attack medicine and physics
CancerDiagnostic tests & proceduresPublic Health
The cancer screening kerfuffle erupts again: "Rethinking" screening for breast and prostate cancer
David Gorski	on November 2, 2009
I see that the kerfuffle over screening for cancer has erupted again to the point where it's found its way out of the rarified air of specialty journals to general medical journals and hence into the mainstream press.
Over the last couple of weeks, articles have appeared in newspapers such as the New York Times and Chicago Tribune, radio networks like NPR, and magazines such as TIME Magazine pointing out that a "rethinking" of routine screening for breast and prostate cancer is under way. The articles bear titles such as A Rethink On Prostate and Breast Cancer Screening, Cancer Society, in Shift, Has Concerns on Screenings, Cancers Can Vanish Without Treatment, but How?, Seniors face conflicting advice on cancer tests: Benefit-risk questions lead some to call for age cutoffs, and Rethinking the benefits of breast and prostate cancer screening. These articles were inspired by an editorial published in JAMA last month by Laura Esserman, Yiwey Shieh, and Ian Thompson entitled, appropriately enough, Rethinking Screening for Breast Cancer and Prostate Cancer. The article was a review and analysis of recent studies about the benefits of screening for breast and prostate cancer in asymptomatic populations and concluded that the benefits of large scale screening programs for breast cancer and prostate cancer tend to be oversold and that they come at a higher price than is usually acknowledged.
For regular readers of SBM, none of this should come as a major surprise, as I have been writing about just such issues for quite some time. Indeed, nearly a year and a half ago, I first wrote The early detection of cancer and improved survival: More complicated than most people think. and then followed it up with Early detection of cancer, part 2: Breast cancer and MRI. In these posts, I pointed out concepts such as lead time bias, length bias, and stage migration (a.k.a. the Will Rogers effect) that confound estimates of benefit due to screening. (Indeed, before you continue reading, I strongly suggest that you go back and read at least the first of the aforementioned two posts to review the concepts of lead time bias and length bias.) Several months later, I wrote an analysis of a fascinating study, entitling my post Do over one in five breast cancers detected by mammography alone really spontaneously regress? At the time, I was somewhat skeptical that the number of breast cancers detected by mammography that spontaneously regress was as high as 20%, but of late I'm becoming less skeptical that the number may be somewhere in that range. Even so, at the time I did not doubt that there likely is a proportion of breast cancers that do spontaneously regress and that that number is likely larger than I would have guessed before the study. Of course, the problem is that we do not currently have any way of figuring out which tumors detected by mammography will fall into the minority that do ultimately regress; so we are morally obligated to treat them all. My most recent foray into this topic was in July, when I analyzed another study that concluded that one in three breast cancers detected by screening are overdiagnosed and overtreated. That last post caused me the most angst, because women commented and wrote me asking me what to do, and I had to answer what I always answer: Follow the standard of care, which is yearly mammography over age 40. This data and these concerns have not yet altered that standard of care, and I am not going to change my practice or my general recommendations to women until a new consensus develops.
Before I discuss the JAMA analysis, let me just sample a bit of how this issue is being portrayed in the popular press:
This week the conversation garnered national attention when physicians at the University of California, San Francisco, and University of Texas, San Antonio, published an analysis in the Journal of the American Medical Association expounding on their concerns about the ongoing harm inflicted on patients who undergo prostate and breast cancer screening. One problem is that screening often picks up harmless or non-life-threatening tumors that could have gone unnoticed for a lifetime, leading to unnecessary and aggressive treatment for patients. In the days before widespread mammography, for instance, physicians rarely encountered ductal carcinoma in situ (or DCIS), a type of low- to intermediate-grade breast cancer that grows slowly and may even regress, meaning that the body may rid itself of the tumor. But today DCIS accounts for nearly 30 % of all breast cancer diagnoses (more than 60,000 cases a year), leading to an untold amount of treatment and patient distress. (TIME Magazine)
The purpose of screenings is to detect cancer early, when treatments are most likely to be effective, and to save lives. That the tests do so for colon, breast and cervical cancer has been well-established for middle-age adults but is not indisputable for those who are older, as most studies have been done in people younger than 65. Research on routine PSA screening has yet to prove a definitive benefit at any age. On the other side are the potential costs, which can include unnecessary treatments for cancers that never would have become life-threatening, the anxiety and distress associated with cancer diagnosis, the complications associated with screening procedures or therapies, and medical expenses. (Chicago Tribune)
Call it the arrow of cancer. Like the arrow of time, it was supposed to point in one direction. Cancers grew and worsened. But as a paper in The Journal of the American Medical Association noted last week, data from more than two decades of screening for breast and prostate cancer call that view into question. Besides finding tumors that would be lethal if left untreated, screening appears to be finding many small tumors that would not be a problem if they were left alone, undiscovered by screening. They were destined to stop growing on their own or shrink, or even, at least in the case of some breast cancers, disappear. (New York Times)
The American Cancer Society, which has long been a staunch defender of most cancer screening, is now saying that the benefits of detecting many cancers, especially breast and prostate, have been overstated. It is quietly working on a message, to put on its Web site early next year, to emphasize that screening for breast and prostate cancer and certain other cancers can come with a real risk of overtreating many small cancers while missing cancers that are deadly. (New York Times)
The latest kerfuffle is simply the latest bubbling up of a controversy that has been going on for years. It's also a classic example of the difficulty of applying messy science to public health issues. Science is messy, but public health initiatives, to be successful, arguably need a simple message that resonates with the public, a message like
Screening for cancer catches cancers at an earlier stage when they are more easily treated.
Screening for cancer saves lives.
You should get screened if you're older than N years old.
Thus, when evidence that calls into question any of these messages appears, it causes consternation among the lay public. Worse, cranks leap to take advantage of such controversies and make claims, such as "mammography is useless" and publish cartoons like this. Even people who should know better say similar things. As I've said before, science-based medicine is hard and not always clear-cut. Moreover, I would argue that debates such as this demonstrate that the claims of cranks that physicians all march in lockstep or that there is a "conspiracy" to keep shortcomings of SBM from patients are nothing but the rankest nonsense. Finally, cranks often conflate, either through ignorance, willful ignorance, or often outright hostility to "Western medicine," screening mammography with diagnostic mammography. Screening mammography is designed to detect disease in asymptomatic women. If a woman feels a suspicious lump, nothing–I repeat, nothing–in any of this data says that she should not have a physical exam and appropriate imaging, including mammography and ultrasound at the minimum, posthaste. That's diagnostic mammography, and that is not what this kerfuffle is over. Suspicious lumps, especially in women over 40, need to be worked up, period. I can't emphasize that strongly enough.
So what did Esserman et al say? They lay out the problem thusly:
A large fraction of the US population participates in screening for prostate cancer and for breast cancer. About 50% of at-risk men have a routine prostatespecific antigen (PSA) test and 75% have previously had a PSA test.3,4 About 70% of women older than 40 years reported having a recent mammogram.5 Two decades of screening have resulted in a significant increase in detection of early cancers. Prostate-specific antigen testing has nearly doubled the chance that amanwill be diagnosed with prostate cancer in his lifetime: In 1980, a white man's lifetime risk of prostate cancer was 1 in 116; today it is 1 in 6.1 A woman's lifetime risk of breast cancer was 1 in 12 in 1980; today it is 1 in 8.1 If ductal carcinoma in situ (DCIS) is included, the risk of being diagnosed with breast cancer, like prostate cancer, has almost doubled as well.
The increase in early cancers as a fraction of total cancers detected is not necessarily beneficial. The introduction of an optimal screening test should be followed by an increase in the rate of early disease followed by a decrease in regional disease while the overall detection rate remains constant.7 FIGURE 1 illustrates hypothetical optimal, worst case, and intermediate-case scenarios, using 1980 breast cancer incidence rates as a starting point. In the worst case, screening leads to an increase in local disease detection without a corresponding decrease in regional disease, thereby increasing costs and morbidity due to overdetection and overtreatment of non–life-threatening cancers.
They conclude that current screening protocols for breast and prostate cancer result in a case that is intermediate between the worst case and optimal case scenarios. Because their graph illustrates this problem better than words can, I present it here:
(Click for larger image.)
Esserman et al then point out that the incidence of early stage breast and prostate cancers has increased dramatically as a fraction of total cancers detected, while regional and metastatic cancers have decreased dramatically as a fraction of those cancers detected. One example is the form of breast cancer known as ductal carcinoma in situ, abbreviated DCIS. DCIS is considered to be breast cancer, but "pre-invasive" cancer, given that the sine qua non for its diagnosis is seeing cancerous-appearing breast epithelial cells confined within the ducts with no evidence of their having invaded through the basement membrane that separates the ducts from the surrounding tissue. Thus, DCIS has been labeled "stage zero" cancer, and its 5 year survival is close to 100%, although a small proportion of patients does recur and ultimately die of breast cancer. There is even a debate over whether DCIS truly represents a precursor to cancer, as it is unclear what percentage of DCIS lesions progress to frank cancer, making the treatment of DCIS very confusing to patients. Before the mammographic screening era, which began in earnest after 1980, diagnoses of DCIS were uncommon. Now DCIS makes up nearly a third of new breast cancer diagnoses. Again, because we don't know which of these lesions will progress and which won't (or even which ones might regress), we treat them all, and this treatment involves nearly all the treatments used for breast cancer, with the only treatment not used on DCIS being chemotherapy.
As for more advanced regional tumors (tumors with lymph node metastases or involvement of local structures) and metastatic tumors, the absolute numbers have also decreased, but not very much. If screening were having a huge effect on breast cancer, the expectation would be that removing tumors at a smaller stage would, nearly three decades later, ultimately result in fewer advanced cancers being detected, their having been preempted by screening and appropriate treatment by surgical excision and adjuvant therapy. (This is one reason why screening for colorectal cancer has been more effective; removal of polyps is effective at preventing the subsequent development of colorectal cancer.) The conclusion, which again should be nothing surprising for anyone who saw my post from last year, is that screening by nature preferentially detects slower growing, more indolent cancers, which by their very nature tend to be less lethal and some of which may not even ever progress enough to endanger the life of the patient. The fastest-growing, most lethal cancers grow so fast that they seem to "pop up" between screenings. As a breast cancer surgeon, I see such tumors occasionally. A woman will come in with a palpable mass and an very ugly mammogram after having had a mammogram less than a year before that shows no trace of abnormality or, sometimes, an abnormality that is only appreciated in retrospect because it was so subtle. Naturally, such women are distraught because they have "done everything right" and obtained their yearly mammography just as their primary care doctors told them too. However, no practical screening program could catch tumors such as these because it would require too frequent screenings, which would also, unfortunately, increase the chances of finding abnormalities that lead to overdiagnosis. The dilemma, again, was discussed in my post, and, indeed, Figure 3 in the JAMA article is in essence the same figure that I reproduced to help me discuss length bias, only with much better design sensibilities and a dash of color:
The most distressing conclusion of Esserman's analysis is just how many patients need to be screened to save one life. It is well-accepted that screening women over age 50 for breast cancer does decrease mortality due to breast cancer by approximately 30%, but what does this reduction really mean? This:
Essentially, mammography reduces the odds of a 60-year-old woman dying of breast cancer in the next decade by 30%. Sounds impressive, until you look at her absolute risk: by getting her annual mammogram, her chances of dying from breast cancer are whittled from 0.9% to 0.6%. Overall, for every 1,000 women in their 60s screened for breast cancer in the next 10 years, mammograms will save the lives of 3 people but 6 others will still die. (The numbers edge up or down in lockstep with a woman's age.)
Esserman et al conclude that, to avert one death from breast cancer with mammographic screening for women between the ages of 50-70, an age range intentionally chosen because it is the age range for which mammographic screening is the least controversial, 838 women need to be screened over 6 years for a total of 5,866 screening visits, to detect 18 invasive cancers and 6 instances of DCIS. The additional price of this was estimated to be 90 biopsies and 535 recalls for additional imaging, as well as "many cancers treated as if they were life threatening when they are not." For prostate, to prevent one death from cancer, 1,410 men need to be screened over 9 years, for a total of 2,397 screening visits and 48 cancers detected. In other words, screening takes a lot of effort for, on an absolute basis, not as many lives saved as we had hoped. The authors also point out that technology exacerbates this problem. We push for ever greater sensitivity at the cost of lower specificity, which means that we detect more and more lesions that turn out not to be cancer or to be indolent, non-lifethreatening cancer. Yet all of these cancers end up being treated the same, again because we cannot distinguish between them. The addition of hypersensitive modalities such as MRI only exacerbate this problem even further. As Esserman et al state:
After 2-1/2 decades of screening for breast and prostate cancer,conclusions are troubling: Overall cancer rates are higher, many more patients are being treated, and the absolute incidence of aggressive or later-stage disease has not been significantly decreased. Screening has had some effect, but it comes at significant cost, including overdiagnosis, overtreatment, and complications of therapy, problems likely to be exacerbated as the US population ages. Additional gains are unlikely with the current approach and may inadvertently add to the burden of treatment and diagnosis for relatively indolent disease.
So what's the solution? Should we give up on screening?
I'll answer the second question with an emphatic absolutely not. There is no doubt that screening can save lives; the problem is that we're doing it "bigger" not and not necessarily better or smarter, using more or less a one-size-fits-all approach when a more tailored approach is necessary. In essence, we're using 20th century technology, when 21st century technology can find a way out of these conundrums. Here's where I (mostly) agree with Esserman et al. We need to change how we screen for common cancers using the latest advances in biochemistry, molecular biology, and technology to differentiate which lesions are most likely to be cancer and which are not, which cancers are likely to progress to the point of threatening a patient's life in their remaining lifespan and which ones are not. To this end we need the following tools:
Develop biological and genetic markers of risk for various cancers. It's long been known that screening is more effective in populations where a disease is common. The more common the disease (to a point) the more likely screening is to save lives. Thus, if we could develop better tools to identify women most at risk for breast cancer, for example, we could start screening such women at a younger age and perhaps screen them more frequently. We already do this to some extent, but our current tools (family history to identify higher risk populations, a handful of genetic markers, and ethnicity) are crude measures at best. Equally important would be to find similar markers to identify women at lower risk, who might start screening at an older age and undergo screening less frequently. Most important of all, identifying high risk patients more effectively would open up the possibility of prevention to a far greater extent than is possible now.
Identify biological markers to differentiate minimal risk cancers from cancers with significant risk. Again, we have begun to do this to some extent. For example, tests such as the Oncotype DX multigene test are now clinically used to determine which women with early stage, estrogen receptor-positive breast cancers would and would not benefit from chemotherapy. It's relatively crude as a tool goes, but we will do better. Indeed, we are beginning to develop the tools that would allow us to go much further. It would not surprise me if, in a decade or so, we routinely do a battery of such tests on newly diagnosed tumors and then use the results to decide which ones can undergo "watchful waiting," which ones need aggressive therapy, and which ones can undergo less aggressive therapy. Eventually, we will be using therapy targeted to the individual genetic alterations in an individuals tumor. Yes, I know I've expressed skepticism over such "targeted" therapy before, but that's not because I don't believe the concept will work; it's because the boosters of targeted therapy often seem not to realize just how difficult and complex it is and will be to identify targets and how hard it will be to turn the targeting of proteins and genetic alterations into therapies that result in decreased mortality from cancer. In other words, I believe that genetic and biomarker analysis will ultimately allow us to tailor therapy to an individual's tumor; I just think that getting there will take a lot longer and be a lot messier than the current generation of biotech cheerleaders either understand or will let themselves admit.
Develop better markers of response to therapy. Right now our primary assessment of whether a therapy is working or not boils down to one of two things: survival rates (the ultimate endpoint) or whether a tumor shrinks or not in response to therapy. Both are very crude. The former is the ultimate measure, but it is not helpful in the case of an individual patient, who wants to know if her treatment worked or not before she dies. The latter is notoriously inaccurate, with tumor shrinkage only showing a weak correlation to overall survival. This is one area where tests similar to the ones being developed to determine the aggressiveness of a tumor could potentially help.
Develop better imaging tools to assess risk. This is the one area that Esserman et al neglected almost completely in their review. This is an area that I have only recently become very interested in for our research program. Technology is progressing to the point where it soon may well be possible to identify a lesion as cancerous or not with greater than 90% using a "virtual biopsy" and then further to determine its risk of causing the death of the patient, when combined with biochemical and genetic analyses. Such tools would allow us to choose patients who might be appropriate for "watchful waiting" with far less anxiety over whether we're doing the right thing than there is now and, even better, potentially to identify tumors that are likely to regress spontaneously.
None of this will be easy, and the process of getting from where we are now to this futuristic vision will be be full of fits and starts. At times it will be controversial. Advances in science-based medicine always are. Moreover, there will be significant barriers to its implementation, not the least of which is the relatively simplistic thinking that has come to predominate how we view cancer screening, not in the rarified air of the research institutions that actually study the pros and cons of screening, but "in the field" at the level of individual practitioners trying to put these screening recommendations into practice. It is not the fault of these practitioners that these fine points are not more widely known. Rather, the reason derives from the conflict between the need for simple messages to tell patients and the difficulty of applying epidemiological science to individual patients.
So what do we do in the meantime? One thing I am not going to say is to stop undergoing screening. Women should continue to undergo mammographic screening as recommended according to the standard of care. Whether men should continue to undergo routine prostate screening as previously recommended is now controversial, with less evidence to support it than mammography except in high risk populations. Until a consensus develops for changing the current regimen, it is the best we have, and, for all its shortcomings, mammography remains the best and most cost-effective tool we have to detect cancer. (The case for routine mass PSA screening, in contrast, now strikes me as considerably weaker.) Even in the worst case scenario that breast cancer screening results in the appearance of a benefit all due to lead time bias (which it isn't; I present this as a worst case scenario), treatment of cancers caught at an earlier stage is still less invasive and less morbid. True, this has to be weighed against overtreatment of lesions that would never progress, but even so the benefit of using less invasive therapy is one that is routinely discounted by the new generation of what I am sometimes tempted to call "screening nihilists," a group that seems to think that nearly all screening is worthless.
The problem is not that screening is worthless. For example, even mammography under the most pessimistic estimates routinely touted as evidence that it's pointless to screen clearly does save lives. A relative reduction in risk is not meaningless or of insignificant worth, either, as some nihilists will suggest. Also, don't forget that even small absolute percentages when applied to millions upon millions of people, can result in large numbers of lives saved. That being said, I also think that these recent trials suggest that we are not doing an adequate job of informing patients of the down side of screening, such as excessive biopsies, a down side many might well be willing to accept if they were informed of it and understood it. After all, as I've mentioned before with regards to breast cancer, I've seen polls where most women say they would agree to undergo cytotoxic chemotherapy to improve their odds of surviving breast cancer by 1%. To save more lives relative to the number of women overtreated and overdiagnosed, we need to learn how to screen smarter. Our current screening regimens are not good enough. Improving them will require science-based medicine, and, unfortunately, will be controversial. Watching the jolting, jarring journey to a new scientific consensus about something as important as screening for breast and prostate cancer is much watching the proverbial sausage being made. I also can't help but point out the contrast to "alternative" medicine, which rarely questions and almost never changes practice in the face of new data. It may well be that our hopes for screening for breast cancer and prostate cancer were too high, but in science-based medicine evidence eventually wins out. Screening practices will likely change in the face of new evidence, and that is a good thing. Certainly it's better than the alternative, namely not changing and not improving.
Posted in: Cancer, Diagnostic tests & procedures, Public Health, Science and the Media
Tagged in: breast cancer, prostate cancer, screening
Posted by	David Gorski
Dr. Gorski's full information can be found here, along with information for patients. David H. Gorski, MD, PhD, FACS is a surgical oncologist at the Barbara Ann Karmanos Cancer Institute specializing in breast cancer surgery, where he also serves as the American College of Surgeons Committee on Cancer Liaison Physician as well as an Associate Professor of Surgery and member of the faculty of the Graduate Program in Cancer Biology at Wayne State University. If you are a potential patient and found this page through a Google search, please check out Dr. Gorski's biographical information, disclaimers regarding his writings, and notice to patients here.
Join The Society for SBM
Buy an e-book:
SBM e-Books for Kindle
SBM e-Books for iBooks
SBM e-Books for Nook
Dr. Hall's video SBM course
Faith Healing & Spirituality
Health Fraud
Medical Academia
Neuroscience/Mental Health
Politics and Regulation
Science and the Media
View all recent comments View all recent comments
Donate to support science!
If you prefer to make a one time donation, please use the link below
(Powered by Google Translate)
English Afrikaans Albanian Amharic Arabic Armenian Azerbaijani Basque Belarusian Bengali Bosnian Bulgarian Catalan Cebuano Chichewa Chinese (Simplified) Chinese (Traditional) Corsican Croatian Czech Danish Dutch Esperanto Estonian Filipino Finnish French Frisian Galician Georgian German Greek Gujarati Haitian Creole Hausa Hawaiian Hebrew Hindi Hmong Hungarian Icelandic Igbo Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish (Kurmanji) Kyrgyz Lao Latin Latvian Lithuanian Luxembourgish Macedonian Malagasy Malay Malayalam Maltese Maori Marathi Mongolian Myanmar (Burmese) Nepali Norwegian Pashto Persian Polish Portuguese Punjabi Romanian Russian Samoan Scottish Gaelic Serbian Sesotho Shona Sindhi Sinhala Slovak Slovenian Somali Spanish Sudanese Swahili Swedish Tajik Tamil Telugu Thai Turkish Ukrainian Urdu Uzbek Vietnamese Welsh Xhosa Yiddish Yoruba Zulu
When homeopaths attack medicine and physics
All Medicines Are Poison!
Development by Digital Gravity Media | © 2020 Paperback Theme by Array.


text:
The Source of Best
PAPER-MARKET.COM!
Home Free Essays Wal Mart Violates Workers Basic Rights Philosophy Essay
Wal Mart Violates Workers Basic Rights Philosophy Essay
Share this: Facebook Twitter Reddit LinkedIn WhatsApp
As an organisation progresses and becomes more significant, the greater impact it has on the people, social and environment. In this competitive environment, business ethics plays an important role in an organisation to determine its reputation and sustainability as it acts as a guide to influence and reinforce an organisational behaviour. Managing an organisation is not solely about self-interest of making profits anymore, decisions and actions have to be exercised with consideration to align with its principles and standards. Although there might be issues about what is the right ethical decision to make, however there is no standard definition as it differs from every individual perspective within and around the organisation.
This essay would assess the ethical theories with different perspective with a case study and evaluate the positive and negative impact among the theories. Then determine an ethical theory that is considered most useful in guiding an organisation decision.
Case Study: Wal-Mart Violates Workers Basic Rights
In this case study, it discusses the ethical practice of an America multinational retailer organisation, Wal-Mart. Despite being the world's largest retailer with more than 2 million workers operating in 8,500 stores worldwide, the organisation has yet to develop a proper policy for its labour condition. Concerns were raised by the trade unions, in regard to the severe violation of workers basic rights in the Wal-Mart factories from Indonesia, China and Bangladesh. Wal-Mart's exploitation of its workers' right extended to forced labour, minimum wages paid, healthcare violations and even bathroom break limitation. Such extreme labour policy reflects on the ethical practices of the organisation. By applying the ethical theories based on the case study, it would show the different perspective of how the theories view an ethical practice.
Normative Theories on Ethics
In business ethics, normative theories established some fundamental principles to distinguish the right decisions from the wrong decisions. Kagan (1998) stated that the normative theories emphasis on how people should act, not how people do act; it is the moral beliefs of a society that reflect how people should behave. As business ethics are fostered through different decisions and cultural aspects, the normative theories are characterised into two main perspective: consequence based and non-consequential based.
Consequence Based: Ethical Egoism, Utilitarianism and Virtue Ethics
Consequence based approach is to determine the rightness of a decision based solely based on the result. When the consequence is positive, the decision is right, while when consequence is negative, the decision is wrong. Thus, an organisation with consequence based approach make a decision on the weighing the ratio of the good and bad consequences. However, what would be the consideration and determinant for the consequences? The three consequence based approaches: ethical egoism, utilitarianism and virtue ethics differentiate its views in the following.
Ethical Egoism
Egoism views personal interest as its moral guiding principle. Where decision rightness is defined in terms of its own self-interest. Thus, in a situation when faced with a business dilemma, an egoist would be inclined to make the decision that is believed to serve own interest. Looking at Wal-Mart violation of labour rights, egoism would rather maximise its own advantage in the situation where profit is more important. Thus, it would view Wal-Mart's action as moral as self-interest is met.
Unlike ethical egoism, utilitarianism is concerned with making decision that result in the maximum of good consequence for maximum individuals. When the consequence causes happiness, it is moral; when the consequence causes unhappiness, it is immoral. To derive to a decision, utilitarianism would perform a benefit/cost analysis by identifying the course of action that could be executed and weight the possible benefits for each course. Then, the decision would be chosen based on the action that yields the maximum benefit over the greatest number of people. Based on utilitarianism, Wal-Mart's ethical standard is immoral as the consequences failed to benefit the employees. Although by underpaying the employees, the stakeholders would gain more profits, however it has resulted in significant unhappiness among the majority of the factories' employees. Thus, the cost has overweighed the benefit and causing the action to be considered unethical.
Virtue ethics
In contrast to ethical egoism and utilitarianism, virtue ethics emphasis on the person and virtue in the morality rather than the consequence of an action. The decision would be regarded as moral if the person making the decision is virtue with the aspect in sense of justice, fidelity, self-care and prudence. In the Wal-Mart's case, the action would be deemed as immoral. Looking at the dire consequence caused, it is apparent that the person making the decision lacks the aspects of virtue.
Non-Consequential Based: Kantianism and Justice Ethics
While non-consequential based approach views decision made whether if its right or wrong as an independent factor of the consequences. What matters for non-consequential based approach is the nature of intention, not the consequences. For the non-consequential based approach, there are the Kantianism and Justice Ethics.
Kantianism expresses that morality is based on the intention behind the decision instead of the consequences. It is based on the abstract of universal principles where an action is moral only if it is motivated by moral obligation. Kantianism encourages the focus on doing the right like being righteous, honest and upholding integrity. Since Kantianism views an organisation as a moral community where individuals are related to achieve common goals, it is immoral for organisation to exploit employees or customers for profit to satisfy the stakeholders. Applying Kantianism in the Wal-Mart's case study, it is evidently immoral. Wal-Mart failed to operate as a moral community and employees were exploited.
Justice ethics
On the other hand, justice ethics based moral on the fairness of action. Justice ethics is closely associated with the fair treatment of people and reward based on the merit to comply with the ethical standards. Hartman and Desjardins (2008) matched justice ethics and utilitarianism in the similarity that both follows benefit/cost analysis. However, unlike utilitarianism, justice ethics abide to fairness and equity in the distribution of good. For Wal-Mart, it failed to comply with the ethical standard of providing fair treatment toward its employee and discriminated the employees' right. Thus, in justice ethics is considered as immoral.
Comparison between the Ethical Theories
In favour of ethical egoism, Wong (2006) stated that every individual has the right to pursue their own interest and placing their desire over others. Where deemed that if every individual pursue own self-interest in a competitive society, it would benefit the society as a whole eventually. However, many philosophers have constantly refuted egoism and refuse to accept it as an ethical theory. According to Quinn (1974),who claimed against the standard of ethical egoism, since self-interest might not be equivalent to the rational morality, egoism might face conflicts of interest resulting in potentially unethical conduct. Moreover, if ethical egoism was to be practiced by all individual, it would places oneself in a higher importance than the rest and would result in inequity and imbalance of harmony in the society. Ethical egoism might not be a well-accepted application in business, but it is more commonly practice by small companies where the lack of social obligation is unlikely to threaten the livelihood of business.
Unlike ethical egoism, Allhoff and Vaidya (2005) stated that utilitarianism is based on an equity approach to fulfil the interest of all individuals as it considers the interest of all relevant individual like the stakeholders, employees, customers and business partners. Andre and Velasquez (1989) mentioned that even though utilitarianism is a popular ethical theory among organisations, it is challenging to rely on it solely for moral decision-making. Also, Preuss (2000) argued that utilitarianism is not practical as it is complex to determine what is considered happiness to different individual, especially when it involves different parties with different interest, it might not be possible to satisfy every needs. Furthermore, utilitarianism might have the tendency to introduce biasness to elevate majority interest over the expense of the minority interest. Even so, among the theories, utilitarianism is considered to be the prevalent form of consequentialism among organisations.
According to Ferrell, Fraedrich and Ferrell (2012), virtue ethics would be considered as a functional theory since it assesses the character of the person who is making the decision instead. Virtue ethics focus on developing good virtue, where the virtue in turn would lead the person to make the right decision. It is believed that it is important for an organisation to focus on building virtues in all individuals, thus to nurture the individuals' character and morality in making the appropriate decision (Ferrell, Fraedrich and Ferrell 2012). It is common that organisation applies virtue ethics through its statement of value, or code of conduct to enforce certain regulation that is believed to be moral. For example, Google's belief in 'Don't do evil' suggests that all the individual within the organisation to comply. However, Nelson and Trevino (2010) countered on the reliance on virtue ethics alone does not guide how one should act when a situation arises. Similarly, the standard of virtue that one should possess might differ in different organisation and culture as well.
As for Kantianism, Rendtorff (2009) believes that it promotes democracy and equity in the business environment and would help to create harmony and peace. In contrary, Smith and Dubbink (2011) feel that given the nature of Kantianism, it is impractical to apply to the business context as the fact that Kantianism encourages organisation to be motivated by its moral duty instead of profit, which is seemingly unrealistic.
While Mandal (2010) believes that justices ethics is useful in an organisation to eliminate any form of inequity and discrimination, thus protect the interest of individual who lacks in influence. Then again, Mandal (2010) also agrees that it is quite difficult to attain a fair distribution as every individual judges fairness with different criteria.
After examining the ethical theories and case study, it is obvious that the five theories have its own distinctive viewpoint about ethical behaviour. However, based on evaluating the arguments of the theories, utilitarianism would be considered the most useful in guiding company decisions. Since no theory is faultless, the idea of a useful theory should be practical and able to contribute to an optimal extent. Utilitarianism provides a consistent approach for moral-decision-making and a systematic approach that aims to achieve the highest desirability and balance of possible happiness. Although it comes with controversy about the ethical aspect, however in comparison with the other theories like Kantianism, utilitarianism would seem like a more realistic approach in an organisation context. Nevertheless, an organisation should not base its decision-making process entirely on the same ethical theory for every different situation. When evaluating a theory on its own, there is shortcoming. Thus, the rational approach is to assess the strength of each theory and synchronize in a way that would be effective for the particular situation. Furthermore, it is necessary to take into consideration that an ethical theory that applies for an organisation in United States might not have the same effect for an organisation in China due to the different business environment, culture, moral standards etc.
In conclusion, business ethics is a complex field that is significant in every organisation. Despite having much controversy about the ethical theories, philosophers have concluded that the ethical theories have its own positive and negative consequences. Comparing the ethical theories, even though utilitarianism is determined to be the most useful in guiding company decisions, it might not work for every organisation or decision-making process. Thus, it is important to be critical towards the ethical approaches to foster an appropriate corporate and social responsibility that is best suited to its own interests toward the stakeholders, employees, customers and business partners. Nevertheless, with an appropriate ethical combination well in placed with the organisation objective, it could
Osho And Human Rights Philosophy Essay
…."According to Osho "In thousand years we have not been able to do so many…
Acknowledging And Respecting A Persons Human Rights Philosophy Essay
Share this: Facebook Twitter Reddit LinkedIn WhatsApp Every person has dignity and value. One of the ways that we recognise…
Human Rights And The Western Concept Sociology Essay
Since the beginning of Human Rights until recent Human Rights issues, the interpretation and concept…
Free Essays (10,958)
General Studies (205)
Beauty Therapy (41)
European studies (33)
Find and hire trusted writing services to write your essays, research papers, dissertations or other assignments for ethical personal use.
This website and its content is copyright of © Paper-Market.COM 2018. All rights reserved.


Breast cancer is a cancer that begins in breast tissue. Other than skin cancer, breast cancer is the most common cancer among women in the United States. Screening can increase your chance of survival with breast cancer if you catch it early.
Breast cancer is a cancer that begins in breast tissue. Other than skin cancer, breast cancer is the most common cancer among women in the United States. Some women are at higher risk of breast cancer than others because of their personal or family history.
Do I need to be screened for breast cancer?
Screening can increase your chance of survival with breast cancer if you catch it early. For women 50 to 74 years old who are not at high risk, screening every other year is recommended. Women over 40 may want to talk to their doctor about when to begin screening.1 Breast cancer screening is done using a mammogram.
Learn more about mammogram screening on our Mammograms page.
Find a Food and Drug Administration (FDA)–certified mammogram facility.
Learn more about breast cancer from the Centers for Disease Control and Prevention.
U.S. Preventive Services Task Force. (2018). Final Update Summary: Breast Cancer Screening.


text:
\section*{Appendix}

\subsection{Metropolis--Hastings}
\label{sec:app:MH}

Metropolis--Hastings (MH) is an MCMC method that allows for sampling when
one only has access to an unnormalized version of the target distribution~\citep{gilks1995markov}.
At a high-level, one attempts iteratively proposes local moves from the current location of a sampler
and then accepts or rejects this move based on the unnormalized density.  Each iteration of this process
is known as a MH transition.

The unnormalized targets
distributions of interest for our problem are $\gamma_k(\mathbf{x})$ where
\begin{align*}
\pi_k(\mathbf{x})  &\propto \gamma_k(\mathbf{x}) =
p(\mathbf{x})\mathbbm{1}_{\{s(\mathbf{x})\ge L_{k}\}}, \ \ \ \ k=1,2,\ldots,K.
\end{align*}
A MH transition now consists of proposing a new sample using a proposal  $\mathbf{x}' \sim g(\mathbf{x}'\mid\mathbf{x})$, where $\mathbf{x}$ indicates the current state of the sampler and
$\mathbf{x}'$ the proposed state, calculating an acceptance probability,
\begin{align}
	A_k(\mathbf{x}'\mid\mathbf{x}) &\triangleq \min\left\{1, \frac{\gamma_k(\mathbf{x}')g(\mathbf{x}\mid\mathbf{x}')}{\gamma_k(\mathbf{x})g(\mathbf{x}'\mid\mathbf{x})} \right\},
\end{align}
and accepting the new sample with probability $A_k(\mathbf{x}'\mid\mathbf{x})$,
returning the old sample if the new one is rejected. The proposal, $g(\mathbf{x}'\mid\mathbf{x})$, is a conditional distribution, such as a normal distribution centred at $\mathbf{x}$ with fixed covariance matrix. Successive applications of this transition process
generates samples which converge in distribution to the target $\pi_k(\mathbf{x})$ and whose
correlation with the starting sample diminishes to zero.

In our approach, these MH steps are applied independently to each sample in the set,
while the only samples used for the AMLS algorithm are the final samples produced from the resulting
Markov chains.

\subsection{Implementation Details}
\label{sec:app:implementation-details}
Algorithm 1 has computational cost $O(NMK)$, where the number of levels $K$ will depend on the rareness of the event, with more computation required for rarer ones. Parallelization over $N$ is possible provided that the batches fit into memory, whereas the loops over $M$ and $K$ must be performed sequentially.

One additional change we make from the
approach outlined by \citet{GuyaderEtAl2011} is that we perform MH updates on all chains in Lines~\ref{line:mh-1},
rather than only those that were previously killed off. This helps reduce the build up of correlations over
multiple levels, improving performance.

Another is that we used an adaptive scheme for $g(\mathbf{x}'|\mathbf{x})$ to aid efficiency.  Specifically, our proposal takes the form of a random walk, the radius of which,
$\epsilon'$, is adapted to keep the acceptance ratio roughly around $0.234$ (see \citet{RobertsEtAl1997}). Each chain has a separate acceptance ratio that is average across MH steps, and after $M$ MH steps, for those chains whose acceptance ratio is below 0.234 it is halved, and conversely for those above 0.234, multiplied by 1.02.

\subsection{Additional results}
\subsubsection{Varying $M$ for fixed $\rho$ on {\scshape CollisionDetection}}
\label{sec:var-M}
Whereas the exact value of $M$ within the range considered
proved to not be especially important when $\rho=0.1$, it transpires to have
a large impact in the quality of the results for larger values of $\rho$ as shown
in Figure~\ref{fig:other-rho}.
\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textwidth]{{collision_exp2_varying_M_rho=0.25}.pdf}\includegraphics[width=0.5\textwidth]{{collision_exp2_varying_M_rho=0.5}.pdf}
	\caption{Mean AMLS estimate relative to naive (unbiased) MC estimate for different $M=$ holding $\rho$ fixed to $0.25$ (left) and $0.5$ (right), for those properties whose naive MC estimate was greater than $\log_{10}\mathcal{I}=-6.5$ such that they could be estimated accurately.
	\label{fig:other-rho}}
\end{figure}

\subsubsection{Per-sample robustness measure during robust training}
\label{sec:app:persample-robustness}
Figure~\ref{fig:all-36} illustrates the diverse forms that the per-sample robustness measure can take on the
$40$ datapoints averaged over in
Experiment \S5.3.  We see that different datapoints have quite varying initial
levels of robustness, and that the training helps with some points more than
others.  In one case, the datapoint was still not robust add the end of
training for the target perturbation size $\epsilon=0.1$.
\begin{figure}[h]
	\includegraphics[width=\textwidth]{mnist_robust_curves.pdf}
	\caption{Convergence of individual datapoints used in forming
		Figure~\ref{fig:robust-training}.
		\label{fig:all-36}}
\end{figure}
\section{Motivating Examples}
\label{sec:mot}
 \vspace{-4pt}
To help elucidate our problem setting, we consider the {\scshape AcasXu} dataset \citep{KatzEtAl2017} from the formal verification literature. A neural network is trained to predict one of five correct steering decisions, such as ``hard left,'' ``soft left,'' etc., for an unmanned aircraft to avoid collision with a second aircraft. The inputs $\mathbf{x}$ describe the positions, orientations, velocities, etc. of the two aircraft. Ten interpretable properties are specified along
with corresponding constraints on the inputs, for
which violations correspond to events causing collisions.
Each of these properties is encoded in a function, $s$, such that it is violated when $s(\mathbf{x})\ge0$. The formal verification problem asks the question, ``Does there exist an input $\mathbf{x}\in\mathcal{E}\subseteq\mathcal{X}$ in a constrained subset, $\mathcal{E}$, of the domain such that the property is violated?'' If there exists a counterexample violating the property, we say that the property is satisfiable ({\scshape sat}), and otherwise, unsatisfiable ({\scshape unsat}).

Another example is provided by adversarial properties from the deep learning literature on datasets such as {\scshape mnist}. Consider a neural network $f_\theta(\mathbf{x})=\text{Softmax}(\mathbf{z}(\mathbf{x}))$ that classifies  images, $\mathbf{x}$, into $C$ classes, where the output of $f$ gives the probability of each class.
Let $\delta$ be a small perturbation in an $l_p$-ball of radius $\epsilon$, that is, $\Vert\delta\Vert_p<\epsilon$. Then $\mathbf{x}=\mathbf{x}'+\delta$ is an adversarial example for $\mathbf{x}'$ if $\argmax_i\mathbf{z}(\mathbf{x})_i\neq\argmax_i\mathbf{z}(\mathbf{x}')_i$, i.e. the perturbation changes the prediction.
Here, the property function is $s(\mathbf{x})=\max_{i\neq c}\left(\mathbf{z}(\mathbf{x})_i - \mathbf{z}(\mathbf{x})_c\right)$, where $c=\argmax_j\mathbf{z}(\mathbf{x}')_j$ and $s(\mathbf{x})\ge0$
indicates that $\mathbf{x}$ is an adversarial example.
Our approach subsumes adversarial properties as a specific case.
\section{Discussion}
\vspace{-4pt}
We have introduced a new measure for the intrinsic robustness of a neural network,
and have validated its utility on several datasets from the formal verification and
deep learning literatures.  Our
approach was able to exactly emulate formal verification approaches for satisfiable properties
and provide high confidence, accurate predictions for properties which were not.  The two
key advantages it provides over previous approaches are: a) providing an explicit and intuitive
measure for \emph{how} robust networks are to satisfiable properties; and b) providing
improved scaling over classical approaches for identifying unsatisfiable properties.

Despite providing a more informative measure of how robust a neural network is, our approach may not be appropriate in all circumstances. In situations where there is an explicit and effective adversary, instead of inputs being generated by chance, we may care more about how far away the single closest counterexample is to the input, rather than the general prevalence of counterexamples. Here our method may fail to find counterexamples because they reside on a subset with probability less than $P_\text{min}$; the counterexamples may even reside on a subset of the input space with measure zero with respect to the input distribution. On the other hand, there are many practical scenarios, such as those discussed in the introduction, where either it is unrealistic for there to be no counterexamples close to the input, the network (or input space) is too large to realistically permit formal verification, or where potential counterexamples are generated by chance rather than by an adversary. We believe that for these scenarios our approach offers significant advantages to formal verification approaches.

Going forward, one way the efficiency of our approach could be improved further is by using a more efficient base MCMC kernel in our AMLS estimator, that is replace line 12 in Algorithm 1 with a more efficient base inference scheme. The current MH scheme was chosen on the basis of simplicity and the fact it already gave effective empirical performance. However, using more advanced inference approaches, such as gradient-based approaches like Langevin Monte Carlo (LMC)~\citep{RosskyEtAl1978} and Hamiltonian Monte Carlo~\citep{Neal2011}, could provide significant speedups by improving the mixing of the Markov chains, thereby reducing the number of required MCMC transitions.
\section{Experiments}
\label{sec:experiments}
\vspace{-4pt}
\subsection{Emulation of formal verification}
\vspace{-4pt}
In our first experiment\footnote{Code to reproduce all experimental results is available at \url{https://github.com/oval-group/statistical-robustness}.}, we aim to test whether our robustness estimation framework is able to
effectively emulate formal verification approaches, while providing additional robustness
information for {\scshape sat} properties.  In particular, we want to test
whether it reliably identifies properties as being {\scshape unsat},
for which $\mathcal{I}=0$, or {\scshape sat}, for which $\mathcal{I}>0$.  We note that
the method still provides a formal demonstration
for {\scshape sat} properties because having a non-zero estimate for $\mathcal{I}$ indicates
that at least one counterexample has been found.  Critically, it further provides a measure for how robust
{\scshape sat} properties
are, through its estimate for $\mathcal{I}$.

We used the {\scshape CollisionDetection} dataset introduced in
the formal verification literature by
  \citep{Ehlers2017}.
  It consists of a neural network with six inputs that has been trained to classify two car trajectories as colliding or non-colliding. The architecture has 40 linear nodes in the first layer, followed by a layer of max pooling, a ReLU layer with 19 hidden units, and an output layer with 2 hidden units. Along with the dataset, $500$ properties are specified for verification, of which $172$ are {\scshape sat} and $328$ {\scshape unsat}.
This dataset was chosen because the model is small enough so that the properties can be formally verified.
These formal verification methods do not calculate the value of $\mathcal{I}$, but rather confirm the existence of a counterexample for which $s(\mathbf{x})>0$.

We ran our approach on all 500 properties, setting $\rho=0.1$, $N=10^4$, $M=1000$ (the choice of these hyperparameters will be justified in the next subsection),
and using a uniform distribution over the input constraints as the perturbation model, along with a uniform random walk proposal.
We compared our metric estimation approach against the naive Monte Carlo estimate using $10^{10}$ samples. The generated estimates of $\mathcal{I}$ for all {\scshape sat} properties are shown
in Figure~\ref{fig:exp2-contrasting-results}.

Both our approach and the naive MC baseline correctly identified all of the {\scshape unsat}
properties by estimating $\mathcal{I}$ as exactly zero.
However, despite using substantially more samples, naive MC failed to find a counterexample for
8 of the rarest {\scshape sat} properties,
thereby identifying them as {\scshape unsat}, whereas our approach found a counterexample for all the
{\scshape sat} properties. As shown in Figure~\ref{fig:exp2-contrasting-results}, the
variances in the estimates for $\mathcal{I}$ of our approach were also very low and matched the unbiased
MC baseline estimates for the more commonly violated properties, for which the latter approach
still gives reliable, albeit less efficient, estimates.
Along with the improved ability to predict rare events, our approach was also significantly
faster than naive MC
throughout, with a speed up of several orders of magnitude for properties where the event is not rare---a single run with naive MC took about 3 minutes, whereas a typical run of ours took around 3 seconds.

\input{figure-collisiondetection-results}

\subsection{Sensitivity to Parameter Settings}

As demonstrated by \citet{BrehierEtAl2015}, AMLS is unbiased under the assumption that perfect sampling from the targets, $\{\pi_k\}_{k=1}^{K-1}$, is possible, and that the cumulative distribution function of $s(X)$ is continuous. In practice, finite mixing rates of the Markov chains and the dependence between the initialization points for each target means that sampling is less than perfect, but improves with larger values of $M$ and $N$.  The variance, on the other hand,
theoretically strictly decreases with larger values of $N$ and $\rho$ \citep{BrehierEtAl2015}.

In practice, we found that while larger values of $M$ and $N$ were always beneficial,
setting $\rho$ too high introduced biases into the estimate, with $\rho=0.1$ empirically
providing a good trade-off between bias and variance.  Furthermore,
this provides faster run times than large values of $\rho$, noting that the smaller values of
$\rho$ lead to larger gaps in the levels.

To investigate the effect of the parameters more formally,
we further ran AMLS on the {\scshape sat} properties of {\scshape CollisionDetection}, varying $\rho\in\{0.1,0.25,0.5\}$,
$N\in\{10^3,10^4,10^5\}$ and $M\in\{100,250,1000\}$, again comparing to the naive MC estimate for $10^{10}$ samples.  We found
that the value of $N$ did not make a discernible difference in this range
regardless of the values for $\rho$ and $M$, and thus all presented
results correspond to setting $N=10^4$.
As shown in Figure~\ref{fig:exp2-varying-rho}, we
found that the setting of $\rho$ made a noticeable difference to the estimates for the relatively rarer
events.  All the same, these differences were small relative to the differences
between properties.  As shown in Figure~\ref{fig:exp2-varying-M},
 the value of $M$ made little difference when $\rho=0.1$,.  Interesting though,
  we found that the value of $M$ was important for different values of $\rho$, as shown in
Appendix~\ref{sec:var-M}, with larger values of $M$ giving better
results as expected.

\input{figure-adversarial-mnist}

\subsection{Convergence with higher-dimensional inputs and larger networks}
To validate the algorithm on a higher-dimensional problem, we first tested adversarial properties on the {\scshape mnist} and {\scshape cifar--10} datasets using a dense ReLU network with two hidden-layer of size $256$. An $l_\infty$-norm ball perturbation around the data point with width $\epsilon$ was used as the uniform input model, with $\epsilon=1$ representing an $l_\infty$-ball filling the entire space (the pixels are scaled to $[0,1]$), together with a uniform random walk MH proposal. After training the classifiers, multilevel splitting was run on ten samples from the test set at multiple values of $\epsilon$, with $N=10000$ and $\rho=0.1$, and $M\in\{100,250,1000\}$ for {\scshape mnist} and
$M\in\{100,250,500,1000,2000\}$ for {\scshape cifar--10}. The results for naive MC were also evaluated using $5\times10^9$ samples---less than the previous experiment as the larger network made estimation more expensive---in the cases where the event was not too rare. This took around twenty minutes per naive MC estimate, versus a few minutes for each AMLS estimate.

As the results were similar across datapoints, we present the result for a single example in the top two rows of Figure~\ref{fig:exp2-mnist}. As desired, a smooth curve is traced out as $\epsilon$ decreases, for which the event $E$ becomes rarer. For {\scshape mnist}, acceptable accuracy is obtained for $M=250$ and high accuracy results for $M=1000$. For {\scshape cifar--10}, which has about four times the input dimension of {\scshape mnist}, larger values of $M$ were required to achieve
comparable accuracy.
The magnitude of $\epsilon$ required to give a certain value of $\log(\mathcal{I})$ is smaller for {\scshape cifar--10} than {\scshape mnist}, reflecting that adversarial examples for the former
are typically more perceptually similar to the datapoint.

To demonstrate that our approach can be employed on large networks, we tested adversarial properties on the {\scshape cifar--100} dataset and a much larger DenseNet architecture \citep{HuangEtAl2017b}, with depth and growth-rate $40$ (approximately $2\times10^6$ parameters). Due to the larger model size, we set $N=300$, the largest minibatch that could be held in memory (a larger $N$ could be used by looping over minibatches). The naive Monte Carlo estimates used $5\times10^6$ samples for about an hour of computation time per estimate, compared to between five to fifteen minutes for each AMLS estimate. The results are presented in the bottom row of Figure~\ref{fig:exp2-mnist}, showing that our algorithm agrees with the naive Monte Carlo estimate.

\subsection{Robustness of provable defenses during training}
We now examine how our robustness metric varies for a ReLU network as
that network is trained to be more robust against norm bounded
perturbations to the inputs using the method of \citet{WongKolter2018}.
Roughly speaking, their method works by approximating the set of outputs resulting
from perturbations to an input with a convex outer bound, and minimizing the worst case loss over this set.
The motivation for this experiment is twofold.  Firstly, this training provides a series of networks with
ostensibly increasing robustness, allowing us to check if our approach produces robustness estimates consistent
with this improvement.
Secondly, it allows us to investigate whether the training to improve robustness for
one type of adversarial attack helps to protect against others.  Specifically, whether training for
small perturbation sizes improves robustness to larger perturbations.

We train a CNN model on {\scshape mnist} for 100 epochs with the standard cross-entropy loss, then train the network for a further 100 epochs using the robust loss of \citet{WongKolter2018}, saving a snapshot of the model at each epoch. The architecture is the same as in \citep{WongKolter2018}, containing two strided convolutional layers with 16 and 32 channels, followed by two fully connected layers with 100 and 10 hidden units, and ReLU activations throughout.
The robustification phase trains the classifier to be robust in an $l_\infty$ $\epsilon$-ball around the inputs, where $\epsilon$ is annealed from $0.01$ to $0.1$ over the first $50$ epochs. At a number of epochs during the robust training, we calculate our robustness metric with $\epsilon\in\{0.1,0.2,0.3\}$ on $50$ samples from the test set. The results are summarized in Figure \ref{fig:exp4-robust-summary} with additional per-sample results in Appendix \ref{sec:app:persample-robustness}.
We see that our approach is able to capture variations in the robustnesses of the network.

\input{figure-robust-training}

As the method of \citet{WongKolter2018} returns the maximum value of the property for each sample over a convex outer bound on the perturbations, it is able to produce certificates-of-robustness for some datapoints. If the result returned is less than $0$ then no adversarial examples exist in an $l_\infty$ ball of radius $\epsilon$ around that datapoint. If the result returned is greater than $0$, then the datapoint may or may not be robust in that $l_\infty$ ball, due to fact that it optimizes over an outer bound.

Though we emphasize that the core aim of our approach is in providing richer information for {\scshape sat} properties,
this provides an opportunity to see how well it performs at establishing {\scshape unsat} properties relative
to a more classical approach.  To this end,
we compared the fraction of the $50$ samples from the test set that are verified by the method of~\citet{WongKolter2018},
to the fraction that have a negligible volume of adversarial examples, $\mathcal{I}=P_\text{min}$, in their $l_\infty$ $\epsilon$-ball neighbourhood. The results are presented in Figure \ref{fig:exp4-robust-certificate}.

Our method forms an upper bound on the fraction of robust samples, which can be made arbitrarily tighter by taking $P_\text{min}\to0$. \citet{WongKolter2018}, on the other hand, forms a lower bound on the fraction of robust samples, where the tightness of the bound depends on the tightness of the convex outer bound, which is unknown and cannot be controlled.
Though the true value must lie somewhere between the two bounds, our bound still holds physical meaning it its own right
in a way that \citet{WongKolter2018} does not: it is the proportion of samples for which the prevalence of
violations is less than an a given acceptable threshold $P_{\min}$.

This experiment also highlights an important shortcoming of \citet{WongKolter2018}. The memory usage of their procedure depends on how many ReLU activations cross their threshold over perturbations. This is high during initial training for $\epsilon=0.1$ and indeed the reason why the training procedure starts from $\epsilon=0.01$ and gradually anneals to $\epsilon=0.1$. The result is that it is infeasible (the GPU memory is exhausted)---even for this relatively small model---to calculate the maximum value of the property on the convex outer bound for $\epsilon\in\{0.2,0.3\}$ at all epochs, and $\epsilon=0.1$ for epochs before $32$. Even in this restricted setting where our metric has been reduced to a binary one, it appears to be more informative than that of \citet{WongKolter2018} for this reason.

\section{Introduction}
\label{sec:intro}
The robustness of deep neural networks must be guaranteed in mission-critical applications where their failure could have severe real-world implications.
This motivates the study of neural network verification, in which one wishes to assert
whether certain inputs in a given subdomain of the network might lead to important properties
being violated \citep{ZakrzewskiEtAl2001, BunelEtAl2018}.  For example, in a classification task, one might want to ensure that small
perturbations of the inputs do not lead to incorrect class labels being predicted \citep{SzegedyEtAl2013, GoodfellowEtAl2015}.

The classic approach to such verification has focused on answering the binary question of
whether there exist any counterexamples that violate the property of interest.  We argue that this approach has two major drawbacks.  Firstly,
it provides no notion of how robust a network is whenever a counterexample can be found.
Secondly, it creates a computational problem whenever no counterexamples exist, as formally
verifying this can be very costly and does not currently scale to the size of networks used in
many applications.

To give a demonstrative example, consider a neural network for classifying objects in the
path of an autonomous vehicle.  It will almost certainly be infeasible to train such a network that is
perfectly robust to misclassification.  Furthermore, because the network will most likely
need to be of significant size to be effective, it is unlikely to be tractable
to formally verify the network is perfectly robust, even if such a network exists.
Despite this, it is still critically important to assess the robustness of the network, so that manufacturers
can decide whether it is safe to deploy.

To address the shortfalls of the classic approach,
we develop a new measure of intrinsic robustness of neural networks
based on the \emph{probability} that a property is violated under an input distribution model.
Our measure is based on two key insights.  The first is that for many, if not most, applications,
full formal verification is neither necessary nor realistically achievable, such that one
actually desires a notion of \emph{how} robust a network is to a set of inputs, not just a binary answer as to
whether it is robust or not.  The second
is that most practical applications have some acceptable level of risk, such that it is sufficient
to show that the probability of a violation is below a certain threshold, rather than confirm that
this probability is exactly zero.

By providing a probability of violation, our approach is able to address the needs of
applications such as our
autonomous vehicle example. If the network
is not perfectly robust, it provides an explicit measure of exactly how robust the network is.
If the network is perfectly robust, it is
still able to tractability assert that a violation event is ``probably-unsatisfiable''.   That is it is able to
statistically conclude that the violation probability is below some tolerance threshold to true zero,
even for large networks for which formal verification would not be possible.

Calculating the probability of violation is still itself a computationally challenging task, corresponding to
estimating the value of an intractable integral.  In particular, in most cases, violations of the target
property constitute (potentially extremely) rare events.  Consequently, the simple approach of
constructing a direct Monte Carlo estimate by sampling from the input model and evaluating
the property will be expensive and only viable when the event is relatively common.
To address this, we adapt an algorithm from the Monte Carlo literature,
adaptive multi-level splitting (AMLS) \citep{GuyaderEtAl2011, Nowozin2015}, to our network verification setting.
AMLS is explicitly designed for prediction of rare events and our adaptation means that we are able to
reliably estimate the probability of violation, even when the true value is extremely small.

Our resulting framework is easy to implement, scales linearly in the cost of the forward operation of the neural network, and is agnostic both to the network architecture and input model. Assumptions such as piecewise linearity, Lipschitz continuity, or a specific network form are not required. Furthermore, it produces a diversity of samples
which violate the property as a side-product.
To summarize, our main contributions are: \vspace{-3pt}
\begin{itemize}
	\setlength\itemsep{0.05em}
	\item  Reframing neural network verification as the estimation of the probability of a violation, thereby providing a more informative robustness metric
	for non-verifiable networks;
	\item Adaptation of the AMLS method to our verification framework
	to allow the tractable estimation of our metric for large networks and rare events;
	\item Validation of our approach on several models and datasets from the literature.
\end{itemize}
\vspace{-6pt}
\section{0pt}{8pt plus 2pt minus 2pt}{4pt plus 2pt minus 2pt}
\setlength{\parskip}{0.4em}

\begin{document}

\maketitle

\begin{abstract}
	\vspace{-4pt}
\input{abstract}
\end{abstract}

\input{introduction}
\input{related}

\input{background}
\input{robustness_metric}
\input{metric_estimation}
\input{experiments}
\input{discussion}

\subsubsection*{Acknowledgments}
We gratefully acknowledge Sebastian Nowozin for suggesting to us to apply multilevel splitting to the problem of estimating neural network robustness. We also thank Rudy Bunel for his help with the {\scshape CollisionDetection} dataset, and Leonard Berrada for supplying a pretrained DenseNet model.

SW gratefully acknowledges support from the EPSRC AIMS CDT through grant EP/L015987/2. TR and YWT are supported in part by the European Research Council under the European Union's Seventh Framework Programme (FP7/2007–2013) / ERC grant agreement no. 617071. TR further acknowledges support of the ERC StG IDIU. MPK is supported by EPSRC grants EP/P020658/1 and TU/B/000048.

\section{Metric Estimation}
\label{sec:est}
\vspace{-4pt}
Our primary goal is to estimate (\ref{eq:integral}) in order to obtain a measure of robustness.  Ideally, we also
wish to generate example inputs which violate the property.   Unfortunately,
the event $E$ is typically very rare in verification scenarios.
Consequently, the estimating the integral directly using Monte Carlo,
\begin{align}
\hat{P}_{X\sim p\left(\cdot\right)}\left(s(X)\ge0\right) = \frac{1}{N}\sum\nolimits^N_{n=1}\mathbbm{1}_{\{s(\mathbf{x}_n)\ge0\}}, \quad \text{where} \quad \mathbf{x}_n \overset{\text{i.i.d.}}{\sim} p(\cdot)
\end{align}
is typically not feasible for real problems, requiring an impractically large number of samples to achieve
a reasonable accuracy. Even when $E$ is not a rare event, we desire to estimate the probability using as few forward passes of the neural network as possible to reduce computation.  Furthermore, the dimensionality of $\mathbf{x}$ is typically large for practical problems,
such that it is essential to employ a method that scales well in dimensionality.
Consequently many of the methods commonly employed for such problems,
such as the cross-entropy method \citep{Rubinstein1997, DeBoerEtAl2005}, are inappropriate
due to relying on importance sampling, which is well known to scale poorly.

As we will demonstrate empirically, a less well known but highly effective method from the
statistics literature, adaptive multi-level splitting (AMLS) \citep{KahnHarris1951,GuyaderEtAl2011},
can be readily adapted to address all the aforementioned computational challenges.
Specifically, AMLS is explicitly designed for estimating the probability
of rare events and our adaptation is able to give highly accurate estimates even
when the $E$ is very rare.  Furthermore, as will be explained later, AMLS also
allows the use of MCMC transitions, meaning that our approach is able to scale effectively in the
dimensionality of $\mathbf{x}$.
A further desirable property of AMLS is that it
produces property-violating examples as a side product,
namely, it produces samples from the distribution
\begin{align}
	\pi(\mathbf{x}) &\triangleq
	p(\mathbf{x}\mid E) =  p(\mathbf{x})\mathbbm{1}_{\{s(\mathbf{x})\ge0\}}\big/
	\mathcal{I}\left[p,s\right].\label{eq:target}
\end{align}
Such samples could, in theory, be used to perform robust learning, in a similar spirit to~\citet{GoodfellowEtAl2015} and~\citet{MadryEtAl2017}.

\subsection{Multi-level splitting}
Multi-level splitting~\citep{KahnHarris1951} divides the problem of predicting the probability of a rare event
into several simpler ones.  Specifically, we
construct a sequence of intermediate targets,
\begin{align*}
\pi_k(\mathbf{x}) &\triangleq p(\mathbf{x}\mid\{s(\mathbf{x})\ge L_k\})\propto p(\mathbf{x})\mathbbm{1}_{\{s(\mathbf{x})\ge L_k\}}, \ \ k=0,1,2,\ldots,K,
\end{align*}
for levels, $-\infty=L_0<L_1<L_2<\cdots<L_K=0$, to bridge the gap between the input model
$p(\mathbf{x})$ and the target $\pi(\mathbf{x})$.  For any choice of the intermediate
levels, we can now represent equation (\ref{eq:integral}) through the following factorization,
\begin{align}
\begin{split}
	P_{X\sim p(\cdot)}\left(s(X)\ge0\right) &= \prod\nolimits^{K}_{k=1}P\left(s(X)\ge L_{k}\mid s(X)\ge L_{k-1}\right) = \prod\nolimits^{K}_{k=1}P_k,\\
	\text{where} \quad P_k&\triangleq
	\mathbb{E}_{X\sim \pi_{k-1}(\cdot)}\left[\mathbbm{1}_{\{s(X)\ge L_{k}\}}\right].
	\label{eq:multilevel}
\end{split}
\end{align}
Provided consecutive levels are sufficiently
close, we will be able to reliably estimate each $P_k$ by making use of the samples
from one level to initialize the estimation of the next.

Our approach starts by first drawing $N$ samples, $\{\mathbf{x}^{(0)}_n\}^N_{n=1}$,
from $\pi_0(\cdot)=p(\cdot)$, noting that
this can be done exactly because the perturbation model is known.  These samples can then be used to
estimate $P_1$ using simple Monte Carlo,
\begin{align*}
	P_1 &\approx \hat{P}_1 \triangleq \frac{1}{N}\sum\nolimits^N_{n=1}\mathbbm{1}_{\{s(\mathbf{x}^{(0)}_n)\ge L_1\}}
	\quad \text{where} \quad \mathbf{x}^{(0)}_n \sim \pi_0(\cdot).
\end{align*}
In other words, $P_1$ is the fraction of these samples whose property is greater than $L_1$.  Critically,
by ensuring the value of $L_1$ is sufficiently
small for $\{s(\mathbf{x}_n)\ge L_1\}$ to be a common event, we
can ensure $\hat{P}_1$ is a reliable estimate for moderate numbers
of samples $N$.

To estimate the other $P_k$, we need to be able to draw samples from $\pi_{k-1}(\cdot)$. For this we note that if $\{\mathbf{x}^{(k-2)}_n\}^N_{n=1}$ are distributed according to $\pi_{k-2}(\cdot)$,
then the subset of these samples for which $s(\mathbf{x}^{(k-2)}_n)\ge L_{k-1}$ are distributed according
to $\pi_{k-1}(\cdot)$.  Furthermore, setting $L_{k-1}$ up to ensure this event is not rare means a
significant proportion of the samples will satisfy this property.

To avoid our set of samples shrinking from one level to the next, it is necessary to carry out a rejuvenation step
to convert this smaller set of starting samples to a full set of size $N$ for the next level.  To do this,
we first carry out a uniform resampling with replacement from the set of samples satisfying
$s(\mathbf{x}^{(k-1)}_n)\ge L_k$ to generate a new set of $N$ samples
which are distributed according to $\pi_k(\cdot)$, but with a large number of duplicated samples.  Starting with these samples,
we then successively apply $M$ Metropolis--Hastings (MH) transitions targeting $\pi_k(\cdot)$ separately to each sample
to produce a fresh new set of samples $\{\mathbf{x}^{(k)}_n\}^N_{n=1}$ (see Appendix~\ref{sec:app:MH} for full details).
These samples can then in turn be used to form a Monte Carlo estimate
for $P_k$,
\begin{align}
P_k &\approx \hat{P}_k \triangleq \frac{1}{N}\sum\nolimits^N_{n=1}\mathbbm{1}_{\{s(\mathbf{x}^{(k-1)}_n)\ge L_{k}\}}
\quad \text{where} \quad \mathbf{x}_n^{(k-1)} \sim \pi_{k-1}(\cdot),
\end{align}
along with providing the initializations for the next level.  Running more MH transitions decreases the correlations between the set
of samples, improving the performance of the estimator.

We have thus far omitted to discuss how the levels $L_k$ are set, other than asserting the need for the
levels to be sufficiently close to allow reliable estimation of each $P_k$.  Presuming that we are
also free to choose the number of levels $K$, there is inevitably a trade-off between ensuring
that each $\{s(X)\ge L_k\}$ is not rare given $\{s(X)\ge L_{k-1}\}$,
and keeping the number of levels small to reduce computational costs and avoid the build-up of errors.  AMLS
\citep{GuyaderEtAl2011} builds on the basic multi-level splitting process,
 providing an elegant way of controlling this trade-off by adaptively
selecting the level to be the minimum of $0$ and some quantile of the property under the current samples.  The approach terminates when the level reaches zero, such that $L_K=0$ and
$K$ is a dynamic parameter chosen implicitly by the adaptive process.

 Choosing the $\rho$th quantile of the values of the property results in discarding a fraction $(1-\rho)$ of the chains at each step of the algorithm.  This allows explicit control of the rarity of the events to keep them at a manageable level.
 We note that if all the sample property values are distinct, then this approach gives $P_k = \rho, \, \forall k<K$.
 To give intuition to this, we can think about splitting up $\log(\mathcal{I})$ into chunks of size $\log(\rho)$.
 For any value of $\log(\mathcal{I})$, there is always a unique pair of values $\{K,P_K\}$
 such that $\log(\mathcal{I}) = K \log(\rho) + \log(P_K)$, $K\ge0$ and $P_K < \rho$.  Therefore the problem of estimating
 $\mathcal{I}$ is equivalent to that of estimating $K$ and $P_K$.

\setlength{\textfloatsep}{8pt}
\begin{algorithm}[t]
	\small\onehalfspacing
	\caption{Adaptive multi-level splitting with termination criterion}
	\label{alg:multi-level-splitting}
	\begin{algorithmic}[1]
		\State {\bfseries Input:} Input model $p(\mathbf{x})$, sample quantile $\rho$,  MH proposal $g(\mathbf{x}'|\mathbf{x})$, number of MH steps $M$, termination threshold
		$\log(P_{\text{min}})$
		\State Sample $\{\mathbf{x}_n^{(0)}\}^N_{n=1}$ i.i.d. from $p(\cdot)$
		\State Initialize $L\gets-\infty, \quad L_{\text{prev}}\gets-\infty, \quad \log(\mathcal{I})\gets0, \quad k\gets0$
		\While{$L<0$}
		\State $k\gets k+1$
		\State Evaluate and sort $\{s(\mathbf{x}_n^{(k-1)})\}^N_{n=1}$ in descending order
		\State $L_{k}\gets\min\{0, s(\mathbf{x}_{\left\lfloor \rho N\right\rfloor}^{(k-1)})\}$\hfill$\triangleright$ Updating the level
		\State $\hat{P}_{k} \gets \#\{\mathbf{x}_n^{(k-1)}\mid s(\mathbf{x}_n^{(k-1)})\ge L\} \, / \, N$
		\State $\log(\mathcal{I}) \gets \log(\mathcal{I}) + \log(\hat{P}_{k})$\hfill$\triangleright$ Updating integral estimate
				\IIf{$\log(\mathcal{I})<\log(P_{\text{min}})$} \label{line:term-1}
				\Return $(\emptyset, -\infty)$ \EndIIf
				\hfill$\triangleright$ Final estimate will be less than $\log (P_{\text{min}})$
		\State Initialize $\{\mathbf{x}^{(k)}_n\}^N_{n=1}$ by resampling with replacement $N$ times from $\{\mathbf{x}_n^{(k-1)}\mid s(\mathbf{x}_n^{(k-1)})\ge L\}$
		\State Apply $M$ MH updates separately to each $\mathbf{x}^{(k)}_n$ using $g(\mathbf{x}'|\mathbf{x})$ \label{line:mh-1}
		\State [Optional] Adapt $g(\mathbf{x}'|\mathbf{x})$ based on MH acceptance rates
		\EndWhile
		\State \Return $(\{\mathbf{x}_n^{(k)}\}^N_{n=1}$, $\log(\mathcal{I}))$
	\end{algorithmic}
\end{algorithm}

\subsection{Termination Criterion}

The application of AMLS to our verification problem presents a significant complicating factor
in that the true probability of our rare event might be exactly zero.  Whenever this is the case,
the basic AMLS approach outlined in \citep{GuyaderEtAl2011} will never terminate as the quantile of
the property will never rise above
zero; the algorithm simply produces closer and closer intermediate levels as it waits for the event to
occur.

To deal with this, we introduce a termination criterion based on the observation
that AMLS's running estimate for $\mathcal{I}$ monotonically decreases during running.
Namely, we introduce a threshold probability,
$P_{\text{min}}$, below which the estimates will be treated as being numerically zero.
We then terminate the algorithm
if $\mathcal{I}<P_{\text{min}}$ and return $\mathcal{I}=0$, safe in the knowledge that even if the
algorithm would eventually generate a finite estimate for $\mathcal{I}$, this estimate is guaranteed to
be less than $P_{\text{min}}$.

Putting everything together, gives the complete method as shown in Algorithm \ref{alg:multi-level-splitting}. See Appendix \ref{sec:app:implementation-details} for low-level implementation details.
\section{Related work}
\label{sec:related}
\vspace{-4pt}
The literature on neural network robustness follows two main threads.  In the optimization community, researchers seek to formally prove that a property holds for a neural network by framing it as a satisfiability problem \citep{ZakrzewskiEtAl2001}, which we refer to as the classical approach to verification. Such methods have only been successfully scaled beyond one hidden layer networks for piecewise linear networks \citep{ChengEtAl2017, KatzEtAl2017}, and even then these solutions do not scale to, for example, common image classification architectures with input dimensions in the hundreds, or apply to networks with nonlinear activation functions \citep{BunelEtAl2018}. Other work has sought approximate solutions in the same general framework but still does not scale to larger networks \citep{PulinaEtAl2010, XiangEtAl2018, HuangEtAl2017}. As the problem is NP-hard \citep{KatzEtAl2017}, it is unlikely that an algorithm exists with runtime scaling polynomially in the number of network nodes.

In the deep learning community, research has focused on constructing and defending against adversarial attacks, and by estimating the robustness of networks to such attacks. \citet{WengEtAl2018b} recently constructed a measure for robustness to adversarial attacks estimating a lower bound on the minimum adversarial distortion, that is the smallest perturbation required
to create an adversarial example.  Though the approach scales to large networks, the estimate of the lower bound is often demonstratively
incorrect: it is often higher than an upper bound on the minimum adversarial distortion \citep{GoodfellowEtAl2018}. Other drawbacks of the method are that it cannot be applied to networks that are not Lipschitz continuous,
it requires an expensive gradient computation for each class per sample, does not produce adversarial examples, and cannot be applied to non-adversarial properties.  The minimum adversarial distortion is also itself a somewhat
unsatisfying metric for many applications,
as it conveys little information about the prevalence of adversarial examples.

In other work spanning both communities \citep{GehrEtAl2018, WengEtAl2018, WongKolter2018}, researchers have relaxed the satisfiability problem of classical verification, and are able to produce certificates-of-robustness for some samples (but not all that are robust) by giving a lower-bound on the minimal adversarial distortion. Despite these methods scaling beyond formal verification, we note that this is still a binary measure of robustness with limited informativeness.

An orthogonal track of research investigates the robustness of reinforcement learning agents to failure \citep{huang2017adversarial, lin2017tactics}. For instance, concurrent work to ours \citep{uesato2018rigorous} takes a continuation approach to efficiently estimating the probability that an agent fails when this may be a rare event.

\section{Robustness Metric}
\label{sec:metric}
\vspace{-4pt}
The framework for our robustness metric is very general, requiring only a) a neural
network $f_\theta$, b) a property function $s(\mathbf{x};f,\phi)$, and c) an input model $p(\mathbf{x})$.  Together these
define an integration problem, with the main practical challenge being the
estimation of this integral.
Consequently, the method can be used for
any neural network. The only requirement is that we can evaluate the property function, which typically involves a forward pass of the neural network.

The property function, $s(\mathbf{x} ;f_\theta,\phi)$, is a deterministic
function of the input $\mathbf{x}$, the trained network $f_\theta$, and problem specific parameters $\phi$.
For instance, in the {\scshape mnist} example, $\phi=\argmax_i f_\theta(\mathbf{x}')_i$ is the true output of the unperturbed input. Informally, the property reflects how badly the network is performing
with respect to a particular property.  More precisely, the
 event
 \begin{align}
 \label{eq:event}
E \triangleq  \{s(\mathbf{x} ;f_\theta,\phi)\ge0\}
 \end{align}
represents the property being violated.
 Predicting the occurrence of these, typically rare,
  events will be the focus of our work.
We will omit the dependency on $f_\theta$ and $\phi$ from here on for notional
conciseness, noting that these are assumed to be fixed and known for verification
problems.

The input model, $p(\mathbf{x})$, is a distribution over the subset of the input domain that we are considering for counterexamples.
For instance, for the {\scshape mnist} example we could use
	$p(\mathbf{x};\mathbf{x}') \propto
	\mathbbm{1}\left(\lVert\mathbf{x}-\mathbf{x}'\rVert_p \le \epsilon\right)$
to consider uniform perturbations to the input around an $l_p$-norm ball with radius $\epsilon$. More generally, the input model can be used to place restrictions on the input domain and potentially also to reflect that certain violations might be more damaging than others.

Together, the property function and input model specify the probability
of failure through the integral
\begin{align}
	\mathcal{I}\left[p, s\right] &\triangleq P_{X\sim p\left(\cdot\right)}\left(s(X)\ge0\right)
	= \int_{\mathcal{X}}\mathbbm{1}_{\{s(\mathbf{x})\ge0\}}p(\mathbf{x})d\mathbf{x}.\label{eq:integral}
\end{align}
This integral forms our measure of robustness.  The integral being equal to exactly zero corresponds
to the classical notion of a formally verifiable network.  Critically though, it also provides a measure
for \emph{how} robust a non-formally-verifiable network is.

redpajama_set_name: RedPajamaArXiv


IT Disaster Recovery Planning is a critical part of any organisation's IT infrastructure to ensure Business Continuity in the face of a data loss and technology failure. But as organisations rely more and more on IT infrastructure to store data and to run operations, a Disaster Recovery Plan has become more important than a part of IT infrastructure, it is a crucial piece of the business machine, required to reduce the business risk facing the organisation and to adhere to regulatory compliance.
We understand that Disaster Recovery is all about risk mitigation from events that are outside your span of control. An inadequate Disaster Recovery plan could incur the business significant costs and lost time when reestablising normal business activity, or in some cases it could prevent the business from restarting at all. Protecting your data and applications is critical as more than half of all companies who experience major data loss will be forced to close down within two years. Isolating your organisation from such events is crucial to it's survival. Read more statistics on the impact of major data loss on businesses.
At Beyond Technology Consulting we believe that Disaster Recovery and Business Continuity Planning go hand in hand. Having the infrastructure to store business data is not sufficient if the company does not have the appropriate plans, processes and capabilities to get the business up and running again.
We are aware that different organisations have different needs around Disaster Recovery and may be at different stages in the planning process. Our consulting approach in general is to offer bespoke services to clients that are scoped to meet their precise needs. Read more about how Beyond Technology approach Diaster Recovery. Our capabilities around Disaster Recovery encompass many areas and can be scaled to include a number of different elements of varying complexity, from a short compliance check, a rapid or full infrastructure assessment, to a full DR strategy and planning process.