The physicist Richard Feynman defined scientific honesty as:
It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty–a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked–to make sure the other fellow can tell they have been eliminated.
The Lancet Iraqi Casualty Survey is a dishonest piece of work. Setting aside all concerns about its methodology and practical implementation, it is easy to see that the paper was written in an intentionally deceptive manner designed not for scientific clarity but for political impact.
I thought I would detail this dishonesty in the form of fisking. Pack a lunch, it’s a long one.
[Download the PDF of the study so you can follow along and keep me honest]
As I mentioned in my last post, the major deception in the paper is the highly selective inclusion and exclusion of the outlier Falluja cluster throughout the paper. The Falluja cluster proved highly problematical for several reasons.
First, it returned a death-by-violence rate so high that it meant that at least 1% of the entire Iraqi population (>250,000) had died in the war almost all due to Coalition airstrikes. Since this clearly was not the case (you couldn’t hide that many bodies) it was clear that the Falluja cluster was an outlier, a statistical freak that, for whatever reason, returned a nonsensical result.
Falluja was also a problem cluster because it was collected in a wholly different method than all the other clusters. Down in the depths of the paper the authors spend a good half-page explaining how the measurement of the Falluja cluster differed from all the others. [p6 pg 6-7, p7 pg1]. GPS couldn’t be used to randomly select the cluster based on demographics, unlike the other clusters, so they did it manually. They had to be wary of insurgents and did a quick in-and-out. In other clusters, less than 8% of the households were abandoned, in Falluja 44% were, meaning that a lot of the results were actually extrapolation.
This presents the possibility that far more deaths had occurred than were reported and the interviewees that remained were the relatively lucky ones (underestimating mortality), or large numbers of residents had fled elsewhere and were still alive. Thus, the deaths reported by the remaining families might represent a disproportionate number of deaths from the larger community that used to live in the area, leading the interview data to overestimate mortality.
Not even considered in the paper is the very real possibility that the insurgents/jihidists who controlled Falluja at the time were able to influence the results. Even if they did not do so, many of the Sunnis in Falluja were hostile to the Coalition and might have taken the opportunity to exaggerate.
So the Falluja cluster was highly problematical even before it returned its off-the-chart death rate. For an honest researcher, the Falluja data presented three choices. He could: (1) include the cluster in all analysis since he had no a priori reason for excluding it, (2) declare the cluster’s data highly suspect and exclude it from all analysis, while taking to care to inform the readers that he had done so, or (3) provide dual analyses, one of which included the Falluja cluster and one of which did not, again taking great care to differentiate the two.
The authors of this study chose option negative four, i.e., confuse the issue as much as humanly possible. Roberts et al weave the Falluja data though the paper like a poison skein. Virtually every major statement they make is grounded on the Fallujah data. If those data are excluded, most of what they state is not true, yet they almost never make it clear when they are relying on the Falluja data and when they aren’t.
(Let me define some acronyms for convenience. IFC is Including Falluja Cluster. EFC is Excluding Falluja Cluster)
Let the fisking begin.
The paper goes off the rails in its opening Summary [p1] . The summary is super important because it is all that 99% of the world will ever see of the study. The media will report only the summary. The summary is where the spin really happens in a scientific paper.
The Findings paragraph starts well.
The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion.
That’s all kosher.
We estimate that 98000 more deaths than expected (8000-194000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.
So here we see Falluja honestly defined as outlier, but why don’t they include the estimated deaths IFC? Probably because the estimated range would immediately leap up past 250,000, which is an embarrassingly high number. Instead they bury the Falluja number in the depths of the paper.
The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death.
So is this true IFC, EFC or both? How can you tell? You can’t without looking at the study’s internals.
This is only true IFC and only just barely (51%). EFC, deaths from violence caused just 24% of all deaths. [p5 pg3]. Further, the breakdown of deaths by disease type is just pointless and silly, except that it creates the impression that deaths by violence are larger later on. It’s sleight of hand. I’m surprised they didn’t list infected hang nails as a cause of death.
Violent deaths were widespread, reported in 15 of 33 clusters,
IFC or EFC? It’s IFC. EFC is 14 of 32 clusters. The actual distribution of the 21 EFC over the 14 affected clusters is not provided in the study. It’s entirely possible that 13 clusters saw one death and 1 cluster saw 8. In fact, given the comically broad confidence interval of this study this type of distribution is more likely than not.
and were mainly attributed to coalition forces.
Curiously, this is never broken down inside the study, save as a passing reference that is IFC.
Most individuals reportedly killed by coalition forces were women and children.
This is easily the most provocative sentence in the Summary. Strangely, it is not supported even IFC. Table 2 shows that of 73 deaths IFC, 33, or 45% percent, were women and children. Even adding in the two elderly deaths gives us only 48%. Weird. Of more interest is that EFC, 62% of deaths are of military-age males and only 38% are of women, children and the elderly. These data make the exact opposite emotional point that the authors sought to make. The 2-to-1 adult-male to everyone-else ratio is what we would expect to see if most of the targets of Coalition attacks were military.
Correction: dsquared has pointed out in the comments that I made an error here. The sentence actually says that of the deaths caused by the Coalition are mostly women and children so I needed to subtract the 12 deaths identified as being caused by non-Coalition actors. Those deaths are of 11 adult males and 1 adult female. So IFC, the ratio of deaths among women and children must be calculated a percentage of 62 total deaths which is 28 children + 4 women for a total of 32 which is 52%.
Unfortunately, based on the paper it is impossible to know how many of the non-Coalition deaths occurred outside of Falluja so I can’t calculate the percentages EFC.
Fortunately, Les Roberts has apparently provided this information in an email.
I am forced to admit that the additional information does support the Summary’s statement. In fact it supports it to such a degree that I am surprised they didn’t break it out separately as it really helps their case.
The information that Roberts provide says that all 12 of the non-Coalition attributed deaths occurred outside of Falluja. Thats means of the 21 deaths EFC only 9 or 42% are attributed to the Coalition. The gender and age breakdown becomes 3 adult males, 1 adult female, 4 children and 2 elderly. Percentage wise that is 33% adult male, 11% adult female, 44% children, 22% elderly. So 66% of deaths EFC attributed to the Coalition are of women children or the elderly. The ratio of non-combatants to possible combatants is higher EFC than IFC.
Wonder why they never pointed that out in the paper? Plausibility perhaps?
The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8·1-419) than in the period before the war.
This is IFC. EFC is something like 22 times higher.
The Interpretation is the money graf for the entire paper. This is what will enter the worldwide dialog to be enshrined as revealed truth.
Interpretation Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq
This is only a conservative assumption IFC. Apparently, “conservative” means tossing out obviously bad outliers. EFC it is in fact the mainline finding. Indeed, there is a 50% chance that the actual number is lower. An actual “conservative” estimate would be about one interval down, which given the wide CI would fall somewhere in the 60,000-70,000 range. 58% of those would be deaths from violence EFC, so combat deaths would run 34,800-40,600, of which 62% were military-age males. That probably doesn’t have the same impact as a nice round 100,000, though.
Violence accounted for most of the excess deaths
This is true both IFC and EFC. But I think few realize this is in fact a ratio of deaths from violence to deaths from illness or accident. Increasing the health of the general population will make it appear that violence is getting worse even if the total number of people dying drops. Still, it is unambiguously supported by the study.
and air strikes from coalition forces accounted for most violent deaths.
This is only supported IFC. The study does not breakdown causes of deaths due to violence as being IFC or EFC. All such mentions are IFC. In fact, data are presented that could mean that most deaths from violence EFC were not due to airstrikes, but the study does not provide sufficient detail to be sure.
We have shown that collection of public-health information is possible even during periods of extreme violence.
As long as you don’t mind a huge amount of slop in your data.
Our results need further verification…
On that we can agree.
…and should lead to changes to reduce non-combatant deaths from air strikes.
(This last bit captures the utter pointlessness of this study. What in the entire study would help anybody reduce non-combat deaths at all? This study is completely useless in any practical sense. Despite the snotty implication in the conclusion that the US is shows a depraved indifference to civilian causalities, the study provides no guidance or information whatsoever that could be used to improve the situation. It is clearly of no value to anyone save as an anti-Coalition propaganda tool. I would be much more tolerant of the imperfections of the internals of this study if anybody could demonstrate any practical use for it.)
I think it is clear that the Summary for this study, which, again, is what most of the world will see and act on, is deliberately written to conceal the use of the Falluja cluster to make the most attention grabbing claims. EFC is a much, much different report. Honest researchers would have made that clear.
The conflation of IFC and EFC continues in the body of the paper.
From [p5 pg3]:
The main causes of death reported for the 14·6 months before the invasion were myocardial infarction, cerebrovascular accidents, and consequences of other chronic disorders, accounting for 22 (48%) reported deaths (table 2). After the war began, violence was the most commonly reported cause of death, either including (73/142 [51%]) or excluding (21/89 [24%]) the Falluja data, followed by myocardial infarction and cerebrovascular accidents (n=18) and accidents (n=13; table 2)
Violence is only clearly the most common cause of death IFC. It is only the case EFC if illness is needlessly broken up into subcategories. Here the authors are arranging the information to convey a certain impression. EFC 21 of 89 deaths, or 24%, were the result of intentional violence. The remaining 76% of deaths resulted from illness or accident. Is it really honest to say that EFC “violence was the most commonly reported cause of death?”
Let’s try a little quiz.
From [p5 pg4]
Figure 2 shows the number of deaths reported during the study period, disaggregated as non-violent deaths, violence in Falluja, and violence in all other clusters. An increase of violent death was noted during the occupation, and violence was geographically widespread, with violent deaths reported in 15 of 33 clusters (45%).
A) IFC or EFC?
Violence-specific mortality rate went up 58-fold (95% CI 8·1-419) during the period after the invasion.
B) IFC or EFC?
Table 2 includes 12 violent deaths not attributed to coalition forces, including 11 men and one woman. Of these, two were attributed to anti-coalition forces, two were of unknown origin, seven were criminal murders, and one was from the previous regime during the invasion
C) IFC or EFC?
Of the 28 children killed by coalition forces (median age 8 years), ten were girls, 16 were boys, and two were infants (sex was not recorded). Aside from a 14-year-old boy, all these deaths were children 12 years or younger.
D) IFC or EFC?
Hell, they’re all IFC.
I am beginning to wonder if the authors actually understand what an outlier is. I would love to know how many of the “12 violent deaths not attributed to coalition forces” occurred outside of the Falluja cluster. Since only 21 deaths total are EFC, if 12 of them were non-Coalition related that would change the complexion of the study immensely. If only 4 or so were EFC that would alter the assertion that most of excess deaths were the result of Coalition action. The study doesn’t provide this kind of breakdown.
I also like the “Of the 28 children killed by coalition forces…” Wow, fire up the Star Wars imperial march. Of course, EFC that would be “of the 4 children killed by coalition forces.” I guess that lacked punch.
This survey indicates that the death toll associated with the invasion and occupation of Iraq is probably about 100000 people, and may be much higher
How much higher? What about 200,000+ higher! Why didn’t you put that in the Summary? Shouldn’t people know your study says IFC that 300,000+ people have died? Good lord, the bodies must be stacked up like cordwood! Why the soft sell? Oh right, it’s an insanely high number and obviously wrong. Better toss out that cluster. Glad you didn’t use its results everywhere else in the study.
We have shown that even in extremely difficult circumstances, the collection of valid data is possible, albeit with limited precision.
Limited precision. Heh. What’s an order of magnitude or a factor of three? Give them a break! This study is vitally important for reasons nobody can explain to me.
In this case, the lack of precision does not hinder the clear identification of the major public-health problem in Iraq–violence.
I am so glad someone spent money to determine that violence is a major public-health problem in a war zone. And they did it so precisely too!
Several limitations exist with this study.
Please stop. I can’t breath.
Second, the January, 2002, to March, 2003, rate applied to the 366 births recorded in the interview households post-invasion would project 10·4 infant deaths, whereas we noted 21 to have happened.
IFC or EFC? Has to be IFC. The study only recorded 10 nonviolent infant deaths, and only 4 from violence EFC, so to get 21 deaths they would have to be IFC.
Second, as Spiegel and colleagues documented in Kosovo,21 there can be a dramatic clustering of deaths in wars where many die from bombings. The cluster survey methodology we used may have, by chance, missed small areas where a disproportionate number of deaths occurred, or conversely, selected a neighbourhood that was so severely affected by the war that it represents virtually none of the population and thus has skewed the mortality estimate too high. The results from Falluja merit extra consideration in this regard.
Extra consideration obviously means injecting the results into every statement unless expressly noted otherwise.
Despite widespread Iraqi casualties, household interview data do not show evidence of widespread wrongdoing on the part of individual soldiers on the ground. To the contrary, only three of 61 incidents (5%) involved coalition soldiers (all reported to be American by the respondents) killing Iraqis with small arms fire.
IFC or EFC? Has to be IFC since only 21 deaths are reported EFC. Anything over that has to be IFC.
The remaining 58 killings (all attributed to US forces by interviewees) were caused by helicopter gunships, rockets, or other forms of aerial weaponry
I would love to see an actual breakdown of all of this IFC and EFC. 12 of the deaths from violence were from non-Coalition actors. How many of those where IFC or EFC? It is possible to construct a distribution from these hints that 15 of 21 deaths from violence resulted from causes other than airstrikes. If most of those where EFC, it would invalidate the assertion that most people died in Coalition airstrikes.
Many of the Iraqis reportedly killed by US forces could have been combatants. 28 of 61 killings (46%) attributed to US forces involved men age 15-60 years, 28 (46%) were children younger than 15 years, four (7%) were women, and one was an elderly man.
IFC, but you knew that didn’t you?
The Geneva Conventions have clear guidance about the responsibilities of occupying armies to the civilian population they control. The fact that more than half the deaths reportedly caused by the occupying forces were women and children is cause for concern.
Of course, here in the study’s concluding paragraph it is IFC all the way.
I think I have shown that the study’s authors, despite labeling the Falluja cluster an outlier, and identifying it as a cluster measured differently from all of the others, rely on it heavily to draw the overwhelming majority of their conclusions. In fact, they use it in almost every statement or conclusion except for their “conservative” estimate.
Without Falluja most of this study absolutely evaporates. Without Falluja most of the victims of violence are not women and children as the authors asserted both in the opening Summary and the closing paragraph. Without Falluja, violence isn’t the major cause of death, illness and accident are. Without Falluja it is even possible that most of the victims of violence didn’t die in Coalition airstrikes.
Falluja may be a statistical outlier but the authors use it as the heart of the study. They are dishonest to do so.