The reliability of Wikipedia, compared to both other encyclopedias and more specialized sources, is often assessed in several ways, including statistically, by comparative review, and by analysis of the historical patterns, strengths and weaknesses inherent in the Wikipedia process. Because Wikipedia is a wiki, and open to collaborative editing by anyone, assessing its reliability requires also examining its ability to detect and rapidly remove false or misleading information.
A study in 2005 suggested that for scientific articles Wikipedia came close to the level of accuracy of Encyclopædia Britannica and had a similar rate for "serious errors."
An early study conducted by IBM researchers in 2003 found that "vandalism is usually repaired extremely quickly--so quickly that most users will never see its effects", they concluded that Wikipedia had "surprisingly effective self-healing capabilities. Caveat: Wikipedia started in 2001 (see History of Wikipedia), and hence the mentioned IBM study was focusing on a few articles that were created for a relatively short period of time, and on the first few versions of these articles. In science, it is easy to find deeper errors that lie dormant in an article for years before a "Good Samaritan" (if there were any) would come along to correct these errors. And that was for mature (but challenging) fields such as thermodynamics. For emerging scientific fields such as quantum information science, Wikipedia has been characterized as a medium that fosters "anarchy and distortions" by some experts.
Wikipedia's model of knowledge creation is relatively novel, since widespread collaborative projects of its kind were rare until the arrival of the Internet, and are still rare on such a large scale. Over time, Wikipedia has developed many editorial tools that have been found to be useful, based largely upon trial and error.
While Wikipedia has the potential for extremely rapid growth and harnesses an entire community – much in the same way as other communal projects such as Linux – it goes further in trusting the same community to self-regulate and become more proficient at quality control. Wikipedia has harnessed the work of millions of people to produce the world's largest knowledge-based site along with software to support it, resulting in more than ten million articles written, across more than 200 different language versions, in less than eight years. For this reason, there has been considerable interest in the project both academically and from diverse fields such as information technology, business, project management, knowledge acquisition, software programming, other collaborative projects and sociology, to explore whether the Wikipedia model can produce good results, what collaboration in this way can reveal about people, and whether the scale of involvement can overcome the obstacles of individual limitations and poor editorship which would otherwise arise.
Another reason for inquiry is the growing and widespread reliance on Wikipedia by both websites and individuals using it as a source of information, and concerns over such a major source being susceptible to rapid change – including the introduction of misinformation at whim. The proponents of such concerns tend to seek reassurance of the quality and reliability of articles, and the degree of usefulness, misinformation or vandalism which should be expected, in order to decide what reliance to place upon them.
The study also found that the quality of Wikipedia articles varied widely. Some articles were excellent by any reasonable measure—authored and edited by persons knowledgeable in the field, containing numerous useful and relevant references, and written in a proper encyclopedic style. Many articles were amateurish, unauthoritative, and even incorrect, making it difficult for a reader unfamiliar with a given subject matter to know which information to rely upon. In addition, Wikipedia contains many stubs— very short articles that provide a brief mention of a subject, and little else. The Dartmouth study was criticized by John Timmer of the Ars Technica website for an inexact measure of quality of Wikipedia articles.
A common source of reliability criticisms is the open process involved, which means that any article can be modified for better or worse at any time, and the fact that no privileged versions of articles currently exist in the main encyclopedia. This fluidity has been assessed by specialists both positively and negatively, as has Wikipedia's model that focuses upon rapid correction rather than initial accuracy.
The most common praises were:
Nature reported in 2005 that science articles in Wikipedia were comparable in accuracy to those in Encyclopædia Britannica. Out of 42 articles, only 4 serious errors were found in Wikipedia, and 4 in Encyclopædia Britannica, although more than a hundred lesser errors and omissions were found in each and Wikipedia's articles were often "poorly structured." On March 24, 2006, Britannica provided a rebuttal of this article, labeling it "fatally flawed", to which Nature responded.
A web-based survey conducted from December 2005 to May 2006 assessed the "accuracy and completeness of Wikipedia articles. Fifty people (a fairly low response rate) accepted an invitation to assess an article. Of the fifty, thirty-eight (76%) agreed or strongly agreed that the Wikipedia article was accurate, and twenty-three (46%) agreed or strongly agreed that it was complete. Eighteen people compared the article they reviewed to the article on the same topic in the Encyclopædia Britannica. Opinions on accuracy were almost equal between the two encyclopedias (6 favoring Britannica, 7 favoring Wikipedia, 5 stating they were equal), and eleven (61%) found Wikipedia somewhat or substantially more complete, compared to seven (39%) for Britannica. The survey did not attempt random selection of the participants, and it is not clear how the participants were invited.
The German computing magazine c't performed a comparison of Brockhaus Multimedial, Microsoft Encarta, and the German Wikipedia in October 2004: Experts evaluated 66 articles in various fields. In overall score, Wikipedia was rated 3.6 out of 5 points (B-). A second test by c't in February 2007 used 150 search terms, of which 56 were closely evaluated, to compare four digital encyclopedias: Bertelsmann Enzyklopädie 2007, Brockhaus Multimedial premium 2007, Encarta 2007 Enzyklopädie and Wikipedia. It concluded: "We did not find more errors in the texts of the free encyclopedia than in those of its commercial competitors.
Viewing Wikipedia as fitting the economists' definition of a perfectly competitive marketplace of ideas, George Bragues (University of Guelph-Humber), examined Wikipedia's articles on seven top Western philosophers: Aristotle, Plato, Immanuel Kant, Rene Descartes, Georg Wilhelm Friedrich Hegel, Thomas Aquinas, and John Locke. Wikipedia's articles were compared to a consensus list of themes culled from 4 reference works in philosophy. Bragues found that, on average, Wikipedia's articles only covered 52% of consensus themes. No errors were found, though there were significant omissions.
PC Pro magazine (August 2007) asked experts to compare 4 articles (a small sample) in their scientific fields between Wikipedia, Britannica and Encarta. In each case Wikipedia was described as "largely sound", "well handled", "performs well", "good for the bare facts" and "broadly accurate." One article had "a marked deterioration towards the end" while another had "clearer and more elegant" writing, a third was assessed as less well written but better detailed than its competitors, and a fourth was "of more benefit to the serious student than its Encarta or Britannica equivalents." No serious errors were noted in Wikipedia articles, whereas serious errors were noted in one Encarta and one Britannica article.
In December 2007, German magazine Stern published the results of a comparison between the German Wikipedia and the online version of the 15-volume edition of Brockhaus Enzyklopädie. The test was commissioned to a research institute (Cologne-based WIND GmbH), whose analysts assessed 50 articles from each encyclopedia (covering politics, business, sports, science, culture, entertainment, geography, medicine, history and religion) on four criteria (accuracy, completeness, timeliness and clarity), and judged Wikipedia articles to be more accurate on the average (1.6 on a scale from 1 to 6, versus 2.3 for Brockhaus with lower = better). Wikipedia's coverage was also found to be more complete and up to date, however Brockhaus was judged to be more clearly written, while several Wikipedia articles were criticized as being too complicated for non-experts, and many as too lengthy.
In its April 2008 issue British computing magazine PC Plus compared the English Wikipedia with the DVD editions of World Book Encyclopedia and Encyclopædia Britannica, assessing for each the coverage of a series of random subjects. It concluded The quality of content is good in all three cases and advised Wikipedia users Be aware that erroneous edits do occur, and check anything that seems outlandish with a second source. But the vast majority of Wikipedia is filled with valuable and accurate information.
A 2006 review of Wikipedia by Library Journal, using a panel of librarians, "the toughest critics of reference materials, whatever their format", asked "long standing reviewers" to evaluate three areas of Wikipedia (popular culture, current affairs, and science), and concluded: "While there are still reasons to proceed with caution when using a resource that takes pride in limited professional management, many encouraging signs suggest that (at least for now) Wikipedia may be granted the librarian’s seal of approval". A reviewer who "decided to explore controversial historical and current events, hoping to find glaring abuses" concluded "I was pleased by Wikipedia’s objective presentation of controversial subjects" but that "as with much information floating around in cyberspace, a healthy degree of skepticism and skill at winnowing fact from opinion are required." Other reviewers noted that there is "much variation" but "good content abounds."
The library at Trent University, Ontario, Canada states of Wikipedia that many articles are "long and comprehensive", but that there is "a lot of room for misinformation and bias [and] a lot of variability in both the quality and depth of articles." It adds that Wikipedia has advantages and limitations, that it has "excellent coverage of technical topics" and articles are "often added quickly and, as a result, coverage of current events is quite good", comparing this to traditional sources which are unable to achieve this task. It concludes that depending upon the need, one should think critically and assess the appropriateness of one's sources, "whether you are looking for fact or opinion, how in-depth you want to be as you explore a topic, the importance of reliability and accuracy, and the importance of timely or recent information", and adds that Wikipedia can be used in any event as a "starting point.
An article for the Canadian Library Association (CLA) discusses the Wikipedia approach, process and outcome in depth, commenting for example that in controversial topics, "what is most remarkable is that the two sides actually engaged each other and negotiated a version of the article that both can more or less live with." The author comments that:
Information Today (March 2006) cites librarian Nancy O’Neill (principal librarian for Reference Services at the Santa Monica Public Library System) as saying that "there is a good deal of skepticism about Wikipedia in the library community" but that "she also admits cheerfully that Wikipedia makes a good starting place for a search. You get terminology, names, and a feel for the subject."
PC Pro (August 2007) cites the head of the European and American Collection at the British Library, Stephen Bury, as stating "Wikipedia is potentially a good thing - it provides a speedier response to new events, and to new evidence on old items." The article concludes: "For [Bury], the problem isn't so much the reliability of Wikipedia's content so much as the way in which it's used. "It's already become the first port of call for the researcher", Bury says, before noting that this is "not necessarily problematic except when they go no further." According to Bury, the trick to using Wikipedia is to understand that "just because it's in an encyclopedia (free, web or printed) doesn't mean it's true. Ask for evidence .. and contribute."
An empirical study conducted in 2006 by a Nottingham University Business School lecturer in Information Systems, the subject of a review on the technical website Ars Technica, involving 55 academics asked to review specific Wikipedia articles that either were in their expert field (group 1) or chosen at random (group 2), concluded that "The experts found Wikipedia’s articles to be more credible than the non–experts. This suggests that the accuracy of Wikipedia is high. However, the results should not be seen as support for Wikipedia as a totally reliable resource as, according to the experts, 13 percent of the articles contain mistakes [10% of experts reporting factual errors of unspecified degree, 3% reporting spelling errors].
The Gould Library at Carleton College in Minnesota has a web-page describing the use of Wikipedia in academia. It asserts that "Wikipedia is without question a valuable and informative resource", but that "there is an inherent lack of reliability and stability" to its articles, again drawing attention to similar advantages and limitations as other sources. As with other reviews it comments that one should assess one's sources and what is desired from them, and that "Wikipedia may be an appropriate resource for some assignments, but not for others." It cited Jimmy Wales' view that Wikipedia may not be an ideal as a source for all academic uses, and (as with other sources) suggests that at the least, one strength of Wikipedia is that it provides a good starting point for current information on a very wide range of topics.
The Chronicle of Higher Education published an article written by Cathy Davidson, Professor of Interdisciplinary Studies and English at Duke University, in which she asserts that Wikipedia should be used to teach students about the concepts of reliability and credibility.
The academic world's view of Wikipedia has improved during the last few years, as can be inferred from the increase in the number of citations in international scientific journals. As of September 12, 2007 a search in the ScienceDirect database (a large online collection of published scientific research produced by Elsevier) for academic and scientific articles citing Wikipedia yields the following result:
|Year article published||No. of articles citing Wikipedia|
|2008 (as of August 11)||490|
BBC technology specialist Bill Thompson wrote that "Most Wikipedia entries are written and submitted in good faith, and we should not let the contentious areas such as politics, religion or biography shape our view of the project as a whole", that it forms a good starting point for serious research but that:
A study conducted in early 2007 by the Pew Research Center found that 8% of all online Americans consult Wikipedia on a typical day. The study also found that the 36% of all US internet users use Wikipedia, with this fraction increasing with education level. About 22% of those with a high school-level education use Wikipedia, 36% of those with some college use Wikipedia, and 50% of those with a college degree use Wikipedia.
The Supreme Court of India in its recent judgement in Commr. of Customs, Bangalore vs. ACER India Pvt. (Citation 2007(12)SCALE581) has held that "We have referred to Wikipedia, as the learned Counsel for the parties relied thereupon. It is an online encyclopaedia and information can be entered therein by any person and as such it may not be authentic.
In his 2007 Guide to Military History on the Internet, Simon Fowler rated Wikipedia as "the best general resource" for military history research, and stated that "the results are largely accurate and generally free of bias. When rating WP as the No. 1 military site he mentioned that "Wikipedia is often criticised for its inaccuracy and bias, but in my experience the military history articles are spot on.
In July 2008, The Economist magazine described Wikipedia as "a user-generated reference service" and noted that Wikipedia's "elaborate moderation rules put a limit to acrimony" generated by cyber-nationalism.
A further informal assessment by the popular IT magazine "PC Pro" for its 2007 article Wikipedia Uncovered tested Wikipedia by a similar device to those described above, by introducing 10 errors that "varied between bleeding obvious and deftly subtle" into articles (the researchers later corrected the articles they had edited). Labeling the results "impressive" it noted that all but one was noted and fixed within the hour, and that "the Wikipedians' tools and know-how were just too much for our team." A second series of another 10 tests, using "far more subtle errors" and additional techniques to conceal their nature, met similar results: "despite our stealth attempts the vast majority... were discovered remarkably quickly... the ridiculously minor Jesse James error was corrected within a minute and a very slight change to Queen Ann's entry was put right within two minutes." Two of the latter series were not detected. The article concluded that "Wikipedia corrects the vast majority of errors within minutes, but if they're not spotted within the first day the chances... dwindle as you're then relying on someone to spot the errors while reading the article rather than reviewing the edits."
A study in late-2007 systematically inserted fibs into Wikipedia entries about the lives of philosophers. Depending on how exactly the data is interpreted, either one third or one half of the fibs were corrected within 48 hours.
The WikiScanner story was also covered by The Independent, which stated that many "censorial interventions" by editors with vested interests on a variety of articles in Wikipedia had been discovered:
"[Wikipedia] was hailed as a breakthrough in the democratisation of knowledge. But the online encyclopedia has since been hijacked by forces who decided that certain things were best left unknown... Now a website designed to monitor editorial changes made on Wikipedia has found thousands of self-serving edits and traced them to their original source. It has turned out to be hugely embarrassing for armies of political spin doctors and corporate revisionists who believed their censorial interventions had gone unnoticed.
Wikipedia has also developed into a key source for some current new events such as the 2004 Indian Ocean earthquake and related tsunami, and the 2007 Virginia Tech massacre. In the latter case, it cites the New York Times, noting with 750,000 page views of the article in the two days after the event:
Wikipedia has been praised for making it possible for articles to be updated or created in response to current events. For example, the then-new article on the 2004 Indian Ocean earthquake on its English edition was cited often by the press shortly after the incident. Its editors have also argued that, as a website, Wikipedia is able to include articles on a greater number of subjects than print encyclopedias may.
Others taking this view include Danah Boyd, who in 2005 discussed Wikipedia as an academic source, concluding that "[i]t will never be an encyclopedia, but it will contain extensive knowledge that is quite valuable for different purposes", and Bill Thompson who stated "I use the Wikipedia a lot. It is a good starting point for serious research, but I would never accept something that I read there without checking."
Information Today's March 2006 article concludes on a similar theme:
Dan Gillmor, a Silicon Valley commentator and author commented in October 2004 that, "I don't think anyone is saying Wikipedia is an absolute replacement for a traditional encyclopedia. But in the topics I know something about, I've found Wikipedia to be as accurate as any other source I've found."
Seemingly, and "in theory", an unsupervised mass collective effort should not work well. But in practice it does, in part because of the social and psychological structures that motivate participants on both content and maintenance tasks. Sheizaf Rafaeli and Yaron Ariel report how "most people agree that at least the English version of Wikipedia is approaching critical mass where substantial content disasters should become rare."
Likewise, technology figure Joi Ito wrote on Wikipedia's authority, "[a]lthough it depends a bit on the field, the question is whether something is more likely to be true coming from a source whose resume sounds authoritative, or a source that has been viewed by hundreds of thousands of people (with the ability to comment) and has survived.
The Seigenthaler incident demonstrated that the subject of a biographical article must sometimes fix blatant lies about his own life. In November 2005, a user edited the biographical article on John Seigenthaler Sr. so that it contained several false and defamatory statements. The inaccurate claims went unnoticed between May and September 2005 when they were discovered by Victor S. Johnson, Jr., a friend of Seigenthaler. Wikipedia content is often mirrored at sites such as Answers.com, which means that incorrect information can be replicated alongside correct information through a number of web sources. Such information can develop a misleading air of authority because of its presence at such sites:
Then [Siegenthaler's] son discovered that his father's hoax biography also appeared on two other sites, Reference.com and Answers.com, which took direct feeds from Wikipedia. It was out there for four months before Seigenthaler realised and got the Wikipedia entry replaced with a more reliable account. The lies remained for another three weeks on the mirror sites downstream.
Wikipedia has a short biography of me, originally added in February 2004, mostly concerned with my internet civil liberties achievements. After discovering in May 2006 that it had been vandalised in March, possibly by a long-time opponent, and that the attack had been subsequently propagated to many other sites which (legally) repackage Wikipedia's content, the article's existence seemed to me overall to be harmful rather than helpful.
For people who are not very prominent, Wikipedia biographies can be an "attractive nuisance". It says, to every troll, vandal, and score-settler: "Here's an article about a person where you can, with no accountability whatsoever, write any libel, defamation, or smear. It won't be a marginal comment with the social status of an inconsequential rant, but rather will be made prominent about the person, and reputation-laundered with the institutional status of an encyclopedia."
In the same article Finkelstein recounts how he voted his own biography as "not notable enough" in order to have it removed from Wikipedia. He goes on to recount a similar story involving Angela Beesley, previously a prominent member of the foundation which runs Wikipedia.
In another example, on March 2, 2007, MSNBC.com reported that Hillary Rodham Clinton had been incorrectly listed for 20 months in her Wikipedia biography as valedictorian of her class of 1969 at Wellesley College. (Hillary Rodham was not the valedictorian, though she did speak at commencement.) The article included a link to the Wikipedia edit, where the incorrect information was added on July 9, 2005. After the MSNBC report, the inaccurate information was removed the same day. Between the two edits, the wrong information had stayed in the Clinton article while it was edited more than 4,800 times over 20 months.
Attempts to perpetrate hoaxes may not be confined to editing Wikipedia articles. In October 2005 Alan Mcilwraith, a former call centre worker from Scotland created a Wikipedia article in which he claimed to be a highly decorated war hero. The article was quickly identified by other users as unreliable (see Wikipedia Signpost/2006-04-17/Persistent hoax). However, Mcilwraith had also succeeded in convincing a number of charities and media organizations that he was who he claimed to be:
The 28-year-old, who calls himself Captain Sir Alan McIlwraith, KBE, DSO, MC, has mixed with celebrities for at least one fundraising event.
But last night, an Army spokesman said: "I can confirm he is a fraud. He has never been an officer, soldier or Army cadet.
There have also been instances of users deliberately inserting false information into Wikipedia in order to test the system and demonstrate its alleged unreliability. Television personality Stephen Colbert lampooned this drawback of Wikipedia, calling it wikiality.
Wikipedia's radical openness means that any given article may be, at any given moment, in a bad state: for example, it could be in the middle of a large edit or it could have been recently vandalized. While blatant vandalism is usually easily spotted and rapidly corrected, Wikipedia is certainly more subject to subtle vandalism than a typical reference work.
In an informal media test of Wikipedia's ability to detect misinformation, an anonymous blogger tested Wikipedia by inserting subtly erroneous facts into obscure articles, stating that its process "isn't really a fact-checking mechanism so much as a voting mechanism", and that material which did not appear "blatantly false" may be accepted as true. Wikipedians by and large responded with anger at what was considered by many to be an unfair trial which had deliberately focused on obscure, less-reviewed articles; the blogger responded that the test was fair.
"A small but loyal group of fans are lovingly called "The Zany Ones" - they like to wear hats made from discarded shoes and have a song about a little potato."
The Observer reported in December 2007 that Carl Hewitt, associate professor emeritus in electrical engineering and computer science at MIT, was banned from editing Wikipedia. According the article Hewitt is alleged "to have disrupted Wikipedia for more than two years by using it for self-promotion, tampering with his own biography and manipulating computer science articles to inflate the importance of his own research. Senior academics in his field say the changes he made have rendered some entries in effect useless." Wikipedia administrators claimed that Hewitt "was found to be citing his own work in articles where it was not relevant, obscuring points of view at odds with his own theories, and editing his biography to promote his forthcoming public appearances". Hewitt rejected these findings and claimed that he was being harassed and censored. The article concluded that:
The banning of Hewitt shows that the academic community is in fact actively involved in editing Wikipedia, but may be no more reliable and trustworthy than any other group of users.
In April 2008, the Boston-based Committee for Accuracy in Middle East Reporting in America (CAMERA) organized a campaign to correct Israel-related biases and inconsistencies in Wikipedia.
On August 31 2008, The New York Times ran an article detailing the edits made to the biography of Sarah Palin in the wake of her nomination as running mate of John McCain. The editor responsible for adding many flattering details was identified as single-purpose account of a McCain campaign volunteer.
In story covered by the BBC, former Novell chief scientist Jeffrey Merkey claimed that in exchange for a donation his Wikipedia entry was edited in his favor. Jay Walsh, a spokesman for Wikipedia, flatly denied the allegations in an interview given to the Daily Telegraph.
In a story covered by InformationWeek, Eric Goldman, assistant law professor at Santa Clara University in California argued that "eventually, marketers will build scripts to edit Wikipedia pages to insert links and conduct automated attacks on Wikipedia", thus putting the encyclopedia beyond the ability of its editors to provide countermeasures against the attackers, particularly because of a vicious circle where the strain of responding to these attacks drives core contributors away, increasing the strain on those who remain.
Jossi Fresco may bear the most extreme conflict of interest in the history of Wikipedia - and he edits the policy that governs conflict of interest.
Some of the most scathing criticism of Wikipedia's claimed neutrality came in The Register, which in turn was allegedly criticized by founding members of the project. According to The Register:
In short, Wikipedia is a cult. Or at least, the inner circle is a cult. We aren't the first to make this observation.
On the inside, they reinforce each other's beliefs. And if anyone on the outside questions those beliefs, they circle the wagons. They deny the facts. They attack the attacker. After our Jossi Fresco story, Fresco didn't refute our reporting. He simply accused us of "yellow journalism". After our Overstock.com article, Wales called us "trash".