text:
Digital News Publications
Overview & Key Findings
Further Analysis & Comparison
Resources and Charts
Skip to navigate this publication
Editorial Analytics: How News Media Are Developing and Using Audience Data and Metrics
Posted by Federica Cherubini Media Consultant and Editorial Researcher Rasmus Kleis Nielsen Director, Reuters Institute for the Study of Journalism on February 23, 2016
News organisations all over the world have in recent years increased their use of analytics – systematic analysis of quantitative data on various aspects of audience behaviour aimed at growing audiences, increasing engagement, and improving newsroom workflows.
In this Reuters Institute Report, we review how a range of different newsrooms across Europe and North America use analytics. On the basis of more than 30 interviews, we find the following:
Leading digital news organisations are developing distinct forms of editorial analytics tailored to help them pursue their particular goals. These forms of editorial analytics differ from more rudimentary and generic approaches (1) in being aligned with the editorial priorities and organisational imperatives (whether commercial, non-profit, or public service) of specific news organisations, (2) in informing both short-term day-to-day decisions and longer-term strategic development, and (3) in continually evolving to keep pace with a changing media environment.
Globally oriented US- and UK-based news organisations remain ahead of most others in their
development and use of analytics, but market leaders in most countries are developing editorial
analytics suited to their specific priorities and situations. Market leaders across continental Europe frequently have more in common with leading US and UK organisations than with their domestic competitors. Many news organisations across all the countries covered here continue to lag behind best practice.
Because best-practice editorial analytics are tailored to the priorities and goals of a given organisation as well as the context in which it competes, there is no one right way to do analytics or one set of tools that will give an organisation everything it needs. Instead, news organisations need to think about how they can develop their analytics capability by making sure they combine (1) the right set of tools, (2) an organisational structure that incorporates the expertise to use them, and (3) a newsroom culture that embraces data-informed decision-making. Falling short in any one of these areas undermines an organisation's analytics capability.
The most sophisticated audience teams are keenly aware that analytics are not perfect. the data never tell the full story, and quantitative analysis always has to be supplemented by editorial expertise and other forms of qualitative judgement. Even the best editorial analytics continue to be constrained by the difficulties involved in defining and measuring many of the things that news organisations aim to achieve and is beset by a whole range of data-quality and data-access issues, exacerbated by rapid changes in the media environment.
Journalists today not only need analytics to navigate an ever-more competitive battle for attention. Many journalists also want analytics, as an earlier period of scepticism seems to have given way to interest in how data and metrics can help newsrooms reach their target audiences and do better journalism. that is encouraging, because analytics and data metrics will continue to evolve, and if journalists are not part of that process, the tools and techniques developed will continue to reflect and empower commercial and technological priorities more than editorial priorities.
1. The Development of Editorial Analytics
News organisations today are competing for attention in an ever-more competitive and constantly changing media environment. No one can take their audience for granted. the battle for attention is a central challenge for journalism because its public role is premised on connecting with an audience – as is the business model of private news media and the legitimacy of public service media.
In this Reuters Institute Report, we analyse how news organisations across Europe and the United States are developing their use of analytics – systematic analysis of quantitative data on audience behaviour – as part of that battle for attention. We show how analytics, in the past perhaps primarily associated with brands like BuzzFeed, gawker, and the Huffington Post, are increasingly central to how news organisations like the Guardian, the Financial Times, and the BBc do journalism today, and are being taken in new directions by both large news organisations like the Axel Springer-owned German newspaper Die Welt and smaller digital start-ups like Quartz and Ze.tt.
At the centre of this development are people in the newsroom with new job titles like 'audience editor', 'growth editor', 'audience development editor', or 'audience engagement editor'. they are developing and using analytics for editorial purposes that were in the past more narrowly tied to predominantly commercial objectives, using tools and techniques previously rarely used by journalists. We look at how they work with colleagues as part of audience development teams and how they work with rank-and-file news journalists across their newsroom. We see how old metrics like pageviews and unique browsers are increasingly accompanied by new measures of social interactions, engaged time, and loyalty, and how new tools like chartbeat, Parse.ly, and NewsWhip, which aim to help specifically editorial decision-making, supplement more generic tools like Omniture, google Analytics, Facebook Insights, and the like. We examine how short-term optimisation of websites on the basis of article placement and testing of headlines and photos is increasingly combined with a broader effort to extend the shelf-life and distributed reach of quality content and with longer term analysis aimed at developing loyal and engaged audiences and doing smarter, more timely, and more effective journalism.
We show how leading organisations are developing distinct forms of what we call editorial analytics. Editorial analytics aim to help journalists and news organisations become more datainformed, not to replace editorial judgement with the tyranny of numbers. They are distinct from more rudimentary and generic approaches in being
aligned with the editorial priorities and organisational imperatives (whether commercial, non-profit, or public service) of specific news organisations;
used to inform both short-term day-to-day decisions and longer-term strategic development (including product development and work planning); and
continually evolving to keep pace with a changing media environment.
It is important to underline that the fact that the most developed approaches – whether in larger organisations like the Guardian or smaller ones like Quartz – are tailored means that there is no one best way to do analytics, just as there is no one 'god metric' for journalism. Good analytics use a wide variety of different approaches and sources of data to help with day-to-day short-term optimisation and longer term planning. All effective use of analytics depends on defining and measuring performance against the specific goals being pursued.
Editorial analytics are underpinned by technological tools, organisational structures, and cultural components that together define a news organisation's analytics capability. Those interested in developing better analytics will have to consider how each of these components can supplement each other in light of the goals of a specific organisation. All three components are necessary parts of this, and none of them can substitute directly for one another. Technology and tools without an organisational structure that ensures they are used well will not deliver their full return.
A well staffed analytics team with a clear place in the organisational structure but poor tools and little cultural cachet with the wider newsroom will not realise its potential. A pro-data culture in the newsroom will not result in more data-informed decision-making if the right tools are not available and the organisation does not have the talent to analyse the data. To use analytics effectively, an organisation needs to know what it wants to achieve, and develop a combination of tools, organisational structure, and newsroom culture that enables a more data-informed pursuit of these goals.
Editorial analytics represent a significant improvement in news organisations' capacity to understand the media environment in which they operate and an important shift from a time in which newsrooms had far less analytic capability than other parts of their organisation (the commercial side, the strategy unit, the media research department, the IT department, and the product development team). But, as the most sophisticated audience development editors and data analysts are the first to underline, editorial analytics are not perfect. The data never tell the full story, and quantitative analysis always has to be supplemented by editorial expertise and other forms of qualitative judgement.
Analytics are constrained by the difficulties involved in defining and measuring many of the things that news organisations aim to achieve (whether editorial impact, conversion of users to subscribers, or public service goals like making citizens more informed). It is beset by a whole range of data-quality and data-access issues. this means that the effective use of analytics for journalistic purposes involves not only (1) making the step from rudimentary or generic analytics to tailored editorial analytics and (2) the development of analytic capability through technological tools, organisational structure, and newsroom culture. It also involves (3) a keen awareness of the strengths and weaknesses of even the best available analytics. Current approaches are better at dealing with an older internet of desktop web use, homepage traffic, and referrals from search and social than with more recent trends like mobile web use, app/browser proliferation, and distributed content consumed across multiple platforms and devices.
When we discuss specific examples, our main emphasis will be on news organisations widely regarded as examples of best practice. But it is important to underline that many news organisations, including some of the most respected in the world, have been slow to embrace analytics, at least in the newsroom. In 2014, the widely discussed New York Times Innovation Report noted that
We are falling behind [in] the art and science of getting our journalism to readers. We have always cared about the reach and impact of our work, but we haven't done enough to crack that code in the digital era. 1 A broad range of industry leaders agree – 76% of respondents in a recent Reuters Institute survey of news editors, CEOs, and digital leaders from across 25 countries said improving the way in which newsrooms use data to better understand and target audiences is going to be very important for their organisation in 2016 (Newman 2016).
We hope this report will be useful in this process. It aims to provide both an overview of the state-of-the-art in newsroom use of analytics for those relatively new to the area as well as an analysis of some of the central challenges ahead for those who already have an advanced understanding of analytics. It is based on over 30 interviews specifically about analytics and audience development, conducted between October 2015 and January 2016, primarily with audience development editors and newsroom analysts. In addition, we draw on background interviews with editors, managers, and strategists that discussed data and metrics as parts of wider conversations. The interviews cover a range of different kinds of news media, including private legacy media (both national and local/regional newspapers), public service media, digital news media, and third-party metrics vendors. We have covered a variety of countries, the main ones being France, Germany, Italy, Poland, the UK, and the US. We have deliberately looked beyond English-language market leaders to better understand how news media more generally are developing and using audience data and metrics. The aim here is not to provide an in-depth set of two or three case studies or an exhaustive inventory of every approach out there, but to provide an overview of some examples of best practice and some of the most important differences we have observed. A full list of interviewees is provided at the end of the report.
The report is structured in three broad sections. chapter 2 examines a range of cases across different countries. Our emphasis is on discussing several different examples of best practice in some detail while also providing an overview over the wider landscape and how analytics are being developed and used in different contexts. Here we develop the distinction between rudimentary, generic, and editorial analytics. Chapter 3 deals with the interaction between technological tools, organisational structure, and newsroom culture and the notion of analytics capability. Chapter 4 covers the main challenges faced by analytics today, including problems of definition, measurement, and data.
2. How are News Media Developing and Using Audience Data and Metrics?
The idea of integrating analytics into daily editorial work and longer term strategic planning has been central to US-based digital news start-ups like Gawker, the Huffington Post, and BuzzFeed for years. these companies have from the start been proud of their ability to use a more data-informed and evidence-based approach to digital publishing than many older media, and have drawn extensively on analytics developed in the technology sector, marketing, e-commerce, and advertising (Küng 2015; Petre 2015). Faced with these aggressive new players as well the increasing importance of digital intermediaries like search engines and social media as drivers of traffic, US legacy news media like the New York Times, the Wall Street Journal, and NPR (National Public Radio) have themselves built teams to help their newsrooms leverage analytics more effectively in the battle for audiences' attention. New start-ups like Quartz, vox.com, and Mic.com are similarly committed to analytics in the newsroom. The same holds true for leading UK-based brands competing for a global English language audience. the Guardian, Financial Times, BBC, and others are increasingly developing their own tailored approaches to find the right tools, organisational structures, and cultures of data to underpin their specific editorial and organisational priorities. Similarly, market leaders across much of continental Europe – so far faced with much less direct competition from start-ups focused on news because of language barriers – are developing and using analytics to improve their audience reach and engagement.
Across all these markets – including the US and the UK – many news organisations still have a very rudimentary approach to analytics. this is especially true for smaller legacy news organisations like local and regional newspapers and for some public service broadcasters. But it also applies to some bigger private media.
In what follows, we will focus on a detailed examination of what a selection of organisations often highlighted by other interviewees as examples of best practice are doing (the Guardian, Financial Times, BBC, and Huffington Post). We then discuss what sets best-practice examples of editorial analytics apart from more generic and rudimentary analytics and provide a wider overview over some overall trends across the countries in continental Europe where we conducted interviews.
2.1. The Guardian's Ophan, a Path-Breaking Bespoke Software
The Guardian's approach to analytics is anchored around its in-house real-time analytics tool Ophan. Ophan was born in 2012 as the result of an internal hack day and the expertise of Audience Editor Chris Moran (who has an editorial background) and Director of Architecture Graham Tackley (who comes from a technology background). Ophan offers minute-by-minute data on individual articles with a high level of granularity. It is browser based, easily accessible on mobile, and can be accessed with a Guardian email address and a password. In December 2015, the Guardian reported that more than a thousand employees had used the tool in the previous month. 2
Since its first incarnation, the tool has continued to grow and be developed around the audience team's needs and on the basis of input from other parts of the newsroom and developers working at the Guardian. Indeed, one of the main advantages of Ophan highlighted by Moran is that it serves as a learning tool, something that is user-friendly for journalists and that communicates data in a very clear way. Ophan shows not only traditional metrics like pageviews, social shares, and attention-time for each article published in the last two weeks (drawn from various back-end sources). It also shows whether they have been pushed via the Guardian's social media channels (including the exact post or tweet with which they have been promoted) or if they have been promoted on the homepage.
Figure 2.1. A screenshot from Ophan, the Guardian's editorial dashboard. Data can be filtered by time, section, device, country, and referrer. Notice how the dashboard gives data on discovery. In this case, 35% of users found this article via the Guardian, 20% via Facebook, only 0.5% came via Google, and a high percentage (39%) came from unknown referrals (this can include e.g. links send by email).
The data can be broken down by different segments, including time, section, device, browser, country, referrer, loyalty, and attention time. It shows the bounce rate, 3 If the user is logged in, and parts of the
user's journey, including where he/she has come from and where they have gone to next. 4 This makes the conversation between them and the rest of the newsroom easier. Part of the audience team's job is to support subeditors with writing and tweaking headlines – they are constantly in touch via messaging chat – and making sure that all articles get the right promotion on the distribution channels. As Mary Hamilton, the Guardian's executive editor for audience, has explained:
Everybody can see the results that they're having … so if they make a change to a headline, or if they add a link or if they add something on the front, they can now actually see the results that that's having in real time. They can test out a gut instinct and see what happens, rather than just flying purely on that instinct. 5 The stress on 'right' audiences underlines the link between the Financial Times' target (elite) audience and its subscription-based business model which means its goals are somewhat different from the Guardian's aim of building a large global audience around free advertising-funded content.
The Financial Times' audience engagement strategy has as its basic aim growing the reach and impact of FT journalism, driving quality traffic to the website, and growing engagement on and offsite. This traffic, in turn, can potentially be converted into FT subscribers. the aim is to achieve this by transforming the newsroom and the reporters into audience-first journalists, changing the way journalism is produced and distributed to integrate engagement objectives into the commissioning and production process. A clear early example of this is the decision – before Kaplan's arrival – to change the workflow at the FT from being dictated by older print habits, where stories were filed late in the evening, to being designed around audience behaviour, where stories are filed at times of peak traffic to the FT website and app, in the morning, around lunch, and early in the evening (discussed in Newman et al. 2015).
To do this Kaplan is working to ensure collaboration across different parts of the FT, and is bringing new skills into the newsroom. Her team is about ten people and still growing. The roles are quite diverse, including both journalist and non-journalist roles. They include a social media team, with editors in London and New york; engagement editors; a data analyst; a marketing manager, who co-ordinates with the other commercial parts of the FT organisation to grow the impact of the promotional efforts; an SEO expert; and an engagement strategist, who helps structure all the projects with relevant metrics; a digital editor focused on producing bespoke content specifically for social. The team helps co-ordinate strategies across the organisation, aligning the shared objectives of the newsroom with the commercial parts of the business. The shared goal is to grow the impact of FT journalism and the title's ability to engage its target audience, and the team underpins the newsroom's general recent move to invest a bit less in producing content, and a great deal more in expanding its editorially led efforts to ensure that FT content finds an audience across multiple channels and platforms including the print paper, the website, the app, newsletters, and social media.
To facilitate the understanding and the awareness in the newsroom of metrics and performances, the organisation is developing a new bespoke dashboard, called Lantern, a name picked for a tool that is meant to be 'illuminating' and user-friendly for journalists. It translates performance into measures that are meaningful to journalists and editors, moving away from the dominance of pageviews, and towards newer integrated metrics of engagement that take into account things like time spent, recirculation, volume of articles read per visit, and number of comments. The Financial Times has been highlighted as a leading example of analytics in the newsroom for years already, and yet Kaplan and her colleagues are very explicit about the challenges ahead. These include defining engagement metrics for different content types, finding ways of measuring softer elements such as the quality of experience and brand perception, as well as understanding behaviour on third-party platforms. Speaking at an event in Paris in December 2015, Kaplan underlined the challenges associated with understanding the different audiences the same brand may draw across different platforms, for example, across a website, a mobile app, and social platforms like Facebook and twitter. Each channel may require both a different editorial approach and a different approach to analytics.
As with the Guardian, the Financial Times is working to combine (1) a user-friendly dashboard for journalists with more complex back-end analytics, (2) an audience engagement team based in the newsroom itself, and (3) a culture of data where everyone has access to data and can use it to make editorial decisions. It is also clear that the particular objectives defined at the FT, and the approach to analytics that Kaplan and her colleagues are developing, are tailored to the title's particular editorial priorities as well as a business model based on subscriptions rather than advertising.
2.3. Scaling Up: Audience Engagement at BBC News
The BBC is another example of a high-profile UK-based news organisation that has recently made serious efforts to build an audience development team and give data and analytics a more central role in the online newsroom. The multidisciplinary team, established within the last year and headed by Elinor Shields, includes people from different backgrounds. Serving a large newsroom and the entire BBc World Service across 28 different language services, the team – of five people – is trying to support a complex and vast news operation.
As Shields and Technical Lead Jeremy Tarling explained, the BBC aims to combine very practical and tactical approaches to day-to-day audience optimisation with a more strategic goal of putting data-led decision-making into the heart of an editorial culture of a very large media organisation deeply rooted in broadcast journalism. Through training and coaching, the team aims to drive culture change and empower BBC journalists to be able to look at the insights that come from dashboards and data, and take actions based on that. On a daily basis this is done by assisting the editorial team, whether by sharing insights during the morning editorial meeting, or via their inhouse tools for audience engagement, or through a playbook of actions and benchmarking that helps journalists understand the meaning of data and evaluate performance.
Alongside this, the team focuses on growth experiments and deep dives dedicated to examining specific challenges over an extended period of time. Off the back of that they make editorial and workflow recommendations to senior editors and managers. Last, but not least, the team is developing tools and dashboards to support the newsroom in this process of change. (At this stage, the BBC does not have a bespoke dashboard for journalists like Ophan or Lantern, but is working on one. Members of the analytics team use chartbeat and a range of other third-party and bespoke tools and share data with colleagues around specific issues. Chartbeat data are also displayed on a screen in the newsroom.) Elinor Shields says:
The idea is that the first wave is around culture change, so giving [journalists] the tools to understand what to do, and giving them incentives to want to do it. And then, the second stage is around helping to drive performance through those growth experiments and from deeper insights. 6
Especially for a news organisation with the scale, complexity, and public service obligations of the BBC, empowering journalists to be able to act on the data, rather than just delivering them a bunch of numbers on a dashboard, is a crucial and challenging step. Working in an organisation that aims to reach a wide variety of different audiences across many platforms, issues, and countries, Shields and her colleagues are very conscious of the importance of providing targeted insights and tools rather than generic recommendations for the whole organisation.
The audience engagement team at the BBC is multidisciplinary, like that at the FT. Shields explains that data-scientists bring analytical skills to the very wide range of activities of the BBC to help with analysis and benchmarks. Other technical specialists work on visualisations and dashboards for journalists to make these more user-friendly and actionable. Having people with an editorial background as part of the team is crucial as they can translate the data and analysis into more actionable insights on the basis of their editorial expertise and understanding of the BBC News organisation. 'In my experience, data-scientists are only as good as the people who are working alongside them, in as much as a data-scientist needs to know what questions to ask of the data', Tarling says. 7
The BBC is still working on developing more tools for easy access to data for journalists across the BBC Newsroom, but the audience engagement team is an investment in a wide range of different forms of analytics expertise organisationally anchored in the newsroom itself, and the team is working to change the overall culture of BBC News Online to be more data-informed. The BBC team is still looking at inspiration from other players, including both private legacy media as well as digital media companies, but has a more developed approach to analytics than any of the other public service broadcasters we have researched for this report.
2.4. The Huffington Post
The Huffington Post has been a pioneer in integrating audience data in editorial decision-making from the start. It is also an important example of how leading practitioners of editorial analytics are working continuously to develop and refine their approach. Analytics that worked well in 2010 are no longer necessarily fit for purpose in 2015. Jack Riley is Director of Commercial and Audience Development at the Huffington Post UK and leads a team of data experts and product analysts who sit between editorial, product, and commercial, helping to optimise the daily output. Riley's role is indicative of the integration between editorial, commercial, and technological forms of expertise common to many innovative digital news organisations today (Küng 2015). At the Huffington Post UK every journalist in the newsroom has access to the analytics through a personalised Omniture dashboard, which includes all kind of metrics, from the more traditional ones like visits, pageviews, and unique visitors, to more complicated ones, like which pages gets over, or below, a certain threshold, referrals from specific sources, video data, and how many articles each journalist has written.
Figure 2.2. Slide from the presentation by Jimmy Maymann, President, Content and Consumer Brands, AOL and former CEO, Huffington Post, at the Reuters Institute Memorial Lecture 2015 (23 Nov. 2015). As with the Guardian and the Financial times, the Huffington Post is working very consciously to tailor its publication schedule around audience behaviour and to promote content also after first publication.
What I think we're finding at the moment as most effective is a combination of real-time data – which is interesting on a daily level – and periodic deeper dives into specific subjects. 8
Riley underlines that topic-based, in-depth analysis helps in connecting with the journalists who are often interested in comparing the performance of different subjects they write about. In his view, an ideal analytical approach involves a combination of thinking around the data done by the journalists themselves, plus more complex insights delivered by the audience team. These are different combinations of editorial and analytic expertise. Data, Riley highlighted, play an important role in the newsroom, both in terms of daily coverage and long-term planning. 'If there is something that does well, you're reminded of it every time you see it coming up in real-time. So you know it's worth investing time in a longer project.'
The Huffington Post uses an array of different tools. Alongside a customised version of Chartbeat, it employs tools that focus on things like click and module tracking and A/B testing (real-time split testing where an article is posted in two or more different versions with different headlines, pictures, etc., and performance is compared across a random sample of visitors for a period of time, with the best performing version then used). It also employs a personalised recommendation service called Gravity, bought by AOL in 2014, which provides analytics on recirculation, and interestgraphs for individual users. All of these tools help the Huffington Post pursue editorial priorities and organisational imperatives (an advertising-based business model) that are different from those of, for example, the Financial Times or the BBC.
Targets and benchmarking play an important role at the Huffington Post. Monthly and daily targets are arranged at a vertical, device, and referral level; and the Omniture dashboard shows the performance of different days against the target, and how journalists are performing relative to their overall target. Targets help in providing orientation, Riley explained, as otherwise it's difficult to know if things are working or not. However, they are applied in a pretty relaxed way, as it's not that important if a journalist misses a target as long as they are able to understand why. As he says, 'Sometimes, it's also the target that could be wrong.' Riley continues: In my experience it is better to have something and then be smart in how you actually apply it, than not to have anything and then struggle to benchmark things.
2.5. Editorial, Generic, and Rudimentary Analytics
So far, we have discussed four cases in some detail that other interviewees have frequently highlighted as examples of best practice of newsroom use of analytics. These cases are all very different, tailored to various degrees to specific goals and shaped by different legacies. But they also have three things in common that distinguish what we call editorial analytics from more generic and rudimentary forms of analytics. The three things are
an alignment between analytics and the editorial priorities and organisational imperatives (whether commercial, non-profit, or public service) of specific news organisations;
analytics used to inform both short-term day-to-day decisions and longer term strategic development; and
analytics continually evolving to keep pace with a changing media environment.
If we place editorial analytics at one end of a continuum (see Figure 2.3), we have at the other end rudimentary analytics (common especially amongst smaller legacy media and continental European public service broadcasters) involving advertising-oriented metrics and off-the-shelf tools like Google Analytics and Facebook Insights but little in terms of a clear organisational structure or a culture of data, no systematic link to decision-making, and few attempts to update analytics to an evolving environment. In the middle, generic analytics (the status of the majority of the organisations we have talked to) involve a wider range of off-the-shelf tools, some of whom are specifically developed for news (like Chartbeat, Parse.ly, and/or NewsWhip), but have a weak organisational anchor and little newsroom culture around analytics; data are used more for shortterm optimisation than to underpin longer term editorial and organisational priorities.
Figure 2.3. A continuum of forms of analytics used in newsrooms.
2.6 How is Continental Europe Doing?
All the organisations discussed above are from the UK and the US. They are frequently mentioned by our other interviewees as examples of best practice and can serve to illustrate a range of different kinds of editorial analytics. that does not mean that only UK and US news organisations have developed beyond rudimentary and generic analytics.
Across the four other countries covered by the bulk of our interviews – France, Germany, Italy, and Poland – digital market leaders frequently have similarly tailored approaches, though they often still see themselves as falling behind international best practice, and frequently still look to the UK and the US for inspiration. Leading players like Die Welt in Germany and ONet in Poland have more in common with market leaders in English-speaking countries than they have with some of their domestic competitors. Start-ups like Ze.tt (from germany), De correspondent (in the Netherlands), and Denník N (from Slovakia) have all been thinking from day one about how analytics can underpin their editorial mission, help them connect with their audience, and develop their business model.
Ze.tt is a new spin-off project from the publisher of the German newspaper Die Zeit and its digital arm Zeit Online. It launched in beta in July 2015 and is aimed at a younger audience. It is just one of many examples of how unthinkable it is to launch a new news project today without having analytics at the core of the editorial operation. Project lead Sebastian Horn explained how Ze.tt uses analytics mainly for two areas: for the website and for social platforms. 'We follow them both very closely', Horn said. 'We run tests and we try to figure out what works particularly well; we do A/B testing. We're literally just starting out so we're experimenting.' 9 He continues:
Because we're such a small team everyone has access to everything, but I try to make sure that people know how to interpret them correctly. It doesn't help if you have tons of data coming in, if you don't know what you're looking at, what that means or what success is for your newsroom.
For a newly born website like Ze.tt is very important not only to grow audience scale and reach, but also to track if they are building a loyal audience. Horn explains that part of doing this involves being very selective about which data are made available to journalists and constantly evaluating what helps them and what does not.
Part of the reason why Ze.tt was created was to experiment so we're also going to experiment with different data tools. We're going to give them a try and see if they give us useful information. However, we don't want to overwhelm ourselves with too much. We don't want to cover our walls with monitors that show random graphs. They will have to be meaningful numbers that people look at and help them make decisions.
De Correspondent is an online-only journalism platform, launched in the Netherlands in 2013, that broke all the crowd-funding records by managing to raise over €1 million by getting '15,000 people to pledge €60 for a one-year subscription to a news site that didn't yet exist'.10 With now over 40,000 paid subscribers, 11 De Correspondent is entirely funded by members and it's completely ad-free. 'the most important metric we look at is the number of new members', publisher and co-founder Ernst-Jan Pfauth explained. 12
At De Correspondent they look at pageviews more from fascination, than as an actionable metric. Numbers like the amount of people signing up for the free newsletter and the conversion rate to subscriptions play a far more central role. The data are important to sustain the viability of the business model but also as a validation of editorial intuitions. 'We always use the data to validate ideas we come up with, but never the other way around', Pfauth said.
But market leaders and start-ups aside, it is clear that many continental European news organisations are still lagging behind when it comes to analytics. Off-the-shelf tools are quite widespread, most importantly web analytics tools like Omniture and Google Analytics, social media tools like Facebook Insight, and newer tools developed for editorial purposes, like Chartbeat, Parse.ly, and NewsWhip. (Other tools in use include LinkPulse, Webtrekk, At Internet, and EzyInsights.) But what kinds of data and tools are used, how the work is organised, and how clearly and consistently it is linked to primary editorial and organisational objectives vary significantly. There is no point in identifying individual organisations as falling short of best practice. Most do, also in the US and the UK (see e.g. Hindman 2015). Many newsrooms have data and analytics, without having any clear organisational structure or newsroom culture for using them, or even clearly articulated strategic objectives for their use. this is what Stijn Debrouwere has called 'cargo cult analytics', a very rudimentary approach where news organisations have adopted some off-theshelf tools for providing audience data, but have done little to develop a systematic approach to making use of them. 13 As Debrouwere writes:
There's nothing like a dashboard full of data and graphs and trend lines to make us feel like grownups. Like people who know what they're doing. So even though we're not getting any real use out of it, it's addictive and we can't stop doing it.
Several interviewees highlight the problem themselves (anonymised):
To be honest with you, audience development was a very new term for me. … It's something that happened organically. It wasn't the top management who decided that we should use specific tools, or we should have specific parts of the newsroom dedicated to this. It's more like people coming to the company within the flow.
An interviewee from a major continental European newspaper admits:
We have Chartbeat and it is used mainly by those who work on the homepage, with video content, or on the visual desk. We have a big monitor in the newsroom that shows the editorial dashboard and we have it on a screen during the morning meeting. But in reality we use it in a very basic, superficial way, and absolutely not to its full potential.
One interviewee from a major continental public service broadcaster says:
We have an analytics department here. But they are in an old world. They are counting TV viewers and radio listeners. … Journalists get a newsletter and a monthly review of what happened on Facebook and Twitter. That's it.
On the basis of our interviews, we would suggest that these variations in how analytics are used are more organisational than they are national. In most countries, digital market leaders and new start-ups use a wide range of analytics to support and continually develop their editorial and organisational goals. Legacy media, especially smaller newspapers and public service media, often lag behind international best practice. Leading English-language organisations are ahead in part because of the pressures and opportunities that come with competing for a global audience, in part because innovations in tools and techniques disseminate faster in English-language work environments – at a very basic level, the bulk of the terminology and much of the technology that underpins analytics is in English, and developed and marketed in the US and the UK first.
In terms of the evolution of how data and metrics are used in newsrooms, our interviews suggest that, with a few significant exceptions, most continental European newsrooms have adopted editorial analytics in a more incremental and bottom-up manner as various online editors pick up new skills in the course of doing their jobs and less as a result of a strategic decision taken at the top of the organisation. A role, let alone a whole team, fully dedicated to audience development, is not common across continental Europe. Audience development is done, but frequently on the side by people with many other responsibilities. this rarely leads to best practice. Many seem stuck at the level of rudimentary or generic analytics (which is the case in most US and UK newsrooms too). With the partial exception of the german media conglomerate Axel Springer, highlighted by many vendors as a sophisticated operation, we have not come across cases on the continent like the New York Times, the BBC, or the Financial Times where a major, legacy media organisation has clearly recognised it was falling behind in terms of best practice compared to competitors and international standards, and set about to address the problem.
3. Tools, Organisation, and Culture: Developing Analytics Capability
As noted from the outset, analytics are about technology and data, but not only about technology and data. What sets best-practice examples of editorial analytics apart from more generic and rudimentary approaches is that these organisations have developed combinations of tools, organisational structure, and newsroom culture that supplement each other in ways tailored to the particular news organisation's specific goals. News outlets interested in assessing their own analytics capability can think of tools, organisation, and culture as dimensions of a capability assessment triangle (similar to the tool sometimes used to assess employees, see figure 3.1).
Figure 3.1. Analytics capability assessment triangle.
Tools concerns whether an organisation uses the best-available technological means, includin sources of data, software, and interfaces, whether in-house tailored tools or generic off-the-shelf products. Organisation concerns whether the newsroom has a clearly structured approach to using analytics, where specific individuals (in larger organisations dedicated teams) are responsible for helping the whole newsroom and have the expertise to do so. Culture concerns whether the newsroom as a whole, including both senior editors and rank-and-file journalists, is routinely and willingly using analytics and data as part of their editorial decision-making. the area covered by the triangle represents an organisation's overall analytic capability. the benchmarks for each dimension will evolve over time as a changing media environment calls for new tools, new forms of expertise, and new ways of linking analytics with everyday and longer term editorial decision-making.
The triangle draws attention to how all three components are distinct and necessary parts of developing analytics capability in the newsroom. tools, organisation, and culture cannot substitute directly for each other and only work together. A newsroom can have the best-available tools and a strong analytics team with a clear position in the newsroom organisation, but without a culture of data use, it will fail to realise its full potential. Similarly, a newsroom can have good tools and a culture of data, but no in-house analytics expertise, and will thus struggle to do in-depth analysis and use analytics systematically, especially for longer term planning.
The three types of analytics discussed above, editorial, generic, and rudimentary, can be illustrated
using the triangle (Figure 3.2).
Figure 3.2. Analytics capability of organisations with editorial, generic, and rudimentary analytics and different levels of development in terms of tools, organisation, and culture.
What sets best-practice examples of editorial analytics apart from others is as much about their organisation and their newsroom culture as about their tools. A tailored tool like Ophan gives the Guardian an advantage over competitors. But generic tools – both general ones like ComScore and newsroom-oriented ones like Chartbeat – are becoming more powerful. The question is thus not only what tools, but also what kind of organisation and culture a newsroom needs to make the best use of analytics. News organisations aiming to improve their analytics capability ignore these issues at their peril. As more user-friendly tools developed expressly for editorial use, like Chartbeat, Parse.ly, and NewsWhip, become more common and more powerful, people can be the hardest part of developing analytics capability.
In this section, we go through each of these three dimensions of tools, organisation, and culture.
3.1 Tools
The first and most evident sign of the rise of analytics in newsrooms around the world is the spread of tools to track audiences. the image of big screens on walls showing real-time figures of pageviews, unique users, and average time spent on site is increasingly common. In a recent survey of editors, news media CEOs, and digital leaders primarily from Europe the most mentioned tools used by newsrooms are Chartbeat, NewsWhip, and Parse.ly. Interestingly though, a high number – 45% – indicated they use a home-grown system, mostly in addition to other tools.
Figure 3.3. Use of analytics in newsrooms. (Source: Reuters Institute Digital Leaders Survey 2016; n=123).
Off-the-shelf tools include Omniture (Adobe Analytics), ComScore DAX (Digital Analytics), and Google Analytics, as well as more local vendors, tools that serve far wider constituencies well beyond the news industry in e-commerce and corporate communications. They, like newer tools such as Facebook Insight and twitter Analytics, are examples of generic tools that many newsrooms appropriate for editorial purposes. Newer additions are chartbeat, Parse.ly, and NewsWhip, all developed at least in part with editorial priorities and news media's organisational imperatives in mind. (there are many others, like EzyInsights, which is gaining traction in Germany and the Nordic countries as an alternative to NewsWhip.) We will briefly outline just three of the most well-known off-the-shelf tools developed for use in newsrooms: chartbeat, Parse.ly and NewsWhip.
Chartbeat is most known for its real-time analytics that focus on audience attention. Its dashboard informs the way the homepage is structured, helping to sharpen headlines and the formats of articles. Based on the numbers, homepage editors can modify the structure of the page in realtime, leveraging the learning from the real-time reactions of readers by optimising the user experience. Chartbeat has recently introduced opportunities for various forms of A/B testing. When rolled out across large volumes of content this can make a significant difference to overall traffic, especially on sites with many visitors.
Figure 3.4. Chartbeat dashboard showing real-time traffic data from gizmodo.com (screenshot taken on 20 Jan. 2016). Notice e.g. the inclusion of a recirculation rate for how many users go from one piece of content on the site to another rather than just leave.
PARSE.LY
Similarly, Parse.ly tracks real-time data, as well as data from published articles to help identify topics audiences have responded well to in the past. The dashboard also allows newsrooms to track and better understand users' behaviour and discover the journey they make through the content, where readers are coming from, and where they're headed next, through what device, and when they are reading it. Parse.ly's tools are developed for editorial, product teams, data analysts, and sponsored content.
Figure 3.5. Parse.ly dashboard showing the average traffic by day of week and time of day. Like Chartbeat, the figures are aggregated but can be split by device.
NEWSWHIP
NewsWhip tracks social media signals, through indicators like tweets, shares, and comments. It offers tools like 'Spike' and 'Analytics'. Spike is a content discovery dashboard used by breaking newsrooms to stay on top of the trending content. It is commonly used by social media editors, breaking news, and trending stories desks. It is a real-time tool that allows newsrooms to hone in on what their audience is talking about on social media over different time periods and use this to think about their own output.
Figure 3.6. NewsWhip: a view of a predefined panel of news publishers, showing their most popular stories on social media over the last 12 hours (screenshot taken on 19 Jan. 2016). NewsWhip can help newsrooms understand what people are reading on social media.
Most news organisations use several of these tools or others like them, as well as wider generic analytics tools like Omniture, Google Analytics, Facebook Insights, and Twitter Analytics. Many also supplement them with their own home-grown analytics tools. As mentioned earlier, the Guardian's Ophan is regularly cited as an inspiration, but other organisations are developing their own tailored dashboards and tools.
From continental Europe an interesting example of a tailor-built in-house tool comes from the German newspaper Die Welt, owned by the Axel Springer media group. As part of the move towards more data-informed decision-making in the newsroom, Die Welt has developed a tool to calculate and communicate an 'article score' for all articles published. The system is designed to help journalists and editors in a simple and user-friendly way by aggregating data from different analytics tools including Chartbeat. It provides a single score for how each article published has performed. The score goes from 0 to 30 points and can be broken down in five constituent elements: page impressions, time spent on the article, video views, social shares, and bouncing rate. The traffic element can give an article between 0 and 10 points, the four other elements between 0 and 5 points each. The different elements are chosen to reflect Die Welt's overall priorities, for example, not only high traffic, but also engagement and various forms of multimedia and social use.
Figure 3.7. Die Welt's article score: a comparison between scores of different articles, identified by topic. High performance is marked green, low performance marked red.
The article score is Die Welt's attempt at creating a system that gives people in the newsroom simple and clear indications of how their articles are performing while promoting quality content rather than simply click-bait. the scores are featured in a daily email sent by the editor-in-chief to the whole newsroom. The system is designed to help journalists understand how their content performed and where there is room for improvement. 'What went right or wrong with my article? Did I include a video that nobody watched, did they find the article useful and interesting enough to share it?': the article score helps journalists find answers to all these questions, Kritsanarat Khunkham, managing editor at Die Welt, explains. 14
The development of the article score is part of a strategic process that focuses on making sure that Die Welt produces quality content that readers will find compelling enough not simply to click on and glance at, but actually to spend time with, perhaps share, maybe even pay for. It is a clear example of how Die Welt – and Springer more widely – is developing forms of editorial analytics that are aligned with the specific editorial goals (quality content) and business model (metered paywall) of a specific organisation. A tabloid like Bild (also a Springer title), with a greater emphasis on advertising, high volumes of traffic, and a freemium pay model rather than metered paywall, calls for a different approach.
Developing tools to underpin an organisation's analytics capability is about making sure that the newsroom has the right combination in place – including both simpler and more user-friendly tools like the Guardian's Ophan and Die Welt's article score aimed at helping the newsroom at large as well as more specialised tools, often both in-house and general tools like Google Analytics, Facebook Insights, and editorial tools like Chartbeat, Parse.ly, and NewsWhip, for more in-depth and detailed analysis.
3.2. Organisation
In all the best-practice cases discussed above – the Guardian, Financial Times, BBC, and Huffington Post – analytics are rooted in the newsroom (even if the teams involved often include people with non-journalistic backgrounds and draw on resources from other parts of the organisation). It is worth discussing two other examples of how analytics are actually organised to show how both large (the Wall Street Journal) and smaller (Quartz) organisations are integrating specialised expertise in their editorial operations for both day-to-day decision-making and longer term planning.
Carla Zanoni has the title Executive Emerging Media Editor and heads up the Wall Street Journal's audience team, which operates globally out of New York, Hong Kong, and London. Her approach shows how analytics can support both day-to-day work and longer term editorial strategy. Zanoni's team is organised around four different legs: audience engagement, audience development, newsroom analytics, and emerging platforms. All have different responsibilities, but are similar in that they are based on a combination of editorial expertise and quantitative data analysis. The audience engagement part of the team is made up of a group of audience engagement editors (who used to be called social media editors). 15 The team is looking at using different tools throughout the newsroom to inform how the Wall Street Journal engages with its audience, whether this happens on social media or on messaging apps or through blogs and more traditional stories. The audience development part of the team has a more strategic role, focusing on thinking through where different opportunities to reach the audience might lie.
Some of the daily role is involved in partnerships and something I call 'digital hygiene': just thinking through SEO [search engine optimisation] and story-flow and making sure that we're publishing stories at the right time, capturing the best audience that is suited for that story.
The newsroom analytics part of the team is the newest addition, and includes data-scientists as well as people with an editorial background. they look at how stories and different sections are performing, and test hypotheses about what might improve performance.
In the past a lot of these kinds of experiments and hypotheses went untested; now we're actually looking at [figuring out if] our beliefs [are] rooted in facts or if they are just a kind of newsroom myth.
Zanoni describes the fourth part of the team, emerging platforms, as more of a 'creating desk', concretely working on producing content for new, emerging platforms, whether that is an internal platform, a new app the WSJ is launching, or a third-party platform. She stressed how the four legs are deeply interconnected and are built to share learning across the team. Everyone on the audience team has a journalistic background and they work with the other journalists in the newsroom from the story-concept and inception, to promotion, and to resurface it after it has been published. So far the conversation around data and analytics has predominantly involved editors, but it is moving towards including reporters in the discussion. The audience team plays an important role in making sure that everyone in the newsroom understands the data and learns how to build a narrative from the numbers, instead of just handing reporters a bunch of metrics and statistics. Zanoni stresses that:
It's really important to give them the right context, so that they can create that narrative, and through building that narrative, once we're all on the same page, we then all know that there are certain levers that we can pull and there are levers that we would like to be able to pull.
More recent and much smaller start-ups are also working systematically with analytics as part of their editorial operation. One example is Quartz, where the audience team and especially the so called 'growth editors' have been integral to the project from the beginning. Part of the answer to the question behind the launch of Quartz – what would The Economist look like if it had been founded in 2012? – is that it would involve data-informed editorial decision-making. Marta Cooper, Deputy Growth Editor at Quartz in London, explains:
Our role is to help Quartz expand its audience through an editorially focused growth strategy, by which I mean we're not a separate audience development team but we work in tandem with the editorial team. We support the news team in multiple regions to make sure the stories they write find an audience using editorial strategies: this involves running the Quartz social media accounts, workshopping resonant headlines and angles of stories with reporters, and tracking emerging stories across social media platforms. 16
Cooper is part of a four-people growth team that includes a director of growth and two other deputy editors, one of whom focuses on the US and Asia and one who focuses on partnerships. In a global editorial staff of about 80 people, this is a significant commitment from Quartz. For a newsroom with 500 journalists, a proportional commitment would be 25 people on the audience development team, more than twice the size of Kaplan's team at the Financial Times.
At Quartz, Cooper explains, all journalists have access to the analytics platforms, specially Parse.ly, Chartbeat, and Omniture, plus insights from Facebook and Twitter.
We receive daily emails with a snapshot of global traffic and that of our specific regions for both the previous day and month. There's a number of metrics we'll take into consideration, including unique visitors and pageviews, but we're also aware of social shares and other data as well.
Growth editors are specialists with insights in analytics who work very closely with individual
reporters. Cooper explains:
Success comes in different forms besides us reaching our traffic goals. For instance, if a reporter and I spend time going through their story and why they find it compelling and we eventually come up with a strong headline. That to me is a successful process.
The Wall Street Journal and Quartz are both organisations that, like other examples of bestpractice editorial analytics, have teams in place that help with both short-term day-to-day decision-making and longer term editorial strategy development. In both cases, their approach is tailored to the business model in question, subscription and an emphasis on engagement for the Wall Street Journal, native advertising and a search for greater reach via platforms at Quartz.
Many other newsrooms have a far less clearly defined approach to analytics. In many cases, analytics are organised in ways that reflect inherited workflows and the incremental accumulation of new tools and techniques over time more than any thought-through approach. Real-time analysis may be carried out by a homepage editor, social media insights are monitored by a social research department and not in the newsroom. Search engine optimisation is sometimes streamlined across the whole online newsroom, sometimes the province of a SEO editor, sometimes an afterthought.
Developing an organisational structure to support analytics capability is about ensuring that someone in the newsroom – in some case individuals, in most cases teams – has primary responsibility for data, that these people have access to the information and tools they need, that they have the expertise needed to make use of them, and that their relations to the rest of the newsroom are clear.
3.3. Culture
Having all journalists understand the strengths and weaknesses of data and how they can inform editorial decision-making is a crucial part of an organisation's overall analytics capability. If you just give people a bunch of numbers, they are likely either to ignore them or to use them to justify conclusions they would have arrived at without the data. The Guardian's audience editor Chris Moran says:
First of all, give [journalists] lots and lots of data but choose it carefully. It's not just about the
amount of data that you deliver; it's also about the culture that you build around that data.
People have to understand what you're talking about: you have to have an open, honest
and transparent conversation about quality and promotion at the same time. 17
Many of the audience editors interviewed for this report highlighted the development of a culture of data in the newsroom as one of the most important parts of their job.
Based on our interviews, it seems the general response from journalists to analytics has in most cases shifted from resistance to curiosity and interest. this is in contrast to earlier research, which suggested many newsrooms have resisted the introduction of analytics (see e.g. christin 2014; Petre 2015; Zamith 2015). (It is worth mentioning that the interviews have been conducted predominantly amongst those in newsrooms who are active promoters of the use of data and who work primarily with online/digital journalists rather than print or broadcast newsrooms.) What emerges from our interviews is not resistance, but the challenge of helping journalists achieve a real understanding of what the data mean and how to act on this.
The primary issue is explaining what they are looking at, why those metrics are important, and how this affects their work. Elinor Shields, Head of Audience Engagement, BBC News, says:
I think now the issue is much more about demystifying data and giving people ways of
understanding how it will change what they do in a practical way and how it can add value
to what we do.
Her colleague Jeremy Tarling explains further that the crucial point is acting on the data, not just providing people with dashboards. When a dashboard shows a journalist that the average engaged time on an article was only 5 seconds, what does that mean and what can he/she do about it?
WSJ's Carla Zanoni echoes this sentiment:
They [the journalists] are so hungry for data. I worried when I came [to the Wall Street Journal] that my role would have been more evangelical in some way, but that could not have been further from the truth. They are extremely hungry for data, they want tools to be able to measure in real-time whether they are doing the right thing. If anything, I think that the biggest hurdle has been making sure that we are able to provide them with the kind of context and actions for them to be able to do something with the data.
In many newsrooms, a monitoring of performances and achievements is indeed happening, often in the form of periodical or even daily emails, but these numbers rarely inform editorial decision making. They are numbers without meaning and without consequences. Short-term day-to-day optimisation of articles on the homepage and posts on social media are not uncommon, but there is less longer-term strategic use of data to shape editorial priorities and underpin organisational objectives. When this occurs, it often stays within the team handling the job (often not in the newsroom) and is not spread across departments and the whole newsroom. The kind of 'democratisation of data' associated with open and user-friendly dashboards like Ophan or clearly communicated metrics like Die Welt's article score are rarely found in organisations with a more generic or rudimentary approach to analytics.
Clémence Lemaistre, Editor-in-chief for Digital content at the French paper Les Echos, explains:
There is a strong awareness on the online desks of the importance of looking at the data. This doesn't always mean they are doing it perfectly, but they are not recalcitrant. It's much more a question of understanding rather than obstruction. 18
Huffington Post UK's Jack Riley admits it is sometimes easy for journalists to fall back onto the dichotomy that sees audience targets directly opposed to the journalistic value, where the most popular things are those that have the least journalistic value. 'I don't think that's a useful way of thinking about it. If you're writing something you're really proud of … and no one reads it, [it means that] that is actually not having any impact at all.' It's important, Riley says, to reconcile the two concepts and make sure that the content journalists are most proud of also reaches the widest possible audience.
The overarching ambition of all the best-practice cases discussed in this report is to supplement editorial judgement with quantitative analysis of relevant audience data. the goal – to use a mantra repeated in many of our interviews – is to be data-informed, not to be data-driven. 19 The Guardian's audience editor Chris Moran puts it very clearly:
We describe ourselves as data-informed not data-led, and that's critical to me. I'm obsessed with data in the newsroom, but it shouldn't be the only thing that is making the decisions: editorial instinct should do that. So in terms of what [journalists] are going to be writing about, broadly speaking, that is going to be dictated just by the news agenda and by the natural instinct of the newsroom. More importantly perhaps, the data can lead us to understand the kind of journalism that people might want, particularly on different platforms. 20
For this to work, journalists and editors – and not only dedicated audience teams – need to understand the meaning of the data that they are given and how to act upon it, as Marta Cooper from Quartz underlines:
[Audience data] is a central feature of our newsrooms. There is sufficient training for staff on how to make the most out of the analytics tools we use, so it's easy for them to become part of a journalist's daily workflow.
Developing a 'culture of data' in the newsroom to underpin analytics capability is about making sure that journalists and editors who are not part of the audience team are given access to data that are relevant to them, know why – and agree that – these data are relevant for them, and know how to act on it. If data are available but ignored or mostly used to validate decisions already made for other reasons, the culture does not underpin editorial analytics. If data are taken seriously as one of several factors informing decision-making, evaluations of performance, and the development of workflows and new editorial products, the culture does underpin editorial analytics.
4. Metrics, Metrics Everywhere
As is clear from the above, newsrooms have access to and use more and more different kinds of audience data. But some things are harder to define and measure than others, some kinds of data are harder to access than others, and all forms of analytics have to confront problems of data availability, incompatibility between different kinds of metrics, and the limitations involved in using quantitative indicators to understand the messy and diverse realities of how people engage with journalism, why, and what it means. Even the most data-driven technology companies like Facebook and Google are keenly aware that the data never tell the full story and that decisions ultimately always involve qualitative assessments and human judgement. 21 So too with editorial analytics.
4.1. What do you Want to Measure?
Much depends on what you are interested in measuring, how you measure it, and how good your metrics are. there is, as BuzzFeed founder and data enthusiast Jonah Peretti said in an interview with Felix Salmon, no 'god metric' for journalism. 22 A news industry that used to obsess over print circulation and broadcast ratings has circled through a quick succession of preferred digital metrics, from clicks, pageviews, and unique users to engaged time. Most news organisations today work with a range of metrics, including both older ones like pageviews and unique users and newer ones like shares.
Box 4.1. A list of some of the most important metrics
Very simply, the metrics used (unique browsers, time spent) and the underlying phenomenon inferred from them (reach, engagement) define what you see. Analytics teams are looking at audience data and trying to infer people's behaviour, attitudes, experiences from it. As media researchers like to say: there are no audiences, only ways of seeing people as audiences. 23 A unique browser is not necessarily the same as a person using content. Time spent on a site is not necessarily the same as an engaged user. They are proxies. The question is then which proxies are most useful and for what. currently, pageviews and unique browsers are falling out of favour, and attention or engagement is seen as the future.
One of the most public champions of engagement as a key metric for news has been Tony Haile, CEO and founder of the analytics company chartbeat. In an article in Time magazine in March 2014 he wrote:
If you're an average reader, I've got your attention for 15 seconds, so here goes: We are getting a lot wrong about the web these days. We confuse what people have clicked on for what they've read. We mistake sharing for reading. We race towards new trends like native advertising without fixing what was wrong with the old ones and make the same mistakes all over again. … The media world is currently in a frenzy about click fraud, they should be even more worried about the large percentage of the audience who aren't reading what they think they're reading. 24 In 2015, they ranked the top articles by the total combined time readers had spent looking at them. 25
4.2. Strengths and Weaknesses of Different Metrics
Most of our interviewees push back against the idea that there is one metric on which news organisations should focus. The dominant view is that different metrics have different strengths and weaknesses and are suited to different purposes. It all depends on what you want to understand. One way to think about some currently popular metrics like pageviews and attention is to consider how they map on to a continuum between things that can be relatively precisely defined and measured (like reach) and things that are much harder to define and measure (like impact). (See Figure 4.1.) Even the seemingly simplest metrics, like reach, while more clearly defined and measured today than engagement, loyalty, or impact, is still plagued by inconsistencies. In August 2014, the Huffington Post celebrated reaching 115 million global unique visitors as measured by ComScore while also reporting that their internal number was 368 million global uniques. 26 (Sometimes the things that are currently harder to define and measure are more important, both for editorial and organisational purposes.)
Figure 4.1. A range of metrics mapped in terms of relative clarity of definitions and measures.
Different metrics can thus serve different purposes, but some things are currently harder to measure than others. Some will likely always resist quantification. Some sources of data, like sessions, are used in attempts to understand quite different things, like reach versus engagement. Some sources of data, like social media interactions, can be measured very precisely, but can be hard to link to core interests like reach, engagement, loyalty, and impact (what does a 'like' or a comment mean?) (graves and Kelly 2010; Ofcom 2014). Current metrics remain much better at capturing traces of (parts of) what people do with digital news content than why or what it means.
All the metrics share a set of further challenges, of which the people we interviewed for this report are keenly aware.
In most cases, it is very hard to link users to specific demographics (unless a site requires registration and can verify information entered by registered users).
It is hard to integrate metrics not only across different digital channels (website, app, thirdparty platforms), but also with offline media (print, broadcast, etc.).
the most easily accessible data are precisely audience data, meaning that most news organisations have little information about
a) people who do not use their content, including
b) competing news organisations' audience and
c) people's behaviour on platforms like Facebook, twitter, and Snapchat through which they may well encounter an organisation's journalism.
Chris Moran from the Guardian says:
I think the next big challenge for everybody is: How on Earth do you pull this stuff in? So one of the big conversations we're faced with right now [with platforms like Facebook and Apple News] is 'Give us the data.' 27
Sebastian Horn from Ze.tt echoes this:
Generally speaking, the big challenge will be how do we measure reach and impact on social and not just on the website. Every platform will give you data and the first question is how can we trust that data, what does that mean for you and how do you compare the data between platforms. Is an impression on Twitter the same as an impression of a Facebook post? How do you compare a Snapchat video with a three minutes YouTube clip? How do you aggregate this, how do you create a ranking, how do you put a monetising number against these values?
4.3. Good Compared to What?
The difficulties of defining goals precisely, measuring them in reliable ways, and capturing or accessing relevant data also mean that benchmarking is difficult. Many news organisations struggle to define what good looks like – especially at the level of individual journalists or pieces of content. this is particularly the case when comparing content often used in very different ways: a video, audio, a quiz, a multimedia piece, graphics, a live blog, or an investigation in traditional article format. All will perform differently depending on what measure you look at. How do you compare the value of their numbers?
Numbers need context, underlines Nick Petrie, Deputy Head of Digital of The Times of London and Sunday Times. 28 Just having a number of how many people access a piece of content, without the overall reference of how many people access the whole content section, for example, gives you little insight into how well your article did. 'If you just give [journalists] a number and say, "Oh, 300 people have read it every day this week." It's like, "Well, is it good or bad?"', Petrie says. Jeremy Tarling from the BBC makes the same point: 'It's not just about providing people with dashboards. It's the next step, isn't it? "So, this dashboard's telling me that my average engage time was only 20 seconds on this article. What do I do?"'
Throughout, the most developed metrics are those that serve as currencies for most digital advertising – clicks, pageviews, and unique users. those that sit between commercial and editorial considerations – like engagement and loyalty – are also relatively developed. those that currently are primarily of editorial interest – like impact – are poorly developed.
4.4. The Link between Analytics, Editorial Priorities, and Organisational Imperatives
The use of analytics in newsrooms is normally aligned with the business model of a given news organisation. Advertising-supported free sites, subscription-based sites with metered or hard paywalls, social media centred organisations emphasising sponsored content, and public service media organisations with more or less secure public funding all use metrics in different ways (Usher 2013). This is an example of how editorial, technological, and commercial/managerial forms of expertise are intertwined in digital news media (Küng 2015). this is a point made very explicitly and unapologetically by many of our interviewees. As Sebastian Horn from the German start-up Ze.tt puts it:
Ideally there would be a link between the numbers that are important for the newsroom and those that are important for the business side. If we upload a video directly on Facebook, the important number for the newsroom is how many people we reach. We might have no way to monetise that on Facebook, but we've reached a huge audience and from the content side that is very important. But then a business person might tell you 'what are you doing?!'
I think, generally speaking, that for the industry this is the time when we figure out what the relationship with these platforms looks like and what's the deal. And how do we reconcile the success of the editorial goal of reaching huge numbers through the platforms with the commercial success of monetising it?
Similarly, Renée Kaplan from the Financial Times argues that news organisations need to align their editorial priorities and organisational imperatives like running a sustainable business:
The engagement team sits squarely in the newsroom so our objectives aren't commercial. Nonetheless my job, and every job in the newsroom and the job of every one of our 600 correspondents around the world, depends on the viability of the business model. At the end of the day people have to want to pay for the content otherwise we all go away. So everyone is interested in making sure we reach the greatest number of people who are likely to be affected by our journalism and likely, ultimately, to want to pay for it.
Chris Moran from the Guardian echoes the same sentiment.
We produce quality journalism. I want that journalism to be widely read and I am not apologetic about that because journalism exists in the context of its audience and because it's our best way to get new loyal readers as well. 29
4.5. How do you Measure Impact?
The question is whether metrics for impact will be developed as more and more advertisers grow disenchanted with clicks, pageviews, and unique users, which are increasingly seen not only as imprecise proxies for what advertisers want to achieve (move product, burnish their brand), but also as vulnerable to increasingly large-scale fraud and noise from non-human traffic that can be mistaken for actual users. One industry estimate is that as much as 22% of all web traffic is made up of 'impersonator bots' designed to look like human users. 30 This has led to greater advertiser interest in alternative metrics, like attention, engagement, or impact. How this will develop is yet to be seen. Historically, as the experience of audited circulation, broadcast ratings, and earlier generations of web metrics suggest, multiple stakeholders including advertisers, media organisations, and regulators normally get involved in the protracted development of shared, agreed-upon 'audience information systems'. (Journalists have tended to be less involved in these processes, and their considerations less reflected in the metrics developed: Napoli 2011).
But so far, work on effective metrics for the impact of journalism is led by non-profits, philanthropists, and public media in the United States. (European public service media have so far been surprisingly absent from this development.) The Bill and Melinda Gates Foundation and the John S. and James L. Knight Foundation are founding the 'Media Impact Center' at the University of Southern California to create new ways to measure the impact of media. The Knight Foundation has also given $35,000 to NPR (National Public Radio) in order to build Carebot, a tool that aims to measure whether people really cared about content they used. The tool seeks to build on existing metrics like page completion and sharing but also to develop new ones for audience assessment of quality. NPR visual Editor and Project Lead Brian Boyer has explained to the Poynter Institute that Carebot will emphasise a series of metrics that include social engagement (likes, shares, comments), time spent on site, and completion rate, and will aggregate numbers from a variety of sources, including chartbeat, google Analytics, and social networks like Facebook and Twitter. 31 Other US-based non-profits, like the Investigative Reporting Workshop and ProPublica, are also working on metrics for impact. 32
Basically, contemporary forms of analytics are very good at understanding the main ways in which people used digital media in 2010. Homepage traffic, referrals from social and search, and ways to increase pageviews through A/B testing and article placement are well developed. Even the most advanced approaches to editorial analytics, however, still face many challenges when it comes to understanding more recent trends, like use across multiple devices, offsite consumption of content on third-party platforms, and conversion of users into loyal users, and potentially subscribers or members. It also remains hard to link digital data directly with wider consequences like people becoming more informed, engaged, etc. (Audience data remain better at documenting what people do than why or what it means.)
Editorial analytics in their different forms represent a clear step forward, from a journalistic viewpoint, from rudimentary and generic analytics, which are often almost exclusively driven by short-term and often commercial considerations. But it is also clear that, even in their most advanced best-practice examples, contemporary editorial analytics continue to face many challenges.
One set of challenges concerns news organisations themselves: how do you define your primary goals, and how do you ensure you build analytics capability by combining the right tools, organisation, and culture to make sure you can measure how you perform, act on that information, and develop your operation over time? Another set of challenges concerns analytical questions around definition, measurement, and data quality.
News organisations need to think about how they can leverage the strengths of analytics for both short-term and long-term operations while remaining aware of their shortcomings. In our interviews, it is striking that people from organisations frequently held up as examples of best practice are also often amongst those who are most modest about what current forms of data can accomplish.
5. Editorial Analytics: The Journey Ahead
Most newsrooms have adopted the use of analytics in recent years and journalists who in the past may have resisted the introduction of metrics increasingly request information about how people use their content. Data-informed decision-making previously associated with sites like BuzzFeed, Gawker, and the Huffington Post is increasingly central to editorial processes at organisations like the Guardian, the New York Times, and Die Welt as well as leading public service media like the BBC and various start-ups like Quartz and Ze.tt.
There are significant differences in how analytics are used in different news organisations. Many have incrementally adopted a range of rudimentary and off-the-shelf forms of analytics that are often used in an ad-hoc manner to help increase day-to-day traffic and reach, but have done little to develop analytics clearly aligned with editorial priorities and organisational goals, to reorganise workflows, or to ensure relevant and comprehensible data are available and used throughout the newsroom.
Broadly speaking, leading media organisations in the US and the UK still seem ahead in the development and use of editorial analytics, but market leaders in continental European countries like Germany and Poland are developing their own practices and pulling ahead of domestic competitors. the whole publishing industry is still behind leading technology companies in terms of their use of analytics, and could also learn from leading advertising, marketing, and e-commerce companies.
When it comes to specifically editorial analytics, digital-only start-ups as well as a small number of relatively innovative private legacy news media are generally leading the development, with smaller private legacy media and many public service media lagging behind. The sophisticated editorial analytics employed by small start-ups across Europe and North America underline that size and resources are not as decisive a factor as a pro-digital culture, strategic leadership, and a willingness to invest in analytics to help understand and engage the audience in a very competitive marketplace of attention.
To make the move from rudimentary or generic analytics to the kind of editorial analytics practised
by market leaders, news media need to
1. define their editorial priorities and organisational goals,
2. identify the data and metrics most useful for pursuing these effectively, and
3. develop tools, organisational structures, and newsroom cultures that make analytics
actionable both short-term and long-term.
Editorial analytics are powerful but not perfect. Some things are difficult to measure. Some measures have weaknesses and flaws. Some measures are incompatible or contradictory. The data do not speak for themselves. And tools and techniques as well as sources of data have constantly to evolve to keep track with a changing environment. Best-practice editorial analytics are today still primarily concerned with short-term optimisation of onsite traffic and offsite distribution via search engines and social media. Analytics are only beginning to be used for longer term planning including the development of new products, audiences, and newsroom workflows.
Because different news media have different editorial priorities and organisational goals, there is no one right way to do editorial analytics. this also means that analytics are not plug-and-play. Developing effective editorial analytics requires investments in technological tools, organisational reform, and cultural change. This does not need to be expensive (in particular in comparison to the resources already invested in content production, and the risks associated with flying blind), and the returns on investment are in many cases significant. But it does involve change, and change can be difficult.
Editorial analytics are an evolving phenomenon. It is not about identifying a few standard tricks to increase audience reach or engagement, but about developing a process where quantitative evidence supplements more qualitative editorial expertise and enables continuous evaluation of performance and experiments to improve workflows and results. Because editorial analytics work best when aligned with clearly defined ends, changes in, for example, editorial priorities (a decision to develop a new area of coverage or reach a new target audience) or organisational goals (a change in business model from advertising-supported to pay or membership-based) should be accompanied by reformed analytics so that goals and ends remain aligned. An organisation aiming to maximise reach and advertising revenues is different from one seeking to engage a niche of loyal readers and convert them to subscribers (or members). Both are different from a public service media organisation with a guaranteed revenue stream. Similarly, because editorial analytics are fundamentally about helping journalists understand and effectively navigate a changing media environment, new media trends (the rise of distributed content on social media and through messaging apps, the rise of the mobile web, and of online video) require a continuous evolution of new metrics and forms of analysis and changes in the tools, organisation, and culture of analytics. Everyone we have interviewed for this research, including those at organisations often seen as examples of best practice, is keenly aware that we are only at the beginning of the development.
The journey ahead for editorial analytics involves
1. addressing a set of organisational challenges around the mainstreaming of analytics in newsroom tools, organisation, and culture, which requires leadership, investment, and restructuring;
2. better data, especially better data linking use to individuals across devices, browsers/apps, and across distributed environments including offline media as well as social media and messaging apps; and
3. an effort to clearly define and precisely measure priorities like impact.
It is important that journalists are part of this development. Analytics will continue to evolve. As Billy Bosworth, CEO of the software company DataStax has said, 'ten years from now, when we look back … we will be stunned at how uninformed we used to be when we made decisions.' But it is crucial to underline that what we will be more informed about in the future depends critically on who gets involved in developing analytics and metrics. If journalists are not part of this continued evolution, newsrooms will have a poorer understanding of the audience they need to reach and at a competitive disadvantage relative to more sophisticated rivals. It will also mean that the development of data, metrics, and analytics will continue to be entirely shaped by advertising, commercial, and technological priorities with little consideration of journalism, and leave publishing at a huge disadvantage in the wider competition for attention with other, non-news choices like social media, gaming, and the like. As one interviewee, who preferred to be anonymous to speak frankly, put it very bluntly:
You've got to be [data-informed]. You can't say, 'Are we?' Because some news organisations are not going to survive. They're either, in ten years' time, five years' time, they're either going to be greatly reduced or some of them will have ceased to exist. So no news organisation, big or small can arrogantly think that they can keep on going without looking into their data. … If you're not looking at the data, you're blind.
We agree. the promise of editorial analytics is great. the risk of not engaging with it even greater.
Anderson, c. 2011. 'Between creative and Quantified Audiences: Web Metrics and Changing Patterns of Newswork in Local US Newsrooms', Journalism, 12: 550–66.
Christin, A. 2014. 'Clicks or Pulitzers? Web Journalists and their Work in the United States and France', Ph.D., Princeton University.
Graves, L., and Kelly, J. 2010. Confusion Online: Faulty Metrics and the Future of Digital
Journalism. New York: Tow Center for Digital Journalism, Columbia Journalism School.
Hindman, M. 2015. Stickier News: What Newspapers Don't Know about Web Traffic has Hurt
Them Badly: But There is a Better Way. Cambridge, MA: Shorenstein Center on Media,
Politics and Public Policy, Harvard University.
Küng, L. 2015. Innovators in Digital News. London: I. B. Tauris.
Napoli, P. M. 2011. Audience Evolution: New Technologies and the Transformation of Media
Audiences. New York: Columbia University Press.
Newman, N. 2016. Media, Journalism and Technology Predictions 2016. Oxford: Reuters Institute
for the Study of Journalism.
Newman, N., Levy, D. A. L., and Nielsen, R. K. 2015. Reuters Institute Digital News Report 2015.
Oxford: Reuters Institute for the Study of Journalism.
Ofcom. 2014. Measuring Online News Consumption and Supply. London: Ofcom (prepared by
Oliver and Ohlbaum, the Reuters Institute for the Study of Journalism, and the Oxford Internet
Institute).
Petre, c. 2015. The Traffic Factories: Metrics at Chartbeat, Gawker Media, and the New York
Times. New York: Tow Center for Digital Journalism, Columbia Journalism School.
Usher, N. 2013. 'Al Jazeera English Online: Understanding Web Metrics and News Production
When a Quantified Audience Is Not a Commodified Audience', Digital Journalism, 1: 335–51.
Williams, R. 1958. Culture and Society, 1780–1950. London: Chatto & Windus.
Zamith, R. 2015. 'Editorial Judgment in an Age of Data: How Audience Analytics and Metrics are
Influencing the Placement of News Products', Ph.D., University of Minnesota.
https://www.scribd.com/doc/224332847/NYT-Innovation-Report-2014
The New York Times has since invested heavily in analytics and its audience development team. Many other news organisations, both in the United States and Europe, continue to lag behind: they use a few generic tools, fail to tailor them to their editorial priorities and organisational goals, and thus have a poor understanding of their audience and limited analytics capability.
But things are changing. The combination of new opportunities to measure audience behaviour and new challenges in terms of connecting with audiences has led more and more news media organisations to embrace and develop new forms of analytics. As the BBC's Director of News and Current Affairs, James Harding, wrote in his 2015 report on the future of news, the challenge of effectively using data about how content is being consumed will be 'central' for journalism moving forward. [2. http://newsimg.bbc.co.uk/1/shared/bsp/hi/pdfs/29_01_15future_of_news.pdf ↩
https://medium.com/@guardiancomms/behind-the-scenes-ophan-how-the-guardian-democratised-data-36cde3967062#.wqk08qbep ↩
'Bounce rate' and a range of other terms are explained in Box 4.1 below. ↩
https://medium.com/@GuardianComms/behind-the-scenes-ophan-how-the-guardian-democratised-data-36cde3967062#.wqk08qbep%5D
Ophan is part of how Moran and his colleagues seek to build a culture of data in the Guardian newsroom, focused on understanding users' behaviour and on helping individual journalists and editors make data-informed decisions. For a start, everyone in Moran's team – four people in the UK – has an editorial background. There are currently no data-scientists on the team (though there are elsewhere in the organisation). Moran stresses the value of having people on the audience team who don't necessarily speak in the language of data, and have to work a bit to understand it. [6. chris Moran of the Guardian, interviewed by Federica cherubini, 19 Nov. 2015. ↩
https://www.niemanlab.org/2015/01/constantly-tweaking-how-the-guardian-continues-to-develop-its-in-house-analytics-system/
(Hamilton oversees all aspects of the Guardian's online interaction with audiences, including Moran's audience team but also more widely moderation, reach, and optimisation, as well as social media and community journalism.)
As Chris Moran sees it, the real objective of the audience team is to make data understandable and actionable for everyone in the newsroom. The Guardian's approach to analytics involves everyone considering data as part of their evaluation of every piece of content, not just a focus on the homepage, on overall site traffic, or the most-read articles. As there is no one single approach that works for every part of the Guardian's editorial operation, across all markets, and for every piece of content or topic, Moran underlines it is important that journalists develop an approach in which they try something and then use the data, as well as their editorial judgement, to evaluate the outcome. Examples of how analytics inform decision-making includes day-to-day tweaking of headlines, pictures, placement, and promotion across social media as well as changes to workflow, like time of publication. As Moran says, 'the single best piece of advice you can give is "launch it when your audience are awake". It sounds ridiculous, but you would be amazed by the number of news organisations, including us still, which launch things at midnight, as a hangover from a legacy/print experience.'
Other interviewees highlight the Guardian as an inspiration for an approach to analytics that combines (1) Ophan as a user-friendly tool accessible to all (and a wider range of more complex tools available behind it), (2) an audience team organisationally anchored in the newsroom and able to help their colleagues, and (3) a 'culture of data' where individual journalists and editors beyond the audience team make use of analytics as part of their editorial decisions and where everyone is at least in principle open to experimentation and evaluation.
2.2. The Financial Times' Rules of Engagement
In March 2015, the Financial Times hired Renée Kaplan, former chief content officer at Havas WorldWide and previously Editorial Director for France 24 and producer at CNN, for a newly created role as Head of Audience Engagement. Since then, Kaplan has been putting together a team and developing a strategy for moving the Financial Times from 'digital first' to what she calls 'audience first'. In Kaplan's view audience engagement is about building a relationship with readers: 'Audience engagement is about getting our journalism out in front of more audiences, and more of the right audiences.' [8. Discussed during an event at the BBc in Nov. 2015. ↩
Elinor Shields, BBc News, interviewed by Federica Cherubini, 18 Nov. 2015. ↩
Jeremy Tarling, BBc News, interviewed by Federica Cherubini, 18 Nov. 2015. ↩
Jack Riley, Huffington Post, interviewed by Federica Cherubini, 9 Nov. 2015. ↩
Sebastian Horn, Ze.tt, interviewed by Federica Cherubini, 12 Nov. 2015. ↩
http://www.niemanlab.org/2013/04/a-dutch-crowdfunded-news-site-has-raised-1-3-million-and-hopes-for-a-digital-native-journalism/ ↩
40,000 members for a Dutch-language medium in the Netherlands, a country with a population of 17 million, is comparable to having 750,000 subscribers in the US, https://medium.com/de-correspondent/dutch-journalismplatform-the-correspondent-reaches-milestone-of-40-000-paying-members-a203251c2de2#.rh4duaycq ↩
Ernst-Jan Pfauth, De correspondent, interviewed by Federica Cherubini, 8 Dec. 2015. ↩
http://debrouwere.org/2013/08/26/cargo-cult-analytics ↩
Kritsanarat Khunkham, Die Welt, interviewed by Federica cherubini, 12 Nov. 2015. ↩
Carla Zanoni, Wall Street Journal, interviewed by Federica Cherubini, 16 Dec. 2015. ↩
Marta Cooper, email exchange, 14 Dec. 2015. ↩
Speaking in Paris at the Sciences Po' Journalism School conference in Dec. 2015. ↩
Clémence Lemaistre, Les Echos, interviewed by Federica Cherubini, 25 Nov. 2015. ↩
this is in clear contrast to an earlier period where analytics were clearly cast as standing in contrast to journalistic professionalism (see e.g. Anderson 2011). Editorial analytics are about making analytics part of journalistic professionalism. ↩
http://www.slate.com/articles/technology/cover_story/2016/01/how_facebook_s_news_feed_algorithm_works.html ↩
https://medium.com/matter/buzzfeeds-jonah-peretti-goes-long-e98cf13160e7#.rnpao5qbv ↩
the original phrase is from Raymond Williams (1958: 300), who wrote 'there are in fact no masses; there are only
ways of seeing people as masses.' ↩
https://time.com/12933/what-you-think-you-know-about-the-web-is-wrong%5D/
An example of how news organisations have grown increasingly interested in Haile's idea of an 'attention web' is the New York Times. In 2014, they ranked their top articles of 2014 by the number of unique visitors. [28. http://www.nytco.com/the-new-york-timess-most-visited-content-of-2014-2 ↩
http://www.nytimes.com/interactive/2015/12/09/upshot/top-stories.html ↩
http://www.huffingtonpost.com/arianna-huffington/100-million-thank-yous-to-huffposters-around-theworld_b_5822998.html ↩
Interview by Federica cherubini, 19 Nov. 2015. ↩
Nick Petrie, The Times and Sunday Times, interviewed by Federica Cherubini, 29 Oct. 2015. ↩
http://fivethirtyeight.com/features/why-we-still-cant-agree-on-web-metrics/ ↩
http://www.poynter.org/news/mediawire/382681/npr-is-building-an-analytics-bot-that-emphasizes-caring-over-clicks ↩
http://www.niemanlab.org/2014/03/how-can-journalists-measure-the-impact-of-their-work-notes-toward-a-model-ofmeasurement ↩
Navigate this publication
The Development of editorial analytics
How are news media developing and using audience data and metrics
Tools, Organisation, and Culture: Developing Analytics Capability
Metrics, Metrics Everywhere
Editorial Analytics: The Journey Ahead
© 2022 Reuters Institute for the Study of Journalism