Tools that measure the frequency of keywords found on online posts or the patterns of mobile-phone use can give a real-time snapshot of peoples' current environment, says Anoush Tatevossian, spokeswoman at UN Global Pulse, a UN initiative set up to research ways to use big data for development.
The insights from such data may include food prices, prevalence of disease and social attitudes towards interventions such as vaccination programmes, Tatevossian says.
"As technology becomes more user-friendly, there is a real opportunity for digital data analysis to have a significant impact on development," she tells SciDev.Net.
Behaviour on social media, news websites, blogs and forums, along with the less-visible by-products of digital life such as phone records and Internet search patterns, have already been shown to accurately depict events on the ground, she says.
The challenge now is to work out how to use this wealth of information — which would take years to collect via door-to-door surveys — to assist development projects, she adds.
Tracking food prices in Indonesia and access to financial services in Kenya by analysing the frequency of keywords used on social media are just two of the applications that the Global Pulse team is exploring.
“Social media can give you a 'pulse' that you can then use to investigate deeper.”
UN Global Pulse
Although potentially not as accurate nor as detailed as traditional methods, digital data analysis can provide immediate feedback to those running development projects, allowing them to see what is working and tailor their approach accordingly, says Tatevossian.
"Social media can give you a 'pulse' that you can then use to investigate deeper," she adds.
Big data is a hot topic in the discussions on the post-2015 development agenda, with two influential reports — from the UN High Level Panel of Eminent Persons on the Post-2015 Development Agenda and the Sustainable Development Solutions Network — highlighting the need to use it more effectively.
This acceptance of the need for real-time information and feedback has seen a notable increase in interest from policymakers and development actors, says Tatevossian.
As well as a rise in visibility, a new Global Pulse lab is on its way.
A new centre in Kampala, Uganda, which is due to open its doors later this year, will join those already established in New York, United States, and Jakarta, Indonesia.
Having local labs to test-drive potential analysis tools is essential, says Tatevossian, as different factors, such as the type and scale of mobile phone and Internet use, determine the effectiveness of data mining in any given situation.
This variability in technology access means that analysis of big data cannot shed light on every situation, she says. But if applied intelligently to appropriate questions, it is a valuable tool, she adds.
This targeted approach is already showing signs of bearing fruit: the Indonesian food-price tracker is being used by the country's social development ministry to monitor commodity fluctuations.
But the field is still in its infancy, says Chris Danforth, an associate professor at the University of Vermont, United States, who has helped develop a tool for analysing Twitter users' emotional states.
The scientific community is still at the point of seeing what is possible, and does not yet have the evidence to make policy suggestions, he tells SciDev.Net.
Furthermore, solutions for issues regarding personal privacy and access to commercial data need to be solved if the full benefits of big data are to be realised, he adds.
Tatevossian is quick to acknowledge the importance of getting the private sector involved in "data philanthropy", but insists that the signs are promising.
Several large companies, such as telecoms giant Orange, have shown a willingness to provide access to their data banks to help development projects, she says.
See below for an animation about the UN Global Pulse initiative: