By Charles Moreira
“Big data” – or huge volumes of unstructured data, including words and sentences, documents, GPS location and other telemetry data, photos, graphics and videos, is one of the latest buzzwords in enterprise IT today.
Unlike traditional relational database systems, where data is organised in a highly structured manner into fields, records and files which are relatively easy to store, index and query, big data requires very much more powerful processors and more complex algorithms to analyse, and lots more storage, especially for bulky pictures and videos.
Yet it must be able to produce analyses results within an actionable timeframe, or almost immediately, and not have users wait a month for information services to produce a report.
Extraordinary revolution
While perhaps still not quite understood by many, big data represents “an extraordinary knowledge revolution that is sweeping almost invisibly, through business, academia, government, healthcare and everyday life,” according to Rick Smolan, co-founder and chief executive officer of Against All Odds Productions, a company which specialises in the design and execution of large-scale, crowd-sourced global projects which combine compelling story telling with state-of-the-art technology.
Smolan and his partner Jennifer Erwitt kicked off their global The Human Face of Big Data project to help raise public awareness of big data.
It began on 12 March, 2012 with the dispatch over 100 photographers, editors and writers worldwide to explore the world of big data to determine if it truly represents “a never before seen toolset for humanity, which helps us address the biggest challenges facing our species,” as claimed by many of its proponents in the industry.
On 26 September, Smolan made a free Android mobile application available for the public to download from the project’s website at http://thehumanfaceofbigdata.com, followed by an Apple iOS app a few days later. They are available in English, Mandarin (simplified), Japanese, Portuguese, Spanish, French, Korean and Russian.
The app recorded passive data using the phones’ sensors throughout the day, such as how far and how fast their users travelled, or which time of the day they were most active. It also gather more subjective data such as their users’ answers to questions about themselves, their families, trust and safety, their sleep and dreams as well as dating and sex life.
It also lets them submit photos, maps of their daily travel, share their lucky rituals, share their daily travel patterns on Facebook and Twitter, the one thing they would like to do during their lifetimes and to find their “Data Doppelgänger” – i.e. a fellow participant who is their closest match in appearance, age, lifestyle, opinions, interests, likes, dislikes and so on.
And the results
Then on 2 October, simultaneous Mission Control events were held in Singapore, London and New York City to provide members of selected media with analyses, visualisation and interpretation of the data gathered.
These massive volumes of unstructured data were anonymised to protect users and will be made available to researchers, data scientists and educators to study as a snapshot of a week in the life of humanity in 2012. Enterprise IT News covered the Mission Control event in Singapore.
The event included practical demonstrations of how big data can transform various areas of our lives and the kinds of information which can be derived from it, though not all of them were based upon data from The Human Face of Big Data project.
 |
| Tapping into the Twitter Firehose |
On how it transforms public safety, crime, city life and government, the input from over a billion tweets between 28 August and 6 September were fed from Twitter Firehose into an EMC Greenplum Unified Analytics Platform and were used to produce a chord diagram showing different types of interest in different parts of the world.
In the run-up to the U.S. presidential elections in November, the tweets rated Obama’s popularity at 61% versus Romney at 31%, but while Obama leads in the U.S. Romney is more popular in France. In the U.S. Romney also leads Obama 57% to 42% in finance and commerce.
On safety and crime, the analysis also showed the safest and worst cities in the world, and based upon key words, can also show public sentiment and even predict the possibility of a riot breaking out.
 |
| Changes in sentiment over time |
 |
| Taking charge of how we eat |
Also based upon the input from Twitter Firehose, the analysis showed changing public confidence in business in the U.S., wit negative sentiment growing due to political campaigning.
Another application lets users view societal attitudes by gender on charts based upon several questions. For example, more women objected to their spouse having a side affair than men were. The analysis also found that people who had liberal parents tended to feel safer at home, while those who had strict parents tended to have a more negative view of life.
In health, wellness and the environment, the analysis found more people wanted to take charge of their lives, with lots of interest in eating local produce around the time of the Global Food Summit on 09 April.
Meanwhile, Kristian Kloeckl, Singapore-MIT Alliance for Research and Technology (SMART) research scientist and Real Time City Group Lead, Senseable City Lab, told the event how big data is used in Singapore to explore the connection between people, digital technology and the city in real time to help public decision makers.
One of its applications accessible on multi-touch tables lets users monitor various bus routes, traffic conditions, number of passengers, where they board and where they alight, at different times of the day to better understand bus usage.
In an earlier instance, SMART senior research engineer Oliver Senn spend five months working on a joint initiative to provide real-time data and insights to citizens to help them improve their city.
During his first assignment, Senn compared two months of weather satellite data with 830 million GPS records of 80 million taxi trips, he discovered from GPS records that many taxis in a fleet of 16,000 were stationary during rainstorms.
Since it’s known that taxis in Singapore are hard to come by during rainstorms, he interviewed the drivers and learned that the company which owned most of these taxis would withhold Sin$1,000 of their pay immediately after they had an accident until it could determine who was at fault – a process which took months to resolve – so to avoid the risk of having an accident, the drivers pulled over until the rain stopped. His discovery, based on analysis of big data, enabled the resolution of this problem affecting the public, the drivers and the taxi company.
Meanwhile, according to Clive Gold, EMC Australia marketing CTO for Australia and New Zealand, the kind of quick feedback such as provided by big data analytics results in more effective response.
“For example, it was found that electricity meters which immediate consumption resulted in lower usage,” he said.
Also, more people now realise that leaving home electronics and appliances on standby consumes about 20% of their power when fully on, so are now switching them off. Also, while average power consumption in Australia is reducing, hence lower revenue for power companies; peak power is rising, which results in increased cost, so big data can help power companies better cope with these two contradictory trends.
Meanwhile, police in the U.S. use big data for face and number plate recognition and are able to track people from when they enter an airport, till when they leave.
While most of the above examples are based on unstructured textual data, Gold admits that working with images is a bigger challenge for big data analysis but still do-able.
Moving on
The Human Face of Big Data’s next stop is Big Data Detectives on 8 November in association with TED Youth. Here, thousands of students worldwide aged 6 to 12 and their teachers engage in a series of data collection activities to analyse and map their worlds.
The project’s highlight will be the unveiling on 20 November of The Human Face of Big Data large format book, illustrated by a fascinating array of over 200 photographs, compelling essays by noted writers and dramatic info-graphics from renowned designer Nigel Holmes.
Copies of the book will be delivered that same day to 10,000 of the world’s most influential people, including world leaders, CEOs of Fortune 500 companies, and Nobel laureates across 30 countries. It will also be available in bookstores and as an e-book on the Apple iPad.
A documentary on The Human Face of Big Data will be released in 2013.
Big brother is watching you?
While, there already are concerns that big data analytics can result in a “big brother” kind of public surveillance, Smolan is confident that it won’t be a threat if handled well.
“Like all new tools, it (big data) carries the potential for unintended consequences. But if we are careful and wise, in the not too distant future this new set of technologies may have an impact on humanity as great as those of language and art,” Smolan said.
The editorially independent project was supported by EMC Corporation as main sponsor, with supporting sponsorship by Cisco, VMware, Tableau and Originate.
EMC can help enterprise customers with an end-to-end infrastructure to support all aspects of big data, including its EMC Isilon scale-out NAS (network-attached storage) and EMC Atmos Cloud Storage, EMC Greenplum data warehousing and analytics platform and the EMC Documentum xCP actionable insights, while it can also help ensure security and good governance of big data with RSA Security, Compliance and Risk Management.
(Enterprise IT News was in Singapore as EMC’s guest to this event)
