Data.

"Big data is the new business black" ... enterprises will invest more than $US120 billion by 2015 in data capture and analysis across hardware, software and services. Photo: istock
Customer information is suddenly worth billions.
There's another mining boom you may have missed. It too involves paying young people six-figure salaries in their first jobs, and exploring deeper for resources which may have been previously overlooked. But it's not about driving trucks or digging holes. It's about building algorithms and crunching facts and numbers. It's mining for data.

Big data is the new business black. It's a catch-all phrase for the billions of transactions and other bits of information about their customers, suppliers and operations logged by businesses and governments the world over every day. Yesterday's storage problem has become today's strategic asset. Turns out there's gold in them thar files.

''This is the biggest industry that people are only now starting to talk about,'' says Anthony Goldbloom, a 28-year-old former Reserve Bank of Australia statistician who has moved his start-up data analytics business, Kaggle, to Silicon Valley where NASA is among its clients.
''The whole place is big data mad,'' he says. ''Industries like banking, insurance, and increasingly pharmaceuticals are competing on the back of predictive models that get built [by mining data].''
Enterprises are using data analysis not just to improve their everyday business processes, but also to build predictive models of consumer behaviour. Retailers, telcos, airlines, hotels, health care and credit card companies are among those with information-rich customer data. In Australia, ''only really leading companies have realised this as an opportunity'', Goldbloom says. To his knowledge, Telstra, Myer, the University of Melbourne and the NSW Roads and Maritime Services are among those known to have applied large-scale data analysis to their operations.

A 2011 report by McKinsey and Company said using big data to drive efficiency and quality in the US health care industry could create $US300 billion value a year, including cost savings. Using personal location data to sell services to individuals could ''capture $US600 billion in consumer surplus'', it said.

Global market intelligence firm IDC has estimated enterprises will invest more than $US120 billion by 2015 in data capture and analysis across hardware, software and services. IBM is investing heavily, now boasting almost 9000 ''business analytics and optimisation consultants'', 400 researchers and a global network of ''analytics solutions centres'', aiming for $16 billion in business analytics revenue by 2015.

''There is no end in sight. I think it is an endless opportunity,'' the company's chief data analytics guru, Jeff Jonas, says. Among other things, he is working on what might best be described as a digital concierge service. Customers who opt to give the service access to their personal information - such as one or more of their social media accounts and their digital personal organiser - would receive timely and, according to Jonas, helpful prompts to guide their actions and purchases.

Imagine, Jonas says. ''It says, 'I can see from your Facebook that you like Metallica, and I can see you will be in New York next week, and Metallica is there on the same night. Would you like to get a ticket?'

''You could subscribe to a service that would be so precise in its prescriptions that you could feel like it saves you time and money,'' says Jonas, IBM Fellow and chief scientist, IBM Entity Analytics. He acknowledges people's possible discomfort at the idea of unsolicited advertisements, but insists users of the service would come to see it as ''friendly'' and ''genius''.

Another company capitalising on the growing expanse of data, and one of IBM's rivals, is the Silicon Valley data-sorting firm Tibco.

The company's chief technology officer, Matt Quinn, says Tibco's products aim to give clients the right information, at the right time, in the right place and in the right context.

One of its more unusual client groups, casino operators, use the company's software to gauge gambler satisfaction. The system collects data each time a gambler uses their casino loyalty card - be it for gambling, purchasing food in the restaurant or paying for snacks from their room minibar - to create a pattern of their behaviour.

The system can see, for example, that the last few times Gambler X visited the casino, they stayed for two days and lost between $200 and $300, then promptly left the gaming floor, spent no more money and went home the next day.

''As a casino operator you don't want that, you want to make people happy and spend more money,'' Quinn says.
Enter Tibco's event processing software. When the system detects the customer is falling into a particular pattern, such as the consistent losing streak that caused them to leave during their last visit, it sends an automatic note to a gaming floor attendant to offer the person a free meal, or ticket to a show.

The idea is to distract the gambler long enough that they'll come back later and continue to play and lose money, albeit in more palatable amounts. While no Australian casinos use Tibco's products, they are beginning to show interest, Quinn says.

Department stores, whose loyalty programs amass gigabytes of data on their shoppers' every purchase, have also begun using Tibco's event processing software to model their customer's spending patterns and predict their next buy.

Quinn says the software would pick up that every year Customer A spent a fair bit of money around the beginning of October, which could suggest they had a few birthdays around that time.
This would be a perfect time for the store to offer this customer a discount or voucher, a more effective and targeted marketing strategy than TV advertising and catalogues, he says.
''The systems we're building are helping [companies] better understand who their customers are.''
Tibco products also analyse data to keep customers loyal.

In late 2010, Tibco bought the technology startup Loyalty Lab, whose software platform specifies the best time for a company to engage customers in their loyalty program based on data it has collected on previous customer interactions.
''It's technology and psychology rolled into one,'' Quinn says.
Trained loyalty scientists build systems that trawl through a company's customer behaviour data to predict certain events, such as what might prompt them to leave.

Loyalty Lab's director of global solutions strategy, Michael Greenberg, says for telcos, for example, there are several events, such as phone calls dropping out, the end of a contract, or more than two complaints, that suggest a customer may be getting ready to take their business elsewhere.
That is when Tibco advises companies make contact with these customers with an offer, he says.

While big businesses make up the bulk of Tibco's clients, Nasdaq, the US Major League Baseball, the Department of Homeland Security and major hospitals also use the company's data-sorting software.
In 2010, scientists at the Walter and Eliza Hall Institute of Medical Research in Melbourne began using Tibco's visualisation software, Spotfire, to help find new drug therapies for diseases like cancer and malaria.

To locate potential drug candidates, the researchers run tests on tens of thousands of compounds each day, generating hundreds of thousands of data points for each potential therapy.
The co-ordinator of the the institute's high-throughput laboratory, biochemist Kurt Lackovic, says the unit is capable of running 80,000 reactions a day.
Spotfire takes the results of each reaction and presents them in diagrams, making it easier for researchers to spot trends, outliers and errors rather than searching for a needle in a rapidly expanding digital haystack.

Hype notwithstanding, big data will turn out to be a useful value adding tool rather than a ''panacea to solve everything'', says Robert Hillard, Deloitte Consulting technology lead partner.
The privacy issues raised by its use have yet to be resolved, he says. It is crucial people are given the choice between opting in and opting out of the use of their identifiable personal data, he says. Companies approaching customers need to be ''utterly transparent'' about how they know what they know about a customer.

IBM's Jonas, whose personal motto on privacy issues is ''don't surprise the customer'', says it's important for people to realise how commercially powerful the data which locates them in time and space is, such as that communicated by GPS devices in cars and mobile phones. People are putting more data online about themselves than ever before, yet when signing up for the latest ''irresistible'' service, few bother to read the terms of use, he says.