The open source development foundation Apache has announced the first major stable release of its much anticipated big data framework, Hadoop.

The Apache Software Foundation (ASF) – comprised entirely of volunteer developers – says the release of Apache Hadoop v1.0 “marks a major milestone six years in the making.”

Having steadily grown into one of the biggest buzzwords of 2011, ‘Hadoop’ is staking a claim early in 2012 to top the perch.

The framework, which is the first major release of its kind, allows users to process and perform analysis on vast and complex data sets by connecting thousands of different servers together to provide ample compute power to complete the tasks quickly and reliably.

Using applications built on the framework, users can work with exabytes of data. For perspective, one exabyte is a billion gigabytes – if stored on physical CDs it would fill 13,000 jumbo jets, according to Microsoft’s Simon May at the 5th Cloud Circle Forum last year.

As such, ASF styles Hadoop as ‘a foundation of cloud computing and at the epicentre of big data solutions’.

Arun C. Murthy, vice president of Apache Hadoop, says: “This release is the culmination of a lot of hard work and cooperation from a vibrant Apache community group of dedicated software developers and committers that has brought new levels of stability and production expertise to the Hadoop project.

“Hadoop is becoming the de facto data platform that enables organisations to store, process and query vast torrents of data, and the new release represents an important step forward in performance, stability and security.”

The companies already using beta – or test – versions of Hadoop (named after a stuffed toy elephant by its original concept developer) include Amazon Web Services, Apple, eBay, Facebook, The New York Times, Twitter, and Yahoo!, while Microsoft and IBM are among those to integrate the technology within their existing products. As an open source framework, they are free to do this without paying royalties to ASF as the developer.

While some of the techniques used in big data analysis have been around for some time, the amount of compute power and the complexity of the algorithms involved meant that it was unviable for the vast majority of companies and even most state bodies. But the advent of cloud computing and frameworks like Hadoop is opening up the possibilities for organisations of all sizes and sectors to spot trends hitherto not visible and to react to market conditions intelligently, in near real time.

In December 2011, The Cloud Circle’s newest sister community, the Big Data Insight Group, was launched. The aim of the new community – which has already signed up over 500 members including the likes of Deloitte, the Ministry of Defence, Royal Bank of Scotland, Tesco and Vodafone – is to provide practical strategic insight on how big data can be used within the enterprise.

The new community has a big data ‘think tank’ event scheduled for 23 February, followed by the release of our 1st Big Data Industry Trends Report in March and our 1st Big Data Forum in April.

If you have an interest in big data you can become a member for free*, gaining access to all of our events and thought leadership insight. Email charlotte.tite@thecloudcircle.com for more information.

*Pre-qualification applies