It has been some time that I blogged and the reason is simple: there was too much going on. If you follow the innovations of the last years one can feel that the Singularity is near – as Kurzweil proclaimed. In my early childhood days a revolutionary innovation was one a decade and now it seems like every week there is something disruptive.
Different strains of innovation influence each other and lead to the exponential curve of innovation. Especially IT is a driver for other branches and enable totally new business models and corresponding new data architectures.
Kevin Kelly, founder of Wired Magazine once said:
Singularity is the point at which "all the change in the last million years will be superseded by the change in the next five minutes."
I personally feel this status is not very far away and I totally agree with Nadella (CEO Microsoft) saying:
„Every industry is now a software industry where they are building these systems of intelligence… provide SaaS services that go along with your products. “
Following the Microsoft way of partnering with Hortonworks to adopt the power of Hadoop and the Hortonworks Data Platform (Microsoft enriches HDP and labels it HDInsight) also ORAYLIS made this major step forward. ORAYLIS is now a System Integrator Partner with Hortonworks and has certified Hadoop Developers (me included), and has concepts for the Modern Data Architecture which brings together the classic Business Intelligence Stack with the power of Hadoop to harnish the Big Data challenge and lead our customers to their Big Business opportunities.
So let’s review my 3 days in Brüssel with the PreCon Day for Hortonworks partners exclusively and the two official Hadoop Summit days.
Hortonworks did a great job with following the 100% Open Source Track (in comparison to e.g. Cloudera) and is now in the emerging Open Data Platform Group a key player along with 19 companies like IBM, Pivotal or SAP.
With ODP the different distributions will have a common core, which means as a customer you’re not stuck with the platform decision. Workloads can easily be transferred and once verified run on any ODP Platform.
Hortonworks bundled the best projects of Apache Hadoop and it’s ecosystem to an "Enterprise Ready" Distribution. The major feature updates of the last year have been amazing including Data Governance with Falcon (and Atlas) or Data Security with Knox and Ranger and of course the most important enhancement: YARN – which can be described as the Data Operating System. With YARN you can not only do the classic MapReduce Job but Real-Time or Interactive Applications all parallel on the the same data in Hadoop.
One of the killer-apps on Hadoop is: SQL. No need for Java MapReduce coding anymore, just use e.g. HQL an SQL adaption in HIVE on Hadoop or Phoenix on Hbase or Kylin for SQL/OLAP.
The SQL approach is fast evolving in several ecosystem projects and is leveraging the adoption in companies, because the same skill sets as with working with RDBMS can now be applied when working with Hadoop.
While 2014 was the year of adding lots of functionality to the Hortonworks Data Platform (HDP) 2015 is the year of the simple product experience, expect more Uis and easy workload transfer between onprem and cloud.
Also remarkable is the clear vision of Hortonworks not only regaring Hadoop and the Plattform but clear visions for each vertical including a business case roadmap for each industry.
Microsoft was prominent at the Summit too – of course. With Microsoft enhancement of HDP to HDInsight users not only can embed Hadoop in the APS Appliance and query both sides (relational/Hadoop) with one SQL and join the two sides in one result set. Polybase does the trick and will hopefully be in the future not only available in the APS Appliance but in every SQL Server. With Microsoft Azure a setup of Hadoop has never been easier: Just type in a number and some minutes later you have a cluster up and running with the specified number of nodes. Hadoop as a Service is key in PoCs or for cloud born data. Microsoft added the possibilities to program your MapReduce Code in C# or LINQ as an example.
Out of the partnership between Hortonworks and Microsoft resulted the first Hadoop version which also runs on the Windows Operating System. You can download the Windows Version from the Hortonworks Website or get an HDInsight deployment by Microsoft. "Hadoop anywhere"
I hope to elaborate on details within the next days, but I want to close this post with one key take away, resulting from the talks of customers who already implement Hadoop as an extension of their classic data plattform. Those customers are British Airways, Dailmer, KPN, or ING.
The first benefit of enhancing the data platform with Hadoop is no use case, but cost saving. The 100% open source Hadoop or ODP compliant distributions like HDP won’t cost any licence fee and can run on commodity hardware. It is difficult to find the customer individual use case in advance to implementing Hadoop. It definitely helps to have ideas about potential use cases, but the common lessons learned of all those companies was: once you integrate all your data and potentially new data sources like Internet of Thing devices, Web-Content like Social Media and equivalent – you see patterns and use cases you couldn’t have thought about in advance. So if you like to be future ready develop a vision and just start. The ROI argumentation can be done by the cost saving factor, all your future use cases and potentially new businesses can hardly be measured in advance. Quote: "the engineer is the marketeer of the future"
Or as the Microsoft’s Vice President: T. K. “Ranga” Rengarajan expressed in his talk "Changing the Art of the Possible With Big Data": “It’s time to dream.”