Today at the Spark Summit in San Francisco I announced the latest milestone in our initiative with Apache Hadoop* and Apache Spark* communities and their ecosystems to accelerate Big Data analytics. Moore’s Law has driven us to a tipping point. Computing technology has become so small and affordable that one can integrate intelligence into anything from bracelets to building A/C units. At the same time, we are transforming the datacenter bring pervasive analytics and insights to our emerging digital service economy. There is more data than ever before, and as we build out the Internet of Things – a global network of devices and machines – our opportunity to sense, understand, and react to our world in motion is exploding.
Across a number of industries, demand for real-time, streaming analytics has greatly increased. To maximize results and competitiveness, enterprises want to visualize and monitor their business in real-time and take immediate actions. They want to listen to reactions in social media to better understand their consumers’ points of view. Governments want to make cities smarter, air cleaner, and traffic smoother. And businesses and governments alike need to identify and contain malware and cyberattacks before the damage is done.
This is why I announced the open source release of Streaming SQL for Apache Spark. These usage models require data to be analyzed in the moment, as it streams in. Intel Software is investing resources in Apache Spark as part of a larger effort to develop a complete open-source framework for streaming analytics and make these capabilities pervasive. Apache Spark provides in-memory cluster computing that is well suited to extracting insights from streaming data.
One of the key challenges we are addressing is the time it takes for businesses to achieve these insights. In the case of Spark, we have found that one barrier to adoption is the need for special programming skills. According to KDnuggets, which recently surveyed 3,000 members of the data mining community on 93 different tools, SQL continues to be very popular. It ranked as the #3 software tool, used by more than 30% of respondents. Also according to a 2015 survey by Stack Overflow , 48% of developers know SQL – the second most popular coding language after javascript. Java was 3rd at 37.4%. Scala didn’t make the list. The challenge is that SQL-savvy data analysts who don’t know Java* or Scala* need to hire developers or purchase proprietary software to leverage Spark.
Streaming SQL for Apache Spark is a query language which makes it easy and efficient for data analysts to write stream processing applications. There is no need to learn new programming language nor to rely on infrastructure developers for streaming applications.
For example, we recently provided this capability to JD.com. JD.com is a Chinese electronic commerce company headquartered in Beijing. It is one of the largest online retailers in China by transaction volume. They want to provide the real-time statistics as a service for JD.com’s Cloud service. Their SQL-savvy analyst had to go through the cumbersome of working with infrastructure developers to write a streaming application in Scala or Java and store the data into a database. Only then could the analyst query data and develop business logic. This process took days or weeks. The solution was for the infrastructure team to provide the analyst with a direct SQL interface using Streaming SQL for Apache Spark. This enabled the analyst to new applications in just a few hours. “It sometimes took weeks to write new business applications. It was a cumbersome process involving two different teams to analyze real time streaming data, ” Said Xiaohui, Lia, Senior Development Manager. “With Streaming SQL for Apache Spark, our Analysts were able to develop business applications in a matter of hours – a huge boost to productivity.”
Our overarching goal is to make streaming analytics easy and efficient for data analysts everywhere. We want to see businesses, governments, and even consumers to be able to react quickly and intelligently to events as they happen. As intelligent devices begin to pervade our world, smart objects and digital assistants will also rely on these streaming capabilities to help you find the things you need, discover things that you want, and live a life that more productive and stress-free. We strive to make this happen by giving software developers the tools they need to create.
*Other names and brands may be claimed as the property of others.
No computer system can be absolutely secure. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.
The post Real-time Intelligence: Accelerating Time-to-Insight appeared first on Intel Software and Services.
