3 Trends that are Changing the World of Data

In my last post, I wrote about the new data integration requirements. In this post I wanted to share a few points made recently in a TDWI institute interview with SnapLogic founder and CEO Gaurav Dhillon when he was asked:

What are some of the most interesting trends you’re seeing in the BI, analytics, and data warehousing space?

There are three trends that I believe are fundamentally changing the world of data.

The first is the shift to the cloud. Rapid provisioning, ease of use, and cost are just a few of the drivers as data gravity continues to shift.

(Data gravity means that when a large data set is sitting in a large Hadoop or Splunk instance in an on-premises system, it doesn’t make sense to load all that data into the cloud to run analytics functions. Instead, one would ship the function to the data and return results. Being able to do this seamlessly can greatly simplify integration pipelines.)

Also driving this trend is the fact that cloud data warehousing and analytics have moved from rogue departmental use cases to enterprise deployments.

The second trend is the data lake and how to complement, extend — and in some cases replace — the traditional data warehouse with a reference architecture that is built to handle all new and future sources and enable more proactive and predictive analytics.

The third trend is the Internet of Things (IoT). It’s already happening today in some industries with data velocity, variety, and, of course, volume. What’s more important, though, is the kinds of analytics and insights that will become possible because of IoT sensors, wearables, and devices — once organizations figure out how to separate signal from noise through the right data management techniques.

He was also asked about the drivers for modern integration platforms, whether you call it iPaaS or HIP or something else:

1. The end of the ETL and ESB cycle
2. The need for speed
3. Hybrid infrastructure

Be sure to check out the entire interview and share your feedback.

Addressing the Last Mile – Middleware in Focus

Check out this post by Glenn Donovan: The History of Middleware. From the enterprise service bus (ESB) to the integration competency center (ICC), he has a lot to say about the good, bad and ugly of legacy enterprise application integration (EAI) and extract, transform and load (ETL) technologies. The central theme – it’s about the need for simplicity and speed, which has become even more critical for successful cloud integration deployments. Here are a few noteworthy observations in the post:

On the bureaucracy of the ICC:

“Sure you could build competency centers and “factories” but to this day, such approaches end up creating more bureaucracy, more dependencies and complexity while adding less and less value compared to what a developer can build him/herself with RESTful services/micro-services, and most things one wants to integrate with today already have well defined APIs so it’s often much easier to connect and share data anyway.”

On the ESB:

“Along the way, a problem became obvious. The cost, expertise, complexity and time involved in building such elegantly designed and governed systems frameworks ran counter to building systems fast. A good developer could get something done that worked and was high quality without resorting to using all those WS standardized services and conforming to its structure.”

On the need for a new enterprise platform to replace the legacy ESB:

“I think many are ready to dump all that highly complex and expensive overhead which came along with messaging buses when an enterprise class platform comes along that enables them to do so.”

Simplicity = IT agility:

“This is all coming together now, so you will see growing interest in throwing out the old integration server/message bus architectures in organizations focused on transformation and agility as core values.”

Check out the entire article to understand the author’s point of view. As a veteran of the middleware industry, he’s looking at modern integration platform as a service (iPaaS) vendors like SnapLogic as having: “the potential  help IT with the “last mile” of cloud build-out in the enterprise, not just due its features, but rather because of the shift in software engineering and design occurring that started in places like Google, Amazon and Netflix – and startups that couldn’t afford and “enterprise technology stack” – and is now making its way into the enterprise.”

[Infographic] Cloud Integration Drivers and Requirements

Here’s an an infographic from SnapLogic that reviews some the drivers and requirements for cloud-based integration, also known as Integration Platform as a Service (iPaaS), as well as reasons why legacy technologies won’t be able to keep up with the need for speed.

[Infographic] Cloud Integration Drivers and Requirements in 2015 from SnapLogic, Inc.

7 Things You Should Know About SnapLogic’s Elastic Integration Platform

In a series of recent posts on the SnapLogic blog, I’ve been reviewing the primary requirements of a modern integration platform. In this post I outlined some of the key principles behind SnapLogic’s Elastic Integration Platform, as well as the most popular posts on the blog. (Not surprisingly, 3 of the 5 most popular posts were written by the company’s Chief Scientist.)

The presentation below provides an overview of 7 things you should know about SnapLogic’s elastic integration platform as a service (iPaaS):

7 Things You Should Know About the SnapLogic Elastic Integration Platform from SnapLogic, Inc.

6 Requirements of a Modern Integration Platform

In 2009 I worked with data and application integration guru David Linthicum on a whitepaper called, “What to Look for When Evaluating Cloud Integration Solutions.” The 6 requirements were:

1. True multitenant versus hosted offering
2. Ease of use
3. Try and buy / rapid deployment
4. IT or LOB usability
5. Scalability
6. Vendor viability

While I don’t think I’d change this list too much in 2014, I’ve been putting together a series of posts on the SnapLogic blog summarizing the requirements of a modern integration platform. Now commonly known as integration platform as a service (iPaaS), the 6 primary requirements are:

1. Fully-Functional Cloud-based Service (based on a Software-Defined Architecture)
2. Single Platform for Big Data, Application and API Integration
3. Elastic Scale
4. Built on Modern Standards (REST, JSON)
5. Broad Cloud and On-Premises Connectivity
6. Self-Service for Citizen Integrators

Let me know if you agree or disagree with the list. I’ve embedded a demonstration of the SnapLogic Elastic Integration Platform below if this is an area of cloud computing that is new to you.

Cloud Application, Analytics and Integration Drivers and Barriers

Last week I hosted a webcast that reviewed the results from a recent SnapLogic TechValidate survey on cloud integration. You can download the complete results of the research here. The webcast also featured a deep dive demonstration of the SnapLogic Integration Cloud. I’ve embedded the recording below.

Why the ESB is the Wrong Approach to Cloud Integration

This week Maneesh Joshi from SnapLogic posted an article on Wired Insights called: Why Buses Don’t Fly in the Cloud: Thoughts on ESBs. It’s a pretty deep post, summarizing the vision of a services oriented architecture (SOA) and why the concept of the enterprise service bus (ESB) has reached its limits in the era of Social, Mobile, Analytics, Cloud and the internet of Things (SMACT). Here are a few snippets:

“Long implementation cycles, inability to absorb change, and high costs have made it difficult for these ESB solutions to keep up with fast evolving business requirements and often resulted in unmet expectations.”

“The ESB as an agile integration layer has been exposed as the long pole in project plans and customers are looking for alternatives.”

“On-premise ESBs or cloud integration platforms that are natively XML-based but apply translations to JSON at its extremities to keep up are going to fall short in the world of SMAC.”

“REST and JSON together are increasingly replacing SOAP and XML, making ESBs less relevant in today’s enterprise SMAC architecture.”

“Sticking with legacy technologies such as ESBs will only hamstring organizations from innovating rapidly and capitalizing on emerging opportunities.”

Do you agree? Do you see integration platform as a service (iPaaS) as complementary or a long-term replacement of the ESB as more and more of the applications and platforms are delivered in the cloud?

Better Business Analytics in the Cloud

I moderated a SnapLogic webinar today focused on the market shift to business intelligence, analytics and enterprise performance management delivered in the cloud. We discussed the trends, shared some industry analyst perspectives and demonstrated the importance of cloud integration to an overall business analytics as a service strategy. Here’s the presentation. You can watch the recording when it’s posted here. As always, feedback welcome.

Here are some of the links referenced in the presentation:

• What’s Driving the Shift to On-Demand BI
• Gartner predicts Business Intelligence and Analytics will Remain the Top Focus for CIOs through 2017
• Insights from Findings in Dresner Advisory Services’ 2013 Wisdom of Crowds Business Intelligence Market Study
• Big Business Intelligence Trends for 2014
• Petabyte Scale Data Warehousing with Amazon Redshift
• Business Intelligence Advances Into the Cloud with Some Caveats
• BI in the Cloud: Find Your Bliss
• SnapLogic Named EMA Vendor to Watch
• Ovum on the Radar: SnapLogic

Cloud Integration on the Whiteboard

Check out this chalk-talk series with the head of engineering at SnapLogic talking about application and data integration delivered as a cloud service (aka iPaaS):

Going Beyond Point-to-Point Cloud Integration

SnapLogic Integration Cloud Architecture in Review

The series of whiteboard presentations is posted on the SnapLogic blog.

Embedded Cloud Integration and the 3 V’s of SaaS Applications

I recorded an interview recently as part of the Cloud Channel Summit in Mountain View.  Industry analyst Jeff Kaplan from THINKstrategies posted his views on the event here and here.  In my interview I talk about the need for SaaS application and platform (PaaS) providers to either build, buy or partner when it comes to cloud integration. Everyone talks about the 3 V’s when it comes to Big Data. My belief is that IT organizations are equally overwhelmed with the volume, variety and velocity of SaaS applications coming into the enterprise.  This is why the topic of cloud integration will continue to it’s rise in relevance and strategic importance in 2013.

Related articles

• Platform as a Service emerges from shadows of SaaS and IaaS (zdnet.com)