Integration as an on-demand service
This blog is focused on cloud data and application integration, with a lot of posts about analytics along the way. This presentation introduces the key cloud monitoring requirements for delivering enterprise digital intelligence in the world of increased cloud adoption, microservices, containers, serverless architecture, and artificial intelligence. You can read the full article here.
As the data visualization, big data, Hadoop, Spark and self-service hype gives way to IoT, AI and Machine Learning, I dug up an old parody post on the business intelligence market circa 2007-2009 when cloud analytics was just a disruptive idea.
Here’s the post in it’s entirety (with apologies to Billy Joel). Lots of companies here that I had forgotten about. I’m sure Claudia Imhoff has their coffee mugs at the #BBBT headquarters in Boulder…
–––––
This week the very funny Bubble 2.0 video was removed from YouTube for copyright reasons. In honor of the video, which was based on Billy Joel’s We Didn’t Start the Fire, here’s my tribute to the business intelligence market. Note, I’m no poet and it’s Friday! Thanks to The OLAP Report for lots of great market materials. Anybody want to shoot a video?
“We Couldn’t Get the Answers”
HP Lune, EF Codd, Chris Date, Stonebrake
APL, Express, Forest and Trees.
Comshare, Pilot, Metaphor, watch out here comes some more:
OLAP, ROLAP, HOLAP, MOLAP now my head hurts.
Ingres, Sybase, DB2, Informix, Oracle, Tandem too
Relational, ODBMS, how about we call it DSS?
EIS, client server, bring on the Web and SQL Server
OLAP for the masses, gents? No, let’s call it business intelligence.
We couldn’t get the answers
The tools took too much learning
For simplicity we’re yearning
We couldn’t get the answers
IT couldn’t find them
But they kept on buying them
MetaCube, Lightship, Beacon, Pillar, Holos, I’m starting to weaken
OLAP Services, TM1, Pablo, Wired, and Crystal fun.
Zambonini, Liautaud, Michael Saylor still unknown
Codd again, Netz bros, Kimball, Inmon, then Arbor goes.
Ad hoc query, data mining, information I’m still not finding
Data warehouse, what the hell! Just don’t forget about ETL.
Balanced scorecards, GIS, analytic apps, extranets
BI portals, real-time, dashboards and visualization look fine.
We still couldn’t get the answers
The tools took too much learning
For simplicity we’re yearning
We couldn’t get the answers
IT couldn’t find them
But they kept on buying them
Panorama, Andyne, Pilot, WhiteLight what a find!
Maximal, Gentia, and Next Action Technologies.
Sterling, Platinum and IA eventually all go to CA
Showcase, SQRIBE all get imbibed and don’t forget OLAP@ Work.
Comshare, Brio, AlphaBlox, Crystal’s on the chopping block
Siebel goes to Oracle, would Hyperion be the next vendor culled?
BOBJ on a shopping spree, Acta, Cartesis, SRC
Gartner bandwagon of CPM leads Cognos to Adaytum.
We couldn’t get the answers
The tools took too much learning
For simplicity we’re yearning
We couldn’t get the answers
IT couldn’t find them
But they kept on buying them
SOA, MDM, running out of acronyms
ERP, FASMI, MDX, and BAM
Still searching for the one truth, data silos hard to remove
Consolidate, standardize, “trough of disillusionment” lies.
Competency centers, shelfware, disparate data everywhere
Pay and pay, day after day, there has to be a better way!
We couldn’t get the answers
The tools took too much learning
For simplicity we’re yearning
We couldn’t get the answers
IT couldn’t find them
But they kept on buying them
Oracle buys Hyperion, SAP is on the run
Outlooksoft and Pilot go, now they’ve joined Netweaver’s show.
Waldorf / Paris HQs unite, “business users” told to delight
IBM buys Cognos too, what’s next for the SAS and MicroStrategy crew?
Open source, on-demand, innovation from a brand new land
Infrastructure, integration, what happened to my business question?
Gartner quads, Forester Waves, so many analytic predictions made
On-premise platform wars, will the SaaS model even the score?
We couldn’t get the answers
Will this business challenge go on and on and on….
In my last post, I wrote about the new data integration requirements. In this post I wanted to share a few points made recently in a TDWI institute interview with SnapLogic founder and CEO Gaurav Dhillon when he was asked:
What are some of the most interesting trends you’re seeing in the BI, analytics, and data warehousing space?
There are three trends that I believe are fundamentally changing the world of data.
The first is the shift to the cloud. Rapid provisioning, ease of use, and cost are just a few of the drivers as data gravity continues to shift.
(Data gravity means that when a large data set is sitting in a large Hadoop or Splunk instance in an on-premises system, it doesn’t make sense to load all that data into the cloud to run analytics functions. Instead, one would ship the function to the data and return results. Being able to do this seamlessly can greatly simplify integration pipelines.)
Also driving this trend is the fact that cloud data warehousing and analytics have moved from rogue departmental use cases to enterprise deployments.
The second trend is the data lake and how to complement, extend — and in some cases replace — the traditional data warehouse with a reference architecture that is built to handle all new and future sources and enable more proactive and predictive analytics.
The third trend is the Internet of Things (IoT). It’s already happening today in some industries with data velocity, variety, and, of course, volume. What’s more important, though, is the kinds of analytics and insights that will become possible because of IoT sensors, wearables, and devices — once organizations figure out how to separate signal from noise through the right data management techniques.
He was also asked about the drivers for modern integration platforms, whether you call it iPaaS or HIP or something else:
Be sure to check out the entire interview and share your feedback.
This week SnapLogic posted a presentation of the 10 Modern Data Integration Platform Requirements on the company’s blog. They are:
I’ve embedded the presentation below.
SnapLogic published Eight Data Management Requirements for the Enterprise Data Lake. They are:
The post originally appeared on ITBusinessEdge as a slide show. The company also recently hosted a webinar on Democratizing the Data Lake with Constellation Research and published 2 whitepapers from Mark Madsen.
Last week SnapLogic posted the company’s mission on their blog, which is focused on accelerating how enterprise integration technology is delivered in the enterprise. The company is growing rapidly and hiring. Here is the their mission statement:
It’s time to rethink integration.
Integrating data and applications with legacy products is slowing down your business.
What you’re running now is no longer running fast enough for today’s business needs.
But what if you could eliminate the pain of integration and get to its promise?
To prioritize outcomes over process.
To enable your enterprise with a modern solution.
With SnapLogic you can.
SnapLogic brings all your data together, at incredible speeds and
with an ease never known before.
Data, applications, and APIs—from any source, anywhere.
Instead of coding, you merely drag and drop intelligent connectors in the cloud.
They fit together seamlessly.
In a snap.
You’ll connect faster and more efficiently, enabling you to become a more agile business.
Whether it’s maximizing investments in big data, or getting new products to market faster.
Whatever your enterprise strategy, SnapLogic gets you there easier and quicker.
And wasn’t that what integration was supposed to do in the first place?
Don’t let your legacy integration solution be your legacy.
It’s time to let go of those old ideas.
It’s time to operate at the speed of modern business.
You can learn more about SnapLogic on the company’s website.
Gaurav Dhillon, the co-founder and CEO of SnapLogic, who also co-founded Informatica in the early ’90’s and was CEO of that company for 12 years, posted 4 data predictions and 1 market prediction this week on LinkedIn and the SnapLogic blog. His predictions are as follows:
I shared some of my 2016 predictions in my previous post and summarized the most popular 2015 SnapLogic blog posts here.
Happy New Year!
I can’t help it. I enjoy the end of the year technology predictions, even though it’s hard to argue with this tweet from Merv Adrian:
By 2016, 99% of readers will be utterly sick of predictions.
— Merv Adrian (@merv) December 19, 2015
//platform.twitter.com/widgets.js
With that in mind, I recently participated in a SandHill.com Q&A on predictions and the year in review. Here are my answers and the links to the other responses on the Sand Hill site [updated on January 4th 2016 to add a new answer and more prediction links]:
Imagine that you have a time capsule to fill. What is the most important news item about a software company that occurred in 2015 that belongs in the capsule, and why?
The resurgence of Microsoft as a cloud company was big news in 2015. Under the leadership of Satya Nadella, the company has once again become relevant, despite a continued decline in PC sales. On its most recent earnings call, financial analysts noted that, “cloud is the epicenter of the growth story,” which represents the ultimate tipping point for enterprise cloud computing adoption.
Who was the biggest tech disruptor in 2015?
Microsoft’s resurgence is the biggest news of 2015, but Amazon, and specifically AWS, has done the most to disrupt traditional approaches to delivering (and purchasing) software in the enterprise. From AWS Aurora and Redshift for database management and data warehousing, to AWS GovCloud, which brings public cloud options to US government agencies, AWS continues to set the cloud computing standard for enterprise IT organizations and independent software vendors (ISVs).
Aside from the Internet of Things, which of the following software areas will experience the most change in 2016 – big data solutions, analytics, security, customer success/experience, sales & marketing approach or something else?
2016 will be the year of the data lake. It will surround and, in some cases, drown the data warehouse, and we’ll see significant technology innovations, methodologies and reference architectures that turn the promise into a reality. At the same time, big data solutions will mature (read: security, governance, metadata management) and go beyond the role of being primarily developer tools for highly skilled programmers.
In 2016, which software company will be the biggest game-changer for the long term?
From a cloud applications perspective, we continue to see Workday, ServiceNow and, of course, Salesforce expand their footprint and disrupt their markets. From a data management and analytics perspective, more and more Hadoop projects are moving into production with Cloudera, Hortonworks and MapR while also looking to Spark and Databricks, which are gaining significant mindshare for real-time, in-memory computing as well as cloud-based deployments. Meanwhile, Microsoft, AWS and Google continue to push the boundaries of the cloud, each now offering data management / cloud analytics solutions. With that as the background, it’s difficult to pick one software company as the biggest game-changer for the long term. On the other hand, I have to give credit to Amazon for continuing to change the game in each market it enters.
Which software company executive will be in the news headlines the most during 2016?
Does Elon Musk count? The Hyperloop concept is amazing.
Which software company will be the biggest rainmaker in attracting new customers in 2016?
Is Netflix considered a software company these days? What about Uber? Amazon Prime is taking over, but I’m not sure if that counts either. Software is truly eating the world, which is why it’s now easy to confuse a car manufacturer, a media company, a retailer and a ride-sharing service as software companies in 2016. It’s software plus business model disruption and the right mix of timing that will lead to massive customer attraction in 2016.
What are the biggest risks for software vendors in 2016?
With so much hype around the promise of new technologies and disruptive business models, the risk is that the so-called unicorns become ‘unicorpses.’ (I’ve also heard the term ‘dinocorns.’) New technology platforms must demonstrate a fast time to value and quantifiable ROI or they will no longer be relevant.”
What are the biggest risks for software buyers in 2016?
Complacency – thinking that you can continue to just work with the same handful of legacy software vendors that you’ve known for 10-20 years and hoping that they’ll be able to deliver the level of innovation and agility that newer technology companies and technology platforms were built to deliver. Digital transformation is at the top of the hype cycle, powered by the promise of cloud computing, big data and the Internet of Things; and software buyers must be open to new thinking, new approaches and new technologies.
Thanks to Sand Hill for the opportunity to participate in the series. Here are some of the predictions that I’ve enjoyed the most (so far) this holiday season:
Check out this post by Glenn Donovan: The History of Middleware. From the enterprise service bus (ESB) to the integration competency center (ICC), he has a lot to say about the good, bad and ugly of legacy enterprise application integration (EAI) and extract, transform and load (ETL) technologies. The central theme – it’s about the need for simplicity and speed, which has become even more critical for successful cloud integration deployments. Here are a few noteworthy observations in the post:
On the bureaucracy of the ICC:
“Sure you could build competency centers and “factories” but to this day, such approaches end up creating more bureaucracy, more dependencies and complexity while adding less and less value compared to what a developer can build him/herself with RESTful services/micro-services, and most things one wants to integrate with today already have well defined APIs so it’s often much easier to connect and share data anyway.”
On the ESB:
“Along the way, a problem became obvious. The cost, expertise, complexity and time involved in building such elegantly designed and governed systems frameworks ran counter to building systems fast. A good developer could get something done that worked and was high quality without resorting to using all those WS standardized services and conforming to its structure.”
On the need for a new enterprise platform to replace the legacy ESB:
“I think many are ready to dump all that highly complex and expensive overhead which came along with messaging buses when an enterprise class platform comes along that enables them to do so.”
Simplicity = IT agility:
“This is all coming together now, so you will see growing interest in throwing out the old integration server/message bus architectures in organizations focused on transformation and agility as core values.”
Check out the entire article to understand the author’s point of view. As a veteran of the middleware industry, he’s looking at modern integration platform as a service (iPaaS) vendors like SnapLogic as having: “the potential help IT with the “last mile” of cloud build-out in the enterprise, not just due its features, but rather because of the shift in software engineering and design occurring that started in places like Google, Amazon and Netflix – and startups that couldn’t afford and “enterprise technology stack” – and is now making its way into the enterprise.”
I recently participated in an online web conference called Cloudcon 2015: Integration and Web APIs where I reviewed 5 signs you need better cloud integration:
It’s a topic I wrote about in this blog post last year. Here’s the recording of the presentation, which also includes an overview of the SnapLogic Elastic Integration Platform and a Q&A session with Vance McCarthy from Integration Developer News.
In(tegrate) the Clouds RSSDefine the Future of Data Management
I like punk rock and storage arrays
Trying to keep up with Big Data Vendor Landscape
Focusing on AI & Machine Learning's Impact On The Future Of Enterprise Software
Accelerate Your Integration. Accelerate Your Business.
products, development, management...
Perspectives on the SMB Technology Market
In(tegrate) the Clouds 