Build Your Own Integration – Don’t Be Dumb!

Image representing ebizQ as depicted in CrunchBase

Image via CrunchBase

Recently Hollis Tibbetts has been writing extensively about all aspects of data integration (with a heavy does of cloud) on ebizQ. His most recent post pulls no punches:  Building Integration Yourself – Possibly the Dumbest Idea You’ve Had in a Long Time.

The article wraps up with a series of questions to consider before you jump into the “tarpit” of hand-coding your data integration. I suggest you pose these questions to anyone in your IT organization who tells you they’re “just going to write some scripts” or “simply develop Web  Services” when it comes to cloud integration:

1)  In the SaaS world, APIs are updated on average 4-12 times a year. What is the impact of that on your custom code? What if a document format (e.g. and EDI document) changes? Will you even know in advance of these changes, or will the change happen and suddenly your system stops working and you have a crisis on your hands?

2)  Are you prepared to handle latency and unavailability issues, timeouts, etc.?

3)  Have you budgeted for building a sufficiently robust logging system for errors, as well as for when data ends up somewhere it shouldn’t and you need to undo the situation?

4)  Can you guarantee that data won’t get lost when something “bad” happens?

5)  How will you monitor what’s going on in the system?

6)  Coding transformations and business logic in Java, C#, C++ or any other programming language is very time consuming. Transformations and business logic change a LOT. How will you support that? Most integration products support simple or standard scripting languages, drag and drop, reusable objects, etc.

7)  How do you plan to implement mapping – especially between something like a Web Service and a Relational Database (where one can be hierarchical in nature and the other a collection of tables). What happens when one of those things changes? Have you thought about transactionality and serializability? Do you need to support that?  How will you do it?

8)  Many applications require the use of proprietary SDKs for integration. Are you trained in those? Prepared to support changes in the SDKs?

9)  What levels of performance are required? How do you plan to meet those? What happens if that changes – is scalability built into your solution?

10)  If more sources or targets for integration are added, will your system support that, or did you build something that is a throwaway?

11)  Does your home-built system support concurrent development?

Great questions Hollis and a great article. I hope everyone considering hand-coding their data integration reads and shares it.

0 Responses to “Build Your Own Integration – Don’t Be Dumb!”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 7,151 other followers

Follow Me on Twitter


Sheffield View

Trying to keep up with Big Data Vendor Landscape

A Passion for Research

Focusing on CRM, cloud computing, ERP and enterprise software

SnapLogic Blog

Accelerate Your Integration. Accelerate Your Business.

Learning by Shipping

products, development, management...

Laurie McCabe's Blog

Perspectives on the SMB Technology Market

SaaStr 2018

Getting From $0 to $100m ARR Faster. With Less Stress. And More Success.

Andreessen Horowitz

Software Is Eating the World

%d bloggers like this: