Image via CrunchBase
Recently Hollis Tibbetts has been writing extensively about all aspects of data integration (with a heavy does of cloud) on ebizQ. His most recent post pulls no punches: Building Integration Yourself – Possibly the Dumbest Idea You’ve Had in a Long Time.
The article wraps up with a series of questions to consider before you jump into the “tarpit” of hand-coding your data integration. I suggest you pose these questions to anyone in your IT organization who tells you they’re “just going to write some scripts” or “simply develop Web Services” when it comes to cloud integration:
1) In the SaaS world, APIs are updated on average 4-12 times a year. What is the impact of that on your custom code? What if a document format (e.g. and EDI document) changes? Will you even know in advance of these changes, or will the change happen and suddenly your system stops working and you have a crisis on your hands?
2) Are you prepared to handle latency and unavailability issues, timeouts, etc.?
3) Have you budgeted for building a sufficiently robust logging system for errors, as well as for when data ends up somewhere it shouldn’t and you need to undo the situation?
4) Can you guarantee that data won’t get lost when something “bad” happens?
5) How will you monitor what’s going on in the system?
6) Coding transformations and business logic in Java, C#, C++ or any other programming language is very time consuming. Transformations and business logic change a LOT. How will you support that? Most integration products support simple or standard scripting languages, drag and drop, reusable objects, etc.
7) How do you plan to implement mapping – especially between something like a Web Service and a Relational Database (where one can be hierarchical in nature and the other a collection of tables). What happens when one of those things changes? Have you thought about transactionality and serializability? Do you need to support that? How will you do it?
8) Many applications require the use of proprietary SDKs for integration. Are you trained in those? Prepared to support changes in the SDKs?
9) What levels of performance are required? How do you plan to meet those? What happens if that changes – is scalability built into your solution?
10) If more sources or targets for integration are added, will your system support that, or did you build something that is a throwaway?
11) Does your home-built system support concurrent development?
Great questions Hollis and a great article. I hope everyone considering hand-coding their data integration reads and shares it.
Recent Comments