Data Integration - Is the Future Pain Free?

Simon Kennedy, CIO, The Warehouse Group

Simon Kennedy, CIO, The Warehouse Group

"Good things don’t come easily” is a pretty reliable truism. It’s hard to think of many exceptions and it would certainly extend to include the field of data integration. For now at least.

Data integration is hard. Someone who tells you different is most likely selling you a solution or hasn’t understood your problem. First, there are technical aspects to get right – meta data, virtualised schemas, synchronization, and so forth. And once you have all of that squared away then you can wrestle with the real-world problems relating to data governance, completeness, accuracy, privacy, compliance and ethics. Fun times indeed.

The technologies that support data integration are undoubtedly getting better, and rapidly so. I suspect they are still at what I would describe as the ‘fax machine’ level of maturity though. Remember how we used to love fax machines? No? Me neither. They were clunky, difficult to operate and unreliable to the point of displaying almost human-like petulance. Even though they ruled the roost as the de facto standard for transmitting documents and even though at that time there was no superior commercial alternative, you could sense they were not a very polished solution to the need.

"The nuts and bolts of data integration take such effort to execute that we can lose sight of the business outcome we are trying to achieve "

Today’s data integration technologies do not have the petulance of fax machines and my point is not to downplay some very impressive solutions and capabilities. However, data integration is, at this point of time, is undoubtedly clunky and difficult to operate. You can’t help thinking that given a few more iterations of evolutionary improvement, the technology several years from now will make today’s solutions appear steam-age by comparison.

And what if they do? What then? There’s a scene in the film Apollo 13 in which the supporting ground-crew need to devise a procedure to enable the stricken astronauts to rig up an air vent to give them any chance of a return to Earth. Who knows how close to reality this scene was, but it made for great cinema. Assorted items – anything and everything that could be found on the spacecraft - are dumped onto a table in front of the support crew. The instruction is clear: “Your job is to find way of using THIS, to achieve THAT”. The task appears hopelessly complex, but sure enough the rocket scientists find a solution and get their men home.

Data integration can feel the same - somewhat complex and all a bit rocket science. The compounding problem is that we very often don’t know what the “THAT” is. The nuts and bolts of data integration takes such effort to execute that we can lose sight of the business outcome we are trying to achieve. Or worse, we can fail to identify any business outcome at all, meaning that the data integration mission becomes an end in itself.

My expectation is that this problem will become less of an issue as data integration technologies mature, as surely they will, to the point where the mechanics of plugging tab A into slot B become effectively effortless. Then we can put all the focus where it is most valuable – defining the ‘THAT’ which will drive breakthrough value.

Until then, while it will continue to be hard work, it won’t be a wasted effort. The value of data integration – and of bringing together the science, art and creativity of data disciplines – is a certainty. Putting in the hard yards now, even if it turns out to be inefficient by future yardsticks, is an investment in achieving a flow state that will yield long term returns. Which is good, so… You know the rest.

Established in 1982, The Warehouse is a general merchandise retailer, known for wide range of products from clothing, entertainment, technology and music to sporting, gardening, and many others.