Solving your application and data integration challenges

Because of the poor economy and corporate demands to increase revenue while cutting costs, business users are asking for more information now than ever before, according to Rick Sherman, founder of Athena IT Solutions. The benefits of making more data available via a strong integration process are obvious, Sherman says, but many companies continue to struggle to solve their integration problems.

In this video interview, Sherman discusses the pros and cons of different integration technologies as well as service-oriented architectures (SOA) and enterprise resource planning (ERP) software, and gives his take on the most successful integration tools for businesses. He also discusses how to overcome application and data integration challenges, build a business case for integration projects and show return on investment (ROI) on your integration implementations.

In this 11-minute videocast, viewers will learn:

  • Why application and data integration challenges are so prevalent in companies today, and the benefits of successful integration implementations
  • Why technologies such as SOA and ERP have failed to deliver on their promises of integration harmony
  • The types of data and application integration technologies that are successful and making a difference in businesses
  • Why hand-coding is a less desirable approach than using packaged extract, transform and load (ETL) tools
  • How technical and IT professionals should tackle integration challenges and problems
  • How to build a business case and show ROI for data and application integration projects

Related resources on data integration and application integration:

  • Make sure you've brushed up on the top eight integration definitions and buzzwords
  • Learn how efficiencies can be gained at the intersection of middleware and data integration
  • See what upstarts are challenging Informatica and IBM for the top spots in the data integration market
  • Learn why interest in open source data integration is growing
  • Find out how data integration for SaaS BI software is evolving

Read the full text transcript from this video below. Please note the full transcript is for reference only and may include limited inaccuracies. To suggest a transcript correction, contact [email protected].    

Solving your application and data integration challenges

Hannah Smalltree: Welcome to a video cast, about data integration and solving the integration challenge. My name is Hannah Smalltree. I'm the editorial director for TechTarget's Enterprise Application's Media Group. With me today is Rick Sherman, the founder of Athena IT Solutions, a Boston based independent consulting firm. Now Rick has over 20 years of information management and business intelligence experience, and he's a popular speaker, writer and industry expert. Thank you so much for being with me here today Rick.

Rick Sherman: You're welcome Hannah.

Hannah Smalltree: Now Rick, this is a little bit of a slow pitch I know, but why is integration such a huge challenge for companies today, and what's the payoff for getting it right?

Rick Sherman: Well in this day and age, in this economy, business users are demanding information. We have government regulations, we have government and industry pressures that are causing a greater demand for information, and quite simply a lot of us are walking around with Blackberry's and iPhones in our belt. We're used to being able to get information instantaneously about everything. Then the business folks who had the Blackberry's go to their desk and can't find out what the sales numbers were for last week. So there's a huge pent up demand for more information in our society, in particular in business today. The payoff certainly is to better run the company. Right now there's a lot of cost cutting involved, but there's also a lot of looking for more revenue. You can't do that unless you know what you're selling, how you're selling it, and who you're selling it to. It's sort of obvious why there's an information demand. The good news is that in the last ten years most of the work I've been involved in is being generated by business people, as opposed to IT folks. So that tells a story about the fact that business is demanding the information.

Hannah Smalltree: Now whatever happened to the promise of ERP and more recently SOA's? Weren't those supposed to solve all of our integration problems?

Rick Sherman: I've heard that story through the 90's with enterprise resource planning systems and then earlier in this decade with SOA being the magic bullet. Everybody kept hoping that somebody else would solve their data integration problems. The issue came with ERP systems is that almost nobody implements just one. They have multiple enterprise applications, and most of the large enterprise application vendors have bought other companies and put them into their products. So their product suites, despite what they say aren't really integrated either. So you have companies that have dozens of enterprise applications, homegrown, and built. SOA is a great technology, absolutely used in a lot of the BI, business intelligence and ETL technology out there. But all it is is just a way to communicate data to move back and forth. That's sort of a data access layer. What our issue is, is data integration, and you don't just do that by moving the data. You have to get the data consistent, work on data quality, all those little issues that take the hard work.

Hannah Smalltree: So what kind of technologies are really making a difference in how companies can solve their integration challenges?

Rick Sherman: The key one, the bedrock of this is really the data integration tools, the extract, transform and load tools. They've taken what a lot of people had been doing, hand coding and in some cases unfortunately still do, and being able to code it systemically with tools. I teach at a local engineering college in a master's program. What I tell the students is ten years ago you picked up these ETL tools, these basic tools, and all they really did was automate what you coded in SQL. Nowadays there are a lot of different best practices in data integration. Things that the people listening probably won't understand, but slowly changing dimensions, change data into capture. Once you get into the technology you start hearing all these buzz words and these processes. You used to have to code them, now they're built into the ETL tools. It's just sort of drag that little icon over that does all that for you. So it's automated a lot of the basic practices. Then on the higher end practices data quality, data cleansing, data profiling, a lot of things have now been built into the higher end tools. Again, what you used to have to do by hand or generally you didn't have enough time and you didn't do it at all. So this data integration has been this bedrock of the technology that has moved in there.

You mentioned SOA, the ETL technology has incorporated SOA web services into their brand too. So it's not just batch oriented, do the updates over night. You can do it real time, you can do it overnight, you can use messaging. So this technology has expanded greatly, and a lot of it incorporates best practices into the technology so that you don't have to do hand coding.

Hannah Smalltree: Ok, so talking about hand coding, it's free, it works, what's the issue? What is really the problem with companies using hand coding for integration?

Rick Sherman: Hey, I used to code too, so I like doing it. The problem is you start off doing one little code, one little program and a few lines of code. A month later it's 5000 lines of SQL code that's not documented. I used to manage a software engineering group-- data developers. Everybody's got their style, the first thing that comes on when the next programmer comes in is he changes half the code of what he inherited because he doesn't like the style, it doesn't add to the value of the code. So hand coding is quick, but one of the issues is the fact that you get a lot of it, it's not documented, and you waste a lot of time changing things. But more importantly, I mentioned a lot of the best practices that are now built right for you. So why do you have to come up with this slowly changing dimension that I mentioned. It takes a lot of code to be able to do it. Why not use a tool where you just grab-drag the icon and it does it for you. There's hundreds of examples of this kind of functionality that's embedded in these tools. So you're much, much more quicker to use the ETL tools then you are to doing it by hand. There's still a lot of hand coding you can do, but not in data integration.

Hannah Smalltree: So tactically speaking, how do you recommend that IT professionals begin to tackle this integration problem?

Rick Sherman: I think if they haven't' been doing data warehousing or data integration, I think the first thing they need to do is educate themselves as to what that means. They've probably been coding, they've probably been able to move data, and access data, that's not the problem. The problem is understanding what's the data architecture, and what you need to do in order to make data consistent, and the thing like I mentioned slowly change dimension. You need to read about it, or take some courses to understand what is involved in data integration. Then you can go to the tools and use them. I had one of my clients go off to training for a week in ETL tool. He came back and said I know how to use the tool, but I don't know anything at all about ETL. I think that's a classic one that I'd like to tell people, you really have to understand what the processes are and what you're trying to do. It's sort of like playing a sport. You can pick up a hockey stick but if you can't skate, and don't know what the rules are, what good is it. You need to understand what you're doing, and then you do it.

The second thing is regardless of how much money you have, you can pick up an ETL tool, or a data integration tool to do the work, so do that, get the tool. Even if you're going to migrate up to another tool more sophisticated later on, get some experience with at least one of the tools so you know how to apply that knowledge.

The third thing is realize that data integration isn't just coding, but it's talking to the business and figuring out what data quality is, what's consistent data, and then embedding that knowledge into the code. So you still have to talk to business people despite what a lot of technical vendors tell you. Being able to converse and understand and define things is one of the critical areas of success.

Hannah Smalltree: Now funding can be a definite issue here. How do you recommend that companies prove the ROI of integration technologies, and get the budget approved?

Rick Sherman: Historically we have an issue with IT folks underestimating how much time it's going to take. It's easy to figure what the price of a tool is, and how much hardware you need, that's the easy part for IT folks. But they need to figure out how much time it's going to take, and how many people they need to get going with it. So that's sort of the cost element. The benefit, the other part of the ROI, what you get out of it is, as we started off the discussion. Most business people need more information, and they understand the value of the information. Do some work with financial services companies or retailers. They'll easily justify spending money on information in order to increase the effectiveness of their marketing campaigns, or up selling their customers to buy more products. So for those folks it's pretty easy to do it.

I never had an issue with a business person not stepping up to the plate to want to fund the effort, it's just a question of how much it is, and if it is a lot of money or time, then explaining to the business people why it will take him that amount of time to do it. So that's the really the trick, the cost side, and that's mainly the time side they have to understand. IT folks always underestimate, especially when they get started in a project. I mentioned about learning about ETL, or data integration, that takes a little bit of time. It's not just cranking the tool out, and IT folks have a tendency to underestimate how much that will take.

Hannah Smalltree: Any other advice for IT professionals tasked with solving their organizations integration challenges?

Rick Sherman: As I mentioned, get some training. I think reading up a little bit about it and if they have peers from other companies that have been involved in it. It's always good to talk with your peers. There's a lot of different user groups and a lot of different organizations that may be local to you, or just some of your friends or peers that you have in other companies. It's always good to get some advice from people who have been through it, particularly if it's your first time through it. Not necessarily the vendor brochures, but to get real live experience with some folks giving you some sort of understanding of what it is you're going to have to go through.

Hannah Smalltree: Thank you so much for your insight here today Rick.

Rick Sherman: You're welcome.

Hannah Smalltree: You can learn more about Rick and check out his Data Doghouse blog at You can also see more of Rick's writing right here at Until next time I'm Hannah Smalltree, thank you for joining us, and have a great day.

View All Videos
Business Analytics
Content Management