Integration between SAP Data Services, Business Warehouse on the rise

Improvements in DS 4.0, BW 7.3

Integration with SAP sources including BW took a big jump with Data Services 4.0 and was further refined with the recent release of Data Services 4.1.

The newest version of Data Services can read directly from SAP's "business content" extractors in the ERP, CRM and SRM applications. The table structure in these applications can be extremely complex. Extractors act as semantic views on top of these tables and can provide additional capabilities like delta change capture. This functionality was introduced in Data Services 4.0, and parallel processing was improved in version 4.1.

Data Services 4.0 also supports native loading of BW DataSource objects, and Data Services is better integrated with BW as a Source System type. This new integration makes it very easy to pull data from Data Services sources directly into BW DataSources, all from the BW interface. (DataSources is BW's data ingestion construct.) These capabilities are available in BW version 7.3 or later.

SAP Data Services 4.1 also introduces improved parallel loading when using BW's OpenHub tables as a source for Data Services. This is an important functionality for many customers because efficiently moving data out of BW and into other systems for analysis and reporting can be a challenge.

Advantages Data Services can offer now

These improvements result in a compelling case for using BW and Data Services together.

SAP Business Content extractors, originally developed for use with BW, are the semantic representation of ERP data for analytic uses. Now they are available directly for non-BW use via Data Services. These extractors should work without interference even when simultaneously used for BW extraction, and can now be used to feed other data marts or an operational data store instead of, or in parallel with, BW loads.

Data Services is attractive for current BW shops because it is much better at some types of transformations than BW. Pivot transformations, for example, are difficult in BW, often requiring custom-coded routines. In some environments, skills to develop complex transformations may be easier to come by when developing these transformations in Data Services' custom development language (Python) than in ABAP. Doing custom transformation development in Data Services, rather than BW, may make custom development more reusable, while reducing the complexity of the BW environment.

Data Services also enables a different software lifecycle management approach. BW changes are often limited to preplanned transport windows every week or month. This ensures a stable system but can be problematic for transformation logic that needs to closely track changes in source systems. Segregating source-system-oriented transformations in Data Services could allow a software lifecycle management regime with faster reactions to source-system changes or errors.

BW still has advantages over traditional databases when used as a Data Services data store, and current Data Services shops may want to consider it for this reason. Some of the advantages are outlined in this article comparing the differences between BW and HANA.

Still some rough edges

There are still some rough edges in the Data Services integration with SAP applications and BW. Data Services support for business content extractors is extensive, but it’s limited to extractors for "Operational Data Provisioning," which doesn’t cover every scenario. You can find the full list of supported extractors in note 1558737 (login is required).

The integration between BW Source Systems, BW DataSources, Data Services Targets and Data Services Jobs is also confusing. When you create a Data Services source system in BW, you can only use Data Services as a "dumb pipe" to load data to BW. There is no ability to transform data by using Data Services in this scenario. This approach throws out much of Data Services' power in the name of simplicity.

However, all is not lost. If you configure the Data Services connection to BW, and then create a External System source system in BW (instead of a Data Services source system), you can create an arbitrarily structured BW DataSources and BW InfoPackages configuration to run Data Services jobs of your choosing.

If the Data Services job's target has the same structure as the BW DataSource, everything works. Surprisingly, the load will work even if the target of the Data Services job appears to be a different DataSource. This is a nice functionality because it allows people to leverage any Data Services transformation logic directly from BW. But it's also dangerous because there are no restrictions forcing you to match structures, though the capability is likely to be found only by more adventurous users.

Figure 1: Different Source System types for Data Services behave differently in BW. “Data Services” source systems allow you to create DataSources that exactly mirror the structure of a Data Services “Source”. “External System” source systems allow you to create DataSources with arbitrary structures that are filled using any Data Services Job you choose.

Data Services can now make use of a lot more of the functions that both SAP applications and BW have to offer. This should make life a lot easier for those trying to use Data Services in an environment with other SAP applications, and offer some especially interesting new options to BW shops. That said, this is a relatively new functionality, so it suffers from bugs and implementation inconsistencies that can be confusing. Remaining aware of the shortcomings and hidden functionality can ease integration headaches.