Data mesh success depends on more than architecture

Why data mesh appeals to data leaders

Traditional centralized data architectures gather data from disparate sources, such as spreadsheets and software applications, and then collect, manage and store it in a unified repository, such as a data warehouse or data lake. But now, organizations are turning to decentralized data architectures, including data mesh, because centralized models can overload data teams with dashboard requests, pipeline changes and other data-related needs from across the entire business.

Introduced in 2019, data mesh is a "socio-technical" approach that embraces a "divide and conquer" mentality. Data mesh distributes data ownership across business domains rather than concentrating responsibility within a central data team. In data mesh, domain teams manage and are accountable for their data, said Noel Yuhanna, vice president and principal analyst at Forrester.

According to Yuhanna and other data leaders, placing accountability with domain teams that best understand the data and its regulations, restrictions and governance needs makes sense. This approach can yield higher-quality, more accessible data without sacrificing security, privacy and compliance, they said.

"That's a nirvana state that people want to get to," Yuhanna said. "But getting to that state is not trivial."

Why data mesh adoptions stall

Technology supports data mesh, but no single platform or tool can deliver the operating model on its own. Yuhanna described the move to data mesh as a socio-technical approach that requires changes to processes, culture and people.

"[Data mesh] requires people to think differently," Yuhanna said.

This change requires strong change management to help teams adopt new responsibilities and workflows.  

Executives should anticipate resistance to the new approach, he added, noting that domain teams sometimes don't want to share data or collaborate with other domains or IT. Executives also might resist moving to data mesh after years of investment in centralized data management approaches, experts said.

An organization's existing data ecosystem can also be a barrier to data mesh adoption, with legacy technology, existing data silos and complexity often stymying efforts, Yuhanna said. Some organizations get only a fraction of their data -- as low as 1% -- into a data mesh, which limits the value, he added.

Conventional approaches to funding data management can further frustrate attempts to implement data mesh. Because this model involves shared work across domains, organizations must find ways to fund priorities that span multiple departments.

"If you have a centralized funding model and not cross-functional funding, that can be a challenge," McClelland said.

Many domain teams lack the necessary data engineering, product management and governance skills to own data products used in a data mesh model. McClelland said data mesh forces less technical domain teams, such as marketing or HR, to operate like software product teams. Training in this area can address that gap.

"Most domain teams lack data engineering expertise, and forcing them to manage data pipelines without proper training leads to broken infrastructure and poor data quality," McClelland said.

Some organizations still fall short on data management maturity, which is necessary for a effective data mesh adoption. Automated data classification, security controls, compliance checks and metadata standards should be built into the platform from the start to avoid an unmanageable "data swamp," McClelland said.

He stressed that data mesh doesn't eliminate enterprise data governance, noting that an overarching governance framework sits above it.

Assessing readiness

Data mesh can deliver benefits, but ROI depends on the organization's starting point, Yuhanna said.

A distributed data approach needs clear management structures and accountability, otherwise data mesh is unlikely to work, he said. Yuhanna advised executives to assess their data culture and evaluate their organizational structure and maturity around data accountability, security and governance. A higher level of maturity makes moving to data mesh less taxing.

Organizations must also determine how well IT and business teams collaborate, as well as the level of communication among CDOs, CIOs, engineers, business analysts, data stewards and business users, Yuhanna said. A more collaborative environment makes implementation easier.

McClelland said leadership should also assess their maturity around data-related processes, using the following tests:

• Time-to-data audit. McClelland advised businesses to measure how long it takes a business unit to request, get approval for and ingest a new external data source. "If the bottleneck is purely bureaucratic rather than technical, the culture is highly resistant to self-serve models," he said.
• Domain capability mapping. This test audits the technical literacy of business units by classifying domain teams into tiers based on tech capability, McClelland said. "If no business units have the skills to manage their own data endpoints, a mesh is premature," he said.
• "Who pays?" test. "Propose a scenario where Domain A needs a feature built by Domain B's data team," McClelland said. "If your current corporate structure has no mechanism for cross-domain funding or shared priority alignment, the operational model will stall."

How leaders can prepare for data mesh

The scope and complexity of the challenges aren't insurmountable, especially if executives keep some best practices in mind as they move to a data mesh architecture, experts said.

Pre-implementation

First, stakeholders must formalize a plan. Ad hoc adoption is not an option, Yuhanna said, and emphasized the need for defined policies, standards and requirements.

During planning, organizations should embed security and governance from the start, Yuhanna added. But McClelland warned against relying on documentation or human committees to enforce governance. Instead, automated governance is preferable.

"Bake compliance, masking rules and access control directly into the self-serve platform deployment templates so domains get compliance 'for free,'" McClelland said.

Implementation

Start with a high-value use case that spans several domains, recognizing the shift is evolutionary, not a big-bang moment, said Bhath Thota, a partner at consulting firm Kearney. Begin with domains with more mature data management practices and staff capable of championing the shift, he added.

Alternatively, start by moving small datasets to data mesh, Yuhanna said.

Either way, organizations should fund the platform as a product with domain developers treated as customers, McClelland said.

"If the self-serve platform is difficult to use, domains will bypass it and build their own rogue infrastructure," he added.

Treat the move to data mesh as a culture shift. That shift requires organizations to replace data hoarding habits with an open marketplace that ties data quality to business performance, McClelland said. Technology should support that behavioral shift, he added.

"Treating data mesh as a technology migration is the number one reason implementations fail," McClelland said. "Data mesh is inherently a sociotechnical shift."

Embedding data product owners into business units will help. Instead of reporting to central IT, employees report to the business domain lead to help align data initiatives with business strategy, McClelland said.

And don't aim for perfection, Thota said. Data mesh doesn't have to be all-or-nothing transformation. Many organizations use a hybrid approach that combines data mesh principles with data lakes and data lakehouses.

"I have yet to see any company with 100% in data mesh," Yuhanna said.

Mary K. Pratt is an award-winning freelance journalist specializing in enterprise IT, cybersecurity strategy and data management.