Enterprise Data Mesh

A hot topic in enterprise software, data mesh is a new approach to thinking about data based on a distributed architecture for data management. The idea is to make data more accessible and available to business users by directly connecting data owners, data producers, and data consumers. Data mesh aims to improve business outcomes of data-centric solutions as well as drive adoption of modern data architectures.

From the business point of view, data mesh introduces new ideas around “data product thinking.” In other words, thinking about data as a product that fulfils a “job to be done”, for example, to improve decision-making, help detect fraud or alert the business to changes in supply chain conditions. To create high-value data products, companies must address culture and mindset shifts and commit to a more cross-functional approach to business domain modeling.

From the technology side, Oracle’s view on the data mesh involves three important new focus areas for data-driven architecture:

1. Tools that provide data products as data collections, data events and data analytics
2. Distributed, decentralized data architectures that help organizations who choose to move away from monolithic architectures, towards multi-cloud and hybrid-cloud computing, or who must operate in a globally decentralized manner
3. Data in motion for organizations who can’t depend solely on centralized, static, batch-oriented data, and instead move towards event-driven data ledgers and streaming-centric pipelines for real-time data events that provide more timely analytics

Other important concerns such as self-service tooling for non-technical users and strong federated data governance models are just as important for data mesh architecture as they are for other, more centralized and classical data management methodologies.

A data mesh approach is a paradigm shift to thinking about data as a product. Data mesh introduces organizational and process changes that companies will need to manage data as a tangible capital asset of the business. Oracle’s perspective for the data mesh architecture calls for alignment across organizational and analytic data domains.

A data mesh aims to link data producers directly to business users and, to the greatest degree possible, remove the IT middleman from the projects and processes that ingest, prepare, and transform data resources.

Oracle’s focus on data mesh has been in providing a platform for our customers that can address these emerging technology requirements. This includes tools for data products; decentralized, event-driven architectures; and streaming patterns for data in motion. For data product domain modeling and other sociotechnical concerns, Oracle aligns with the work being done by the thought leader in data mesh, Zhamak Dehghani.

The sad truth is that the monolithic data architectures of the past are cumbersome, expensive, and inflexible. Over the years, it’s become clear that most of the time and costs for digital business platform from applications to analytics are sunk into integration efforts. Consequently, most platform initiatives fail.

While data mesh is not a silver bullet for centralized, monolithic data architectures, the principles, practices, and technologies of the data mesh strategy are designed to solve some of the most pressing and unaddressed, modernization objectives for data-driven business initiatives.

Some of the technology trends that led to the emergence of data mesh as a solution include:

A mindset shift is the most important first step toward a data mesh. The willingness to embrace the learned practices of innovation is the springboard towards successful modernization of data architecture.

These learned practice areas include:

• Design thinking—a proven methodology for solving "wicked problems," applied to enterprise data domains for building great data products
• Jobs to be done theory——applying a customer-focused innovation and an outcomes-driven innovation process for ensuring that enterprise data products are solving real business problems

Design thinking methodologies bring proven techniques that help break down the organizational silos frequently blocking cross-functional innovation. The jobs to be done theory is the critical foundation for designing data products that fulfill specific end-consumer goals—or jobs to be done—it defines the product’s purpose.

Although the data product approach initially emerged from the data science community, it is now being applied to all aspects of data management. Instead of building monolithic technology architectures, data mesh focuses on the data consumers and the business outcomes.

While data product thinking can be applied to other data architectures, it is an essential part of a data mesh. For pragmatic examples of how to apply data product thinking, the team at Intuit wrote a detailed analysis of their experiences.

Products of any kind—from raw commodities to items at your local store—are produced as assets of value, intended to be consumed, and have a specific job to be done. Data products can take a variety of forms, depending on the business domain or problem to be solved, and may include:

• Analytics—historic/real-time reports and dashboards
• Data sets—data collections in different shapes/formats
• Models—domain objects, data models, machine learning (ML) features
• Algorithms—ML models, scoring, business rules
• Data services and APIs—docs, payloads, topics, REST APIs, and more

A data product is created for consumption, typically owned outside of IT and requires tracking of additional attributes, such as:

• Stakeholder map—Who owns creates and consumes this product?
• Packaging and documentation—How is it consumed? How is it labeled?
• Purpose and value—What is the implicit/explicit value of the product? Is there depreciation over time?
• Quality and consistency—What are the KPIs and SLAs of usage? Is it verifiable?
• Provenance, lifecycle, and governance—Is there trust and explainability of the data?

Decentralized IT systems are a modern reality, and with the rise of SaaS applications and public cloud infrastructure (IaaS), the decentralization of applications and data is here to stay. Application software architectures are shifting away from the centralized monoliths of the past to distributed microservices (a service mesh). Data architecture will follow the same trend toward decentralization, with data becoming more distributed across a wider variety of physical sites and across many networks. We call this a data mesh.

A mesh is a network topology that enables a large group of nonhierarchical nodes to work together collaboratively.

Some common tech examples include:

• WiFiMesh—many nodes working together for better coverage
• ZWave/Zigbee—low-energy smart home device networks
• 5G mesh—more reliable and resilient cell connections
• Starlink—satellite broadband mesh at global scale
• Service mesh—a way to provide unified controls over decentralized microservices (application software)

Data mesh is aligned to these mesh concepts and provides a decentralized way of distributing data across virtual/physical networks and across vast distances. Legacy data integration monolithic architectures, such as ETL and data federation tools—and even more recently, public cloud services, such as AWS Glue—require a highly centralized infrastructure.

A complete data mesh solution should be capable of operating in a multicloud framework, potentially spanning from on-premises systems, multiple public clouds, and even to edge networks.

In a world where data is highly distributed and decentralized, the role of information security is paramount. Unlike highly centralized monoliths, distributed systems must delegate out the activities necessary to authenticate and authorize various users to different levels of access. Securely delegating trust across networks is hard to do well.

Some considerations include:

• Encryption at rest—as data/events that are written to storage
• Distributed authentication—for services and datastores, such as mTLS, certificates, SSO, secret stores, and data vaults
• Encryption in motion—as data/events that are flowing in-memory
• Identity management—LDAP/IAM-type services, cross-platform
• Distributed authorizations—for service end-points to redact data
  For example: Open Policy Agent (OPA) sidecar to place policy decision point (PDP) within the container/K8S cluster where the microservice endpoint is processing. LDAP/IAM may be any JWT-capable service.
• Deterministic masking—to reliably and consistently obfuscate PII data

Security within any IT system can be difficult, and it is even more difficult to provide high security within distributed systems. However, these are solvable problems.

A core tenet of data mesh is the notion of distribution of ownership and responsibility. The best practice is to federate the ownership of data products and data domains to the people in an organization who are closest to the data. In practice, this may align to source data (for example, raw data-sources, such as the operational systems of record / applications) or to the analytic data (for example, typically composite or aggregate data formatted for easy consumption by the data consumers). In both cases, the producers and consumers of the data are often aligned to business units rather than to IT organizations.

Old ways of organizing data domains often fall into the trap of aligning with the technology solutions , such as ETL tools, data warehouses, data lakes or the structural organization of a company (human resources, marketing and other lines of business). However, for a given business problem the data domains are often best aligned to the scope of the problem being solved, the context of a particular business process, or the family of applications in a specific problem area. In large organizations, these data domains usually cut across the internal organizations and technology footprints.

The functional decomposition of data domains takes on an elevated, first-class priority in the data mesh. Various data decomposition methodologies for domain modeling can be retrofit to data mesh architecture including classical data warehouse modeling (such as Kimball and Inmon) or data vault modeling, but the most common methodology currently being tried in data mesh architecture is domain-driven design (DDD). The DDD approach emerged from microservices functional decomposition and is now being applied in a data mesh context.

An important area where Oracle has added to the data mesh discussion is to elevate the importance of data in motion as a key ingredient to a modern data mesh. Data in motion is a fundamentally essential to take data mesh out of the legacy world of monolithic, centralized, batch-processing. The capabilities of data in motion answer several core data mesh questions, such as:

• How can we access source-aligned data products in real-time?
• What tools can provide the means to distributed trusted data transactions across a physically decentralized data mesh?
• When I need to make data events available as data product APIs, what can I use?
• For analytic data products that must be continuously up to date, how would I align to data domains and ensure trust and validity?

These questions are not just a matter of “implementation details” they are centrally important to the data architecture itself. A domain-driven design for static data will use different techniques and tools than a dynamic, data in motion process of the same design. For example, in dynamic data architectures, the data ledger is the central source of truth for data events.

Ledgers are a fundamental component of making a distributed data architecture function. Just as with an accounting ledger, a data ledger records the transactions as they happen.

When we distribute the ledger, the data events become “replayable” in any location. Some ledgers are a bit like an airplane flight recorder that is used for high availability and disaster recovery.

Unlike centralized and monolithic datastores, distributed ledgers are purpose-built to keep track of atomic events and/or transactions that happen in other (external) systems.

A data mesh is not just one single kind of ledger. Depending on the use cases and requirements, a data mesh can make use of different types of event-driven data ledgers, including the following:

• General purpose event ledger—such as Kafka or Pulsar
• Data event ledger—distributed CDC/replication tools
• Messaging middleware—including ESB, MQ, JMS, and AQ
• Blockchain ledger—for secure, immutable, multi-party transactions

Together, these ledgers can act as a sort of durable event log for the whole enterprise, providing a running list of data events happening on systems of record and systems of analytics.

Polyglot data streams are more prevalent than ever. They vary by event types, payloads, and different transaction semantics. A data mesh should support the necessary stream types for a variety of enterprise data workloads.

Of course, there are more than just three attributes of a data mesh. We’ve focused on the three above as a way to bring attention to attributes that Oracle believes are some of the new and unique aspects of the emerging modern data mesh approach.

Other important data mesh attributes include:

• Self-service tooling— data mesh embraces the overall data management trend towards self-service, Citizen developers will must increasingly come from the ranks of the data owners
• Data governance— data mesh has also embraced the long-standing trend toward a more formalized federated governance model as championed by chief data officers, data Stewards and data catalog vendors for many years.
• Data usability — delving in to principles of data mesh, there is quite a lot of foundation work around ensuring that data products are highly usable. Principles for data products will be concerned with data that is valuable, usable, and feasible to share.

Looking beyond 'lift and shift' migrations of monolithic data architectures to the cloud, many organizations also seek to retire their centralized applications of the past and move toward a more modern microservices application architecture.

But legacy applications monoliths typically depend on massive databases, raising the question of how to phase the migration plan to decrease disruption, risks, and costs. A data mesh can provide an important operational IT capability for customers doing phased transitions from monoliths to mesh architecture. For example:

• Sub-domain offloading of database transactions, such as filtering data by 'bounded context'
• Bidirectional transaction replication for phased migrations
• Cross-platform synchronization, such as mainframe to DBaaS

In the lingo of microservices architects, this approach is using a bidirectional transaction outbox to enable the strangler fig migration pattern , one bounded context at a time.

Business-critical applications require very high KPIs and SLAs around resiliency and continuity. Regardless of whether these applications are monolithic, microservices or something in between, they can’t go down!

For mission-critical systems, a distributed eventual-consistency data model is usually not acceptable. However, these applications must operate across many data centers. This begs the business continuity question, “How can I run my apps across more than one data center while still guaranteeing correct and consistent data”

Regardless of whether the monolithic architectures are using ‘sharded datasets’ or the microservices are being set up for cross-site high availability, the data mesh offers correct, high-speed data at any distance.

A data mesh can provide the foundation for decentralized, yet 100% correct data across sites. For example:

• Very low latency logical transactions (cross-platform)
• ACID capable guarantees for correct data
• Multi-active, bidirectional, and conflict resolution

A modern, service mesh–style platform uses events for data interchange. Rather than depending on batch processing in the data tier, data payloads flow continuously when events happen in the application or datastore.

For some architectures, microservices need to exchange data payloads with each other. Other patterns require interchange between monolithic applications or datastores. This begs the question, “How can I reliably exchange microservice data payloads among my apps and datastores?”

A data mesh can supply the foundation technology for microservices-centric data interchange. For example:

• Microservice to microservice within context
• Microservice to microservice across contexts
• Monolith to/from microservice

Microservices patterns, such as event sourcing, CQRS, and transaction outbox, are commonly understood solutions; a data mesh provides the tooling and frameworks to make these patterns repeatable and reliable at scale.

Beyond microservice design patterns, the need for enterprise integration extends to other IT systems, such as databases, business processes, applications, and physical devices of all types. A data mesh provides the foundation for integrating data in motion.

Data in motion is typically event-driven. A user action, a device event, a process step, or a datastore commit can all initiate an event with a data payload. These data payloads are crucial for integrating Internet of Things (IoT) systems, business processes and databases, data warehouses, and data lakes.

A data mesh supplies the foundation technology for real-time integration across the enterprise. For example:

• Connecting real-world device events to IT systems
• Integrating business processes across ERP systems
• Aligning operational databases with analytic datastores

Large organizations will naturally have a mix of old and new systems, monoliths and microservices, operational and analytic data stores; a data mesh can help to unify these resources across differing business and data domains.

Analytic data stores may include data marts, data warehouses, OLAP cubes, data lakes, and data lakehouse technologies.

Generally speaking, there are only two ways to bring data into these analytic datastores:

• Batch/micro-batch loading—on a time scheduler
• Streaming ingest—continuously loading data events

A data mesh provides the foundation for a streaming data ingest capability. For example:

• Data events from databases or datastores
• Device events from physical device telemetry
• Application event logging or business transactions

Ingesting events by stream can reduce the impact on the source systems, improve the fidelity of the data (important for data science), and enable real-time analytics.

Once ingested into the analytic datastores, there is usually a need for data pipelines to prepare and transform the data across different data stages or data zones. This data refinement process is often needed for the downstream analytic data products.

A data mesh can provide an independently governed data pipeline layer that works with the analytic datastores, providing the following core services:

• Self-service data discovery and data preparation
• Governance of data resources across domains
• Data preparation and transformation into required data product formats
• Data verification by policy that ensures consistency

These data pipelines should be capable of working across different physical datastores (such as marts, warehouses, or lakes) or as a “pushdown data stream” within analytic data platforms that support streaming data, such as Apache Spark and other data lakehouse technologies.

Events are continuously happening. The analysis of events in a stream can be crucial for understanding what is happening from moment to moment.

This kind of time-series-based analysis of real-time event streams may be important for real-world IoT device data and for understanding what is happening in your IT data centers or across financial transactions, such as fraud monitoring.

A full-featured data mesh will include the foundational capabilities to analyze events of all kinds, across many different types of event time windows. For example:

• Simple event stream analysis (web events)
• Business activity monitoring (SOAP/REST events)
• Complex event processing (multistream correlation)
• Data event analysis (on DB/ACID transactions)

Like data pipelines, the streaming analytics may be capable of running within established data lakehouse infrastructure, or separately, as cloud native services.

Those at the leading edge of data integration are seeking real-time operational and analytical data integration from a diverse collection of resilient datastores. Innovations have been relentless and fast as data architecture evolves into streaming analytics. Operational high availability has led to real-time analytics, and data engineering automation is simplifying data preparation, enabling data scientists and analysts with self-service tools.