Interview: What is Data Mesh

Cheuk: What is data mesh?

Luke:

So this is an idea about how organizations should scale their data infrastructure. It’s not just about technology, it’s also about how you think about organizing inside your company. Rather than having one big central data team, you bring domain ownership to data. This allows teams to own their data upstream, and make it available to the rest of the organization.

We think about the way that a lot of organizations manage their data today. You have one big central data warehouse with lots of spokes coming in. These spokes are different downstream teams. And the central hub is always overloaded by stuff coming in from the spokes. So what we need to do is try and move responsibility away from that hub, towards the spokes, and make those teams self-sufficient. Teams can then serve data properly to the rest of the organization.

Cheuk: Why do companies need to do that?

Luke:

That’s a good question.

Some companies don’t. It’s not necessarily all companies need Data Mesh. Some companies can get by, they’re small enough, or they’re structured in such a way that they can have a central data warehouse.

But when you start getting to scale, and the problems are multiplying, because more and more people want reports, and everybody working in data engineering knows their life is full of people shouting at them all day. I want this data in this way, I want this data in that way. So being able to allow those people to access high-quality curated data is really the goal of Data Mesh.

Cheuk: How can companies use TerminusX to achieve Data Mesh?

Luke:

TerminusX is good at building versioned data products. We take organizational data, package it up in such a way so that non-experts can work with it, and curate data locally. The team can build a single data product or multiple data products, that are governed and work together.

One of the big problems with Data Mesh is that you’re at risk of just having loads and loads of silos. For example, all of the teams decide to make their own data products, using their own technology, with no overarching governance. This would end up in a mess.

A strong layer of organizational governance that’s centrally dictated out to decentralized teams is imperative. With TerminusX, you get that interoperability across your data products, so that they can all operate together, and you don’t end up with a mess, you end up with something that’s usable. An ecosystem of integrated version-controlled data products.

Cheuk: When should a company start using TerminusX?

Luke:

I think you should start using it straight away, no matter what size you are.

We’ve been very successful in working with smaller companies that want to build applications. They want to have a really robust data layer between the application and want to build data products that come out of that. But also for large companies, and especially when you’re finding it very difficult to find disparate data across the organization.

So say you have a lot of subscribers, and those subscribers might be in different places. You can build a data product of subscribers that sucks this data up from different places into a single data product. Your data scientists and analysts can then access that data and use it with whatever projects they’re working on.

TerminusX makes a lot of sense with these sorts of use cases. You can stick all the subscriber data into TerminusX and access it from there, and you have a version history of that data. So you can say, hey, I want to know how many subscribers we had on January 1st, 1999. Or, I want to know, did we have anybody in these areas in 2017. Or even, what’s been the subscriber change in area XYZ between 2017 and today. Having a versioned data product like that gives you incredible power and visibility into the movement of change of that data over time.

Cheuk: How can someone get started with TerminusX?

Luke:

Oh, that’s easy.

Just come to our website, get started. You can get started for free. Start building today. Start playing with it, start testing and see it work, see what works for you. Come talk to us. We’re open, we’re on Discord. We have a lot of community support and community events and we’re here for people to come in and give the tires a kick.