Bridging the data and domain divide

Us and them – it’s a powerplay that governments, media, and countless other bearers of power have used to divide and conquer. It is fair to say that this approach is not a constructive way to approach power and one side will always feel like they lost.

Somewhere, sometime, a divide has occurred between IT teams and domains. This widening chasm is stagnating modern businesses. This divide doesn’t have winners, only losers. This article is about communication and collaboration, and attempts to look at how domains and data teams can bridge the divide without radical change.

I will level with you, I write this article firmly from a domain perspective. That does not mean I will use it to cry that IT and data teams have neglected me throughout my career, I will endeavor to use empathy and professionalism to look evenly from both sides.

Data teams and domains want the same thing

Both sides of the gulf have many things in common, they want to achieve:

• Business goals.
• Job satisfaction.
• Career progression.
• Peer acceptance and praise.
• Improved remuneration.

Much of that list can be accomplished by successfully achieving business goals. However, goals set by the business become murky as they travel from the top through various levels of management and land on a foot soldier’s desk.

Do department-level goals pull in the same direction as overall company objectives? You’d hope so, but sometimes it is easy to get lost in your silo to see the bigger picture.

Let’s look at the domain and data team perspectives to see where differences occur.

Domain need for data

Typically, domains get access to data through applications and services. Think Google Analytics for marketing, or Sage for finance. If they need additional data for new services, campaigns, or analysis, they need to request it.

Often this request needs a business case that takes time to put together. The request is made through the IT department or service desk, taking more time to get to the right team. As requests are infrequent, there is no personal relationship between the data or IT team and domain members. Knowing who to speak to is a task in itself, particularly in a large organization that implements a call-center-style service desk, think, ‘you are fourth in the queue…your call is important to us.’

When the domain gets through, the data team is busy and has added the request to the backlog of tasks.

The domain team gets frustrated having to wait for, what seems, a simple request. They have an idea they want to implement that will benefit the business, but the delay is blocking it.

Data team requests from domains

The data team is a central team handling all business domains’ data requests and the maintenance, upgrades, and fixes to the data warehouse/lake and ETL pipelines that have evolved. It is a team of highly skilled data engineers, infrastructure architects, and data scientists.

The centralized team is at capacity and has an enormous workload managing the data requirements of the organization with additional requests coming in from domains on an ad hoc basis. The team has specific KPIs and goals to maintain and achieve but is also expected to assist domain teams to achieve their own goals.

The ETL pipelines are extremely brittle and require constant attention. They also need to focus on the technical debt and reduce overheads where possible. The tech stack has evolved to accommodate new applications, data demands, and technology advancements. None of the team would have built it this way and it is idiosyncratic to the organization with knowledge of it depleting every time a team member leaves their role.

It is a daily struggle to firefight the monolithic architecture and to assist domains with their requirements, many of the team are exhausted.

The animosity between IT and domains

The struggles outlined above cause friction between domains and data teams.

Domains are frustrated by the wait for their data requests.

Data teams are frustrated by their workload and short notice from domain data requests.

Out of despair, domains take action

I’ll tell you a story about my culpability for shortsightedness in trying to achieve my goals.

I worked for a nationwide distribution company that was bought by a global organization. The ERP system was a cobbled-together mess – something that was originally built for timber merchants and reconfigured to apply to our industry. On top of that, there was a customer and sales reporting tool that surfaced numbers and allowed you to export to CSV. I exported the numbers and worked through the thousands of customer and product records to segment them into groups with similar behaviors.  Having run a couple of successful campaigns off the back of the groups, I wanted a quick way to report on the criteria I’d manually found in Excel.

My first port of call was to see if I could get custom reports from the reporting tool. I spoke to several members of the IT team who didn’t know. I spent a long time speaking to different ‘call agents’ at the conglomerate service desk. I spoke to the finance department, who were heavy users of the tool, but they couldn’t help me either. I ended up writing a business case and getting it signed off by the marketing director for an external software company to build a bespoke application where I could feed CSV files into it every month and it would export the segmented groups for me. There were also limited configuration capabilities in the app.

Fast forward several months. IT ran an audit into hardware and software (a result of the takeover). When explaining my bespoke app, it turns out that the reporting tool was more than capable of dealing with my needs. It also transpired that the business changed and as a result, the purpose-built application no longer fulfilled its purpose. So there it sat, somewhere on a server, gathering dust. An echo of pure shortsighted endeavor.

Technical debt increased. Tick.

Wasted marketing budget. Tick.

Scolding from IT. Tick.

IT team’s trust in domains is eroded

Requests for quick turnarounds, rude responses (I was never rude FYI), escalation procedures, and domains taking matters into their own hands thereby increasing technical debt have eroded trust in domains.

For data teams to move forward and be happy in their work, i.e. not exhausted, this trust needs to be found again. Otherwise, we run the risk of groundhog day.

How do we bridge the data and domain divide?

Bigger things are happening in the data world as organizations realize the importance of data. These are born out of the need to improve data access, usability, and quality, and to keep and hire skilled data professionals. An increasingly difficult task. As you’re probably aware, the bigger movements include:

• Data mesh – decentralized domain-oriented architecture focusing on the socio-technical aspect – the nuanced ways that humans and systems work together – this is how Zhamak Dehghani describes it in her book.
• Data fabric – a data architecture framework designed to make data management more agile in a complex, diverse, and distributed environment.
• DataOps –  a process-oriented methodology, adopting Agile practices, to shorten the cycle times of analytic and application development and improve data quality. One of the main pillars of DataOps is removing barriers to improve collaboration between teams.

These are great, and from a database company perspective, the emphasis on ‘the importance of data’ is music to our ears. However, if you’re in a data or domain team experiencing the frustrations outlined in this article, are they going to help you in the short term? Will you be too exhausted if something doesn’t change? The chances are, that your organization won’t be entertaining either data mesh or data fabric just yet. DataOps might be more realistic for the short-term, but it is a cultural shift so may require a little time.

There is one common factor among the three examples I provided. That’s a collaborative approach. How do you achieve collaboration? Easy – communication. By talking, listening, and understanding.

Communication – the short-term bridge to the data and domain divide

As silly as it sounds, communication is the simple and short-term solution to ease some of the friction between IT and domains. Knowledge, awareness, and removing the unknown can build empathy and an understanding of the roles and pressures of the respective teams. With empathy, relationships can blossom and frictions eased.

Domains don’t understand what IT and data teams do

Domains, generally speaking, do not understand your job. Something to do with numbers. A bit of coding. Turning a machine on and off. They have no understanding of the difficulties and skills that are involved. That is not to say that domain personnel are stupid, far from it, they’ve just never studied or experienced your field.

If I, as a domain person, want an extra field for an application or want to know what a piece of particular information means, I expect that to be simple. It’s a little request, a small question. What I don’t know is the intricacies of the application, I don’t know that the data I asked about is with 5 gazillion bytes of other data.

So a good place to start is to make people understand that your job is not as straightforward as it seems. As a great example, an ex-colleague once showed me a Slack notification diagram to help me understand.