Bridging the data and domain divide
Us and them – it’s a powerplay that governments, media, and countless other bearers of power have used to divide and conquer. It is fair to say that this approach is not a constructive way to approach power and one side will always feel like they lost.
Somewhere, sometime, a divide has occurred between IT teams and domains. This widening chasm is stagnating modern businesses. This divide doesn’t have winners, only losers. This article is about communication and collaboration, and attempts to look at how domains and data teams can bridge the divide without radical change.
I will level with you, I write this article firmly from a domain perspective. That does not mean I will use it to cry that IT and data teams have neglected me throughout my career, I will endeavor to use empathy and professionalism to look evenly from both sides.
Data teams and domains want the same thing
Both sides of the gulf have many things in common, they want to achieve:
- Business goals.
- Job satisfaction.
- Career progression.
- Peer acceptance and praise.
- Improved remuneration.
Much of that list can be accomplished by successfully achieving business goals. However, goals set by the business become murky as they travel from the top through various levels of management and land on a foot soldier’s desk.
Do department-level goals pull in the same direction as overall company objectives? You’d hope so, but sometimes it is easy to get lost in your silo to see the bigger picture.
Let’s look at the domain and data team perspectives to see where differences occur.
Domain need for data
Typically, domains get access to data through applications and services. Think Google Analytics for marketing, or Sage for finance. If they need additional data for new services, campaigns, or analysis, they need to request it.
Often this request needs a business case that takes time to put together. The request is made through the IT department or service desk, taking more time to get to the right team. As requests are infrequent, there is no personal relationship between the data or IT team and domain members. Knowing who to speak to is a task in itself, particularly in a large organization that implements a call-center-style service desk, think, ‘you are fourth in the queue…your call is important to us.’
When the domain gets through, the data team is busy and has added the request to the backlog of tasks.
The domain team gets frustrated having to wait for, what seems, a simple request. They have an idea they want to implement that will benefit the business, but the delay is blocking it.
Data team requests from domains
The data team is a central team handling all business domains’ data requests and the maintenance, upgrades, and fixes to the data warehouse/lake and ETL pipelines that have evolved. It is a team of highly skilled data engineers, infrastructure architects, and data scientists.
The centralized team is at capacity and has an enormous workload managing the data requirements of the organization with additional requests coming in from domains on an ad hoc basis. The team has specific KPIs and goals to maintain and achieve but is also expected to assist domain teams to achieve their own goals.
The ETL pipelines are extremely brittle and require constant attention. They also need to focus on the technical debt and reduce overheads where possible. The tech stack has evolved to accommodate new applications, data demands, and technology advancements. None of the team would have built it this way and it is idiosyncratic to the organization with knowledge of it depleting every time a team member leaves their role.
It is a daily struggle to firefight the monolithic architecture and to assist domains with their requirements, many of the team are exhausted.
The animosity between IT and domains
The struggles outlined above cause friction between domains and data teams.
Domains are frustrated by the wait for their data requests.
Data teams are frustrated by their workload and short notice from domain data requests.
Out of despair, domains take action
I’ll tell you a story about my culpability for shortsightedness in trying to achieve my goals.
I worked for a nationwide distribution company that was bought by a global organization. The ERP system was a cobbled-together mess – something that was originally built for timber merchants and reconfigured to apply to our industry. On top of that, there was a customer and sales reporting tool that surfaced numbers and allowed you to export to CSV. I exported the numbers and worked through the thousands of customer and product records to segment them into groups with similar behaviors. Having run a couple of successful campaigns off the back of the groups, I wanted a quick way to report on the criteria I’d manually found in Excel.
My first port of call was to see if I could get custom reports from the reporting tool. I spoke to several members of the IT team who didn’t know. I spent a long time speaking to different ‘call agents’ at the conglomerate service desk. I spoke to the finance department, who were heavy users of the tool, but they couldn’t help me either. I ended up writing a business case and getting it signed off by the marketing director for an external software company to build a bespoke application where I could feed CSV files into it every month and it would export the segmented groups for me. There were also limited configuration capabilities in the app.
Fast forward several months. IT ran an audit into hardware and software (a result of the takeover). When explaining my bespoke app, it turns out that the reporting tool was more than capable of dealing with my needs. It also transpired that the business changed and as a result, the purpose-built application no longer fulfilled its purpose. So there it sat, somewhere on a server, gathering dust. An echo of pure shortsighted endeavor.
Technical debt increased. Tick.
Wasted marketing budget. Tick.
Scolding from IT. Tick.
IT team’s trust in domains is eroded
Requests for quick turnarounds, rude responses (I was never rude FYI), escalation procedures, and domains taking matters into their own hands thereby increasing technical debt have eroded trust in domains.
For data teams to move forward and be happy in their work, i.e. not exhausted, this trust needs to be found again. Otherwise, we run the risk of groundhog day.
How do we bridge the data and domain divide?
Bigger things are happening in the data world as organizations realize the importance of data. These are born out of the need to improve data access, usability, and quality, and to keep and hire skilled data professionals. An increasingly difficult task. As you’re probably aware, the bigger movements include:
- Data mesh – decentralized domain-oriented architecture focusing on the socio-technical aspect – the nuanced ways that humans and systems work together – this is how Zhamak Dehghani describes it in her book.
- Data fabric – a data architecture framework designed to make data management more agile in a complex, diverse, and distributed environment.
- DataOps – a process-oriented methodology, adopting Agile practices, to shorten the cycle times of analytic and application development and improve data quality. One of the main pillars of DataOps is removing barriers to improve collaboration between teams.
These are great, and from a database company perspective, the emphasis on ‘the importance of data’ is music to our ears. However, if you’re in a data or domain team experiencing the frustrations outlined in this article, are they going to help you in the short term? Will you be too exhausted if something doesn’t change? The chances are, that your organization won’t be entertaining either data mesh or data fabric just yet. DataOps might be more realistic for the short-term, but it is a cultural shift so may require a little time.
There is one common factor among the three examples I provided. That’s a collaborative approach. How do you achieve collaboration? Easy – communication. By talking, listening, and understanding.
Communication – the short-term bridge to the data and domain divide
As silly as it sounds, communication is the simple and short-term solution to ease some of the friction between IT and domains. Knowledge, awareness, and removing the unknown can build empathy and an understanding of the roles and pressures of the respective teams. With empathy, relationships can blossom and frictions eased.
Domains don’t understand what IT and data teams do
Domains, generally speaking, do not understand your job. Something to do with numbers. A bit of coding. Turning a machine on and off. They have no understanding of the difficulties and skills that are involved. That is not to say that domain personnel are stupid, far from it, they’ve just never studied or experienced your field.
If I, as a domain person, want an extra field for an application or want to know what a piece of particular information means, I expect that to be simple. It’s a little request, a small question. What I don’t know is the intricacies of the application, I don’t know that the data I asked about is with 5 gazillion bytes of other data.
So a good place to start is to make people understand that your job is not as straightforward as it seems. As a great example, an ex-colleague once showed me a Slack notification diagram to help me understand.
In my head, I would have thought the decision tree would be a simple yes or no. Maybe a few more conditions, I don’t want to sell myself too short. However, this diagram was like a lightbulb moment for me.
It is also worth noting the basic psychology of a person. Nobody likes to feel stupid. There is a deep-rooted fear for most people to be out of their comfort zone. This is what happens to a lot of domain folk when they enter the world of IT. Lots of words and terms that are not part of their vernacular. So while discussing things with domains, be mindful of your prose to convey understanding.
Data teams should have a bigger picture
Domains, in general, are a little easier to understand. A data team may make assumptions about what sales do, what marketing does, operations, HR, and so on. However, the plans, the pressures, and the whys are not clear without listening.
Requests for data don’t come from a bad place, they come from a need to achieve a goal. With more visibility of domain plans and awareness of their objectives, data teams can get an understanding of requirements ahead of time to better plan their workload.
Get out of the silos and talk
Meetings can be a drag, but to achieve a more collaborative working environment between domains and data teams, it makes good sense to schedule regular meetings, perhaps quarterly or biannually.
These meetings are a chance for all parties to listen and to learn. After all, you’re all on the same side. We would suggest setting an agenda along the lines of:
- Introductions – Put a face to a name. Let everyone know your responsibilities and where your expertise lies.
- Overviews – This is more important for the data team as it’s a good opportunity to demonstrate the complexity of the tasks at hand and explain the processes behind data requests. It’s also a chance to establish some ground rules, i.e. give us at least three weeks’ notice.
- Plans – Give a top-level view of your plans. Do you work on a 12-month schedule or quarterly?
- Data teams – Let the domain know about any big plans that could impact your team’s ability to help, such as migrations, planned maintenance, etc.
- Domains – Work through your plan, be it marketing plan, sales plan, operations, etc, and stick to top-level data-heavy plans. The data team doesn’t need to know the day-to-day stuff. Don’t forget to include some context as to why particular data is important, where it’s lacking, and any ideas you have. This is your chance to get feedback from highly skilled colleagues.
An important thing to remember is your audience. Jargon and language make a huge difference to understanding. The last thing anyone wants is for a meeting to happen and everyone walks away none the wiser. Convey messages simply and concisely. Get the meetings to run for 30 minutes or less.
Outcomes of your meetings
Ideally, these meetings will foster empathetic relationships and demystify the roles of data and domain teams. In addition to the love, it would be prudent to also come away with:
- Key dates – Marketing campaigns, financial reporting, operational efficiency, database migrations, or system upgrades – no matter what key event, the respective teams should have a picture of what’s happening in the future. Get them in project boards and/or calendars so that both teams can work with an understanding of workloads.
- Understanding – More applicable for domain teams as they will have a better picture of the complexity of their requests and the processes involved to answer them. It will also set timescales and expectations for requests.
More meetings, more work?
I’m a big advocate of limiting meetings. Only when necessary. There is already a ton of work to do, why add to the burden?
In this case, I would say the benefits of better communication and collaboration will save a great amount of time, for a little upfront cost. Not all domains are equal either, for example, operations might need an annual meeting, whereas marketing might need a quarterly one. Keep meetings short, structured, and focused on helping each other out.
By having key dates and tasks noted, resources can be allocated, timescales improved, and the business can operate collectively.
Could the data team members be allocated domain responsibilities? It’d build relationships and there would be a much-improved domain-focused approach from a central team. A sort of data mesh mini.
If you have already done this, I’d love to hear from you and how it’s gone. Was it successful? Did you start the process for it to fall away? Did you begin with one domain and branch out? I’d love to know. I can be found on Twitter, LinkedIn, email, or our Discord server if you want to share your thoughts.