This is a story about when data teams go bad and what can be done to save them. Set in a dystopian future, the Departments seek freedom from Central, an authoritarian institution that wields all of the power. Out of the darkness is a lighthearted and dramatized take on the warehouse/lake vs data mesh data wars.
Repeat until fade
“You have the requirements, just get it done.”
It was a regular Monday morning. Every face around the huge mahogany table looked glum as they wearily faced the man who sat at the top barking out the instructions for the week.
Everybody knew what the week would hold. They’d work hard; crunch numbers, wade through reams of data and understand nothing. Eventually, they’d supply other departments with what they requested. There’d be complaints about inaccuracies, arguments, requirements that weren’t met. Then it would be Monday and the same process would kick-off. And again. Repeat until fade.
Clarence had accepted his fate. He didn’t question Mr. Peters, he didn’t question Central. He was part of a bigger cog. Integral. Elite. They had the skills that other departments, lesser departments, did not. It was accepted throughout The Regime. That’s just the way it was.
Clarence didn’t enjoy life, he didn’t hate it either. It was a life many would fight for. He was privileged, skilled, and well paid. He had learned not to question it as time had passed. Mr. Peters had been clear; to relinquish control from Central would be anarchy. The other departments’ lack of skills, inability to follow simple governance, and general ignorance would cost The Regime money, incur costs, and most likely, people would starve. Clarence’s life wasn’t perfect, but it was better than anarchy. That much he knew. Or at least he thought he knew.
As the Monday morning briefing came to a close, Mr. Peters having handed out the week’s worksheets, Clarence looked across the table and looked at his fellow workers, all exhausted, all expressionless. All except Zara, who was smiling as she looked at her worksheet. This was highly unusual. Clarence had worked for Central for 14 years: not once had he seen anyone smile during the morning briefing.
Zara looked up and caught Clarence’s gaze, seeing his confusion, her smile dropped and rested to a neutral expression as she shifted her eyes back to the worksheet. She was the first to leave when the meeting finished, slinking out the door.
The week went as expected. Clarence worked through the reams of data, walking virtually through the great halls of the enormous warehouse armed with a loose map, a set of requirements, and lacking a great deal of information to help him meet his quota of tasks. Arguments ensued, meetings between departments ended in disappointment and anger, quotas were missed. Standard procedure.
Clarence usually slept well, but that week, he kept waking, thinking about that smile. Why was Zara looking so damn happy?
Departments
“What we have is wrong, and what we don’t have, we need.”
There were loud cheers of agreement inside the smokey room. Apollonia, who was Principal for the Department for Supply, was adamant the requirements were clear and Central had let them down once more.
“We send request after request. The same thing over and over. What do we get?” “NOTHING” the chorus came back.
Apollonia was a reasonable woman. Her ire was due to her care. She cared about her department and the people she served. She had a good team, she had clever, hardworking minds, and she was certain that if they had more power, more autonomy, more information, that she and her team could make a difference. A real difference to The Regime and to the people she served. The Department for Supply was not afforded that luxury. They were treated with contempt, much like her comrades in the other departments.
Apollonia had attempted to reason, sometimes heated, with Central on countless occasions, but her resistance gradually faded and her fight slowly turned from indignation to acceptance. Can she have virtual access to the warehouse? Can she accompany a member of Central to the warehouse? Could she attend the Monday briefings? No. No. No.
It was clear, departments were not to be trusted with Central’s prize assets. There was protocol. There was a clear way of working. There would be delays. There would be inaccuracies that hampered the department’s planning, preparation, and fulfillment of its missions. But, according to Central, the way of working had been successful for eons and had rescued The Regime from the dark ages, moving them forwards by utilizing the information from the past to pave the way for a better future.
Apollonia wasn’t old enough to remember the times before Central, but her parents had often entertained her with tales of the times before. They were wise and honest folk who had been balanced in their description of the ‘dark ages’. Times were tough, there were hardships, but with ignorance, there was bliss. People didn’t miss what they didn’t know and as such reveled in a freedom that today’s centrally focused regime would not afford to its people. Apollonia wanted to find a balance, a way to take the advances made by Central, but with freedom for her and her people. She wanted control, or some control at least.
Adopting a stance similar to scientists who encountered religious extremists, Apollonia decided not to engage with Central regarding her concerns. Her acceptance was perfunctory. A guise. She had fight, but knew the fight could not be fought with those who were so fundamentally invested and blinkered in their own ideals.
Apollonia had found people like her. She was not alone.
Espionage
Three weeks had passed since Clarance had seen Zara smiling. Monday had rolled around again and all eyes were on Mr. Peters. Glancing over to Zara, a tick that Clarance had developed, he saw her smirking. It was slight, but he could see the glee, that she was happy. The smirk was slight, but real, and it troubled and intrigued Clarance. He was interrupted by Mr. Peters.
“I have an oddity to report. The Department of Supply is down 12 percent in requests this week. It’s most unusual. I’ve summoned their Principle to talk it through. Send her in.”
Clarence, sitting closest to the door, opened it to let Apollonia in. She walked in with purpose, smiled, and nodded to Clarance, walking to the head of the table opposite Mr. Peters. Clarence couldn’t help notice the purpose and confidence in her movements. He was impressed.
Mr. Peters spoke without formalities, “Why are your requests down 12 percent?”
“The data you provided in the previous weeks was high. We have been running with surplus inventory for three weeks now. You should be pleased as we have managed to reduce costs and keep supply to people with efficiency.” Apollonia spoke with confidence.
Mr. Peters didn’t look happy. He didn’t appreciate flaws being pointed out, especially not within the morning briefing – his domain of control. His eyes narrowed as he spoke, “You do know the punishment for going behind Central’s back?”
With these words he clicked on the holoprojector to show a naked man being dragged through the streets, people throwing rotten vegetables at him. He was crying. Humiliated. He carried a placard round his neck which read, ‘I am an anarchist’.
Apollonia didn’t say anything. She knew it was Carlson, the Principle of the Department of Human Affairs. She knew Carlson longed for freedom, but she was aware he was careless and forceful and was now suffering the humiliation that comes with going publicly against the autocratic, and all-powerful Central.
Mr. Peters stopped the holoprojector and said, “You may go.”
Mouth agape, Clarence watched as Apollonia slowly walked out of the room with the same poise and grace as her entrance. That’s when he saw it. The smallest of glances.
Zara and Apollonia knew each other. Were they in cahoots?
–
The day continued without further drama. The bell signaled work was over. Clarence was packing his bag to return home when he saw Zara leaving. Spontaneously, he followed her.
Unwanted Guest
It was immediately clear Zara wasn’t going home. Everyone who worked for Central lived in the same luxurious complex and Zara went in the other direction. She was walking at speed and Clarence struggled to keep her within sight.
He followed her for 20 minutes until she disappeared into a building. It was a grey relic of a bygone age. A concrete monstrosity compared to the modern materials used today.
Reaching the door, Clarence didn’t know what to do. Was unsure what he was up to. He stood there for a couple of seconds and then cautiously, slowly went through the door.
It was dark. The air was musky, Clarence could see light ahead, enough to navigate the debris that cluttered the hallway. He came to another door, where the light was escaping from. Peeking through he saw Zara talking, he pushed the door a little more open, and then saw Apollonia.
Frozen, Clarence couldn’t figure out why Zara and Apollonia were talking in secret and out of the office.
“What do you think you’re doing?” A voice from behind made Clarance jump as he felt strong hands push him forcefully into the room.
“Clarence?
“Oh Clarence, you shouldn’t be here. What are we to do with you?” Zara’s voice was worried. “I followed you. I’m sorry.”
“Why did you follow me?” “Curiosity.”
“Come in and sit down.” Apollonia commanded and Clarance did just that.
“You’re going to get us into trouble aren’t you? We can’t have that.” The large man in the doorway said in an intimidating tone.
“No, I was just curious.”
“You want to know why I’m here don’t you? I’ve seen you looking at me. What do you want, Clarence?” Zara’s tone was searching but more friendly.
“What are you doing here? What are you and Apollonia planning?
“You have it so good, well remunerated, you’re in control. Why would you jeopardize all that?” Clarence was flustered.
This time Apollonia stepped forward, closer to Clarence, smiling, “Clarence, have you never questioned the control Central has over all the departments? Have you never questioned what it is you’re looking for when you go into the vast warehouse armed with so little context? Have you ever questioned the output of your work and whether it has a positive or detrimental impact upon the recipients of that information?”
“Of course I have. But there is no other way.”
“There’s always another way and we’ve found it. There was a reason for that 12 percent drop. It was because we were 12 percent less reliant on Central. Next week we’ll be 20 percent, the week after more, until we won’t need Central at all.”
“Mr. Peters won’t let that happen. It can’t happen, you don’t have access to the warehouse.”
“We won’t need access soon. Zara here is getting us our historical information direct from the warehouse. Soon we’ll have it all and we’re piping all of the information we need directly to our department now, from our department and from our friends in the other departments.
“You don’t have the skills to pull this off.”
“That’s where you’re wrong, we’re smaller than Central. We need fewer resources, Zara here is guiding us and we’re following guidelines to roll our way to the other departments. Soon, they’ll all be in control of their own destinies, like us, and we’ll work together to share our information so we can serve our people to the best of our ability.
“When you walk through your warehouse, it’s reams and reams of data, you don’t know where anything is and when you do find it, you don’t even know what it is. You then have to work through what you find to establish meaning. Without intimate knowledge of that data you make too many assumptions and those guesses have cost my people their lives. People have starved, people have suffered, all because of Central’s inflexibility to relinquish control. To trust us. All the data we hold in the department is relevant to us, anything else we need, we borrow from our friends, we know where it is and what it means. We can predict, we can innovate, and make the right decisions for our people. Do you think you’ve helped The Regime deliver these things?”
Clarence was silent. In truth, he had no idea.
“Do you understand the more the population grows the more information we need to understand our production requirements, our forecasts, our modeling? If we don’t move away from Central we’ll stagnate, problems will escalate and suffering will increase. We need to adapt or die.
Apollonia was an impressive woman. She had purpose, a vision, and she was compelling. Clarence didn’t have a response. He knew she was right. He was frightened, would they let him go? Was he to be held captive, or worse?
Freedom
“Where’s Mr. Smith and Ms. Fernandez?” Silence filled the room.
“Ms. Patel, buzz their apartments. “I’ll begin anyway-”
Mr. Peters began the Monday morning briefing as per usual. Barking orders. He seemed agitated. He looked tired. Worried. This was unlike Mr. Peters who was normally so self-assured.
The worksheets were handed out and those who sat around the large mahogany table noticed immediately. They were short. The number of requests was down. Not just the Department of Supply, across the board.
Nobody said anything. There were glances exchanged, but with the tension that filled the air from Mr. Peters’ edgy demeanor, Central’s representatives knew better than to raise the issue. Less work wasn’t an issue for them in general, it was a change, a welcome relief.
–
The following Monday Mr. Peters looked physically unwell. He was jumpy and irritable. The morning briefing lasted barely five minutes. There was no sign of Clarence, and four other representatives were missing too. It made no difference to the workload, it was minimal, two sheets of requests for each person. Maybe four hours work maximum.
Mr. Peters stormed out of the room once the worksheets were handed out.
–
Mr. Peters was walking to the Monday morning briefing the following week. He put his hand on the door handle to enter but was interrupted by a friendly command, “The Chancellor would like to see you Mr. Peters. Please follow me.”
He exhaled slowly. He had been expecting this. Something had changed, quickly. The weeks previous had the same pattern and it all started with Apollonia.
Mr. Peters grimly followed the Chancellor’s aid through the brightly lit corridors. He was finished. He knew it.
The door was already open. Waiting for his demise. How many year’s service had he given to Central? All those hours, all his experience, all his knowhow. All for the greater good of The Regime.
Swallowing back his rage, Mr. Peters walked into the large, exquisitely decorated room and looked around.
Clarence, Zara, Apollonia, along with the other Central representatives who had been missing the previous week. Joining them at the table were the other Principles of the departments. This was a firing. An execution squad.
“Sit down please George.” The Chancellor said warmly, pulling out a chair.
Clarence nearly choked. George, 14 years and he hadn’t known Mr. Peters’ first name.
“Thank you for joining us George. As you will be well aware, Central has not had the same quota in recent weeks. It has been brought to my attention that the departments aren’t entirely happy with how their requests have been handled. Poor quality, missed deadlines, that kind of thing.
Mr. Peters sensed now wasn’t the time to talk and remained silent as the Chancellor pressed his fingers together.
“Apollonia sought counsel, which she has been attempting to do for some time now. I granted her the time last week as she made a compelling argument about relinquishing the power of Central and applying some trust in the departments over their own affairs.
“Would you like to know more George?” “Yes sir.”
“Well, Apollonia, with the help of Zara here, and lately Clarence, have been working to give the Department of Supply more autonomy over their work and their data. Apollonia has been arguing the case for better service for her people for some time now. I was loath to listen as I believed in Central’s approach and skill set. I was guided by you and your team. I was misguided.”
“If you will let me explain sir.”
“This isn’t a discussion George, this is a sermon. Please listen and watch carefully and you will understand that there is another way. A sense of freedom and a better way for The Regime to represent its people.
The Chancellor pressed play on the holoprojector and some charts and tables appeared.
“You will see the statistics here George. Over here are the production requirements, the forecasting model, and the population predictions. Here you will see production costs, the happiness index, and importantly, the shortfall chart that counts zero. Do you know the last time we went without a shortfall?
“Don’t answer that question, it was rhetorical. We have never gone without a shortfall, people have always gone without. That is until now.
“By taking control, albeit in an unorthodox and deceitful way, Apollonia has been able to use the information we have to predict the future and meet the needs of the people, without inflicting harm to them. Do you know why?”
Mr. Peters knew not to answer.
“Apollonia cares about her data. She cares about the results of the data. She understands her data. And she is passionate about her cause. Her purpose.
“With the close bonds between departments, she can extract and utilize data, for instance, the Department of Human Affairs provides their birthing model and provides an explanation of their data for Apollonia’s understanding. She is able to combine this shared knowledge for the good of the people. For the good of The Regime.
“What is more. She has been able to do this in a narrow timeframe and at a significantly lower cost. Human and fiscal.”
Mr. Peters closed his eyes. He knew what was coming.
“George, you have been the driving force behind Central for nearly 50 years. You have dedicated your life to us. You have made a difference and dragged The Regime from the mud to a place where we have served the people to the best of our ability and made a real difference.
“It would be remiss of me to forget that. You have an in-depth understanding of The Regime and the processes that keep our population alive. But Central has reached its capacity. Central can be no more. Representatives will be distributed to the departments to decentralize the power.”
Tears formed in Mr. Peters’ eyes.
“You have a choice George. Go your own way, we will honor your service. Or, and this will be painful for you, you can help the departments, govern them, set the rules, and guide them across all departments so that we work together in harmony. In unison. We have a chance to really make a difference and make things better for The Regime and its people.
“What do you say?, will you join us?”
–
