The Hidden (Obvious) Cost of Not Testing

"First of all, Are you serious?"

Deadly serious, I'm afraid. Unit, Mocking, Stubs, Load Testing, Integration, if any of these mean anything to you, congratulations you've been doing software engineering long enough while still keeping your sanity, if you dont know these words, then this article is for you.

So, I want to present a scenario, meet Dave, Dave is a "Get Stuff Done" guy, goes above and beyond each time, Dave does overtime, everyone in our hypothetical company can call on Dave, he works nights, weekends, early to work, you name it, Dave is your guy, however Dave doesnt write a lick of tests, nothing.

He's all about the point-point-click-click-until-it-works school of engineering babyyy, each feature Dave pushes out requires a full point-point-click session to confirm that things are still working, "No Biggie" he thinks, this has worked for years for Dave, and he's still managed to deliver several big systems for the company.

The company values Dave, and management sends all the critical issues to him which eventually get resolved, you get the picture.

Recently, management has decided that their systems are too isolated and need to be integrated, for argument's sake, let's say we have three hypothetical systems.

Mango, Berry, and Pear systems were singlehandedly implemented by our get-stuff-done guy Dave, woohoo Dave!

Management, being the good managers they are decide to bring in more engineers to assist Dave with the integration and to speed up the process, they need more business insights!

Dave takes a look at the scope of the project and gives management a 3-month timeline to have all the systems happily talking and laughing with each other, "it's GO time" he thinks, he is ready.

The First Challenge

The first hiccup in the big project is that the external engineers brought in have started cranking out features on Berry at light speed and have started to change most of what Dave has written, a bead of sweat rolls down his temple, "that's cool, I can just scrape some time to go back over testing that part" he thinks. Dave continues to work assiduously towards connecting Mango and Pear, he does his usual point-point-click routine and breathes a sigh of relief, it seems to be working, "Awesome! Time to send these off" he shouts with a grin of confidence, everything should be good, after all, he did test it himself. He loads up his changes and straight to production it goes.

As Dave decides to take lunch after a productive morning, his phone rings, it's someone from Order Management, which the Berry Platform is responsible for, "Uh oh", he thinks,

"Hey Dave, the Orders we're getting from Berry are coming over with mismatched invoices and some dollar figure calulations are way off, this is impacting revenue, we need you to look an ASAP!"

Dave does a quick U-turn, no time for lunch, back to his desk, he has to fix this now, and he starts another point-point-click session, generating Orders on the Berry Platform takes forever and he really doesnt have the time, he holds his head down and he continues. After a couple of hours, he finally made headway on solving the problem, a function he had written behaved differently when passed different inputs, he cursed floating point math in his head and silently implemented the fix, up to production it goes and the crisis is averted for now.

It's way past closing and Dave is exhausted from all the Orders he had to manually generate for testing the Berry platform, "It's over" he thought as he heads home for the day.

The Next Challenge

Unfortunately for Dave, this continued for the duration of the three-month timeline he gave management, he would make a change, point-point-click, the external dev team would make a change, point-point-click. Bugs came from Dave,bugs came from the team, back and forth they went.

The three months quickly zoomed by, management came knocking..

"Hey Dave, we've scheduled a small demo for this week for you to show us the progress, looking forward to it!"

At this point, management had exhausted its budget on this external dev team, so expectations were high. Dave's heart began to race, the integration of the systems was not going as planned, they spent most of the time ensuring that the existing systems still worked as expected and no major integration features were shipped.

Dave felt his eyes tingle, dry as firewood, he's been going through multiple cycles of testing between Mango, Berry and Pear. Orders are being generated in Berry but not being shown coming over in Pear, and worse so, Mango is generating incorrect orders altogether now. He needs to break the news to management.

"Three more months?!?! Are you crazy? We spent all our budget and we needed this integration! Our shareholders are expecting us to deliver something big for Q3!"

Dave continues to push, he tries his best, he pushes out more code to account for several edge cases and point-point-clicks away at testing the system. "I can get it done," he thinks.

Suddenly he gets a call from Operations

"Hey Dave, we arent getting printout for receipts from Mango, can you take a look please?"

"Mango?!" he screams to himself, "I JUST finished doing a full testing session with that, the external dev team must have pushed some changes" he ponders.

"No, sorry Dave, we havent touched the Mango repository"

He sets off to find the root cause of what changed, it takes him literal days, the orders pile up, Operations come under fire, Dave continues his search...

After 4 days, Dave realizes that the problem wasnt actually from Mango, all that time wasted, a malformed payload he didnt account for in Berry caused the outage, management is furious.

The Final Challenge

It continues like this for several months, extension after extension, at this point the simple integration project is a year and a half overdue, the company's share price took a downward turn as a result of failing to deliver meaningful features, Dave is at a breaking point.

His final deadline, management's patience has ran out, they decide to give Dave one final extension, after this it's do or die, they must go to production with these integrations.

He cracks his knuckles and straightens his back, months of being slumped over has done a number on his posture and wrists, his final frontier. Dave begins a rigorous testing session on Berry, Mango and Pear, and decides that he will finish all the integrations in two weeks. The first week is smooth, it appears he's making progress, the external dev team is point-point-clicking around the clock, they're at capacity too, it's the home stretch.

Two days before their go live date, Dave notices a small blip on his phone, an email from Operations "No time for that" he thinks, he must push on, he's almost there.

Another blip
And another blip

"What is nowww" he groans, surely it can wait..

"Hey Dave, I know you have your deadline but Im seeing something weird on Berry.."

The orders on Berry are disappearing after the status is changed, "seems like a small change" he thinks, "I'll get to it after go-live".
Dave pushes on to Go Live date and he's finally made it, all systems tested and accounted for and Integration seems ready.

The team all meets in the conference room for the big Go-Live launch, Dave is confident he has account for all the scenarios, after all it's been a year and a half of mostly manual clicks, surely he's covered everything.

"3, 2, 1...We're Live!"

All the team members cheer, it's finally over, they can take a break now, a collective sigh of relief. The integration is just in time for the shareholders Annual General Meeting, they made it.

Blip
Blip
Blip
Blip
Blip

"It's Operations, I wonder what happened" Dave thought with baited breath, it might be that small issue from earlier

"Dave! Our old olders are being deleted!!!"

Dave's blood runs cold, the shareholder meeting is almost here, how can he fix it, he just went live, "Can we roll back? Can we pause? Testing took up all my time", he didnt have the time to implement a roll back plan, he waits for the calamity to start. Dave panics, he feels his heart race, his vision gets blurry.

Dave slumps over in his chair defeated, "What can I do now?" he's out of options, he has to tell management.

Aftermath

Surely as usual, Dave managed to fix the problem with Berry, another use case he didnt account for which changed some previous behaviour. The Shareholder meeting was a disaster, the company had expended all of it's operating capital on the Integration project, missed it's timelines and delivered a half baked project to save face. The share price took a nose dive and layoffs ensued, Dave somehow survived, only a demotion and a pay cut was the price he had to pay. "Never again" he thought, "Never again".

Conclusion

This is but a cautionary tale, the cost of not testing systems (You'd think) would be obvious, what started as an extra 5 minutes of manual testing, quickly sprawled into a company wide failure and devaluation. As engineers, we often dont see the totality and downstream of bad engineering decisions, especially when we arent in a position to (Not in management).

It's quite easy to get wrapped up in the nuances of delivering and the false sense of pride from writing something only ourselves can understand.

We. Must. Test.

Had Dave had unit, integration and end to end tests in place, he could have saved his well-being and potentially the company, had he written tests that showed how functions and by extension the entire system behaved under certain circumstances, he could have collaborated easily with his external dev team, knowing instantly when things were broken and when behaviour had changed. He could have saved his weekends, his nights, his hobbies, his life could have, quite literally been different.

I harp on this point because, if you're reading this and you have aspirations of becoming an engineer, you MUST include testing as apart of your journey, not as chore, but as a trusty companion that saves you COUNTLESS of hours, that Dave unfortunately could not.

Learn it, embrace it, think about the consequences, we must test.