Originally posted on , until its entire network, which was still based on automated phone routing systems, crashed. Planes and crews had to be flown home empty, the worst possible inefficiency, just to give its systems a place from which to start over. A literal “have you tried turning it off and on again”?
The Southwest example is one of truly ancient architecture, but the problem of prioritizing easy solutions over quality affects modern microservice architectures as well. In the world of microservice architecture, we see engineers valuing the speed of and consistency can often be solved by “quick fixes” that work well for a given scale.
What follows is a list of the ways that platform teams take shortcuts to make testing and releasing code to “just work”’ as they increase in scale, and how those shortcuts come back to bite you.
How Platform Teams Prioritize Speed Over Quality
I’d like to go over some of the failure modes we see in modern architecture teams.
Over-Rotating to Unit Testing
Talking to multiple platform engineers, one of the recent themes has been a renewed emphasis on unit testing. Unit testing is an appealing option since, generally running on a developer’s laptop, it runs quickly and efficiently.
In an ideal world, the service each developer is working on would be nicely isolated from others, and with a clear spec for the performance of the service, unit tests should cover all test cases. But sadly we develop in the real world, and interdependence between services is common. In cases where requests pass back and forth between related services, unit tests struggle to test in realistic ways. And a constantly updated set of services means that even efforts to document requirements can’t stay up to date.
(or whatever other environment you deploy to before production). When developers push their pull requests without being certain they’ll work, their testing is faster, but the time to get real feedback is slower. As a result, the developer’s feedback loop is slower. Developers wait longer to find out if their code passes integration testing, meaning that implementation of features takes longer. Slower
Setting up multiple lower-order environments (that is, environments that are smaller and easier to manage than staging) has a number of drawbacks, the biggest being test quality. When tests are run with mocks and dummy data, the reliability of passing tests can become quite low. We run the risk of maintaining (and paying for) environments that really aren’t usable for testing.
Another concern is synchronization; with many environments running clones of a service, it’s very difficult to keep all those services updated.
In a recent case study with Fintech company Brex, platform developers talked about a
This does absolutely solve the problem of test quality. When these tests run, we are truly sure that they’re accurate. But we now find we’ve gone so far in pursuit of quality that we abandoned speed. We’re waiting for every merge and tweak to be done before we run a massive suite of tests. And worse, we’ve gone back to a state where developers are waiting hours or days to know if their code is working.
The Promise of Sandboxes
The emphasis on quick-running tests and a desire to give .
What It Takes To Implement Sandboxes
Let’s go over the requirements for a sandbox.
for the sake of speed often leads to costly consequences down the line. While duplicating environments might appear to be a quick fix for ensuring consistency, the financial burden of maintaining these setups can escalate rapidly.
The temptation to rush through testing, skip comprehensive checks or rely on incomplete staging setups is understandable under pressure. However, this approach can result in undetected issues, unstable releases and eventually, more time and resources spent fixing problems in production. The hidden costs of prioritizing speed over thorough testing are felt in delayed projects, frustrated teams and lost customer trust.
At Signadot, we recognize that effective testing doesn’t have to come with prohibitive costs or slow down development cycles. Using strategies like dynamic provisioning and request isolation, we offer a way to streamline the testing process while keeping infrastructure costs under control. Our shared test environment solutions allow teams to perform safe, canary-style tests without duplicating environments, resulting in significant cost savings and more reliable staging setups.
SOCIAL SHARE CARD GENERATOR