4 minutes read

I’m still not sure who likes these End-2-End testing environments. When I start with a rant about the environments in presentations I see a lot of people nodding, putting their thumbs up and smiling. Actually nobody came to me and told me they are in favor of these environments. Maybe it’s because of my rant, but I hope I’m accessible enough to speak up and share your thoughts even if they don’t align with mine.   While talking with people about their setup of End-2-End test environments, they ask me what I think about their setup. All kind of setups are suggested and I always start my answer with: ‘It depends’. It might be a good fit for your organization now, but they are never the best solution. Maybe it is not worth the effort of getting rid of it, but that doesn’t make them a good thing. If I’m not the only one that thinks bad about these environments, then why are they still around?

So, what is wrong with these environments from hell….. - Test data hell - Not so “production like” - Test scope - Unstable tests - Slow Feedback time - Increased time-to-market - Costs

In this and the following blog posts, I want to briefly explain these points.

Test data hell

The test data in these kinds of environments are “precious”… The quality of the (production like) test data in End-2-End test environments is in a large part accountable for the quality of the tests. Services are communicating to ‘real’ test systems, no stubs or mocks. Because the test data is being served by code/deployments that are similar to production, the quality of the test data is likely to be better than test data coming from stubs/mocks. This is one of the reasons people rely on End-2-End test environments.

So where is the Hell in this? Well, the problem is not using it, the problem is creating and maintaining it.   Because, believe it or not, the test data currently used in the End-2-End test environments is once created. Assuming End-2-End environments consist of multiple systems, owned by multiple teams, this means that the data in these systems is also owned and created by multiple teams. When new data and/or changes are needed, you need to align this with multiple teams and systems. Everything needs to be correlated between these systems. You can imagine that this takes a lot of effort in a larger setup. The larger the End-2-End environment, the harder it will be.   How about consuming the data? If you consume/change the data by doing a test, how are you going to reset it back to the state before the test? How are you going to make sure you have stable tests that are not flaky? And how are you going to automate this? These things are much harder when a team executing the tests is not in control and systems and data belong to different teams.

Not so “production like”

The End-2-End test environments might not be as production-like as expected. Although it is sometimes called “Pre Production” it is not as production-like as the name suggests. With multiple systems and multiple release cycles, the environment is changing often. Teams using these environments are deploying their releases on the End-2-End environment before deploying it to production. The time between deploying it to an End-2-End environment and on production breaks the “Production like” state of the environment for others. Things like Continues Deployment and automation will make it worse. When the teams are doing incremental small changes and deployments multiple times a day/week, the pace of changing will make the End-2-End test environment less production like for everybody. It seems every team is doing is utterly best to make it less production like for other teams. Passing tests ran against these not so production like environment will give false confidence, which might be worse then not running tests at all….

Recent posts