Tips for successful debugging of automation tests

A set of simple techniques that I use personally to make the debugging of automation tests as smooth as possible

It may sound like a cliche, but the idea of this article came to me in a dream. I was dreaming about speaking at a conference and the topic I was talking about was the ways of debugging your automation tests. I woke up and noted down the idea for the article. Today, around 2 months later, I finally found time to share my thoughts with you.

Debugging skills matter

So, why do they matter? Simply, because during your work you will spend more time debugging tests than time creating them. So, while writing test code the main focus should be on keeping this code well maintainable and debuggable for the sake of future you and your teammates.

Debugging is not an art, but just a set of techniques that, applied thoroughly, will help you find a bug in code. It may be surprising, but not all of these techniques are straightly related to debugging code.

Without further ado, let’s proceed to discuss these techniques.

Clean code and best practices

I started with this one because, in my opinion, it’s the most important rule of all. It may be not so obvious, but let’s imagine a situation, you are investigating a broken test and have narrowed down the area of your search to the following method.

Poor code readability makes it much harder to debug an issue

It’s not that clear what this function does and what could be broken, so you have to spend more of your time to further debugging an issue.

Now let’s look at the same function but using meaningful variable names and much well structured.

With clean code, you clearly see an issue after a first look

Now you can obviously see that function reads construction years from the search results page, converts them to integers, and checks that all construction years are equal to or greater than a base year value provided in the test.

And now you can clearly see the error in this expression: if (year > baseYear) return false; Obviously, we should return false when year < baseYear according to the function name. So, as you can see, poor code readability may not only harden your debug experience but also be a cause of mistakes in the first place.

The same goes for best practices and engineering principles, like Page Object, DRY, KISS, SOLID, etc. It’s much easier to find a problem in a class that has a single responsibility or that is structured accordingly to some rules assumed by the team working on the project. This is applicable for using standard design patterns as well. If a new person has a small code base knowledge and has to do debugging in a stressful situation when they see commonly used patterns like Wrapper or Factory the time to understand the code and where the issue drastically decreases.

Extensive reporting

This tool is the only one here that is unique for automation tests and it could be an ace in a hole. The original function of test reports is to display results of testing activities to product and business people and give a broader set of people the ability to interpret test results and give them more clearance on what the quality team is doing.

But you can (and you have to) utilize this report for your debugging needs as much as possible. Add screenshots and videos in case the test failed, try to log all the actions, and don’t forget to include all available data in this log(incoming state, action, outcome).

The next code snippet utilizes Allure report annotations to give more information about what the method is doing and in case there would be something wrong with incoming returned values you can narrow your search down to several lines of code.

Verbose logging

The next thing I want to talk about is logging. Having well-structured verbose logging is crucial for debugging every piece of software, not only automated tests. Its usage may be a bit different for automated tests when compared to some production code because usually for test automation we don’t use logging and monitoring tools like DataDog or Grafana.

When it comes to debugging failed tests on CI often you will be reading a log from a container where your tests were executing. And this circumstance puts even more pressure on how well your logs should be structured and verbose. I like my logs to have all the information that is available for the method. i.e. input values, actions performed in the function, and the outcome. Very often I log the same information as I add to the test report.

It may look like an overhead, but there were situations in my career where such logs helped me to identify the issue where reports were not useful at all. (i.e. once there were flaky tests using sharing test data and it was not identifiable with test reports, only after investigation of test logs I could define the issue).

Clear failure message

I’d like to stress this out as, in my opinion, the most important part of test logging. When a test has failed this message is the first what you will see in the console and having something meaningful might (and very often will) save you a lot of time on debugging.

Of course, the existence of such a message by itself doesn’t help you a lot if the message is ambiguous and doesn’t have any useful information. A good failure message should have expected and actual results and also some context to help understand a failure better.

Assertion libraries like Hamcrest force you to write assertions in a comprehensive manner and build understandable failure messages depending on them. But you can go even further and provide a custom reason message that will make s failure message even more clear.

For example, if the next assertion fails

assertThat("User is logged in", homePage.hasUserLoggedIn, is(true));

the displayed failure message will give you all the information you need to quickly understand the issue

java.lang.AssertionError: User is logged in
Expected: is <true>
     but: was <false>

Know your tools

When it comes to debugging code, all modern IDEs give you plenty of tools to improve your experience. Debuggers are very smart and it would not be clever not to use them.

When you start working in a new IDE, spend some time in a calm situation and learn how the debugger works there, try to simply debug several functions to build an understanding of the functions that debugger has, what elements you have to control the process and how to debug some complex cases, i.e. multi-thread debugging. You may also want to learn keyboard shortcuts to improve your experience.

Sometimes there are situations when you cannot reproduce an issue locally. In this case, you might want to investigate what tools allow you to perform remote debugging. For example, enabling the VNC option in Selenoid will allow you to connect to a Selenoid container and debug your distributed test in real-time using your IDE.

In addition to previously discussed tools like logging and reporting which are very useful for remote debugging, you may not want to limit your toolset to them but rather explore other opportunities.

Conclusion

Everything I was talking about just reflects my own experience that I gained making a lot of mistakes and fixing them during my career and I am not in any way claiming that you have to accept my thoughts without hesitation. Critical thinking and healthy skepticism are the wind that pushes progress ahead.

Don’t be afraid of debugging. Exercise the above-mentioned techniques, come up with your own ones, constantly extend your toolset and you will find out that debugging is not as scary as you thought.

Thanks for reading this article! If you like it please clap and subscribe if you want to receive notifications about my new articles.

If you have questions or just want to connect, you can find me on LinkedIn.