In previous entries in this series, we covered how we used GitLab Duo to generate tests for our code as well as what we learned while using GitLab Duo for generating automated tests. We also shared some of the ways we addressed making changes to GitLab Duo generated tests. This last article in the series will cover a test we ran to validate the impact of GitLab Duo on our team’s automated testing and discuss the impressive results we have achieved thus far.

Validation testing results

To validate that our usage of GitLab Duo to generate tests was adding value the way we expected, we challenged ourselves and GitLab Duo to replace and increase test coverage. The team removed all previously written tests to get our test coverage to 0% and then methodically went through the repository and created new test files to store GitLab Duo-generated tests.

From this starting point, the team followed the steps outlined in the first blog to generate tests. Tests and test files were unmodified by humans to provide a stable control group and a Tests Generated by Duo comment at the top of them were suffixed by duo.py to indicate where the tests came from.

All iterations of the tests were only done through interactions with GitLab Duo through the Generate Tests and GitLab Duo Chat window as outlined in the second blog in the series. As we shared, we requested GitLab Duo to make updates based on encountered errors, test failures, and example code snippets for GitLab Duo to use as added context.

At all times. when testing with GitLab Duo, we were running tests and coverage reports so we could see if our GitLab Duo-generated tests were increasing testing coverage and adding value as we expected. Taking advantage of GitLab's test coverage visualization, we were able to continuously monitor the results of our work.

Ultimately, after using GitLab Duo to regenerate tests for code previously covered through our mostly manual testing, we were able to achieve test coverage of 84%. This was a great accomplishment for the team because:

1. It was a significant improvement from prior coverage, which was at 74%.
2. It took approximately two days by one engineer to achieve 84%, compared to the approximately four weeks across multiple engineers that the 74% had taken.

Since this experiment, the team has increased coverage even further to 89% with the help of GitLab Duo, while continuing to introduce new features.

Using GitLab Duo allowed for increased testing efficiency and coverage, and also allowed developers with lower context around existing code to write valuable tests, quickly. This has resulted in increased confidence on the team to develop new features without worrying about introducing errors.

If you'd like to try GitLab Duo, sign up for a free, 60-day trial today!