Quality assurance: autotests that bring profits

Autotests represent a tool that one needs to apply properly. To do so, we have to outline the goals that we pursue when implementing such practice. I would highlight the following ones:

• Reducing the cost and time spent on manual testing;
• Speeding up software releases;
• Increasing quality and reducing the number of defects and risks.
  

Let's look at how we can achieve each of these goals using autotests.‍‍

Reduced manual testing costs

A constant expansion of the project, introduction of new features, framework’s version updates, etc. increases the load on the manual test team. If the workload is increasing and the manual testing team requires more and more members, then you should implement test automation in your processes. Over time, autotests will cover old stable features and free up time for manual testing. This way, you won’t need to expand your team - instead, you’ll save money. Besides, manual QA specialists will actually do the testing and not “conduct test cases.”

Reducing the time of manual testing results in faster delivery time

A business needs to deliver changes to users as quickly as possible. As autotests are launched frequently, bugs will be found and fixed at an early stage. That practice reduces the time spent on testing and fixing prior to the eventual release. Second, the speed of autotests is faster than manual checks. Implementing automated testing into processes can significantly reduce the time it takes to release updates. This way, you can stay ahead of the competition with the release of a new killer feature.‍

Erik Fogg, COO at ProdPerfect expands the idea: “The value of automated testing comes from re-testing your application or services end-to-end for little more than the cost of computation. Compared to completely manual testing, you will save not only money but also time. It becomes possible to deploy small changes independently without having to wait for a scheduled release day. This increased development speed allows you to move your business much faster and flexibly. Developer velocity gains are hard to measure but easy to feel, and the benefit is huge”.

Increasing quality and reducing the number of defects and risks

Of course, manual QA specialists are very skilled and won’t let a single bug pass undetected. However, let's suppose you have a mobile bank application (or a billing system) that contains a huge number of fields with 15-digit numbers, dates, and transactions on each screen. Never forget about the human factor: a mistake in such an application can be really costly. When checking a vast amount of data, you can always miss an error due to the so-called effect of losing the freshness of vision. In such cases, autotests are indispensable. And guess what: manual QA specialists will only be happy to shift this routine to autotests.

Here’s Erik Fogg, COO of a data-driven testing service ProdPerfect on the matter: “The biggest benefit of automated testing is the early detection of issues and trust in the build. Both enable us to increase the speed of delivery, as well as developer productivity, significantly. Fundamentally, one could achieve equal test coverage with manual tests, but it’s expensive and slow to do so as your team and codebase grow”.

How do you know when to switch from manual to automated testing?

Long-term projects. For short-term projects with a small number of releases, it will be more profitable to perform manual testing several times.

The functionality and environment are quite stable. The more stable the functionality is, the lower the cost of maintaining the tests turns out to be.

The functionality should be tested regularly and frequently. The more often functionality is tested, the faster its automation pays off.

Technical ability to test using autotests. There are many test cases that are technically impossible to automate. Besides, automating a certain functionality is sometimes labor-intensive and such autotests are hard to maintain. In this case, it is more profitable to use the manual testing.

Сross-platform or cross-browser testing. Automation tools allow you to use the same code for different browsers and platforms. Therefore, having spent the resources to develop a single autotest, you can run it on different platforms (browsers and device versions). 

Brian Stewart, Project Manager at ProsperoWeb, shares his experience: “We create web applications, and they need to be tested on the basis of cross-platform and cross-browser compatibility. We do use autotests for the same and it does save a lot of time.”

In regard to these points, a question needs to be asked. Can auto testing replace manual testing?

‍Dave Lane from Twin Sun explains: “I believe automated testing and manual testing serve different purposes. Automated testing should ensure that your code technically fulfills the requirements so far as the developer understood them. Manual testing should ensure that your software functions in an intuitive and predictable way for your average user. A team with a good appreciation of automated testing will not replace manual testing: instead, they'll shift the responsibility of manual testing up higher in the value chain. Manual testers will not need to spend as much time verifying that the application works. Instead, they'll begin focusing on the overall user experience and think about how to improve the software”.

There are so-called ROI (Return on Investment) calculators. Based on a number of parameters, they help to determine at what point automated testing becomes more profitable than manual testing.

However, it is essential to understand that the calculations will be very approximate since it is extremely difficult to predict and take into account everything with high accuracy. In addition, automation only replaces some of the manual checks. At the same time, automation itself is a product that asks for a technical debt to be repaid, and this should also be taken into account.

Bobby from Blue Label Labs sums it up perfectly: “While automated testing can catch some issues, it often requires a considerable amount of intervention to configure specifics for use scenarios as well as to make sure it’s performing up to speed. A lot of times, automated testing ends being about as methodical and effective as a Roomba in a messy house”.

To calculate the effectiveness of test automation for the selected functionality, you need:

• Manual testing costs (in person-hours);
• The cost of developing autotests (in person-hours);
• The number of testing iterations.

All in all, test automation reduces costs for the business, and the economic benefits increase steadily over time. The main thing is to know how exactly a business may apply this powerful tool with maximum efficiency.

‍Artur Shaikhutdinov, QA Automation Engineer at Deutsche Bank, sums it up: “The initial investment in test automation can be very high. The acquisition of software, the cost of training to work with it, the design and creation of automated tests - all this requires a lot of time and money. However, once you start developing more and more new features in your product, manual testing ends up being more expensive and automated testing cheaper.”