Smart Steps towards Test Automation

Greg Paskal, Director of Quality Assurance - Automation, Ramsey Solutions

Greg Paskal, Director of Quality Assurance - Automation, Ramsey Solutions

Your technology leadership is talking about test automation again and why it needs to become part of your IT testing strategy. On the surface, test automation sounds good, maybe even too good. The tools, techniques and methodologies of test automation are varied and can lead to some very expensive mistakes resulting in nothing to show for it other than spent effort and budgets.

The “Why” of Test Automation—As shared in the exceptional TED Talk by Simon Sinek (How great leaders inspire action), if we don’t have a clear understanding of the “Why” of test automation, we’re likely to make the wrong decisions regarding the “What” of test automation.

The best “Why” of Test Automation is the following. By creating and leveraging a consistent, reliable suite of automated tests, you’ll gain insights into quality and confidence that your tested product is performing to your standards and is ready for release.

Notice there’s no mention of savings in time or money and that’s for good reasons. Honestly, few test automation efforts initially result in saved time or money. The reason’s simple; test automation development is software development coupled with sounds testing practices. Where people are involved, time and money will be spent. Focusing on the wrong “Why” such as time and money savings will result in the revelation that it’s taking a lot of time and money to build what could simply be tested manually. Consistent execution of a reliable automated test suite, in the hands of a skilled manual tester is like giving a doctor the telemetry to know their patient is healthy.

"If we don’t have a clear understanding of the ‘Why’ of test automation, we’re likely to make the wrong decisions regarding the ‘What’ of test automation"

“How” to use Test Automation—Test automation works well as a tool in the tool belt of a manual test engineer. They leverage test automation to give insights into the overall health of an application. If the automation suddenly fails, yet has been running well up to this point, then they have added insight, something’s changed. When a manual tester executes test automation daily (regardless of a new build) they begin to understand the “personality” of an application and how their automation tools interact with it. Just like a parent noticing their child acting different one morning, a manual tester will identify automated test running different (failing, running slower, flaky) and know something’s up with the application. I call this the “Sick Kid Syndrome” and have found this approach invaluable in uncovering more defects than any other approach in automated testing. The symptom of something running differently, in most cases, leads to finding an issue in the application being tested by the automation.

“What” to Automate—A smart automation development strategy is built upon a solid manual testing strategy. Every automated test case should be traceable back to a manual test case. Automate the most critical and high priority tests and avoid lower priority tests. Implement a simple, “automation evaluation” process for new areas you are considering to automate. This ensures your tool can reliably interact with the application before committing to fully automate new functionality.

•Tip: Not everything makes a good automation candidate. Ensure the results from the automation evaluation indicate your automated test will be reliable. Know when to move forward and when to pull the plug on automated test development that leads to unreliability.

•Warning: Don’t automate applications or functionality that are likely to change soon. If you’re automating a new application, allow time for the application to mature before spending time and money to automate it.

•Warning: Be cautious of letting the test automation engineer simply automate whatever they believe is valuable. Just because screens are getting filled out with data and interacted with does not mean anything is getting tested. Be intentional about the test you want automated.

Build In-house or by Consultant—A big mistake, I see, made time and again is to hire a consultant to automate a bunch of test automation with no plan for how it will be maintained in the future. Consider early on, your sustainability plan, for the test automation to be built.

•Tip: Plan on hiring a test automation engineer: Building test automation means maintaining test automation going forward. Anticipate 10-15 percent of the test automation will need maintenance on an ongoing basis. If you do bring on a consultant to build test automation, ensure they have a reputation of building sustainable test automation and a plan for teaching your test automation engineer how to maintain it going forward.

•Tip: Have realistic expectations, it typically takes months to get the initial test automation suite up and running. Focus on automating a dozen or so of the most critical test cases in the initial effort.

•Warning: Never plan to automate the entire, manual regression suite. This is one area you’re likely to hear and unlikely to fulfill. The reason’s simple, test automation requires ongoing maintenance. The moment you build it, maintenance and updates will be necessary it to keep it working.

Nothing’s Free in Test Automation —In a survey, I conducted in 2016, the majority of test automation engineers indicated they chose their automation tool based on it being “free” verses it being the right tool for the job. Personally, I’ve yet to see anything “Free” when it comes to test automation. Do your homework when selecting an automation tool that is right for the work you need to get accomplished. Consider some of these important points.

•Does the tool provide historical reporting capabilities?
•Does the person executing the automation need to be a Unix Guru?
•Is the tool being continually developed and improved upon?
•Is it easy to upgrade when a new version is released?
•What secondary dependencies are necessary to make the tool work?
•What training is available, so the tool is used as designed, not as discovered?

•Tip: When evaluating your test automation tool, identify a source that can provide training and ongoing support. Even with Open Source tools, you can find experts that provide published training materials so you have a consistent and proven methodology to follow. If you buy an off the shelf tool, purchase training from the tool vendor and send an automation engineer who will be responsible for learning the materials and capable of teaching them to the rest of the automation team.

•Tip: When hiring new automation engineers, ensure they follow the training and usage methodology you’ve decided upon and are not introducing secondary methodologies. It’s important to hire teachable automation engineers that have a history of building sustainable, maintainable automation that’s been in use for at least a couple years or greater.

•Warning: Noting is free regarding test automation. While the tool may appear to be free, the talent to develop it and the tools necessary to use it will likely cost something, probably more than you expect.

Applying these tips and being aware of common problem areas will lead you in the right direction, to build and leverage test automation well.