Maximize Test Automation ROI with Test Data Generation
Test Data Generation (TDG) is the perfect companion technology for test automation. By their nature, test automation tools require that test data be provisioned at high velocity. With TDG, provisioning speed is not a problem because test data can be generated instantly when the automated tests run.
With the help of TDG, realizing the full benefit of continuous testing can be a very achievable goal. However, transitioning from traditional test data provisioning to TDG can be perplexing. For many QA professionals, it’s something of a “where do I begin” dilemma. And it’s a classic change management challenge to overcome.
Change is always difficult and the inertia of current practices can be hard to break. One of the principles of successful change management is to identify quick wins that establish early success and reinforce the team’s commitment to a new direction. Quick wins combine achievable outcomes and measurable benefits with the use of best practices that can be used and expanded over time.
Selecting quick wins for TDG deployment should be guided by your test automation deployment. Which tests you choose to automate first will have a profound effect on productivity gains. Because some categories of testing provide greater ROI than others in terms of time and cost savings, it’s best to identify areas of testing where automaton and test data generation will have the greatest impact.
According to a survey of 2,291 QA professionals conducted by ToolsQA, functional and regression testing are the most common areas for the application of test automation.
Functional testing is an essential role performed by QA to verify the system or application under test fully meets the requirements of its functional specification.
Non-functional testing exercises the performance, reliability, scalability, compatibility, security, and other non-functional aspects of the software.
The Software Test Pyramid provides a useful way to look at the operational nature of testing. Lower on the pyramid are tests performed at high volume and on a smaller code base. Tests that are higher on the pyramid are performed at a lower volume and on a larger code base.
Functional testing can be performed at all layers of the software test pyramid – Unit Testing, Service Testing and User Interface Testing. Functional tests performed at the unit testing layer, often referred to as white box testing, are often performed by developers who examine source code internals to ensure they conform to the business logic the software is meant to execute.
Functional testing performed at the Service and UI layers is often referred to as black box testing and requires no knowledge of internal code structure or logic. The concept is to provide inputs to a “black box” and compare actual results with expected results. This kind of functional testing lends itself to the practice of Data Driven Testing, an approach to testing where an external data source is used to feed a test case with a large number of test data variations in a controlled fashion in order to fully exercise the code. This technique enables positive testing (based on likely, valid and complete input data) and negative testing (based on unlikely, invalid and missing input data).
The middle of the software test pyramid represents a sweet spot for test automation tools integrated with test data generation. Service layer tests for functional, performance and regression testing all share three important attributes that maximize ROI. That’s because service layer tests represent tests that:
- Require a high-volume of test cases
- Involve tests that are highly repetitive
- Require no knowledge of source code internals
- Are data-intensive and require controlled test data inputs
Software tests that are performed manually, and on a repeated basis, should be automated first to save time, reduce project cost and apply data driven testing principles that maximize coverage and quality.
Based on this analysis, the best types of tests to start deploying TDG with automated testing are:
- Data-driven functional testing
- Functional tests repurposed for performance testing
- Functional and performance tests that rerun for regression testing
This approach provides the best path to quick wins because each functional test can be repurposed for multiple test categories with test data configured by the TDG platform based on reusable test data scenarios.
An Example of Test Automation, TDG and Data-Driven Testing
Here is a simple example that illustrates the power of test automation, TDG and data driven testing. Let’s assume that we want to test a login system having multiple input fields with 1,000 different data sets.
To test this, you can take following different approaches:
Approach 1) Create 1,000 scripts one for each dataset and run each test separately one by one.
Approach 2) Manually change the value in the test script and run it multiple times.
Approach 3) Import test data from an excel sheet, fetch data from excel rows one by one and execute the script.
Approach 4) Create a Test Data Scenario using GenRocket’s TDG platform to replace multiple test scripts and generate 1,000 iterations of patterned, conditioned, controlled test data – or any volume of data needed at the rate of 10,000 rows per second.
Clearly Approach 4 is the preferred method for a test automation environment. An investment of 10 minutes to create a Test Data Scenario for the TDG platform translates to test execution time of just a few seconds with a test case that can be reused for performance testing and regression testing. Following is an example of the kind of test data that can be defined and generated by the TDG platform. Notice the many variations that are possible and the controlled manner in which the test data can be used by test cases.
The sample test data in the table above represents just 10 rows of data. Think of the time savings and coverage improvement when 1,000 rows (or more) are generated in a matter of seconds. Now consider the ability to run load, stress, or performance tests with a million rows of data generated by a TDG platform in a matter of minutes.
Combined Functional, Performance and Regression Testing
One way to maximize the ROI of your test automation investment and get some quick wins during your transition to TDG is to combine multiple test operations into a single testing process. Start by automating your functional test cases and configure your test data scenarios to maximize coverage. Specify test data for both positive and negative testing purposes. Now include the assertions that validate code functionality and identify defects.
Then as a separate test case, change your assertions to gauge the performance of the application by increasing load conditions. With GenRocket, the volume of your test data can be easily scaled up or down simply by changing the loop count for a given test data scenario. You can vary the load in a structured, methodical way to systematically assess the impact on performance.
Additionally, you can perform automated regression testing simply by rerunning the functional and performance tests for each new release. Regression testing must always be performed with consistent test data and with GenRocket, test data is always fresh as it is generated new each time for every test. And if the “Seed” feature is used the identical data (even if it is random data) is generated every time.
The diagram below illustrates the concept of a consolidated test suite that simultaneously performs functional tests with controlled test data and also performs performance testing by varying the application load with increasing data volume.
If your performance testing calls for a variety of test cases, you can still leverage your functional tests by repurposing them, and their test data scenarios, for various types of performance testing such as:
- Capacity Testing: Increasing the number of application users
- Load Testing: Increasing the number of simultaneous transactions
- Volume Testing: Increasing the volume of data handled by the system
- Stress Testing: Testing for application behavior at loads beyond expected capacity
- Soak Testing: Also called Endurance Testing, refers to examining the impact of load over time
Regardless of the type of test, the important concept to keep in mind is this: With TDG, the speed and effectiveness of automated testing is no longer dependent on the availability of test data. Any volume or variety of data can be configured in a test data scenario and generated in real-time by the GenRocket TDG engine. Carefully selecting which test procedures to automate first and using the power and flexibility of TDG to enable multiple categories of testing is the best path to maximizing your test automation environment ROI.
If you would like to learn more about integrating TDG with test automation tools, just schedule a demonstration.
