Test Data Automation Performance Criteria

The GenRocket engine is designed to run on almost any platform. It supports Windows, OSX and Linux and runs on the latest high-performance systems as well as legacy PC architectures. Its benchmark performance standard of 10,000 rows per second* was gauged using a modest PC typical of any tester’s desktop or laptop system. While faster machines will deliver higher performance, there is no requirement to upgrade existing hardware to operate GenRocket.

At the same time, the transition to test automation places new demands on test data provisioning. In the case of manual testing, reducing test data provisioning time from days and weeks to same-day provisioning is certainly game-changing. However, the advent of test automation requires data provisioning to be performed in minutes or seconds. GenRocket’s Test Data Automation platform is fully capable of meeting this demand.

Three environmental factors impact the performance of the GenRocket engine – the client’s hardware environment, the network environment and the database server environment. Each factor plays a role in performance. Optimizing the speed of test data generation requires attention to each one.

The Hardware Environment

Starting with the hardware environment, the client machine running GenRocket software plays a major role in performance. The CPU speed, number of cores, memory size and storage media are all involved in the speed of test data generation. However, the performance experienced by users depends on their application and use case.

Low volume test data generation scenarios of a few hundred to a few thousand rows of data require less sophisticated hardware. High volume requirements of 1,000,000 rows (or more) require faster hardware. Test Data Scenarios involving sequential database operations or large-scale data migration testing require the most advanced system requirements.

Although GenRocket is able to run on nearly any machine that supports Java, we recommend a minimum system with the following characteristics for basic test data provisioning applications.

• System Hardware: 4 Core CPU with 8GB RAM
• Operating System: Linux, macOS, and Windows
• Browser: Chrome, Firefox, Safari and others* (no IE)
• Java: 1.7 (JDK) GenRocket does not yet support Java 1.8 u20 or Java 1.9+

However, for modern test automation environments we recommend your machine meets or exceeds the following specifications:

• 4 Core CPU
• 16 GB RAM
• SSD Hard drive
• Linux/Unix based operating system

In addition to increasing the system resources of the target machine, testers can also use GenRocket’s Partition Engine to scale performance beyond the capacity of a single system. The Partition Engine is used to balance the test data generation load across multiple servers to generate millions or billions of rows of data in a very short span of time. For example, 10 servers each running 10 instances can generate 1 billion rows of test data in less than 20 minutes. Learn more about the GenRocket Partition Engine.

The Network Environment

The second key performance factor is the network environment. If your test operation and its need to access data are confined to a local area network, then your performance should not be impacted by the network. However, it’s not unusual for test procedures to involve access to remote resources, especially if testing is performed by outsourced staff and over an Internet-based VPN connection.

Remote access introduces the elements of:

• Bandwidth
• Utilization
• Latency

Bandwidth over a network path can vary significantly. Each network segment may provide a different level of bandwidth. The amount of traffic sharing that bandwidth, fluctuates with changing levels of network utilization. And as the number of hops along the path increases, so does the amount of latency, or delay in end-to-end data delivery.

These factors are further complicated by a global network connection where more variables are in play. Periodically running an end-to-end speed test can provide insights into the available bandwidth and degree of latency for a given connection. Scheduling tests to avoid busy hours can help to mitigate the impact of network-related performance issues.

The Database Environment

The third performance factor is the database environment. Data access takes many forms. A software test may query an external database as part of an application workflow. Data migration testing may involve large amounts of data transferred between systems. GenRocket’s Test Data Queries may be used to blend stored production data with real-time synthetic test data. Or a GenRocket Test Data Scenario may be generating an extremely large test file with hundreds of thousands of records.

Here are performance issues related to the database environment:

• The use of indexing for database access
• The size of the database or test files in use
• The server used for database queries or data insertion

Indexes are data structures that allow rapid access to data tables in place of sequentially examining each record to find a given row of data. However, poor indexing can be a source of poor performance for data-intensive applications. Adding indexes without proper analysis can cause insert, update and delete functions to take longer when a large number of indexes need to be updated. And if the file is not indexed, each database operation must sequentially scan the entire data file to perform the right operation on the right record.

Database size is a factor that multiplies the impact of every other performance issue. The performance issues caused by slow hardware, a slow network or a poorly indexed database will be amplified when large volumes of data are involved.

Finally, the server hardware itself can be a source of performance issues. Hardware-related performance problems are unlikely for a database maintained on a production system. However, production data that is moved to an under-resourced test server will result in a performance hit.

Performance Recommendations

Start by sizing GenRocket client systems for an automated test environment as opposed to manual test environment. Even if your initial level of automation is low, it will certainly expand over time. Be ready for your transformation to test automation by establishing a self-service Test Data Automation capability. Determine if and when partitioning is necessary for large test data generation requirements. Finally, consider all of the factors described above – hardware, network and database – when developing your test plans to optimize the performance and efficiency of testing operations. We’re here to help. Contact us with any questions about how to optimize the performance of your GenRocket Test Data Automation solution.

*The performance benchmark (10,000 rows per second) is based on a machine that meets the minimum system requirements running a Test Data Scenario with one domain and 10 attributes.