Tips for managing automation test data

Hello QA folks,

I would like to share my experience on one of the challenging problems for QA automation testers: how to manage test data efficiency. Especially for API testing, when there are several complex request and response payloads to be handled. Moreover, the legacy code that the test data dependency is all over the place. This may be blocked you from applying automation parallel test tools such as robotframework-pabot to speed up your testing time. If you are in the same shoes as me or a new joiner in this area, I encourage you to try the tips below. I believed your test data would be more manageable.

Tips:

1. Independence is the key
2. Keep it tidy
3. Automate over manual
4. Speed up the test
5. Teamwork makes progress a success

Independence is the key:

• “The key to happiness is independence.” Marty Rubin said, which is a big YES for Test data management. Why? because when test data is independent, you can speed up testing time by running the test parallelly and test results are still correct. Imagine you have 30 test suites. If you are running it sequentially, it may take 30 minutes to finish. However, if you run 30 test suites 5 processes parallelly then it will probably be 6 minutes. Therefore, you should split test data at least at the Test Suite level. It is easy for you to start to do the right thing for the fresh project that will benefit you last. For the legacy project, there may have existing independent test data which take time to fix. If you have to add on a new test suite, I suggest starting separate test data for your new test suite first. The legacy can be improved later. Don’t reuse test data like the legacy code did even though it is convenient. Remember, you are finding the way out of this maze not losing in the illusion path again.

Keep it tidy:

• Once you have many test data, you properly need to keep it tidy. Here is the test data structure that I have used. Keeping it in the .yaml file is quite convenient for API testing. You can keep and retrieve them as variables, dictionaries, and lists.

File Structure

Data Structure

Automate over manual:

• Test data could be prepared automatically. Especially, for the test environment that the data storage often cleans up. So, we don’t want to re-create everything manually again. Better write the script to prepare them. Additionally, the benefit of the script is the knowledge is shared by itself. Even though manual steps have been documented, they could be outdated or lost when QA leaves the company. Therefore, an automated preparation script would help everyone in the team (including developers) can understand how to create test data.

Example of prepare test data script

• Normally, there are perfect sections to set up and teardown test data in Robot framework such as Suite or Test Setup and Suite or Test Teardown. Those steps will be running every time with functional test cases. So, there is another factor to be considered; time for running those steps in the whole regression testing every single time. Is it worth the wait?

Speed up the test:

• It is important to speed up the regression testing time because if the regression testing is fast means we can release software faster. If the preparation test data step significantly slows down regression test time. You should consider the below approaches.

1st approach:

“Shall we prepare and clean up test data every single time when running regression test?”

2nd approach:

“Shall we prepare and clean up test data separately by running script only when they need to be created or removed?”

Please note that the 2nd approach does not include a must-have Setup and Teardown required for the stableness and correctness of the script. Those should always be there.

Which approach is suitable for your project? The answer is we need to calculate the possible testing time first.

• Imagine you have 10 test suites. Running time excludes preparing test data steps is about 1800 seconds (30 mins). Your test data preparation is not very complex. This process could spend 20 seconds per one test suite. Therefore, the total execution time plus test data preparation time is [(1800) + (20*10)]/60 = 33.33 mins which are not very significant. You are in the perfect automation test scenarios. Go ahead and proceed with the 1st approach.
• Imagine you have 10 test suites. Running time excludes preparing test data steps is about 1800 seconds (30 mins). Your test data preparation is typically very complex. This process could spend 80 seconds per one test suite. Therefore, the total execution time plus test data preparation time is [(1800) + (80*10)]/60 = 43.33 mins which are quite significant for the project that test cases are increasing every day. Thus, your testing time will be increasing too. Therefore, you probably go for 2nd approach.

Teamwork makes progress a success:

• All tips are required a lot of effort and discipline. So, everyone involved in the automation project should be agreed to maintain them. Especially for the legacy project that a lot of test data are messed up, I bet you can’t improve them all alone while you have regular tasks on hand. Don’t forget, alone we can do so little; together we can do so much. Start sharing the plan with your teammates (include Developers) and do it little by little then it will be much successful. For an individual contributor, you may create a prototype and share it with the development team first. Therefore, at least you can adopt an adventurous attitude for upgrading things in the workplace.

I hope these 5 tips for managing automation test data could at least give you some ideas. Ultimately, could help resolve your current test data management problems. Cheers !🖖