Continuous Testing (CT) is the act of running tests on a schedule. The source of the code base or the tests isn’t necessarily specified. It could be a static, standalone instance that has the ability to execute Cycle Feature or a Playlist on a schedule.
Requirements for Continuous Testing
- Development team writes a Cycle Scenario to validate each new feature, improvement, or bug fix they develop.
- Data and operations planning is required to avoid contention for functions running simultaneously across agents.
- A task scheduler runs the full set of Cycle Features or Playlists at a regular interval.
- Fewer bugs get shipped to production as regressions are captured early by the automated tests.
- Building a release is easy as all integration issues have been solved early.
- Less context switching as developers are alerted as soon as they break the test build and can work on fixing it before they move to another task.
- Testing costs are reduced drastically – a CT server can run hundreds of tests in the matter of seconds.
- The QA team spends less time testing and can focus on significant improvements to the quality culture.
How many agents running Cycle are needed?
These major factors will determine how many agents your CT setup will require.
- How long the test suite takes to run:
- If we need to run our tests every night, and the full test suite takes 20 hours to run on a single agent - that's too long to run daily. Add agents and divide up the test suite until the time it takes to run the full test suite is suitable for daily execution.
- How many different platforms are being targeted:
- If we need to test on Windows versions x,y,z - Linux flavor a,b,c - etc - we would likely need a separate agent for each of those platforms.
- Necessary frequency/timing of test suites:
- Once we use Jenkins/TeamCity as part of the code merge process, every time a developer needs to merge code, they need to run through Jenkins/TeamCity. If there is a bottleneck there, more agents will reduce it so that multiple people can get their tests done and get code merged more quickly.