Automated Test Need Maintenance
What stakeholders hate to hear is the news “Automated test will not work in this environment need maintenance”. In the current industry trend many organizations will manage multiple versions of code going through different kind of changes or unique configurations settings. I am sure many of you in test automation area may be looking for one tool which supports sustenance of automated tests % while product is going through changes,
When to start focusing on test automation?Some industry veterans recommend to look at automation effort only when product is stable. But some people say automated tests always increase speed of going to market so start anytime it is worth. Who is right? we are not sure. But challenge of sustaining automated test % in a given environment is always there for all of us.
Multi stage of product
A build developed goes through development environment, QA environment, Pre-production environment and production environment. On top of it if you have a key customer configuration environment where support traffic is tough to handle increases challenges for final deployment decision.
The dynamically changing code is on development environment. Here we always wish to see that smoke test suite is automated for all feature and it runs every day. This is also a stage where person responsible for writing code ensure it is robust. Maximum issues are found later are traced back to escape from this stage. I know it requires a revolutionary change for developer to arrest few issues when they write code (Look at STAG Software one of the offerings in this zone. Very nicely articulated on trend and challenges here).
Next is QA environment where we push the build from development environment to QA as per project plan. Here we wish even basic interfaces are tested with automated tests and it should work. Many new features that released to QA may get tested with automated tests. What we group as regression tests also executed here. % automated regression tests is a vital key metrics organization to watch.
Next to this is pre-production or staged environment where product is all most tested in a real production environment kind of situation. More of smoke test suite is what we prefer though regression is welcome here. From my experience I witnessed many organization doing regression test suite running on QA environment itself. The
Final is a production environment. This is a short window of production downtime to deploy the new version and hand over to support team. Automated smoke test full or partial to be executed based on history of deployment related issues from the field. Many times the package of release will have some scripts to be executed to ensure protection of existing configuration status, db changes impact etc. This event is closely watched by many and rollback option must be available if risks are seen after deployment. Failure here is a painful issue to manage for all levels in the organization.
The configuration of key customers whose support issue traffic was painful to handle in previous release must be tried at pre-production environment automated test results must be available for final decision on these configurations in particular. With testing on cloud possible now we should focus on firing automated tests on different key configurations settings. Therefore automation tool should support cloud execution also.