Release process covers 3 distinct consecutive test phases for which product teams (devs and SSEs) are in charge:
1) Design and usability testing
2) Continuous testing
3) Static/release testing
Each team must have a clear test plan that will allow for systematic advancement between subsequent test phases culminating in a product release
In a good product release scenario a team should never regress from one test phase to a previous one - some guidelines follow to explain each test phase
1. Design and usability testing
During this phase the team is testing new features, changes in a product, and is working towards locking down the product interaction design
Usability testing helps make sure the interaction design is optimal and customer friendly - the team should leverage feedback from as many external resources as possible: company stakeholders, power-users, other product teams, etc.
It is ok to update interaction design, make functional UI changes during this phase, but not after this phase is locked down
2. Continuous testing
Once interaction design is locked down, the team must focus on finding as many bugs as possible and fixing them all as soon as they are found and reported
Usually this phase will help to clean up product functional bugs, regression bugs, and make sure the build is stable enough to go on to the next and final phase of the release process (for more details about Continuous testing vs. static testing read this article)
At this time we should have no usability or interaction design bugs reported, however if it happens that such a bug is found then the team must stop continuous testing and immediately regress back into #1 Design and usability testing phase
3. Static/release testing
At this time the team cannot make multiple product builds meaning the code must be fully locked down, and that only one build can be tested
If the previous two testing phases were covered well then the static testing will only reveal limited number of functional bugs
For static testing round #2 the team should fix only the most critical bugs:
a) Regression/broken bugs
b) Bugs that will clearly make the new build considered worse than the public one
If done correctly there should be no static testing round #3, but if it does happen there can be no round #4 so product release is mandatory after that
If it happens that there are design bugs or multiple "critical" bugs found during static testing then the team must stop static testing and immediately regress back into corresponding previous test phase
At any time when a process regression happens this only means that the previous testing phases weren't done well and the team must update/improve owned testing plan in order to prevent process regression from happening in the future
Remember that each team gives its own product release recommendation at the end - good litmus test question for a release: "Is this product build better than the previous one?"
The only thing that a team must follow at all times is this clear release process and must be able to provide feedback to all stakeholders about:
a) What is the current product testing phase
b) What is the ETA to go into the next phase / release
