We've been cutting labels and sending products to testing for years - both developers and QA know the rules implicitly without even looking down at written guidelines. What scared me the most this week is that when system rules are violated and there is a massive confusion on both developer and QA sides, one of the arguments was "we've been doing it like that since I joined the company"
Nobody likes reading the rules, operation manuals and workflow guidelines, but still they are there for a reason. If you forget about them, you'll be hearing from me soon
Let's get a few things straight:
1) Devs, there's no more Continuous testing; you cannot just send in a new major build without some internal testing and expect to have QA determine if the product can start or not, then find/fix a few bugs each day until the build is stable for real testing; instead conduct a few days of internal testing (plan it in your Sprints) to pick the low hanging fruit, resolve critical issues immediately (product doesn't start, main features don't work) and write down all non-critical ones to forward to QA along with the first label to log down to bug tracking system
2) Devs, make sure you always send modified product version to testing; we cannot have two builds with identical versions that are functionally different just because engine takes too long to rebuild. Consider updating the way engines are referenced and versioning strategy, suggest solutions proactively; Everyone leads - proactively identify problems and solve them, don't ask questions but make recommendations
3) Devs, always send an official request for testing when a new build is ready; specify testing areas as detailed as possible even when it means you have to roll back your sleeves and write detailed guidelines for back-end high-tech stuff that QA might not understand on their own; Everyone shares - communicate widely and effectively, educate your customers
4) QA, it IS your job to determine if the product can start or not; Everyone serves - treat everyone as a customers and take personal ownership of their issues vs. "it's not my job" attitude; if you got a version that doesn't start at all assume that it did work on devs' side and that it is your job to isolate on which systems product works and on which it doesn't, then report Bugs
5) QA, report all found Bugs professionally following the established procedure without sending additional emails with built up emotions and no concrete analysis or actionable suggestions
6) QA, use your right to cut testing short if there are too many obvious Critical issues found, however always follow the procedure and send in testing summary at the end and allow devs to fix the Bugs to surprise you in the next test round vs. hate-mailing about the build quality
Upside from this week's confusion related with our new product is that almost everyone managed to transition to constructive discussions with actionable suggestions lead by analysis and pros/cons; we need more of this in all aspects of our work
Friday, September 28, 2012
Tuesday, September 25, 2012
Think, simplify, consolidate
We must consolidate new Bugs and reduce administrative overhead and by doing so help all teams when dealing with Bugs. Having multiple Bugs that all relate to the same cause, result in the same violation, expose the same incorrect behavior will allow for them to be fixed independently and can cause inconsistencies or needlessly take more time to create, review and update
Some examples encountered recently:
Issue 1: GUI standards issue: several menu / ribbon items have inconsistent casing and are named differently across different products
Bug 1: Have only one Bug for each of the products and list all related UI issues and standard violations within; this way all issues will be fixed at one time vs. risking to introduce new inconsistencies
Issue 2: Product UI has multiple issues when using 120 DPI system font resolution
Bug 2: Have only one Bug for the product at one time vs. one Bug per dialog or even one Bug per visual issue; there is no reason why some should be fixed and some not - they must all be fixed in order to consider having support for specified system font resolution
Issue 3: Product stops responding during a specific operation (any general perceived performance issue)
Bug 3: Have only one Bug explaining how to reach the non-responsive state; don't create individual Bugs for everything that no longer works after the product is already in the non-responsive state as the core issue must be fixed to resolve this
Issue 4: Multiple usability or obvious design issues
Bug 4: Have one Bug per group of issues that should all be resolved together, i.e. per UI dialog or group of dialogs where they are discovered
Obviously it's not possible to list all cases encountered so far and all possible cases that will be found in the future. Note the following best practices and apply:
1) If Bugs found should be resolved all together to ensure complete functionality, consolidate
2) If resolving found Bugs individually (even if they are all reported with equal severity) can produce inconsistencies, consolidate
3) If just by grouping similar issues in a single Bug you can save much time vs. explaining them individually, consolidate
4) If there are multiple unwanted consequences resulting from a single cause, consolidate
5) When in doubt, always think and try to simplify: "Think critically, objectively and with full information before you answer, act or deliver"
Some examples encountered recently:
Issue 1: GUI standards issue: several menu / ribbon items have inconsistent casing and are named differently across different products
Bug 1: Have only one Bug for each of the products and list all related UI issues and standard violations within; this way all issues will be fixed at one time vs. risking to introduce new inconsistencies
Issue 2: Product UI has multiple issues when using 120 DPI system font resolution
Bug 2: Have only one Bug for the product at one time vs. one Bug per dialog or even one Bug per visual issue; there is no reason why some should be fixed and some not - they must all be fixed in order to consider having support for specified system font resolution
Issue 3: Product stops responding during a specific operation (any general perceived performance issue)
Bug 3: Have only one Bug explaining how to reach the non-responsive state; don't create individual Bugs for everything that no longer works after the product is already in the non-responsive state as the core issue must be fixed to resolve this
Issue 4: Multiple usability or obvious design issues
Bug 4: Have one Bug per group of issues that should all be resolved together, i.e. per UI dialog or group of dialogs where they are discovered
Obviously it's not possible to list all cases encountered so far and all possible cases that will be found in the future. Note the following best practices and apply:
1) If Bugs found should be resolved all together to ensure complete functionality, consolidate
2) If resolving found Bugs individually (even if they are all reported with equal severity) can produce inconsistencies, consolidate
3) If just by grouping similar issues in a single Bug you can save much time vs. explaining them individually, consolidate
4) If there are multiple unwanted consequences resulting from a single cause, consolidate
5) When in doubt, always think and try to simplify: "Think critically, objectively and with full information before you answer, act or deliver"
Friday, September 21, 2012
Scrum tales - Part 7
Almost all developer teams have encountered this issue with their Sprints - how to define Sprint tasks and estimates when it is almost impossible to know all the individual steps needed upfront to get to the defined PBI goal - i.e. you just don't know where to start?
To some extent this also applies to other teams who cannot define their PBI individual tasks upfront during the Sprint planning meeting, so please read ahead and don't "guestimate" too much
1. Understand your goal / PBI - what exactly do you need to accomplish?
a) Improve performance - where, by how much %, can this be perceived performance improvement vs. hacking open Windows drivers?
b) Fix a stubborn Critical severity issue that you have no idea what is causing it - is there a workaround you can implement instead, can you at least fix it so that it is at least no longer considered Critical severity, can you research a bit to define how to proceed next?
c) Build a new product or product feature - do you at least know the feature set needed if you don't have full specs or UI mockups?
2. Research - one thing that was rarely used or was used incorrectly are research tasks:
Yes - you can add research tasks to closer define actual tasks that will lead to the goal accomplishment
No - you cannot have an always open research task to work on during the whole length of the Sprint
Start by creating a finite research task that will allow you to figure out your exact next tasks towards the PBI endgoal. Limit yourself in research which must result in:
1) New research tasks that are closer defined and can be shared amongst more team members
2) Specific finite tasks to start working towards the goal
Specifically related to the hypothetical goals mentioned above:
a) To improve performance start by defining tasks to:
- test current version vs. previous versions of the product and isolate speed difference
- test comps and isolate speed difference
- research code to isolate bottlenecks
- research bottlenecks to prioritize them starting with easy fixes (perceived performance improvements) to complex driver hacks
b) To fix a bug when you have no idea what is causing it, define tasks to:
- research code to find workarounds or at least new more specific research areas
- research new specific areas for possible fixes and list them from easiest (quick workarounds) to most complex ones (redesign engines) and estimate each one
c) To build a new product knowing only a closed feature set, define tasks to:
- research comps and create new design tasks
- research features functionality how to technically implement them and what all will be needed to do this; define design tasks
3. Define achievable tasks with real estimates
Scrum is different from the Waterfall model in that it doesn't require you to know all the How specifics before starting to sprint. Scrum team must clearly understand the goal (PBI) but can figure out how to achieve it during the sprint itself
It is important however to note that research tasks must also be finite and they must result in new research tasks or concrete development tasks leading you one step closer to the goal
To some extent this also applies to other teams who cannot define their PBI individual tasks upfront during the Sprint planning meeting, so please read ahead and don't "guestimate" too much
1. Understand your goal / PBI - what exactly do you need to accomplish?
a) Improve performance - where, by how much %, can this be perceived performance improvement vs. hacking open Windows drivers?
b) Fix a stubborn Critical severity issue that you have no idea what is causing it - is there a workaround you can implement instead, can you at least fix it so that it is at least no longer considered Critical severity, can you research a bit to define how to proceed next?
c) Build a new product or product feature - do you at least know the feature set needed if you don't have full specs or UI mockups?
2. Research - one thing that was rarely used or was used incorrectly are research tasks:
Yes - you can add research tasks to closer define actual tasks that will lead to the goal accomplishment
No - you cannot have an always open research task to work on during the whole length of the Sprint
Start by creating a finite research task that will allow you to figure out your exact next tasks towards the PBI endgoal. Limit yourself in research which must result in:
1) New research tasks that are closer defined and can be shared amongst more team members
2) Specific finite tasks to start working towards the goal
Specifically related to the hypothetical goals mentioned above:
a) To improve performance start by defining tasks to:
- test current version vs. previous versions of the product and isolate speed difference
- test comps and isolate speed difference
- research code to isolate bottlenecks
- research bottlenecks to prioritize them starting with easy fixes (perceived performance improvements) to complex driver hacks
b) To fix a bug when you have no idea what is causing it, define tasks to:
- research code to find workarounds or at least new more specific research areas
- research new specific areas for possible fixes and list them from easiest (quick workarounds) to most complex ones (redesign engines) and estimate each one
c) To build a new product knowing only a closed feature set, define tasks to:
- research comps and create new design tasks
- research features functionality how to technically implement them and what all will be needed to do this; define design tasks
3. Define achievable tasks with real estimates
Scrum is different from the Waterfall model in that it doesn't require you to know all the How specifics before starting to sprint. Scrum team must clearly understand the goal (PBI) but can figure out how to achieve it during the sprint itself
It is important however to note that research tasks must also be finite and they must result in new research tasks or concrete development tasks leading you one step closer to the goal
Tuesday, September 11, 2012
Scrum tales - part 6
Finally managed to finish a Sprint and you're happy that all/most/some of the goals (PBIs) have been completed successfully? Check everything from the list below before saying that again:
1) Deliverable specified in the PBI has been actually delivered to the intended customer (internal or external) and the Product owner has seen this. I.e.:
a) Product new build has been approved for production
b) Enough new bugs have been found for a product build and testing summary has been sent
c) Patch build has been sent to Support team
d) All customers/leads have received intended emails
2) New or updated document required by the PBI has been reviewed by everyone with vested interest (future users of the document) and all have signed off on it; Product owner has also seen the document and has approved it
3) Content created as specified in the PBI has been reviewed by all higher level owners. I.e.:
a) Release notes have been technically reviewed and approved by CTO and content rules-wise by Operations manager
b) Article is signed off by copy-editor and by the Tech marketing team as usable for SEO
4) All feedback received back from the reviewers / Product owner for the deliverable (testing feedback, emails sent out, created document, etc.) has been incorporated in the work and resubmitted; Product owner had no further comments
If you answered Yes to all of the above, you can say with confidence that your Scrum team's Sprint goals have been successfully accomplished - congrats to your team for doing the right things right way
If there is at least one No answer and the Sprint is over, then the goal (PBI) is not done and must be specified as such in the Sprint review. How to fix this:
A) Move the unfinished PBI into the new Sprint; work with your team to add corresponding tasks to the goal to be able to answer positively to all of the above before successfully finishing the goal
B) Prevent the same situation from happening again by reviewing completed work with Product owner before the end of the next Sprint
1) Deliverable specified in the PBI has been actually delivered to the intended customer (internal or external) and the Product owner has seen this. I.e.:
a) Product new build has been approved for production
b) Enough new bugs have been found for a product build and testing summary has been sent
c) Patch build has been sent to Support team
d) All customers/leads have received intended emails
2) New or updated document required by the PBI has been reviewed by everyone with vested interest (future users of the document) and all have signed off on it; Product owner has also seen the document and has approved it
3) Content created as specified in the PBI has been reviewed by all higher level owners. I.e.:
a) Release notes have been technically reviewed and approved by CTO and content rules-wise by Operations manager
b) Article is signed off by copy-editor and by the Tech marketing team as usable for SEO
4) All feedback received back from the reviewers / Product owner for the deliverable (testing feedback, emails sent out, created document, etc.) has been incorporated in the work and resubmitted; Product owner had no further comments
If you answered Yes to all of the above, you can say with confidence that your Scrum team's Sprint goals have been successfully accomplished - congrats to your team for doing the right things right way
If there is at least one No answer and the Sprint is over, then the goal (PBI) is not done and must be specified as such in the Sprint review. How to fix this:
A) Move the unfinished PBI into the new Sprint; work with your team to add corresponding tasks to the goal to be able to answer positively to all of the above before successfully finishing the goal
B) Prevent the same situation from happening again by reviewing completed work with Product owner before the end of the next Sprint
Monday, September 10, 2012
JIT testing, or what to test when you think there is nothing to test
More than once it happened that you have a goal to test a new build of a product but it turns out the build is not available in time; you could:
a) Test a product that has lower priority in your current Sprint
b) Write some obscure automated test scripts for a product that isn't even ready yet and you haven't seen it at all but you did have 1 training session where you understood squat about the product
c) Play Windows Solitaire until the build gets ready
Choosing any of the above is not the right answer and will quickly get you off the train where we're headed as a company. What to do?
Enter JIT testing
Your goal says you need to find bugs score in minimum of 250 and verify resolved bugs for a new product build. Ok - you cannot verify the resolved bugs as you still don't have the new product build, this part of the goal is externally blocked. However you can find bugs score of minimum 250 by testing previous product build you have available:
1) Take the latest product build you have available; this can be a public build approved for production or it can be previous testing round build that was rejected due to a few found Critical bugs; it can also be a build recently created internally by the automated nightly build script - check the FTP yourselves and check the build version - don't wait for explicit notifications about such builds
2) Test the hell out of the build and create new bugs that are worth 275 score points (10% over your goal); chances are that all the new bugs you find will still be there in the official new build when it is eventually submitted to testing
Once the new product build is ready, verify the existence all the bugs you found during the testing of the previous available build; I bet that most of the issues will still be there and that in the worst case scenario you may only need to find a few bugs only to exceed the score goal of 250
Other things to do while "waiting"
Don't forget that you'll still get ad-hoc requests to handle new installer testing, new website content to be checked - these are all higher priority and can be handled in parallel with the active Sprint
Developer teams will also create Patch testing PBIs. If a patch testing PBI contains Critical severity bug fixes, test it now and don't let it linger; if a patch testing PBI contains only High severity bug fixes, also test it now - there are only a few bug fixes per patch to test anyway so it won't take too much of your ad-hoc testing/support time
Things NOT to do while "waiting"
What I don't want to see is QA mailing excessively developer teams asking "When will we get the new build ready for testing" - such questions should be directed to QA Product owner (CTO) or not asked at all unless you are planning your next Sprint
a) Test a product that has lower priority in your current Sprint
b) Write some obscure automated test scripts for a product that isn't even ready yet and you haven't seen it at all but you did have 1 training session where you understood squat about the product
c) Play Windows Solitaire until the build gets ready
Choosing any of the above is not the right answer and will quickly get you off the train where we're headed as a company. What to do?
Enter JIT testing
Your goal says you need to find bugs score in minimum of 250 and verify resolved bugs for a new product build. Ok - you cannot verify the resolved bugs as you still don't have the new product build, this part of the goal is externally blocked. However you can find bugs score of minimum 250 by testing previous product build you have available:
1) Take the latest product build you have available; this can be a public build approved for production or it can be previous testing round build that was rejected due to a few found Critical bugs; it can also be a build recently created internally by the automated nightly build script - check the FTP yourselves and check the build version - don't wait for explicit notifications about such builds
2) Test the hell out of the build and create new bugs that are worth 275 score points (10% over your goal); chances are that all the new bugs you find will still be there in the official new build when it is eventually submitted to testing
Once the new product build is ready, verify the existence all the bugs you found during the testing of the previous available build; I bet that most of the issues will still be there and that in the worst case scenario you may only need to find a few bugs only to exceed the score goal of 250
Other things to do while "waiting"
Don't forget that you'll still get ad-hoc requests to handle new installer testing, new website content to be checked - these are all higher priority and can be handled in parallel with the active Sprint
Developer teams will also create Patch testing PBIs. If a patch testing PBI contains Critical severity bug fixes, test it now and don't let it linger; if a patch testing PBI contains only High severity bug fixes, also test it now - there are only a few bug fixes per patch to test anyway so it won't take too much of your ad-hoc testing/support time
Things NOT to do while "waiting"
What I don't want to see is QA mailing excessively developer teams asking "When will we get the new build ready for testing" - such questions should be directed to QA Product owner (CTO) or not asked at all unless you are planning your next Sprint
Friday, September 7, 2012
Scrum tales - part 5
Patch requests have started piling up in the form of Product backlog items (PBIs) in developer team Product backlogs. Some developer teams have a new sprint starting out soon so incorporating these new PBIs into new Sprint is easy. However some developer teams have just started sprinting; what do we do now:
a) Make customers wait to get their patches until the next Sprint in 2+ weeks so we can follow our Scrum development process rules
b) Close one eye and make a small exception within the Scrum rules, add the new PBIs to existing sprint
The answer is c) neither of the above - we need to both get patches to the customers in a timely manner and not violate the Scrum rules. Here's how...
Case 1: A new developer team Sprint is about to start in under a week's time
The answer is simple - just get the new patch request PBIs approved and prioritized on time by your Product owner and incorporate it in the new Sprint
Case 2: A new Sprint has just started and it will be another 2 weeks until it is finished; approved PBI requires a patch containing High severity bug fixes only
This means that the patch will take up to 3 weeks to be finished but it will be ok as none of the defects are Critical severity in the first place
Also use common sense - if you have 2 simple High severity defects to patch which would take a few hours of ad-hoc time, do so now and don't let it linger on for weeks
Case 3: A new Sprint has just started; approved patch requesting PBI contains Critical severity bug fixes that can't wait for the next Sprint
Pause the work on your Sprint and get this patch done ASAP; you already have allocated ad-hoc time for such work outside of the Sprint time
Ad-hoc working Q/A
Q: I've spent my 2 hours of allocated ad-hoc dev time for today; should I go back to my Sprint tasks even though Critical severity bugs patch isn't finished?
A: No - focus on completing one goal at a time; this means you should work whole day if needed to get that ad-hoc patch PBI wrapped up before going back to the Sprint tasks
Q: We're starting work on ad-hoc PBI - do we need to define individual tasks in the team and possibly make a separate Sprint for this, then sprint two Sprints in parallel?
A: Don't introduce complexity when there is none. The reality is you are working on 2-3 ad-hoc bug fixes to get the patch ready; these should show up as Bug fixing tasks in your Daily status report, i.e. "PBI #12345 - Create patch for Refactor 2012 R1 with 2 bug fixes; TFS #8943 - High - Bug name"
Q: I'm going to work on a Saturday, but should I focus on my ad-hoc tasks since weekends aren't calculated in the Sprint time?
A: Assume that working weekends or holidays are just like any other working day and take your priorities in order. Sprint comes first unless there are higher priority ad-hoc tasks such as a patch requesting PBI for Critical severity bugs
a) Make customers wait to get their patches until the next Sprint in 2+ weeks so we can follow our Scrum development process rules
b) Close one eye and make a small exception within the Scrum rules, add the new PBIs to existing sprint
The answer is c) neither of the above - we need to both get patches to the customers in a timely manner and not violate the Scrum rules. Here's how...
Case 1: A new developer team Sprint is about to start in under a week's time
The answer is simple - just get the new patch request PBIs approved and prioritized on time by your Product owner and incorporate it in the new Sprint
Case 2: A new Sprint has just started and it will be another 2 weeks until it is finished; approved PBI requires a patch containing High severity bug fixes only
This means that the patch will take up to 3 weeks to be finished but it will be ok as none of the defects are Critical severity in the first place
Also use common sense - if you have 2 simple High severity defects to patch which would take a few hours of ad-hoc time, do so now and don't let it linger on for weeks
Case 3: A new Sprint has just started; approved patch requesting PBI contains Critical severity bug fixes that can't wait for the next Sprint
Pause the work on your Sprint and get this patch done ASAP; you already have allocated ad-hoc time for such work outside of the Sprint time
Ad-hoc working Q/A
Q: I've spent my 2 hours of allocated ad-hoc dev time for today; should I go back to my Sprint tasks even though Critical severity bugs patch isn't finished?
A: No - focus on completing one goal at a time; this means you should work whole day if needed to get that ad-hoc patch PBI wrapped up before going back to the Sprint tasks
Q: We're starting work on ad-hoc PBI - do we need to define individual tasks in the team and possibly make a separate Sprint for this, then sprint two Sprints in parallel?
A: Don't introduce complexity when there is none. The reality is you are working on 2-3 ad-hoc bug fixes to get the patch ready; these should show up as Bug fixing tasks in your Daily status report, i.e. "PBI #12345 - Create patch for Refactor 2012 R1 with 2 bug fixes; TFS #8943 - High - Bug name"
Q: I'm going to work on a Saturday, but should I focus on my ad-hoc tasks since weekends aren't calculated in the Sprint time?
A: Assume that working weekends or holidays are just like any other working day and take your priorities in order. Sprint comes first unless there are higher priority ad-hoc tasks such as a patch requesting PBI for Critical severity bugs
Wednesday, September 5, 2012
Patch obstacles course
Before clarifying a few use cases below, make sure to (re)read the clear set of Patching rules we already have defined
1) Support must make sure to clarify newly created PBIs which represent goals/requests for patches:
a) Specify product name and version that must be patched; this doesn't mean to specify product version where bugs were found but to specify actual version of the product that is currently available to customers and must be patched
1) Support must make sure to clarify newly created PBIs which represent goals/requests for patches:
a) Specify product name and version that must be patched; this doesn't mean to specify product version where bugs were found but to specify actual version of the product that is currently available to customers and must be patched
b) List all defects individually that need to be patched - these can be High or Critical severity bugs only - each defect's number, severity and name must be specified in PBI description; OR you can simply link TFS bugs to this new TFS PBI
c) Send FYI email about created/updated PBI to developer team AND to Product owner; if you send to developer team only then they will have to pull Product owner's sleeve to get this approved which introduces more emails into the otherwise direct process
2) Developers must review the new PBI immediately; then:
a) Know your Scrum rules – you cannot put a new and especially unapproved PBI in ongoing Sprint – do so and you will violate core Scrum rules which is primarily bad for your team as your sprint may fail to achieve original goals
2) Developers must review the new PBI immediately; then:
a) Know your Scrum rules – you cannot put a new and especially unapproved PBI in ongoing Sprint – do so and you will violate core Scrum rules which is primarily bad for your team as your sprint may fail to achieve original goals
b) If PBI contains Critical severity defects to patch, estimate if you will be able to get it done in week's time per rules by waiting for the next sprint; if not, you have ad-hoc time allocated for fixing such Critical issues outside of the regular sprint
c) If PBI contains High severity defects only, work with Product owner to get it approved in time for inclusion in the immediate next Sprint
d) Support team doesn't care what code branch you will work on, how to implement the same fixes in multiple branches, etc., they just need the patch build delivered per rules
Subscribe to:
Posts (Atom)