Efficient customer support chain

To have an efficient chain of help from customer and all the way down to developers teams should make sure to not skip steps and seek shortcuts as they will prevent internal growth but also unwillingly consume more of team time on the long run making the support inefficient and time consuming

 

 

What does this mean in practice?

 

   1) Support - don't be a hero, you cannot isolate all bugs by yourselves as this will suck up your time and you won't be able to work on anything else that day; pass complex support cases down to QA to isolate and reproduce the issue when you define the problem itself; not passing cases down to QA will also deny them of the opportunity to grow hard skills and help you better/faster in the future

 

   2) Support - make sure you do define what the problem is about when the customer first contacts you; this means you shouldn't forward received customer email directly to QA if you don't know the answer immediately - try to at least define the problem before pushing it down the chain to QA to isolate and report a bug

 

   3) Support - some of you have tendencies to contact devs and discuss directly with them whether a customer report is a bug and whether or not devs can fix it easily; don't go to devs but go to QA instead because going to devs without a defined bug will only waste time for both teams

 

   4) QA - although you no longer have unattended assistance metric this doesn't mean you should contact devs for every single support case and definitely not repeatedly when similar issues show up - I can hear directly from devs which QA team is doing better job isolating technical support case bugs

 

   5) QA - you can and should contact customer directly when you need additional help to isolate a complex problem; contacting customer through Support will only waste time of both teams and prolong solving the case

   6) Devs - help QA grow by giving them pointers what to research, check and isolate; if you always roll down your sleeves, dig down into code and isolate a problem for them, you'll be stuck doing this indefinitely while we'll have consequential impact on our product releases

Concise bugs for improved revenue

There's a whole chain of inefficiency spawned by each poorly qualified bug:
   1) QA reports a bug with a title not matching the actual issue close enough
   2) The bug slips through spot-checks with incorrect priority assigned due to inconclusive title
   3) Developers spend time to fix lower priority bug or miss fixing higher priority bug
   4) Support team cannot fix/define release note correctly and spends much time including getting help from QA
   5) Marketing team spends time copy-editing poor release note grammar or semantics
   6) Finally customers get lower quality release or delayed release with incomprehensive release notes

Solution - QA

Peer-reviews: if devs can peer-review code, QA can also peer-review new bugs
   a) Make pairs, i.e. team member 1-2 and team member 3-4
   b) Define review schedule - each day, just before sending test summary (this is up to you)
   c) Suggest corrections to your team pair about all bugs that don't look correct

Once we get the final summary, both the bug owner and the bug reviewer will be held responsible for poor bug title, incorrect priority, stds issues, etc.

Apply the same to new test cases

Solution - Support

As prime owners of release notes you shouldn't bow your head down and keep nagging while rewriting poor release notes
   a) For each release note you need to open a corresponding bug in order to rewrite it, write down the bug owner name
   b) Compile a simple rank list (leaderboard) 1, 2, 3, 4 - from most to least frequent bug owner who you needed to correct
   c) Include the leaderboard along with each corrected and approved release notes

Product release notes - why do they matter

Everyone involved in pre or post-production is writing product release notes: developers, QA and Support. In the end release notes for all products appear consistent and are published following carefully constructed set of internal writing standards. Here's an autopilot plan listing release notes creation process in top-down priority by owners


Support

You are the direct owners of product release notes; they serve as proactive help and guidance for our customers by letting everyone know what we improved, changed and fixed in the new product releases. Having good release notes has multiple benefits:
   a) Satisfied existing customers
   b) More new customers
   c) Less need for later reactive support

This means that you have the final and ultimate responsibility to polish the release notes for all products before the release, make them consistent and easy to understand for all and push to production. To do so, use all the help you can from other teams (QA, developers) to clarify required technical details and rewrite the release notes as needed

Release notes standards are there to make all release notes consistent, not to create more bureaucratic overhead; if you see a way to improve it by making release notes simpler and easier for customers to understand - act on it now


QA

You write the most of the release notes from scratch as soon as Bugs are reported which basically makes you the main authors of 90% of all our release notes. Writing good release notes has multiple benefits:
   a) Release note text can also be directly used as Bug title - two flies with one swat
   b) Bug title matching well written release note will make developers understand bugs more easily and will pull on your sleeve less often
   c) Save time for Support which in turn allows them to resolve more reactive support issues, to forward less issues to you and thus save your own time


Developers

You are the first and the last line of defense - you consolidate all release notes for a new product release into one file on start and incorporate them back into product after Support team reviews them

Yes, you still have to write all enhancements and changes as no one else knows what has improved and changed in the product better than you. Additionally you have the responsibility to incorporate final release notes back into your product and to make sure formatting is consistent across all products

As Everyone Serves, your main job is to provide prompt technical assistance to other teams when they need to write a good and simple release note


Common suggestions

   1) When in doubt how to write a release note just put yourself in customer's shoes and see if an outsider will understand it after a single read
   2) Make all release notes as concise as possible - they are not help content but new product version proactive one-sentence guidelines
   3) Describe anything and everything that has changed and especially improved in new product releases, even if you accidentally improved performance by a few percent
   4) If not sure how to write a release note contact other teams for assistance - don't write a poor and inconclusive release note as it will eventually come back to you

Responsible diversification

...or simply put: how to have a cross-functional team without risking Sprint goal failure

The issue

On one hand, Scrum is all for having cross-functional and self-managing teams vs. narrow-specialists, while on the other hand each team member should be able to finish every goal and task within a Sprint no matter how complex it is; this appears to be the most common Scrum paradox and I'm seeing numerous articles and even entire books on how to solve it

We want to:
   a) Make Scrum teams fully self-managing
   b) Have employees of all qualifications, both Jr. and Sr. to focus on the same critical company goals

We don't want to:
   a) Isolate employees by making them run sprints on their own just because they cannot yet work on all tasks in the team
   b) Introduce valid excuses as to why Sprint goals weren't met
   c) Loosen up the Definition of Done and affect team deliverable quantity and/or quality

Solution

Our goal is to always KISS so to fix the mentioned paradox we'll introduce some flexibility by allowing Scrum teams to be collectively qualified to achieve all assigned goals; what does it mean for Scrum teams compared to the practice so far:
   1) Each team member is no longer needed to be able to perform all tasks in the sprint
   2) Each team must still be able to complete all team tasks with sufficient quality
   3) Tasks achievable by select team members only are to be treated as high risk tasks meaning they must be marked as such and must be prioritized within the sprint
   4) Each team must make an Internal redundancy plan for tasks achievable by select team members only and lay out the redundancy pairs in Sprint planning meeting

In practice - teams

   A) Support team can have one part of the team (Writers/Analysts) focus on writing, analysis and other high level tasks first while Product support guys focus on handling the actual customer support and lower level writing, posting tasks

   B) Core developer teams can directly integrate Jr. developers, have Sr. devs focus on new development design and complex bug fixes while Jr. devs work on low/medium complexity bugs and following specs

In practice - individuals

   A) Working exclusively on simpler tasks or taking ad-hoc tasks and insisting there is no more work in the sprint left for you when all low level tasks are done is an excellent way to not advance your career and eventually have your regular review result in DNE

   B) If you can work on all team tasks, you must still roll up your sleeves when there is critical low level / transactional work left; taking lower priority but more complex work in such cases is even worse than not working at all

Patching zen

We must be able to find a perfect balance between administration and efficiency in order to achieve the goal - deliver customer needed fixes as soon as possible. Let's address specific use cases that were all encountered in under 14 days' time

Support
   1) Each week Support check status of customer requested bug fixes, gets patch ETAs from developer teams - this is good and will result in good hard info that will be part of developer team SMART goals. However not all patch and bug fix requests are backed up by Product backlog items (PBIs) - this is not good and is something we must have in order to:
      a) Ensure all needed customer bug fixes are properly prioritized in developer team Product backlogs; remember that dev teams are mainly Scrum oriented
      b) Have hard evidence when and what bug fixes were requested right there in the TFS, easy for all teams to access and see, not scattered around in dozens of emails and forgotten about

   2) If a specific PBI hasn't yet been committed to, add new bug fix request there; this may be a change to how we did it so far (add bug fix requests only if the goal is not yet approved, otherwise create new PBI), but this change will increase efficiency and reduce your dependencies on development Product owner to approve new PBIs

   3) Don't exaggerate - ask for bug fixes for only the most troubling issues for our customers, or if it can directly help our revenue; other bugs will be fixed eventually. The more bug fixes in a patch you request, the longer it will take to be done

   4) Provide development Product owner with weekly product patch priorities; NOT individual bugs but actual product names and their TFS PBIs; this will help Product owner understand customer priorities better and correctly prioritize developer team goals to meet customer needs

   5) Reproduce a bug and put it into TFS before you ask for a patch; if you cannot reproduce it, QA are there to help

Developers
   1) You're self-managing spec-ops now - don't rely on a single person to guide each and every step of the way to deliver the requested patch to customers. There is no central synchronous checklist to follow and wait until someone else completes their action item

   2) Some of the bugs requested in the patch cannot be reproduced, require too many changes and time to do, or look as though they were by design? Contact Support now and clarify them ASAP, discuss each and every special case as these bugs are customer favorites and cannot be pushed aside

   3) Release notes not yet updated and reviewed? Don't wait for this step as it is an asynchronous operation - instead get the patch to the customer now

   4) Always cut a label before sending it to testing; this ensures the code base is pristine after the patch is quick-tested

   5) QA found a newly broken feature bug in the tested patch build? Fix it now and create a new label to send to testing, don't wait for explicit approval as the patch is not done until the customer starts smiling

   6) As soon as the label is quick-tested by QA, send the build link to Support; you don't need release notes, website content or COO, CTO, CEO to approve this - just get the patch over to the customer

   7) Once release notes are updated, update and rebuild the label, then it can also be promoted publically on the website. We determined that there are dozens of "silent downloaders" that do get these patches manually after they are posted on the website so we should help them as well

Think, simplify, consolidate

We must consolidate new Bugs and reduce administrative overhead and by doing so help all teams when dealing with Bugs. Having multiple Bugs that all relate to the same cause, result in the same violation, expose the same incorrect behavior will allow for them to be fixed independently and can cause inconsistencies or needlessly take more time to create, review and update

Some examples encountered recently:
   Issue 1: GUI standards issue: several menu / ribbon items have inconsistent casing and are named differently across different products
   Bug 1: Have only one Bug for each of the products and list all related UI issues and standard violations within; this way all issues will be fixed at one time vs. risking to introduce new inconsistencies

   Issue 2: Product UI has multiple issues when using 120 DPI system font resolution
   Bug 2: Have only one Bug for the product at one time vs. one Bug per dialog or even one Bug per visual issue; there is no reason why some should be fixed and some not - they must all be fixed in order to consider having support for specified system font resolution

   Issue 3: Product stops responding during a specific operation (any general perceived performance issue)
   Bug 3: Have only one Bug explaining how to reach the non-responsive state; don't create individual Bugs for everything that no longer works after the product is already in the non-responsive state as the core issue must be fixed to resolve this

   Issue 4: Multiple usability or obvious design issues
   Bug 4: Have one Bug per group of issues that should all be resolved together, i.e. per UI dialog or group of dialogs where they are discovered

Obviously it's not possible to list all cases encountered so far and all possible cases that will be found in the future. Note the following best practices and apply:
   1) If Bugs found should be resolved all together to ensure complete functionality, consolidate

   2) If resolving found Bugs individually (even if they are all reported with equal severity) can produce inconsistencies, consolidate

   3) If just by grouping similar issues in a single Bug you can save much time vs. explaining them individually, consolidate

   4) If there are multiple unwanted consequences resulting from a single cause, consolidate

   5) When in doubt, always think and try to simplify: "Think critically, objectively and with full information before you answer, act or deliver"

Patch obstacles course

Before clarifying a few use cases below, make sure to (re)read the clear set of Patching rules we already have defined

1) Support must make sure to clarify newly created PBIs which represent goals/requests for patches:

   a) Specify product name and version that must be patched; this doesn't mean to specify product version where bugs were found but to specify actual version of the product that is currently available to customers and must be patched

   b) List all defects individually that need to be patched - these can be High or Critical severity bugs only - each defect's number, severity and name must be specified in PBI description; OR you can simply link TFS bugs to this new TFS PBI

   c) Send FYI email about created/updated PBI to developer team AND to Product owner; if you send to developer team only then they will have to pull Product owner's sleeve to get this approved which introduces more emails into the otherwise direct process

2) Developers must review the new PBI immediately; then:

   a) Know your Scrum rules – you cannot put a new and especially unapproved PBI in ongoing Sprint – do so and you will violate core Scrum rules which is primarily bad for your team as your sprint may fail to achieve original goals

   b) If PBI contains Critical severity defects to patch, estimate if you will be able to get it done in week's time per rules by waiting for the next sprint; if not, you have ad-hoc time allocated for fixing such Critical issues outside of the regular sprint

   c) If PBI contains High severity defects only, work with Product owner to get it approved in time for inclusion in the immediate next Sprint

   d) Support team doesn't care what code branch you will work on, how to implement the same fixes in multiple branches, etc., they just need the patch build delivered per rules

Learn how to patch in 3 easy steps

If you're wondering who owns the product patches, who determines when a patch will be done, what will the patch cover and how it will be delivered, the answer to all these questions is summed up in one word - Support

You don't depend on CTO Development or Operations manager to forward the requests and make developers do the actual work of creating patches. Simply use the Scrum process we have set in place and get your patches delivered just the way you like them - medium rare or well done

Below you will find 3 easy steps to help you help customers

1. Identify the need
As you are directly in contact with customers you can feel their pain and determine what issues trouble them the most; no one else can do this for you, not CTO, not QA, and definitely not developers who created the issues (although inadvertently) in the first place

Ask yourself the following questions:
   a) How many customers have been affected?
   b) How serious is the bug at hand? (see How to determine your own bug severity)
   c) Does the identified bug have a workaround you can offer now to the customer?
   d) How fast does the customer need the fix?
   e) How many "invisible" customers who didn't contact you could have been affected by the bug?
   f) How brand-damaging is the bug?
   g) Will patching the bug now help the company to keep existing customer or gain a new one?

Compile and maintain the list of bugs-for-patching candidates daily; make sure you include only bugs that customers actually need. Remember: not all bugs are created equal - a High severity bug that has a workaround isn't necessarily instant candidate for patching

2. Make the request
Now that you have a neat list of one or more bugs that obviously need to be patched ASAP, create a patch request for corresponding product: just add/update a Product backlog item directly in the product-owning developer team's Product backlog

If you are requesting a new patch, create a new Product backlog item and describe in details what exactly you need delivered - specify all bugs individually, their ID, severity and description

If you have found additional bugs for the same product that need patching, append them to previously created Product backlog item if it is still open

Send an FYI to Product owner and the developer team about added / changed Product backlog item in their Product backlog

3. Deliver
You've done your job - don't worry if or when will you get the actual patch sent to you, this is now up to the developer team (they also have a system and a set of rules to follow). However for reporting needs you should still maintain a list of all patches you requested and when you requested them

As soon as you receive the patch with requested bug fixes, directly contact the customers who reported the issues and make them happy


Q/A

Q: I need to get a patch for this Medium severity bug as customers desperately need the fix. However rules say I can only request a patch for Critical and High severity bugs?
A: Why do customers desperately need the fix? If this is so obvious, are you absolutely sure your bug should be Medium severity in the first place?

Q: I requested Critical severity bugs patched and the customer needs this yesterday. Developer team just started a new two-week sprint without the patch goal included so this patch won't be done in another two weeks - can I yell at developers to speed this up?
A: No - don't yell and don't go pulling on CTO or Ops manager skirt. Developers also have a set of rules to follow and one of the rules is that Critical bug fixes must be delivered in a week's time even if it means working outside of the Sprint during allocated ad-hoc time. You'll get your patch sooner than you think

Q: I always request new patches each Friday once I build up a list of obvious patch candidates but customers don't want to wait 3 weeks to get the patch. What can I do?
A: Start by requesting new patches as soon as you identify bug candidates for patching, not once per week - there's nothing limiting you to do this daily if needed

Self-help book: Define your own defect severity

You've found a new bug but you aren't sure what the correct severity for the defect is? Welcome to the "This is no exact science" club

Here are some general hints how to solve this question that I already shared with some of you over IM:
1) Treat Bug severity standards as guidelines - see the first line in the document, it clearly says that the document is written as guidelines and that ultimate responsibility for bugs you find is yours

2) Worried about how will Critical severity affect production, developers? Are you also worried about global warming and dying rainforests? If so, how does this help you define the severity of the bug you just found? The answer is: it doesn't

3) Wondering why your Low severity defects describing features not working still aren't fixed after two months but are considering to make this one Low severity as well? Rethink this again

4) Put yourself in the customer's shoes. What is your reaction to the newly found bug?
   a) I don't like the product and I won't use it
   b) This is very annoying, I won't use the product because of it... much
   c) I can't use this feature as I wanted to but I see the workaround exists - kind of annoying but I'll get over it
   d) I don't frequently click in the top right corner 3 pixels next to the Ok button and cause the mouse pointer to glow red - I'll forget about this by tomorrow

5) Make sure that you've put yourself into 80% of the customers' shoes - if you are thinking as someone who used to code Commodore 64 games in assembly back in the 80's and know how to get Linux distro kernel version, then a chance is this issue may only be visible to you

6) Contact your QA team members to see if the apparently Critical issue is easily reproducible by at least one more team member; maybe your VM OS stopped working after months of abuse

7) Devs insist on reducing the bug severity you've set and you're scared, best to comply? No - stand by your decision especially with facts you determined before defining the severity

8) Severity can go both ways, both you and devs have solid facts about the case. Contact analysis team for recommendation. However this should be an extremely rare case if you follow #7 above

9) Still not sure? Notice that I haven't answered any of the questions above with definite answers, when to use which bug severity. This is entirely up to YOU. As you have the ultimate responsibility for quality of our products and ApexSQL brand in general, you also have all the power needed to affect the quality with bugs you find

Note: if devs are giving you hard time, let me know directly ;)

Scrum tales - part 1

Developers, Support and Analysis teams have now had their first experience working with new TFS Visual Studio Scrum 1.0 template and I'm seeing Product backlogs forming, Sprint planning meetings underway

I'm still getting good number of questions, both systems (Scrum) and operational (TFS), i.e.:
1) How do I initiate a Sprint when I have 5 large Product backlog items and all are of the same priority?
You can't because you don't know which one to take first - make sure you have Product backlog items all with different priority approved by Product owner before you initiate Sprint planning meeting. No two goals should have the same priority

2) What do I do when one Product backlog item cannot be worked on in parallel, can I start the next one in the Sprint?
Yes - you can organize internally in the team in order to achieve goals you committed to the best way possible

3) Can I add tasks that include waiting on other people/teams to send me something?
No - why would you commit to a goal that you don't know you can finish? Identify impediments early and resolve them before the next sprint. If there are unforeseen obstacles that show up during a sprint, add an Impediment work item to your Sprint backlog and it is now up to the ScrumMaster to close it ASAP. Until an Impediment is closed proceed working on tasks next in line

There are more use cases and I'm sure we'll encounter even more in the upcoming weeks. I'll put them up as they appear. Everyone should feel free to post back thoughts and questions