Developer’s Blog: Learnings from customer testing sessions

[Alan_K]

Welcome back to the second installment of my humble little blog, where I’ll take you behind the scenes to the development of our upcoming support enhancements! I really enjoyed reading the constructive criticism and encouraging comments from the first blog, please keep them coming 🙂  In today’s entry, I want to share what the customer testing sessions were like, what we learned, and what we’re working on coming out of it.

User testing format

In mid-July, my team and I reached out to 18 testers (9 customers, 6 Oracles, & 3 Moderators) to gather their thoughts and feedback. It was important to learn about the experience using the new support forms both from a customer point of view and from a Moderator view, as support is a two-way street. We needed to think about all of the parties involved, and how we go about optimizing the experience. 

We began our testing by asking testers to start their help enquiry within SIMon, our virtual assistant. Fun fact: Did you know SIMon is answering almost 20K questions per week? He’s been crushing his job of sharing answers from Help Articles and the Community for general questions, or directing customers to contact a Moderator for more serious issues - not to brag, but he’s over 90% accurate 😉 

With these tests, I was also experimenting with having SIMon act in the capacity of a front desk agent, where he’ll point our testers to where they need to go. Our time is precious, and I promise we’re not making you use the virtual assistant just for the sake of it! You might get the answer you’re looking for right away, but if not, the super neat thing about SIMon is that based on your short conversation, he’ll be able to remember what you shared with him and pre-fill parts of your ticket submission to a Moderator for you when we launch in mid-August! 

We had 1-3 scenarios set up for our testers depending on the available time.

1. You’re a new customer trying to activate on Public Mobile, but you’re having credit card issues preventing you from completing activation.
2. You’re trying to activate your account online once again, but you’re encountering an error with your SIM.
3. (We asked customers to think back to the last question/issue that they encountered)

Then, we opened up the playing field to our customers to test away!

What we learned from our testers and what we’re doing about it

*All screens shared are not final, and were taken from the same testing screens our testers saw

On using SIMon…

Our testers navigated through SIMon easily, using quick reply buttons to quickly share their initial issue details, and were pleasantly surprised when SIMon surfaced relevant tips and Help Articles.

SIMon surfacing a Help Article for the tester.

Here were some verbatims that customers shared:

• “I don’t see SIMon, maybe because I’m used to asking for help in the Community.”
• “It’s [my account problem] not something I would think I could ask SIMon”

What we’re focusing on for launch:

• Help customers find SIMon easily (better visual cues)
• Clearly communicate what SIMon can do (i.e. take you to where you need to go for your answer + guided way to submit a ticket to a Moderator)
• Continually work towards making SIMon simple and easy to use with quick replies
• We also recognize the virtual assistant is not for everyone, so don’t worry- you don’t have to talk to SIMon before submitting a ticket- private message is still sticking around, but for those looking for a quicker and guided experience, I’d say SIMon will be a great choice after this launch.

On using the smartforms…

Testers reacted very positively to the added structure that the smartforms offered in contrast to the current free-form text entry of private messaging to contact a Moderator for added support. Folks were glad that their earlier conversation with SIMon wasn’t for naught, but actually helped them to save time by pre-filling fields in the Smartform for them. Moderators and customers were both delighted about authentication happening upfront, without any need to follow-up when authentication was done. Some insightful comments were offered particularly by the Oracles and Mods in terms of fields that we missed, or verbiages that needed some more fine-tuning to be crystal clear at launch 🙂

SIMon will pre-fill your issue category and subcategory for you in Step 1.

Here’s what people had to say:

• “It’s nice to have assisted guidance to resolution. I like it.”
• If Self-Serve password/username authentication was selected, Public Mobile accounts using email aliases (Gmail function) could not be allowed through [Oracles]
• For customers encountering payment issues, if they have a pre-registered credit card, we can collect their authorization to execute the transaction they couldn’t perform in the form to save time [Moderators]

What we’re focusing on for launch:

• Email aliases bug has been addressed in staging and will be ready for launch
• Payment authorization has been included in the form for customers who are comfortable with this option
• Continually monitor customer feedback through surveys to make improvements on a rolling basis; action quick wins and build a backlog of higher lift changes

On the ticket submission confirmation page...

We tested two different versions of the confirmation page- version one being short and concise, whereas version two was longer and had more information related to next steps. Most customers felt reassured as long as they saw a reference number, but wanted to know what should be done with it afterward, and how long they should specifically be waiting. Ultimately, the short form prevailed in terms of popularity.

Short version vs Long version 

Here’s what people had to say:

• “The checkmark is good because it marks that I’m done.”
• “Besides the reference number, I just need to know what my wait time will be. How long do I have to wait?”
• Confusion as to what the next steps are and what to do with the reference number after submission is complete [Oracles]

What we’re focusing on for launch:

• Simplified reference number
• Explicit call outs that a copy of the ticket is available in the customer’s Community inbox, including a direct link, and an option to print the confirmation page (including reference number)
• Future goal: add in up-to-date response wait time estimate

Final thoughts

As always, co-building these support enhancements with our customers gives my team and me lots of joy, and we are always taken aback by the passion and breadth of ideas that everyone shares with us. There’s a ton out there we can (and will) do to keep improving upon these to make your experience even better, and this is just the tip of the iceberg that we can tackle in the remaining development time we have before launch time.

Oh boy, that was a super long post! If you made it this far, thanks for sticking around and I hope this gave you the behind-the-curtains look into what happened during testing week! I’ll be back soon with another update.

Chat soon,

Alan

[darlicious]

It seems the most important issue that needs resolving at the moment is wait time and prioritization of customer issues to be handled by a moderator. Surely this cannot be this difficult to operate it similar to a triage or the Service Canada set up. First statistically review the percentage of issues in past: activations 50%, payments, plans&add ones 25%, self serve, phones& hardware15% and rewards, retail/promo&other 10% as an example. Appropriately staff each of these departments based on the urgency, importance and numbers in the queue. Obviously someone having issues with completing their activation or porting their number should have precedence over someone missing a refer a friend reward. Like Service Canada  Simon or the department on your ticket would put your ticket in the appropriate queue. The reference number of your ticket should give the user the ability to go into a query box and allow you to add more information, withdraw your ticket or mark it as resolved, check what number is being served, what number is up next and the average wait time in your department. That's how it works the old fashioned way with government brick and mortar service models. Moderators could pulled from a slow department to a busier one depending on demand in the system. A gentle reminder to customers that "staying on the line is faster than redialing" would help in keeping multiple tickets dealing with the same issue from one customer to a minimum. I think for the most part even though it seems annoying when the person behind you gets served first....because they ordered a salad and you ordered a well done steak....most people understand the guy having a stroke is more important than your finger sprain. But my lack of technology savvy always amazes me when something i think should be easy for computer to accomplish hasnt been achieved yet. Im still waiting to be able to talk to computer and tell it what to do. One of the reasons i still have a flip phone..i want my brain to be smarter than my phone. Keep working hard pm you may some day "in the near future" surpass the collective brain of the community.

[mperras]

You could likely devise a way to filter repeated questions of the same nature from the same user. Perhaps giving them a prompt before the post to confirm that they really do want to post the same question again. 

[tyedye00]

Exciting times indeed.

I look forward these new tools that will be soon available to the "public". Go SIMon!

Let us know if we can further help/test in any way.

[ChuckYeah]

I'm wondering how often problems are fixed by the time a Moderator looks at the original message/ticket? Will there be a way that the client can mark the ticket as completed so that the Moderators don't spend unnessary time should passing time or the community be able to solve the issue?

[Psygineer]

---

@srlawren wrote:

---

@mimmo wrote:

@srlawren  if you get a ticket number of 500 and then find out ticket 400 was worked on. And later find ticket 425 is worked on you know roughly when yours will get addressed.  Assuming  answered in order received 

---

@mimmo I mean, sure.  But let's say you're on hold with your home TV/internet provider.  You get in the queue, and they say "estimated wait time is approximately 85 minutes", they don't say "you are customer 123 in line, and we are currently assisting customer 62".  Unless you know an average time per customer (ticket velocity) and how many agents are working (team velocity), you really have no way to know what your position in the queue might translate to in reality.  For example, maybe you're number 201 and they're on number 199, so you think, any second now!...but then it turns out that each request actually takes an hour and a half....

---

True but, what about this situation. If we assume they process the tickets in the order they arrive (First in First out) and you ticket number is 1701 and they are currenly working on ticket 1686 you know their likely will be some wait. After a few days of waiting you check back to see your progress and the display says they are now working on ticket 1742 then you know that either your ticket was missed, still being processed by higher level, or completed. Checking your ticket status flag Ex. Open, Closed, Under Review, Escalated, Awaiting User Response etc. would let you know if you need to take any actions yourself. So if your ticket is still sitting in the Open status and they are far past yours in the queue odds are the Mod who got your ticket got distracted and moved on accidentally without doing anything.

The only complication I can see with the spot in queue isn't exactly processing time based, it's the fact that multiple tickets can be actioned at once due to there being multiple mods. If Mod A has 1699, Mod B has 1700, Mod C has 1701 and Mod B finishes their 1700 ticket, they would move on to 1702 which may be a redundant ticket because someone was spam happy and didn't get a response within 5 seconds so they sent another identical ticket. Since Mod B recognized the ticket as being the same as 1700, they close that ticket and go on to 1703. This would imply that 1704 is next (which it truly is) but also that 1699 is likely done  (which it is not). So now the person who issued 1699 might be watching the queue counter due to knowing theirs is being worked on and with the queue rapidly advancing beyond their number might pull the trigger and create ticket 1776 for the same topic as 1699 despite it being actively worked on at the moment. 

I likely didn't explain that all that well, but I tired. I am so tired. 

[mimmo]

@will13am  that can be handled via the smart form. Assuming it's smart enough.  It should be designed is a way to handle all situations. Ie activation errors (where no self serve account is created). It should/could still submit even without verification. 

The reason I suggested removing private messages as a primary/initial contact method was so that everyone gets a ticket number everyone goes through same process. And this avoids having multiple access points for help.

This is also assuming there is a way to access the form without Simon. 

[will13am]

The lively discussion here is great.  The only thing that has been mentioned which I don't agree with is removing the current private message method of moderator team interaction.  My biggest concern with identity verification is the lack of a fail safe option.  If an existing customer forgot their password, cannot get into the self serve to pay up on a suspended account would have a real problem on their hands.  Likewise the customer trying to activate and need moderator team assistance isn't going to be able to navigate the identity verification.  For all the faults with the current interaction system, it can handle all situations.  Unless a replacement is found, the current model has to be retained as a backup.  

[cavemantoronto]

Next?  U.S. roaming text bug after arriving home, suspended account on reneway day, and fix 2 step procdure of add-on purchase?

[Poogzley]

Hope to see more updates like this in the near future!

[srlawren]

---

@mimmo wrote:

@Alan_K  another thing that i believe would be benificial is to have a proper understanding how the backend of the private messages works:

 <trimmed the details for brevity>

@mimmo this is a great ask, it would sure help reduce the opacity of the current (and by extension: near-future) process

[mimmo]

@srlawren  i dont disagree with you i believe i started that sentence or possibly,  as kind of a last ditch method of getting some info from the abyss, since the eta accoriding to alan was a in the near future (and we all know what that means).  

I would love to see something on the lines of: working on messages submitted day x at time y toronto time.  working on tickets from 48 hours ago. again assuming FIFO

in the end i think the ticket number is possibly red herring, if the form works in getting accounts verrified and all the needed info, then i think the effecincy of the process will improve.

how failed activations will work, or people who did not setup a selfserve account with the form will be interesting

[srlawren]

---

@Anonymous wrote:

All I know is that no one likes to be completely in the dark as to where they stand.

---

@Anonymous amen!!!!!

[srlawren]

---

@mimmo wrote:

@srlawren  if you get a ticket number of 500 and then find out ticket 400 was worked on. And later find ticket 425 is worked on you know roughly when yours will get addressed.  Assuming  answered in order received 

---

@mimmo I mean, sure.  But let's say you're on hold with your home TV/internet provider.  You get in the queue, and they say "estimated wait time is approximately 85 minutes", they don't say "you are customer 123 in line, and we are currently assisting customer 62".  Unless you know an average time per customer (ticket velocity) and how many agents are working (team velocity), you really have no way to know what your position in the queue might translate to in reality.  For example, maybe you're number 201 and they're on number 199, so you think, any second now!...but then it turns out that each request actually takes an hour and a half....

[mimmo]

@Alan_K  another thing that i believe would be benificial is to have a proper understanding how the backend of the private messages works:

•  how do multiple private messages get handled.  
• are requests prioritised  (ie no service issues before say i didnt get my autopay) of fifo
• what happens if I say   submit 2 tickest on differeing issues  or on for differing accounts are they lumped into 1 thread  (which seems to be what happens now) or are they considered seperate
• if someone wats to add more info do they need to reply with the ticket number or is all that visable in the backend  as one big thread 
• how non ticketed private messages are handled (see my  first post)

also is there a way to cancel a ticket?

with a proper understanding active members can more easily direct

[Psygineer]

The ticket number being worked on currently would definately be welcome, I am on Day 8 currently of my ticket wait time and don't want to submit another ticket, but I have no idea if my ticket has been missed or forgotten at this point. Some indicator if it was active and pending or closed or anything to show that it will eventually be action would be welcome. I'm also on week 6 of only having service 20% of the time I need it (That's what my ticket was about.).

[wetcoaster]

@srlawren 

IMHO the "ticket number" goes hand in hand with estimated wait times.

If I'm pulling the physical ticket number 674 at the post office, or the BC Service Centre, and the "calling up" display shows number 655, I know that I might be waiting for a considerable amount of time. If, a quarter of an hour in, the "ping" is going off the hook with the next five or six numbers, I can draw a rough estimate how long I might be waiting. But I'm also reassured that someone is working and will eventually get to me.

I agree with you all that a properticket system is long overdue, MUST be launched ASAP, and cannot be a nice to have feature that "we get to implement when we get around to it" (which, by Public Mobile standards, might as well be "never"... Yep, call me disillusioned...)

However...

All these improvements are good and these updates hopefully keep the crew accountable. I strongly believe that  just dealing with the symptoms and not the cause is enough, though.The wait times are so long because there are so many problems with the self serve platform / structure / patch work coding, and it seems that everytime they fix something, they break something else in the process.

I'm not an IT professional, so I might look at this with some rose tinted glasses, but in my field, if something is broken in so many ways and is hold together with some old shoestrings, duct tape and mainly good faith, you go and build a new system from scratch, with a proper, streamlined and up to date structure, with corrections of known old problems (Canada texting not working on day 11 after buying a US add-on, anyone?) and new features that reflect current needs (early renewal of the current plan by the customer, etc). 

It might be costing more upfront, but in the long run there can be huge savings by paying less support staff.

And yes, I'm disillusioned enough that I don't hold my breath that that is going to happen in this century.

[mimmo]

@srlawren  if you get a ticket number of 500 and then find out ticket 400 was worked on. And later find ticket 425 is worked on you know roughly when yours will get addressed.  Assuming  answered in order received 

[Anonymous]

mimmo has an interesting idea.

You are behind 27 others.

Or currently serving # 12. You are 48.

Or use the 611 and check your position and what is being served. In the form response you are #48. Check 611 and it says serving #12. Or a web page with the same thing. Queue-checker.html. Refresh. Serving #13. You already know you're #48.

Then we can all socially shame those that are dumping all over the queue 🙂

Whatever. I dunno. All I know is that no one likes to be completely in the dark as to where they stand.

[srlawren]

---

@mimmo wrote:

4) possibly post what ticket was recently completed. 

---

@mimmo this is the only part I'm not sure I see the value in, but agree with the rest of your post.  Espeically the part about the wait times needing to be sooner than later!

@Alan_K thanks for the second installment.  While I still shudder at the use of the term "Developer" in the title, this episode is definitely more "meaty" than the first, and you've provided some good info.  I'm pleased to see how seriously you're (the collective you) taking the feedback from the testing sessions, and it sounds like you're mostly on the right track here.  I can't stress enough how important the estimated wait time figure is, though.  This has been a recurring theme in the community for the better part of a year or so, with people frustrated at not having any idea how long they might have to wait, or what the expectation should be.

[mimmo]

@Alan_K  great update.

1) The mod reply time should not be a future goal but a next goal. 

2) there should be a form not linked to Simon that is accessible from community. I find it silly someone comes to community, and we say ask Simon for help.   Forms are now smart enough that fields can be populated conditionally

3) private messages should be axed as a way to make first contact with mods. Else we will still have the same issue people will use form and then use private messages when they get no eply in 24 48 72 hours. 

4) possibly post what ticket was recently completed. 

[Anonymous]

 @Alan_K 

An estimated wait time is just that. Place in queue might be more useful. Hopefully all of this will reduce those customers pounding the pm with a bunch of submissions thereby causing the queue to be even more delayed.

Thanks for the update.