cancel
Showing results for 
Search instead for 
Did you mean: 

Developer’s Blog: Learnings from customer testing sessions

Alan_K
Public Mobile
Public Mobile

Welcome back to the second installment of my humble little blog, where I’ll take you behind the scenes to the development of our upcoming support enhancements! I really enjoyed reading the constructive criticism and encouraging comments from the first blog, please keep them coming 🙂  In today’s entry, I want to share what the customer testing sessions were like, what we learned, and what we’re working on coming out of it.

 

User testing format

 

In mid-July, my team and I reached out to 18 testers (9 customers, 6 Oracles, & 3 Moderators) to gather their thoughts and feedback. It was important to learn about the experience using the new support forms both from a customer point of view and from a Moderator view, as support is a two-way street. We needed to think about all of the parties involved, and how we go about optimizing the experience. 

 

We began our testing by asking testers to start their help enquiry within SIMon, our virtual assistant. Fun fact: Did you know SIMon is answering almost 20K questions per week? He’s been crushing his job of sharing answers from Help Articles and the Community for general questions, or directing customers to contact a Moderator for more serious issues - not to brag, but he’s over 90% accurate 😉 

 

With these tests, I was also experimenting with having SIMon act in the capacity of a front desk agent, where he’ll point our testers to where they need to go. Our time is precious, and I promise we’re not making you use the virtual assistant just for the sake of it! You might get the answer you’re looking for right away, but if not, the super neat thing about SIMon is that based on your short conversation, he’ll be able to remember what you shared with him and pre-fill parts of your ticket submission to a Moderator for you when we launch in mid-August

 

We had 1-3 scenarios set up for our testers depending on the available time.

  1. You’re a new customer trying to activate on Public Mobile, but you’re having credit card issues preventing you from completing activation.
  2. You’re trying to activate your account online once again, but you’re encountering an error with your SIM.
  3. (We asked customers to think back to the last question/issue that they encountered)

 

Then, we opened up the playing field to our customers to test away!

 

What we learned from our testers and what we’re doing about it

*All screens shared are not final, and were taken from the same testing screens our testers saw

 

On using SIMon…

Our testers navigated through SIMon easily, using quick reply buttons to quickly share their initial issue details, and were pleasantly surprised when SIMon surfaced relevant tips and Help Articles.

 

screen1.PNG

SIMon surfacing a Help Article for the tester.

 

Here were some verbatims that customers shared:

  • “I don’t see SIMon, maybe because I’m used to asking for help in the Community.”
  • “It’s [my account problem] not something I would think I could ask SIMon”

 

What we’re focusing on for launch:

  • Help customers find SIMon easily (better visual cues)
  • Clearly communicate what SIMon can do (i.e. take you to where you need to go for your answer + guided way to submit a ticket to a Moderator)
  • Continually work towards making SIMon simple and easy to use with quick replies
  • We also recognize the virtual assistant is not for everyone, so don’t worry- you don’t have to talk to SIMon before submitting a ticket- private message is still sticking around, but for those looking for a quicker and guided experience, I’d say SIMon will be a great choice after this launch.

 

On using the smartforms…

 

Testers reacted very positively to the added structure that the smartforms offered in contrast to the current free-form text entry of private messaging to contact a Moderator for added support. Folks were glad that their earlier conversation with SIMon wasn’t for naught, but actually helped them to save time by pre-filling fields in the Smartform for them. Moderators and customers were both delighted about authentication happening upfront, without any need to follow-up when authentication was done. Some insightful comments were offered particularly by the Oracles and Mods in terms of fields that we missed, or verbiages that needed some more fine-tuning to be crystal clear at launch 🙂

 

screen2.PNG

SIMon will pre-fill your issue category and subcategory for you in Step 1.

 

Here’s what people had to say:

  • “It’s nice to have assisted guidance to resolution. I like it.”
  • If Self-Serve password/username authentication was selected, Public Mobile accounts using email aliases (Gmail function) could not be allowed through [Oracles]
  • For customers encountering payment issues, if they have a pre-registered credit card, we can collect their authorization to execute the transaction they couldn’t perform in the form to save time [Moderators]

 

What we’re focusing on for launch:

  • Email aliases bug has been addressed in staging and will be ready for launch
  • Payment authorization has been included in the form for customers who are comfortable with this option
  • Continually monitor customer feedback through surveys to make improvements on a rolling basis; action quick wins and build a backlog of higher lift changes

 

On the ticket submission confirmation page...

 

We tested two different versions of the confirmation page- version one being short and concise, whereas version two was longer and had more information related to next steps. Most customers felt reassured as long as they saw a reference number, but wanted to know what should be done with it afterward, and how long they should specifically be waiting. Ultimately, the short form prevailed in terms of popularity.

 

screen3.PNG

Short version vs Long version 

 

Here’s what people had to say:

  • “The checkmark is good because it marks that I’m done.”
  • “Besides the reference number, I just need to know what my wait time will be. How long do I have to wait?”
  • Confusion as to what the next steps are and what to do with the reference number after submission is complete [Oracles]

 

What we’re focusing on for launch:

  • Simplified reference number
  • Explicit call outs that a copy of the ticket is available in the customer’s Community inbox, including a direct link, and an option to print the confirmation page (including reference number)
  • Future goal: add in up-to-date response wait time estimate

 

Final thoughts

As always, co-building these support enhancements with our customers gives my team and me lots of joy, and we are always taken aback by the passion and breadth of ideas that everyone shares with us. There’s a ton out there we can (and will) do to keep improving upon these to make your experience even better, and this is just the tip of the iceberg that we can tackle in the remaining development time we have before launch time.

 

Oh boy, that was a super long post! If you made it this far, thanks for sticking around and I hope this gave you the behind-the-curtains look into what happened during testing week! I’ll be back soon with another update.

 

Chat soon,

Alan

21 REPLIES 21

mperras
Great Neighbour / Super Voisin

You could likely devise a way to filter repeated questions of the same nature from the same user. Perhaps giving them a prompt before the post to confirm that they really do want to post the same question again. 

darlicious
Mayor / Maire

It seems the most important issue that needs resolving at the moment is wait time and prioritization of customer issues to be handled by a moderator. Surely this cannot be this difficult to operate it similar to a triage or the Service Canada set up. First statistically review the percentage of issues in past: activations 50%, payments, plans&add ones 25%, self serve, phones& hardware15% and rewards, retail/promo&other 10% as an example. Appropriately staff each of these departments based on the urgency, importance and numbers in the queue. Obviously someone having issues with completing their activation or porting their number should have precedence over someone missing a refer a friend reward. Like Service Canada  Simon or the department on your ticket would put your ticket in the appropriate queue. The reference number of your ticket should give the user the ability to go into a query box and allow you to add more information, withdraw your ticket or mark it as resolved, check what number is being served, what number is up next and the average wait time in your department. That's how it works the old fashioned way with government brick and mortar service models. Moderators could pulled from a slow department to a busier one depending on demand in the system. A gentle reminder to customers that "staying on the line is faster than redialing" would help in keeping multiple tickets dealing with the same issue from one customer to a minimum. I think for the most part even though it seems annoying when the person behind you gets served first....because they ordered a salad and you ordered a well done steak....most people understand the guy having a stroke is more important than your finger sprain. But my lack of technology savvy always amazes me when something i think should be easy for computer to accomplish hasnt been achieved yet. Im still waiting to be able to talk to computer and tell it what to do. One of the reasons i still have a flip phone..i want my brain to be smarter than my phone. Keep working hard pm you may some day "in the near future" surpass the collective brain of the community.

Need Help? Let's chat.