cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Public Mobile

Developer’s Blog: Learnings from customer testing sessions

Welcome back to the second installment of my humble little blog, where I’ll take you behind the scenes to the development of our upcoming support enhancements! I really enjoyed reading the constructive criticism and encouraging comments from the first blog, please keep them coming Smiley Happy  In today’s entry, I want to share what the customer testing sessions were like, what we learned, and what we’re working on coming out of it.

 

User testing format

 

In mid-July, my team and I reached out to 18 testers (9 customers, 6 Oracles, & 3 Moderators) to gather their thoughts and feedback. It was important to learn about the experience using the new support forms both from a customer point of view and from a Moderator view, as support is a two-way street. We needed to think about all of the parties involved, and how we go about optimizing the experience. 

 

We began our testing by asking testers to start their help enquiry within SIMon, our virtual assistant. Fun fact: Did you know SIMon is answering almost 20K questions per week? He’s been crushing his job of sharing answers from Help Articles and the Community for general questions, or directing customers to contact a Moderator for more serious issues - not to brag, but he’s over 90% accurate Smiley Wink 

 

With these tests, I was also experimenting with having SIMon act in the capacity of a front desk agent, where he’ll point our testers to where they need to go. Our time is precious, and I promise we’re not making you use the virtual assistant just for the sake of it! You might get the answer you’re looking for right away, but if not, the super neat thing about SIMon is that based on your short conversation, he’ll be able to remember what you shared with him and pre-fill parts of your ticket submission to a Moderator for you when we launch in mid-August

 

We had 1-3 scenarios set up for our testers depending on the available time.

  1. You’re a new customer trying to activate on Public Mobile, but you’re having credit card issues preventing you from completing activation.
  2. You’re trying to activate your account online once again, but you’re encountering an error with your SIM.
  3. (We asked customers to think back to the last question/issue that they encountered)

 

Then, we opened up the playing field to our customers to test away!

 

What we learned from our testers and what we’re doing about it

*All screens shared are not final, and were taken from the same testing screens our testers saw

 

On using SIMon…

Our testers navigated through SIMon easily, using quick reply buttons to quickly share their initial issue details, and were pleasantly surprised when SIMon surfaced relevant tips and Help Articles.

 

screen1.PNG

SIMon surfacing a Help Article for the tester.

 

Here were some verbatims that customers shared:

  • “I don’t see SIMon, maybe because I’m used to asking for help in the Community.”
  • “It’s [my account problem] not something I would think I could ask SIMon”

 

What we’re focusing on for launch:

  • Help customers find SIMon easily (better visual cues)
  • Clearly communicate what SIMon can do (i.e. take you to where you need to go for your answer + guided way to submit a ticket to a Moderator)
  • Continually work towards making SIMon simple and easy to use with quick replies
  • We also recognize the virtual assistant is not for everyone, so don’t worry- you don’t have to talk to SIMon before submitting a ticket- private message is still sticking around, but for those looking for a quicker and guided experience, I’d say SIMon will be a great choice after this launch.

 

On using the smartforms…

 

Testers reacted very positively to the added structure that the smartforms offered in contrast to the current free-form text entry of private messaging to contact a Moderator for added support. Folks were glad that their earlier conversation with SIMon wasn’t for naught, but actually helped them to save time by pre-filling fields in the Smartform for them. Moderators and customers were both delighted about authentication happening upfront, without any need to follow-up when authentication was done. Some insightful comments were offered particularly by the Oracles and Mods in terms of fields that we missed, or verbiages that needed some more fine-tuning to be crystal clear at launch Smiley Happy

 

screen2.PNG

SIMon will pre-fill your issue category and subcategory for you in Step 1.

 

Here’s what people had to say:

  • “It’s nice to have assisted guidance to resolution. I like it.”
  • If Self-Serve password/username authentication was selected, Public Mobile accounts using email aliases (Gmail function) could not be allowed through [Oracles]
  • For customers encountering payment issues, if they have a pre-registered credit card, we can collect their authorization to execute the transaction they couldn’t perform in the form to save time [Moderators]

 

What we’re focusing on for launch:

  • Email aliases bug has been addressed in staging and will be ready for launch
  • Payment authorization has been included in the form for customers who are comfortable with this option
  • Continually monitor customer feedback through surveys to make improvements on a rolling basis; action quick wins and build a backlog of higher lift changes

 

On the ticket submission confirmation page...

 

We tested two different versions of the confirmation page- version one being short and concise, whereas version two was longer and had more information related to next steps. Most customers felt reassured as long as they saw a reference number, but wanted to know what should be done with it afterward, and how long they should specifically be waiting. Ultimately, the short form prevailed in terms of popularity.

 

screen3.PNG

Short version vs Long version 

 

Here’s what people had to say:

  • “The checkmark is good because it marks that I’m done.”
  • “Besides the reference number, I just need to know what my wait time will be. How long do I have to wait?”
  • Confusion as to what the next steps are and what to do with the reference number after submission is complete [Oracles]

 

What we’re focusing on for launch:

  • Simplified reference number
  • Explicit call outs that a copy of the ticket is available in the customer’s Community inbox, including a direct link, and an option to print the confirmation page (including reference number)
  • Future goal: add in up-to-date response wait time estimate

 

Final thoughts

As always, co-building these support enhancements with our customers gives my team and me lots of joy, and we are always taken aback by the passion and breadth of ideas that everyone shares with us. There’s a ton out there we can (and will) do to keep improving upon these to make your experience even better, and this is just the tip of the iceberg that we can tackle in the remaining development time we have before launch time.

 

Oh boy, that was a super long post! If you made it this far, thanks for sticking around and I hope this gave you the behind-the-curtains look into what happened during testing week! I’ll be back soon with another update.

 

Chat soon,

Alan

Mayor / Maire

Re: Developer’s Blog: Learnings from customer testing sessions

 @Alan_K 

An estimated wait time is just that. Place in queue might be more useful. Hopefully all of this will reduce those customers pounding the pm with a bunch of submissions thereby causing the queue to be even more delayed.

 

Thanks for the update.

Oracle

Re: Developer’s Blog: Learnings from customer testing sessions

@Alan_K  great update.

 

1) The mod reply time should not be a future goal but a next goal. 

2) there should be a form not linked to Simon that is accessible from community. I find it silly someone comes to community, and we say ask Simon for help.   Forms are now smart enough that fields can be populated conditionally

 

3) private messages should be axed as a way to make first contact with mods. Else we will still have the same issue people will use form and then use private messages when they get no eply in 24 48 72 hours. 

 

4) possibly post what ticket was recently completed. 

 

Mayor / Maire

Re: Developer’s Blog: Learnings from customer testing sessions


@mimmo wrote:

4) possibly post what ticket was recently completed. 


@mimmo this is the only part I'm not sure I see the value in, but agree with the rest of your post.  Espeically the part about the wait times needing to be sooner than later!

 

@Alan_K thanks for the second installment.  While I still shudder at the use of the term "Developer" in the title, this episode is definitely more "meaty" than the first, and you've provided some good info.  I'm pleased to see how seriously you're (the collective you) taking the feedback from the testing sessions, and it sounds like you're mostly on the right track here.  I can't stress enough how important the estimated wait time figure is, though.  This has been a recurring theme in the community for the better part of a year or so, with people frustrated at not having any idea how long they might have to wait, or what the expectation should be.


>>> ALERT: I am not a moderator. For account or activation assistance, please click here.
Mayor / Maire

Re: Developer’s Blog: Learnings from customer testing sessions

mimmo has an interesting idea.

You are behind 27 others.

Or currently serving # 12. You are 48.

Or use the 611 and check your position and what is being served. In the form response you are #48. Check 611 and it says serving #12. Or a web page with the same thing. Queue-checker.html. Refresh. Serving #13. You already know you're #48.

Then we can all socially shame those that are dumping all over the queue Smiley Happy

Whatever. I dunno. All I know is that no one likes to be completely in the dark as to where they stand.

Oracle

Re: Developer’s Blog: Learnings from customer testing sessions

@srlawren  if you get a ticket number of 500 and then find out ticket 400 was worked on. And later find ticket 425 is worked on you know roughly when yours will get addressed.  Assuming  answered in order received 

Mayor / Maire

Re: Developer’s Blog: Learnings from customer testing sessions

@srlawren 

IMHO the "ticket number" goes hand in hand with estimated wait times.

If I'm pulling the physical ticket number 674 at the post office, or the BC Service Centre, and the "calling up" display shows number 655, I know that I might be waiting for a considerable amount of time. If, a quarter of an hour in, the "ping" is going off the hook with the next five or six numbers, I can draw a rough estimate how long I might be waiting. But I'm also reassured that someone is working and will eventually get to me.

 

 

I agree with you all that a properticket system is long overdue, MUST be launched ASAP, and cannot be a nice to have feature that "we get to implement when we get around to it" (which, by Public Mobile standards, might as well be "never"... Yep, call me disillusioned...)

 

 

However...

All these improvements are good and these updates hopefully keep the crew accountable. I strongly believe that  just dealing with the symptoms and not the cause is enough, though.The wait times are so long because there are so many problems with the self serve platform / structure / patch work coding, and it seems that everytime they fix something, they break something else in the process.

I'm not an IT professional, so I might look at this with some rose tinted glasses, but in my field, if something is broken in so many ways and is hold together with some old shoestrings, duct tape and mainly good faith, you go and build a new system from scratch, with a proper, streamlined and up to date structure, with corrections of known old problems (Canada texting not working on day 11 after buying a US add-on, anyone?) and new features that reflect current needs (early renewal of the current plan by the customer, etc). 

It might be costing more upfront, but in the long run there can be huge savings by paying less support staff.

 

And yes, I'm disillusioned enough that I don't hold my breath that that is going to happen in this century.

Deputy Mayor / Adjoint au Maire

Re: Developer’s Blog: Learnings from customer testing sessions

The ticket number being worked on currently would definately be welcome, I am on Day 8 currently of my ticket wait time and don't want to submit another ticket, but I have no idea if my ticket has been missed or forgotten at this point. Some indicator if it was active and pending or closed or anything to show that it will eventually be action would be welcome. I'm also on week 6 of only having service 20% of the time I need it (That's what my ticket was about.).

Oracle

Re: Developer’s Blog: Learnings from customer testing sessions

@Alan_K  another thing that i believe would be benificial is to have a proper understanding how the backend of the private messages works:

  •  how do multiple private messages get handled.  
  • are requests prioritised  (ie no service issues before say i didnt get my autopay) of fifo
  • what happens if I say   submit 2 tickest on differeing issues  or on for differing accounts are they lumped into 1 thread  (which seems to be what happens now) or are they considered seperate
  • if someone wats to add more info do they need to reply with the ticket number or is all that visable in the backend  as one big thread 
  • how non ticketed private messages are handled (see my  first post)

also is there a way to cancel a ticket?

 

with a proper understanding active members can more easily direct

Mayor / Maire

Re: Developer’s Blog: Learnings from customer testing sessions


@mimmo wrote:

@srlawren  if you get a ticket number of 500 and then find out ticket 400 was worked on. And later find ticket 425 is worked on you know roughly when yours will get addressed.  Assuming  answered in order received 


@mimmo I mean, sure.  But let's say you're on hold with your home TV/internet provider.  You get in the queue, and they say "estimated wait time is approximately 85 minutes", they don't say "you are customer 123 in line, and we are currently assisting customer 62".  Unless you know an average time per customer (ticket velocity) and how many agents are working (team velocity), you really have no way to know what your position in the queue might translate to in reality.  For example, maybe you're number 201 and they're on number 199, so you think, any second now!...but then it turns out that each request actually takes an hour and a half....


>>> ALERT: I am not a moderator. For account or activation assistance, please click here.