“AI will deflect our customers.”
Everyone who is in the support industry has either heard a colleague say this or has considered it themselves. When the support community started jumping on the AI bandwagon, the first focus was — and often still is — how the support demand can be reduced. How can we do more with fewer people? The key metric becomes “deflection.”
The implication of this deflection approach is that it is extremely company-centered. In this context, AI is the path to reducing support cost. Of course, every company will be glad to keep its costs low, but I would argue that for support, this should not be the primary focus.
When deflection becomes the main goal, customer interactions are a fallback — only to be used when absolutely necessary. The support team suddenly becomes the rare resource that we need to shield from those unwelcome interactions.
AI assistants become the henchmen who do the dirty work, sending customers down rabbit holes and endless loops. Support performance metrics are great, as long as the support teams don’t have to engage.
Of course, this is an exaggerated image, but it represents a very dominant support narrative. At WooCommerce, we also bought into it at first. When we first implemented our AI tools, our first metric was deflection.
However, as soon as we asked ourselves the question of what we really wanted to achieve with our AI tools, it clicked that deflection sent us down the wrong path. Now, we always ask a question more central to our own idea of support: does it make the customer experience better?
This is where we want our true AI goal to lie: not as a way to make it harder to contact us, but as a way to help our customers faster and more effectively. Whenever we are evaluating our use of AI solutions, we always come back to that question: does it make the customer experience better?
Our focus on the customer experience has resulted in several strategic changes in how we provide support.
Conducting quality assessments and continuous improvement
Table of Contents
First, we have ongoing reviews of the quality of our AI replies. On a weekly basis, we review at least 250 AI replies (roughly 20% of the volume we send out). Everyone in our support team gets involved with these reviews, so AI reply quality is continuously improved with a large variety of perspectives and areas of expertise.
We review how well our AI tool categorises the support request and how well it selects resources, and we evaluate the actual replies it gives. This allows us to regularly tweak our prompting, resources, and configuration.
And it’s paying off. When we started this over a year ago, we decided to review and rate the AI assistants based on how helpful and problem-appropriate their responses were. We started at around 60% helpfulness. As of April 2025, our AI responses are helpful over 90% of the time.
Our team carefully reviews feedback that we receive about our AI-powered replies and the customer experience in dealing with our AI assistants. While we cannot always make instant changes, we analyse the feedback and make informed decisions based on this analysis.
For example: When customers submit a support request, the first reply they receive is an AI-generated message containing suggestions and possible resolutions to the request. Customers indicated that it was not clear they could simply reply to the AI email to get in touch with a human. As a result, we changed our template to make that more clear. While the amount of times this feedback is given is now smaller, we will keep tweaking our template until no one mentions this anymore.
Rethinking support resources and documentation
We’ve rethought our entire approach to public information about our product. We used to have over 1,000 internal documents on how to configure and troubleshoot our products. Not only had this become difficult to process for our team members (yes, we built an internal AI assistant for our internal docs), we also realized that shielding much of this information behind our support team created an unnecessarily poor customer experience.
So, we reviewed every single one of those docs and shifted the best resources into our public documentation. The move empowered our customers to find more and better information for themselves — and it broadened the number of resources our AI assistant could pull quality information from.
Some of the shifted documentation includes:
Carefully testing and implementing AI assistants
Before launching a new assistant, we take care to test and iterate against a wide range of known support scenarios. Once launched, we’re not afraid to disable AI assistants if the results aren’t acceptable.
For instance, after implementing our AI solution in emails, we found that the automated replies for WooPayments questions weren’t meeting our standards. We were convinced that sending out AI-generated emails for WooPayments would’ve created a worse customer experience than what we had at the time.
As a result, we excluded AI replies from any WooPayments ticket, even though WooPayments consists of ~25% of our total volume. If our main concern was to reduce the number of support requests to our team, this would’ve been an unwise strategic decision.
In early March 2025, we set up a new AI assistant that focused on these support requests specifically. We are finally confident that our tool can handle the uniqueness and the complexity of our WooPayments product.
Providing clear lines to human support
We work with the constant assumption that AI assistants aren’t perfect — much like none of us are either. So, we want to make it as easy as possible for people to indicate that they want to reach out to a human. In our emails, we invite people to simply hit the reply button to chat with a human. On our support page, customers can contact us directly.
Next to that, we’ve also designed our assistants to err on the side of asking for clarification or handing questions to a human if they aren’t sure. That system isn’t perfect, but as a result, we’ve vastly reduced the number of times when one of our AI assistants very confidently gives the entirely wrong answer.
Always focusing on improving the customer experience
We’re comfortable shipping an imperfect solution. When we launched the AI reply in our emails, the correctness of our AI assistant was not yet where we wanted it to be. But the question we kept coming back to was: “Does it make the customer experience better?”
Before AI-powered email replies, we sent out a static automated message that said something along the lines of: “Hey, we’ve received your request and will get back to you within 24 hours. While you wait, have a look at our docs [insert generic landing page] and our troubleshooting guide [even though we weren’t sure whether you actually needed it].”
The email AI assistant changed a mostly useless reply to an instant email that offered specific documentation and attempted to answer your individual questions. During the time customers waited, our assistant offered a helpful reply in more than half of the times it sent one out. So, it wasn’t a perfect customer experience, but it was definitely better than what we had before.
What’s next?
People across all industries, demographics, and geographies are getting used to processing information with the help of AI. Many of our customers look to an AI assistant for help with processing documentation on WooCommerce.com rather than needing to search for the right answer themselves.
In the first full week of 2025, we launched a conversational AI assistant on our support page to do just that. Our next step is to bring this assistant to many more places on our website: our bot will become your ever-present AI assistant to get help on WooCommerce.com.
While this will initially be an AI-only solution, our phase two goal is providing a smooth path to chatting with a human. We will ship the AI-only solution first, though, because we believe it creates a better customer experience than what we’ve had before.
We’d love to hear about your experiences! We review every piece of feedback that we receive, and there are plenty of ways to get in touch:
- Leaving feedback on your support interaction,
- Tagging me on the Woo community Slack,
- or commenting on this post.
Let us know what you think and how we could improve your experience. We can’t wait to hear from you.
About
Job Thomas
Job Thomas is the Chief Customer Officer at WooCommerce, where he and his team enable customers to thrive with Woo. Before joining Woo in 2014, Job worked in the academic world, focused on educational philosophy and curriculum development. He uses that background to empower customers to run their stores. Job is Belgian, but has been living in Cape Town, South Africa, for over a decade.
