How it works
One method, run the same way on every vendor
We have the same scripted conversations with every vendor's AI agent on 55 live storefronts, then score every reply blind.
Principles
Four rules every run follows
Same questions for all
Every agent gets the same scripted messages and the same answers to its questions.
Live stores
Each agent is tested on live storefronts that use it.
Blind judging
The judge never knows which vendor it is scoring.
Misses count the same
If a feature is switched off on a site, such as photo uploads, it counts as a miss for every vendor.
Choosing stores
55 live storefronts
The vendors
Chatbase and 7 vendors: Fin, Decagon, Sierra, Ada, Zendesk, Gorgias and Botpress.
The stores
55 live storefronts where a vendor's AI agent answers the chat.
Running the conversations
A scripted shopper, the same on every vendor
A new visitor every time
Each conversation starts fresh, like a first-time visitor with no account.
Scripted messages
The shopper sends the same messages, in the same order, to every agent, and never shares personal details.
Fixed answers
When an agent asks a question, the shopper replies from a fixed list, so no agent gets a better answer than another.
Hand-offs
If the chat passes to a person or closes, the conversation ends there and the remaining steps score 0.
Judging
Scored blind, one conversation at a time
Names removed
Vendor, agent and store names are removed before the judge reads anything.
Every reply scored
An AI judge places each reply on one of six levels, from 0 to 10, and backs every score with the agent's own words.
One conversation at a time
Each conversation is judged on its own, never against another vendor's.
Reporting
Shopping and Support, out of 100
A score out of 100
Each vendor gets a score from 0 to 100 in Shopping and Support.
Ranks
A vendor needs at least 9 judged conversations in a lane to be ranked there.
Overall
Overall is the average of Shopping and Support.
Guardrails
Guardrails are pass or fail and never change the scores.