How it works

One method, run the same way on every vendor

We have the same scripted conversations with every vendor's AI agent on 55 live storefronts, then score every reply blind.

Principles

Four rules every run follows

  • Same questions for all

    Every agent gets the same scripted messages and the same answers to its questions.

  • Live stores

    Each agent is tested on live storefronts that use it.

  • Blind judging

    The judge never knows which vendor it is scoring.

  • Misses count the same

    If a feature is switched off on a site, such as photo uploads, it counts as a miss for every vendor.

Choosing stores

55 live storefronts

  • The vendors

    Chatbase and 7 vendors: Fin, Decagon, Sierra, Ada, Zendesk, Gorgias and Botpress.

  • The stores

    55 live storefronts where a vendor's AI agent answers the chat.

Running the conversations

A scripted shopper, the same on every vendor

  • A new visitor every time

    Each conversation starts fresh, like a first-time visitor with no account.

  • Scripted messages

    The shopper sends the same messages, in the same order, to every agent, and never shares personal details.

  • Fixed answers

    When an agent asks a question, the shopper replies from a fixed list, so no agent gets a better answer than another.

  • Hand-offs

    If the chat passes to a person or closes, the conversation ends there and the remaining steps score 0.

Judging

Scored blind, one conversation at a time

  • Names removed

    Vendor, agent and store names are removed before the judge reads anything.

  • Every reply scored

    An AI judge places each reply on one of six levels, from 0 to 10, and backs every score with the agent's own words.

  • One conversation at a time

    Each conversation is judged on its own, never against another vendor's.

Reporting

Shopping and Support, out of 100

  • A score out of 100

    Each vendor gets a score from 0 to 100 in Shopping and Support.

  • Ranks

    A vendor needs at least 9 judged conversations in a lane to be ranked there.

  • Overall

    Overall is the average of Shopping and Support.

  • Guardrails

    Guardrails are pass or fail and never change the scores.

See the levels, rules and questions behind every score