Gorgias' live-store benchmark finds ecommerce AI support automation is not the same as answer quality

Gorgias' October 2026 rolling benchmark currently shows 12,327 captured conversations, 12,322 blind LLM-judged conversations, 18 vendors and 265 live stores. Its public 120-day ranking uses cold sessions and two job-specific composites: shopping automation/quality/speed weighted 40/35/25 and support weighted 50/40/10. Gorgias ranks its own product first overall and in support but fourth in shopping, noting slower shopping answers than competitors. The live numbers change over time.

Operators selecting ecommerce AI support should not treat automation or ticket containment as proof of good answers. A system can avoid handoffs while misunderstanding policy, fail to take action, or stall shoppers. The public rubric offers a practical test design, but the evaluator is also a competitor; procurement should include independent tests on the buyer's own catalog and rules.

At inspection the live page reported 265 stores, 18 vendors, 12,327 conversations captured and 12,322 judged. Cold sessions use store-specific catalog questions. An evaluator never requests a human; handoffs must originate from the agent. At least 15 judged conversations per job are required for ranking. This is a moving snapshot, not a fixed research sample.

Support scores weight automation 50%, answer quality 40% and speed 10%. Shopping weights them 40%, 35% and 25%. Speed earns 100 at three seconds or less and zero at 22 seconds or more. Rankings therefore depend on what the buyer needs an agent to do.

Gorgias reports its support agent at 68% automation and 75/100 support quality in the inspected snapshot, first overall in support but second for quality behind Sierra. It reports roughly 18-second shopping answers, slower than some rivals. These are operator-produced metrics, not independently reproduced measurements.

The benchmark describes authentication walls, repeated order-number requests and clarification loops as frequent causes of failure. It also reports large differences between stores using the same vendor. Buyers should test identity, returns, product information and escalation rules with their own data rather than rely on a vendor leaderboard.

Gorgias sells an agent in the comparison and claims the top overall rank. Its rubric is public, but independent reproduction of runner, sample and scoring has not been established. SaaStr analysed an earlier smaller snapshot while disclosing an investment relationship with Gorgias. That is useful commentary, not independent validation.