Intercom adds an eval-and-release pipeline for changing Fin safely

Intercom launched Evals and Releases for Fin on August 13, 2026. Evals group multi-turn simulated customer conversations and score them against team-defined success criteria. Releases provide a separate workspace for changing content, procedures and guidance, running evals before deployment, and then publishing to everyone, gradually ramping traffic or A/B testing against the current configuration. Intercom’s existing Monitors can score live conversations and feed failures back into future evals and releases.

Customer-facing AI agents are increasingly part of SaaS operations, but many teams still change prompts, knowledge and procedures with weaker controls than they would apply to application code. Intercom is packaging a recognizable test–release–observe loop directly into the support product. That lowers the operational barrier to regression testing and staged deployment for non-ML teams, while also formalizing a new discipline: agent behavior needs versioned changes, representative test suites and production monitoring just like other mutable software systems.

A Fin Eval is a named collection of multi-turn Simulations. Teams can build scenarios that resemble real support conversations—refunds, escalation rules, tone or other recurring cases—and score each run against explicit criteria. The same Eval can be rerun after content, guidance or procedure changes to surface behavior regressions before customers see them.

A Release can bundle changes to Fin content, Procedures and Guidance without immediately modifying the live agent. Teammates can review the bundle, run Evals against it, fix failures and then choose how to ship. Intercom says a release can go to everyone, ramp gradually, or run as an A/B test against the current configuration.

Intercom positions Monitors as the production side of the system: every live conversation can be checked against quality standards, with failures becoming inputs for the next set of improvements. The practical pattern is familiar from software delivery—test before release, constrain rollout, observe production and convert failures into regression cases—but applied to probabilistic customer-agent behavior.

Automated eval scores are not ground truth. A simulated suite can miss rare customers, policy edge cases, tool failures or changes in the distribution of real conversations. Teams also need to know who defines scoring criteria and when a failed monitor should block or roll back a release. Evals therefore reduce operational risk only when paired with representative cases, human review and outcome metrics tied to real support quality.