How to measure whether your AI agent is working

First, pick one specific use case. Answering common questions is not the same as qualifying leads, booking appointments or solving support issues. Each goal needs a different main signal.

You also need a baseline. For a comparable period, record how the channel works today: volume, reply time, handoffs, abandoned conversations and whatever result you already track. Without that snapshot, any improvement may just be a feeling.

For a small business that gets inquiries by chat, text or phone, the mix can be simple: time to first reply, the share of cases that reach the right person with a summary, complete lead details and one follow on action, such as an opportunity created or an appointment requested.

Take a sample of conversations and review it with a consistent rubric. Score whether the reply understood the intent, used approved information, asked for the details it needed, avoided making things up and offered a clear next step.

Group errors by type: wrong information, wrong tone, missing context, late handoff or out of scope request. That breakdown is more useful than a single average because it tells you which rule, source or integration to fix.

A handoff is not just saying "let me connect you with someone". A useful handoff includes the reason for contact, the details the customer already gave, what the agent answered and a suggested next step. That way your team can continue without making the customer repeat everything.

Track how many handoffs arrive complete, how long it takes someone to pick them up and how many get reopened because information was missing. A high handoff rate can be right for sensitive cases. A low one can be a problem if the agent is holding on to conversations it should escalate.

Pick an observable business event and tie it to the conversation. For sales, that could be a qualified lead, a meeting requested or an opportunity logged. For support, it could be a confirmed resolution, a reopened ticket or the time until service is restored.

Do not credit every result to the agent automatically. Compare similar periods, note any changes in campaigns or staff and separate a conversation started, an intent detected and a result actually confirmed.

• Define one use case and its main result.
• Record a baseline before the pilot starts.
• Agree on which sources the agent can use and what it cannot promise.
• Tag conversations as resolved, handed off, out of scope or error.
• Review a sample by hand every week.
• Decide in advance what result lets you expand, fix or stop the flow.

Metrics can mislead you when volume, campaigns, hours or the mix of questions change. You also need to think about privacy, access permissions, how long conversations are kept and each channel's rules for templates and consent.

Meta's official docs describe webhooks for receiving events and message statuses for tracking delivery. That helps you instrument the channel, but it does not replace a quality review or a clear definition of the business result.