LangSmith can now search agent logs for behaviors like "said it finished but didn't". Every client bot needs that review loop

The failures that hurt client bots rarely throw errors. They look like a polite reply that did not do the job.

What happened

LangChain added AI Search to LangSmith. You describe a behavior in plain language, such as "did the user get frustrated?" or "did PII leak?", and a fast decision model reads traces and returns the ones where it happened. LangChain lists other examples: an agent that said it finished a task but didn't, or a user trying to cancel a subscription. It works alongside existing filters, so a search can run only on errored traces or a specific tag.

Under the hood, LangSmith turns traces into a trajectory per session, listing each message, tool call and result once. In a sample of 150 Open SWE conversations, about 2K of the median 72K tokens came from user messages and agent replies, and about 85% was tool output. So AI Search fills an 8K token budget in priority order, user and agent messages first. LangChain says vector search can miss the difference between an agent that did the work and one that only said it did. AI Search is in public beta on LangSmith Cloud at no additional cost, and org admins turn it on in settings.

My take

Most of the support and lead bots I see in small businesses have logs nobody reads. The bot does not crash. It just tells a customer "your booking is confirmed" when the calendar call failed, and nobody finds out until the complaint.

The habit worth stealing is a weekly behavior review. Pick five questions that map to money or trust: claimed an action without a successful tool call, user asked for a human, user mentioned cancelling, bot quoted a price, personal data in a reply. Run them over last week's conversations and read the hits.

If you are not on LangSmith, you can still do a rough version: store each session as one clean record with the tool results, then run a cheap yes or no model over it. Every confirmed failure becomes a test case before the next prompt change.