Thousands of customer-agent conversations hold a clear picture of how a support operation is really performing, but it's buried in unstructured text no one has time to read. We built a system that reads all of it.
A support operation produces thousands of conversations, and almost none of them get read again after the ticket closes. Quality review in most contact centers means a supervisor reading a handful of conversations per agent per month. That small sample decides how agents get coached and how leadership sees the whole operation.
We wanted to know if we could do better: give managers a view of every agent built from every conversation, with no sampling.
We ran the project through CRISP-DM, the standard method we use on every project. Six steps that start with the business and end with something people can use.
Before touching any data, we had to learn how support operations are actually judged. It came down to three things a manager needs to know: what customers are contacting the team about, how well agents handle those contacts, and who needs coaching.
We also learned why the sampling approach survives even though nobody likes it. Reading conversations takes time supervisors don't have. Sampling isn't a choice anyone defends, it's just what fits in the hours available. That mattered for the design: whatever we built had to do the reading itself, or nothing would change.
The raw material was thousands of customer-agent conversations. Before modeling anything, we read a lot of them ourselves, enough to know how exchanges typically start, where they go wrong, and what separates a good resolution from a bad one. Then we cleaned and organized the transcripts so every later step had consistent input to work with.
Two tools, each with one job. Topic modeling (BERTopic) grouped conversations by what customers were raising, without us picking the categories in advance. Then a large language model read each conversation and assessed how the agent handled it, the way a human reviewer would, just across all of them.
What mattered was whether a manager could trust the output, not a model score. So the check was simple to state: when the system assessed a conversation, did its read match what we concluded reading the same conversation ourselves? We kept adjusting until it did.
The final step rolled everything into one management view: what customers raise, how it's being handled, and where each agent is strong or needs coaching. That view was the deliverable. A manager can open it and act on it without knowing anything about the models underneath.
This project is the shape of every engagement we run. For a client, the same steps would run on your operation and your data, and the tool would be handed to your team.
This was graduate capstone work. We're happy to walk through it in detail, including what we'd do differently in a live operation.
That's often where the fastest wins hide. Let's talk.