Creative evaluation and synthetic audiences for Bonnemaison

Bonnemaison is a creative and digital media agency that produces video campaigns for corporate, government and editorial clients. I worked with them as a consultant through Mission Cloud, on the part of their process that is hardest to systematize: judging, before anything is shot, whether a script will land.

The first piece was extracting knowledge from their expertise and encoding it for AI systems to utilize. Ad scripts are highly unstructured. The format varies by client, by medium and by writer, and the craft knowledge that makes one work is implicit in the text rather than labeled anywhere for computers and engineers to exploit. I built custom AI pipelines that read a corpus of scripts and pulled out both the artistic techniques in use and the criteria a reviewer actually applies, turning tacit craft into an explicit rubric someone can read and correct.

The second piece applied that rubric back: automated evaluation of a new script against the extracted criteria, so a writer gets structured feedback criterion by criterion instead of one opaque score. Keeping the criteria as data rather than baking them into a prompt is what makes this maintainable. The rubric can be revised, extended or scoped to a client without rebuilding the evaluator around it. What's more, criteria and evaluations started being accumulated, versioned, evolved and exploited, where knowledge and experience was before kept in people's memory.

The third piece was audience swarms: populations of generated personas used as synthetic focus groups. A script is read by many simulated audience members, with different, parametrized demographics and personality traits, and their reactions are aggregated to surface patterns. Not originally part of the project scope, it turned out to be one of the highlights. It was not intended to replace a real focus group, but it makes a cheap first pass possible at a point in the process where a real one is not.

The recurring engineering problem across all three was evaluation itself. Judgments about creative work are not cleanly right or wrong, as with any v0, so most of the effort went into making the systems inspectable and evolution-compatible, as well as providing a combination of metrics, instead of chasing a single accuracy number.