From "We Have This Research" to "Our Work Reflects It"

The problem: research that lives in a PDF

The Words That Work toolkit, developed by Press Forward and Beekeeper Group, is a research-backed guide to which local news message frames resonate with which audiences, and which words and phrases to use or avoid. It's the kind of evidence-based foundation that could change how newsrooms talk to their communities.

In practice, research like this tends to live in a PDF. It gets read once, referenced occasionally, and applied inconsistently. Not because people don't value it, but because there's no infrastructure that makes it easy to use systematically. The gap between "we have this research" and "our communications work reflects this research" is where most of the value gets lost.

I wanted to build a tool that bridged that gap and made the research actionable.

From researcher to coach, with guardrails

The first part of the tool turns the PDF messaging guidance toolkit into an interactive message building tool. What I wanted was something that communications people would use, not just reference: a tool that could give them specific, grounded feedback at the moment they were drafting a message.

Essentially that is taking the role of the AI from researcher to coach. But if I was going to remove the human bridge between the research and the message, I wanted to make sure the tool had enough guardrails that we didn't see the kind of confident yet completely misguided feedback AI is notorious for. So before I built anything for users, I built for testability. The Message Builder knowledge base is structured so that every frame, every audience mapping, every word on the avoid list is explicit and inspectable. The model can only return frames that exist in the research; it can't invent new ones.

The other AI weakness I wanted to build around is the doesn't-know-what-it-doesn't-know problem. AI is good at pattern matching against a knowledge base. It's not good at knowing when the knowledge base is wrong, when an experiment was biased, or when a new piece of research should change the guidance. To make sure human expertise stays in the loop as the tool evolves, every consequential step requires a human decision: every confirmation of a hypothesis, every approval of a knowledge update, every selection of which experiments to synthesize is vetted by a person. That's not a limitation of the prototype. It's a design choice I'd keep in a production version.

The build: five tabs, one workflow

The tool has five tabs, each representing a step in a communications workflow.

Message evaluation & audience matching

These two let a practitioner get immediate, specific feedback on a draft. Not "this message could be stronger" but "this message uses democracy language for a general public audience, which the WTW research shows consistently underperforms, and here's what a revised version would look like."

Experiment builder

Walks practitioners through designing a real A/B test: hypothesis, control message, measurement approach, sample size, confounders to watch for. The goal was to make rigorous experimental design accessible to people who aren't statisticians, with enough explanation that they understand what they're doing and why.

Results dashboard

Interprets experiment outcomes through the WTW lens, including automated detection of Simpson's Paradox (cases where the treatment wins overall but loses in specific audience segments), so findings don't get missed by looking only at aggregate results.

Synthesis workbench

This is the tab that would enlarge the scope and scale of the findings. It lets a user bring in experiment results from multiple sources, flag outliers and caveats, incorporate new research documents, and generate a proposed update to the WTW knowledge base, section by section, with a before/after comparison showing how practitioner messages would actually change under the new guidance. Nothing gets written until a human reviews and confirms. This is the tab that lets the tool learn and evolve, based on how messages crafted under the current guidance actually perform with audiences over time.

What the full version would be

The prototype is built for a single newsroom. The more interesting version is infrastructure for an organization like the American Journalism Project (AJP).

If experiment results from across a grantee network flow into a shared synthesis workbench, with a program officer selecting which ones meet quality standards, flagging outliers, and incorporating new research as it emerges, the toolkit stops being a static document and starts being a living system. Every experiment a newsroom runs becomes a potential contribution to better guidance for everyone else.

At that scale, the analytics questions get interesting. Does geography affect message effectiveness? Are there temporal patterns tied to current events: frames that work better during election season, or after a local crisis? Which frames travel across audiences, and which are highly specific? The data to answer those questions exists in the experiments newsrooms are already running. The infrastructure to aggregate and learn from it doesn't exist yet.

I think that's a version worth building, and it isn't limited to local news messaging. The same architecture applies anywhere there's research-backed guidance, people applying that research, and data that could improve the process.