When a Label Is Subjective, AI Teams Need to Agree on What It Means

A team building an image classifier may begin with a label that feels obvious: “professional.” Then a photographer tags a candid portrait one way, a recruiter tags it another, and a designer points out that the label quietly encodes a workplace norm. Google Research’s Agile Deliberation treats that disagreement as part of the work. Instead of asking people to label examples against an assumed definition, the system guides a conversation that helps a group make its categories explicit. ^{[1] [2]}

Ask what the category means before scaling it

Many classification tasks have crisp answers: a photo either contains a bicycle or it does not. Others depend on context and preference. “Formal,” “beautiful,” “safe,” and “appropriate” can mean different things to different people, and disagreement may be a clue that the label combines several distinct ideas. A classifier trained on those labels can still produce a clean score, but the score may conceal unresolved human choices. ^{[1] [2]}

Agile Deliberation approaches the task as iterative concept design. Participants refine the broad category into subcategories, consider examples near the boundary, and use a language-model-guided optimization loop to build a visual classifier from the clarified concepts. The method’s premise is that a model can help people explore and articulate distinctions, while humans remain responsible for choosing the distinctions that matter. ^{[1] [2]}

The Google Research post reports evaluation through 18 live sessions, each lasting about 1.5 hours. That format lets researchers observe how people’s intent changes while they work with a system, rather than relying only on a static benchmark. The work appeared as a CVPR 2026 paper; the November post is a fresh explanation of the method, not a newly announced general-purpose product. Results from these sessions do not establish that the workflow will resolve every kind of disagreement or work equally well in every culture or organization. ^{[1] [2] [4]}

Deliberation improves the question, not the authority structure

Making categories explicit is a real improvement over hiding a subjective decision inside a dataset. A group can document its working definition, collect examples that expose edge cases and revisit the standard when the use changes. But a guided discussion can still privilege the people in the room, the examples they see and the language the model proposes. A process that ends in consensus can erase minority views rather than resolve them.

That matters when the classifier will influence employment, school access, content moderation or safety decisions. A team should ask who is invited to define the category, whether affected people can challenge it, and whether the output is used as advice or as a gate. It should also check whether the training examples cover the people and situations where the classifier will operate. NIST’s AI Risk Management Framework emphasizes identifying context, affected parties and risks throughout design and use; no single facilitated session substitutes for that work. ^[3]

A better handoff to production

The practical value of a workflow like this is not that it removes ambiguity. It gives a team a better artifact to hand off: a set of named subcategories, examples, documented edge cases, a record of unresolved disagreement and a plan for review. Teams can then test whether different groups interpret the categories consistently, measure errors across relevant populations and keep a route for correcting labels after deployment.

A classifier may be the wrong tool when the category is contested or when a wrong label carries serious consequences. Sometimes the right result is a queue for human review, a narrower task or no automated ranking at all. In that sense, deliberation can help teams discover a product requirement or a reason not to build the classifier they originally imagined.

The method makes an important point for applied AI: model quality begins before model training. If a team cannot explain what its labels mean, a higher accuracy score may only make an unclear decision faster. ^{[1] [2]}