Before Your AI Agent Bargains for You, Make It Prove It Knows What You Want

Suppose an agent negotiates a remarkable price on a hotel room that faces a noisy street. It may have won the bargain and still failed you. Before an assistant can act in your name, it should show that it understands what “a good deal” means to you.

Anthropic’s Project Swap describes an employee book-trading experiment in which much of the outcome shortfall appeared tied to representing people’s preferences, rather than bargaining mechanics. The sample involved employees, a limited set of goods and cooperative Claude agents. That is a useful research clue, not proof that commercial shopping agents are unsafe or that every market behaves the same way. Source

My view is straightforward: products should make preference testing a required step before granting an agent meaningful authority. A user should be able to see what the system thinks matters, correct it, and set a boundary around actions. The more consequential the commitment, the stronger that check should be.

The goal comes before the deal

Negotiation is a search through possible agreements. To judge an agreement, an agent needs a target: what outcomes are acceptable, what tradeoffs are tolerable, and what conditions are non-negotiable? If that target is wrong, better bargaining may make the failure more efficient.

Imagine a hypothetical traveler comparing a direct flight with a cheaper connection. The traveler wants to arrive rested before a presentation. A price-focused agent might trade away the direct route because it has inferred that cost dominates. A preference-aware agent should ask whether the user values price, travel time, reliability or sleep, then confirm the ranking before making an offer.

The same issue appears in a fictional book purchase. One person may want a signed edition; another needs the lowest-cost readable copy. A system that treats both as “find the best book deal” has not captured either person’s goal. It has only found a phrase that sounds like a task.

Agent capability	Question it answers	What it cannot answer alone
Search	What options are available?	Which option fits this person?
Negotiation	Can terms improve?	Which tradeoffs should be accepted?
Preference model	What does the user value?	Is that model current and confirmed?
Action permission	May the agent commit?	Did the user approve this specific commitment?

Separating these stages would make mistakes easier to catch. It would also give users a vocabulary for correcting an agent without restarting an entire task.

Preference checks should be concrete

Asking “What do you prefer?” is often too abstract. People discover their priorities when they compare real tradeoffs. An agent could present two fictional examples and ask which one better fits: a cheaper option with a long connection, or a more expensive direct route. It can then explain the rule it inferred and invite correction.

The important part is that the user gets to inspect the internal summary before it is used. A short preference card might read: “You currently prioritize arriving before 8 p.m.; you accept one connection only if it saves a substantial amount; do not book without approval.” The user should be able to edit each statement, mark it temporary, or discard it.

This kind of calibration should happen at the level of the decision. A person’s preferences about a weekend trip may not carry over to a work trip. A willingness to accept a used book does not imply willingness to accept used electronics. Persistent profiles can help, but they should remain scoped and reviewable.

An agent should also distinguish stable constraints from inferred tendencies. “No overnight layovers” might be an explicit instruction. “Usually picks the cheapest fare” may be a pattern observed in past choices. Those are not equivalent. The product should not silently convert behavior into permission.

The strongest objection: friction can erase the benefit

Frequent confirmation can defeat the purpose of delegation. If a user must review dozens of options and approve every harmless step, the agent becomes another interface to manage. People may prefer to set broad goals once and let a system handle routine details.

That is a serious objection. The answer is not to ask for approval after every keystroke. It is to scale permission to consequence and uncertainty. The agent could autonomously gather options or ask questions. It might reserve a refundable selection within a user-set budget. A non-refundable purchase, a material change in travel or an action outside the stated preferences should require a preview and approval.

This is a better trade than granting blanket authority. Convenience comes from having the system do the tedious work. It does not require hiding the point at which a choice becomes binding. Users can opt into stronger autonomy after repeated successful preference checks, with a visible way to revoke it.

Treat negotiation as a service contract

Before an agent acts, it should state the objective it believes it has, the constraints it will respect, the actions it may take, and the threshold that triggers another question. After acting, it should report what it did and why. These details make the delegation legible, much like a human assistant repeating back an assignment before committing money.

The evaluation should measure whether the system preserves user priorities under pressure, not only whether it obtains favorable terms. A good deal for the wrong person is a poor outcome. Testing should include changed preferences, ambiguous requests, and cases where the agent should decline to proceed until it knows more.

Preference fidelity and permission need separate checks. The decision-model explainer describes a narrow, testable decision interface. Our agent-authority essay argues that a convincing decision still cannot expand what the user authorized the system to do.

Project Swap’s narrow experiment does not settle how consumers will use shopping agents. It does sharpen a design question that product teams should answer before making bold promises about autonomous bargains. Can the agent tell you what it thinks you want, and can you change its mind before it spends? If not, it is negotiating with a guess wearing your name.