Why AI’s Hardest Engineering Often Starts After the Demo

A prototype can answer a prompt in a meeting room. A production workflow has to survive real data, old systems, edge cases, permission rules, user habits, and the moment an answer is wrong. OpenAI’s Oct. 22 account of its forward-deployed engineering teams describes the labor between those two states—and why that labor matters more as models become easier to call.

The work is a loop, not a handoff

OpenAI describes forward-deployed engineers (FDEs) working alongside customer teams to identify a concrete outcome, understand the business process, connect systems, and test whether an AI workflow delivers useful results. One example is personalized support for farmers using John Deere’s See & Spray technology: the work involves more than answering a question; farmers need configuration guidance, seasonal planning, and help interpreting equipment performance. ^[1]

The company also describes a semiconductor verification project where engineers worked with customer specialists, created evaluations for tasks such as test generation and root-cause analysis, and iterated toward production use. The reported metrics—including time saved and performance improvements—are OpenAI’s own account and should be read as company-reported outcomes, not independent benchmarks. ^[1]

The pattern is familiar in serious software delivery. Someone has to map the existing process, discover where the data comes from, decide which actions the system may take, and build a test set that reflects what users will actually encounter. The AI component is one part of a system that also includes identity, permissions, interfaces, observability, support, and a route back to a human.

Why domain knowledge matters even when the model is capable

In agriculture, an answer about weeds can be plausible and still be wrong for a crop stage, weather condition, equipment configuration, or product label. John Deere’s product description explains that its See & Spray system uses cameras and onboard processing to identify weeds and activate selected nozzles. The company’s stated herbicide-savings figures are tied to its own product configurations and trial conditions, and Deere explicitly says results vary. ^[2]

That context is the difference between a system that sounds informed and one that can help safely. The FDE may not arrive as a domain specialist, but the role needs enough curiosity and access to learn from the people doing the work. Domain experts, meanwhile, can explain which distinctions matter and what a harmful failure would look like.

This is not unique to frontier AI companies. A retailer integrating an assistant into returns still has to encode the actual policy, exception paths, and authority limits. A hospital workflow must preserve clinical review. A software team using code agents still needs tests, maintainers, and a release process. The transferable skill is not “prompting”; it is translating an operational goal into a system with measurable behavior.

Evaluations are the bridge to production

OpenAI’s account highlights building evaluations with customer engineers. That is a practical point often lost in launch demos. If a workflow is supposed to classify incoming incidents, the team needs examples of correct and incorrect routing. If it drafts a repair plan, reviewers need criteria for safety, completeness, and escalation. Model quality should be tested on the same kind of inputs and constraints that will appear in production.

OpenAI’s evaluation guidance recommends defining the task, collecting representative data, and running repeatable evaluations as prompts or models change. ^[3] A useful evaluation does not need to be elaborate to start. It does need clear labels, examples that include difficult cases, and an owner who can explain what counts as acceptable performance.

Evaluation should continue after launch. Real workflows drift: products change, policies are updated, source systems acquire new fields, and users discover unexpected paths. A practical system records enough context to investigate failures, samples successful cases for hidden errors, and provides a safe fallback. An average score can conceal the one category where a system routinely makes a costly mistake.

From custom glue to reusable building blocks

OpenAI says some customer projects later moved from custom interfaces and integrations toward shared Codex primitives, skills, and plugins, reducing custom code in those projects. That is an important second phase: first learn what works in the real environment, then standardize the repeatable parts. The company’s percentage reductions and rollout timings are claims from its own article, not evidence that every enterprise will achieve the same result. ^[1]

The sequence matters. Abstracting too early can bake uncertain assumptions into a platform. Staying custom forever creates maintenance debt. Teams can use a small pilot to discover the stable interface, then extract common connectors, permission checks, evaluations, and monitoring once they have evidence that multiple groups share the same need.

A better question for buyers

Instead of asking a vendor whether its model can perform a task, ask who will help define success, what existing systems must be integrated, how permissions will work, what evaluation set will be used, and who owns the workflow after launch. Ask what is included in support and what remains your team’s responsibility.

FDEs do not remove the need for customer ownership. They can shorten the gap between capability and useful software, but the customer still supplies domain knowledge, policy decisions, data stewardship, and operational accountability. The strongest AI projects treat deployment as product engineering: observe the work, make a narrow intervention, evaluate it, and scale only what holds up.

Related reading: how to evaluate coding models on low-resource languages and AI agents need least-privilege tool access.