JetBrains Built Its Own AI Coding Benchmark Because It Doesn't Trust Anyone Else's

Every AI coding agent on the market ships with a chart proving it’s the best. Cursor cites one benchmark, Codex cites another, and Claude Code’s own release notes lean on a third — and none of them run a single Kotlin task. JetBrains, custodian of the language and the IDE most of its ecosystem lives in, decided that gap was no longer tolerable, and built the missing scoreboard itself.

What exactly did JetBrains release on July 8?

JetBrains shipped three things at once: a 105-task dataset drawn from active open-source Kotlin repositories, an open test harness to run agents against it, and a public leaderboard at kotlinlang.org/benchmark tracking the results. Each task hands an agent a real issue description and an existing codebase, and the agent has to navigate the project, write a patch, and get it merged in spirit — solutions run inside containerized environments, and a task only counts as “resolved” if the patch passes the required test suite. There’s no partial credit and no self-reported score; the harness decides.

The first leaderboard run tested three agent-model pairings. Claude Code with Opus 4.7 xhigh resolved 90 of the 105 tasks — 85.71% — for the top spot. JetBrains’ own coding agent, Junie, paired with Opus 4.7 max, resolved 81.9%, exactly matching OpenAI’s Codex running GPT-5.5 xhigh. JetBrains is upfront that this snapshot is incomplete: the results “reflect the first public iteration of the benchmark and do not yet include the most recent model releases,” and a second iteration is already underway.

Why not just trust SWE-bench or the vendors’ own numbers?

Because neither one was built to answer the question JetBrains actually needed answered. SWE-bench, the methodology the Kotlin Benchmark borrows its structure from, was never a Kotlin benchmark — it’s the reference format for “does an agent resolve real GitHub issues,” and JetBrains had to build its own Kotlin-specific task set from scratch to apply it. And vendor-reported capability claims are, definitionally, marketing: every agent vendor grades its own homework on suites it selects. JetBrains says it plainly: the goal is to give teams “a shared frame of reference for comparing setups on Kotlin tasks instead of relying only on vendor claims.”

That framing cuts in an uncomfortable direction for JetBrains too. Junie is JetBrains’ own commercial coding agent, built into the same IDE ecosystem the company sells subscriptions for — and on the benchmark JetBrains designed, built, and controls, Junie didn’t win. It tied a competitor’s agent for second, 3.81 points behind Claude Code. Publishing that result on your own leaderboard, on launch day, is either an unforced marketing error or a genuine bet that a credible scoreboard is worth more than a favorable one. JetBrains’ repeated emphasis on openness — datasets and harness both public on GitHub, built on the open-source Multi-SWE-bench infrastructure rather than a closed internal pipeline — points toward the latter.

How rigorous is the methodology, actually?

Rigorous enough that JetBrains is treating it as infrastructure, not a one-off marketing asset. The benchmark runs on Multi-SWE-bench, an open-source evaluation framework, rather than a bespoke internal grader — meaning outside teams can inspect exactly how a task is scored, not just trust a published percentage. Verification happens inside containerized environments, and the pass bar is binary: the generated patch either satisfies the task’s required tests or it doesn’t. That’s a meaningfully higher bar than the kind of self-reported “our agent handled X% of our internal eval set” claims the industry has trained developers to discount on sight.

Evaluation asset	What it measures	Layer
Kotlin_HumanEval	Whether a model can write correct Kotlin for a given prompt	Model / syntax
Kotlin_QA	Whether a model understands Kotlin language concepts	Model / knowledge
Kotlin Benchmark	Whether an agent can resolve a real issue in a real repo, patch verified by tests	Agent / task completion

JetBrains is also explicit about the current limits. This is a first iteration — 105 tasks, three agent-model pairings, no coverage yet of Android-specific or Kotlin Multiplatform work, and no metrics beyond pass/fail. Those aren’t small omissions for a language whose single biggest real-world use case is Android development, and JetBrains says as much in laying out what’s next.

What does JetBrains say comes next?

Three concrete expansions, all named directly in the release. First, broader ecosystem coverage: more tasks drawn from Android and Kotlin Multiplatform codebases, plus a wider spread of difficulty levels than the current set. Second, more evaluation metrics: passing tests is “a useful correctness signal, but it is only one part of agent evaluation,” and future runs will also score cost, performance, maintainability, and code quality — dimensions where an agent can pass every test and still hand a team a mess. Third, more agents and model setups, including additional commercial agents and open-weight models beyond the three that ran in this first public leaderboard.

What does this mean for developers picking a Kotlin coding agent?

It means the marketing-claims era of agent selection just got a little harder to hide in, at least for Kotlin teams. A language steward publishing an open, test-verified leaderboard — and leaving its own product in second place on launch day — sets a bar that’s awkward for every vendor to ignore. It also lands amid a wider credibility reckoning for AI benchmarks generally: on the same July 2026 stretch, a gamed safety benchmark forced a hard look at how much a published score can be trusted, and a rival vendor’s “Opus-class” capability claim didn’t hold up once independent rankings came in. The Kotlin Benchmark is JetBrains’ bet that the fix isn’t a better slogan — it’s a public, inspectable scoreboard nobody, including JetBrains, gets to grade themselves on.

That bet only pays off if JetBrains keeps the leaderboard current and keeps publishing results that don’t flatter itself. One honest launch-day number is a good start; a maintained, expanding benchmark across two or more iterations is what would actually earn the trust the whole project is implicitly asking for.