FeatureArc: Benchmarking Coding Agents on Evolving Features
Real features grow and change; most agent benches are static. Tests coding agents on extending prior code under new requirements.
The explainer · 3-minute read · 268 words
The claim
Coding agents solve the first stage of a feature far better than the whole evolving feature.
The problem
Real features grow: each extension builds on code written earlier.
Most benchmarks judge one request on a fixed codebase.
That never tests whether an agent's own earlier code survives the next change.
How it works
FeatureArc builds its tasks from real repository histories.
Each task chains 2 to 10 stages of one evolving feature.
Fail-to-pass tests check the new behavior at every stage.
Pass-to-pass tests check that the old behavior still works.
Agents work in one persistent session, with no rollback or human checks.
Figures
What changes
The benchmark measures the whole feature, not a single request.
It covers four programming languages and 73 manually reviewed tasks.
Pooled success falls from 62.7% at stage one to 28.5% on final code.
Limits
Inherited defects drive many failures, present in 72.9% of failed later stages.
After an early error, agents miss it about 95% of the time.
The study covers 73 reviewed tasks; other codebases may behave differently.
Source · arXiv wire