Main site

Engineering AI agent · Test writing

Test-Writing AI Agent

Trace each generated assertion back to the requirement it came from, mark what was read out of the code instead, and hold the suite for the quality engineer who signs the report.

Typical delivery
4–6 weeks
Deployment
Your stack
Named signer
Trace first
After launch
Agent Care

What this agent does

Writes the test, not the verdict on the build

In

  1. 01

    A suite is read in, and the requirements it was meant to trace to are read in beside it.

  2. 02

    A test is generated, and the record says whether it came from a requirement or from the code.

Reason

  1. 03

    An assertion is written, and its expected value is marked as observed behaviour or as specified.

  2. 04

    A fixture is built from the interface contract rather than from the code that happens to call it.

  3. 05

    An existing human-written test is improved, and it ships only if a mechanical filter shows it better.

Decide

  1. 06

    A module carries an open defect, and coverage over it is reported as a measurement, not as evidence.

  2. 07

    An assertion is changed, and the prior expected value, the new one and the reason are written down.

Out

  1. 08

    A test would be deleted or weakened, and the change is held for a named quality engineer instead.

  2. 09

    Execute write actions only inside the approval boundaries agreed during implementation.

Product statement

Generation, tracing and the assertion record belong to the agent. Approval belongs to a named quality engineer, who signs the test report and answers for it.

Example workflow

One test, requirement to report

  1. 1Suite and source receivedRepository, test runner, coverage report or requirement tracker
  2. 2Basis assembledThe requirement, the interface contract, the existing tests and the defects still open
  3. 3Test draftedThe assertion, the fixture, the trace it claims and confidence
  4. 4Controls appliedTrace checks, derivation marking, fixture-source rules and confidence threshold

No human action required

Stages 1 to 4 run unaided, and nothing is merged at any of them — the agent is drafting tests, and the quality lane opens at the confidence gate.

5DecisionBranches at the confidence threshold
High confidence

Goes to the quality engineer to approve.

Low confidence

Adds a test-engineering read first.

Quality approval

The test is held with its trace, its derivation mark and the confidence.

Approve · Amend · Send to test engineering

Approved — merged into the suite

  1. 6Repository and runner updatedOnly where write access and review policy allow it
  2. 7Outcome evaluatedAmendment rate, trace outcomes, defects caught and corrections after release

Amendments

Each quality-engineer amendment is counted in the evaluation.

What should not run autonomously

Human approval stays in control

Outside the boundary — human approval required

8 items

  • Signing a test report into a regulatory submission.
    • Deleting or disabling a failing test.
    • Deciding that observed behaviour is correct behaviour.
    • Declaring a coverage objective satisfied.

  • Automation boundary

    Agent acts unaided

    • ✓Draft tests from the requirement and the contract.
    • ✓Mark which assertions were derived from the code itself.
    • ✓Record what the assertion expected before the agent changed it.
    • ✓Report coverage as a measurement and hold the suite for review.

    Nothing reaches the suite except by a named engineer, inside the agreed boundaries.

    • Approving a suite as adequate for release.
    • Writing or amending the requirement itself.
    • Judging a mutant equivalent and dismissing it.
  • Changes to authoring policy or approval thresholds.

Example output

One generated test, annotated

For most teams this is a build artefact; where a submission carries a printed name it is a record with a signer. Below is one test as the agent leaves it.

Test-generation output · single assertion

Illustrative example
Unit under test
Dose calculator
Assertion written
Asserts the value the current build returns
Expected value
Observed, not specified
Derived from
The source code
Confidence
88%
Held for
The named quality engineer, by name

As received

Taken from the unit and the run that exercised it — the requirement it should trace to is absent.

What the record holds

  • Interface contract
  • Observed return value
  • Open defect record

Why this is flagged

It records what the code does, not what the requirement demanded.

What the score decides

Below the configured threshold a test picks up a test-engineering read before approval.

Action

ApproveAmendSend to test engineering

Value

Where AI adds value

The same four claims, placed at the point in the workflow where each one applies.

Where the value lands

Value 01 – 04

Each generated test

From the code and the requirement

03Generation

Draft from the requirement

Work from the requirement and the interface contract, and mark what could only have been read out of the implementation.

01Approved path

The fixture became the spec

Pick a permissive fixture because it is convenient, and the run that follows certifies the assumption instead of the code.

02Human review

What was looked for, and not found

No statute, no regulation and no guidance found in this work sets a coverage percentage, requires a test to be meaningful, or forbids deleting a failing one. The gap between a test failing and a record that it failed is unlegislated, and that is where the agent sits.

04

Build an evidence trail

The test, the requirement it traces to and the engineer who approved it stay together.

Integrations

Typical integrations

Five system groups connect to the same agent. Which of them are in scope is decided in discovery.

  • Repositories and forges

    • GitHub
    • GitLab
    • Bitbucket
    • Gerrit
    • Azure DevOps Repos

  • Test runners

    • JUnit
    • pytest
    • Jest
    • NUnit
    • Go test
    • RSpec
    • PHPUnit

  • Coverage and mutation

    • JaCoCo
    • Cobertura
    • coverage.py
    • PIT
    • Stryker Mutator
    • mutmut

Agent

Test writing and maintenance

  • Reads the requirement
  • Drafts the test
  • Holds for approval

  • Requirements and issues

    • Jira
    • Polarion
    • Azure Boards
    • Jama Connect
    • codeBeamer ALM

  • Observability & evaluation

    • OpenTelemetry
    • Langfuse
    • Supported monitoring/evaluation sources

Integration availability depends on the client's existing systems and API access.

Agent controls

Six checks between the model and the suite

Six gauges on one line, the last the finest. What reads true is set out in the map below.

L6 · Outermost — last line of defence

Inward → L1 · closest to the model

L6Rollback / safe modeNarrow the agent to reporting on existing tests when evaluation or production signals degrade.Roll back
L5Version monitoringTrack model, prompt, generation-rule and fixture-source changes.Track
L4TraceabilityRecord the requirement read, the test drafted, the derivation mark and the approval time.Record
L3Quality approvalHold tests for the named quality engineer; the hold governs merge, not whether the assertion is right.Gate
L2Authoring guardrailsTest each draft against its trace, its fixture source and the assertion-change record; a failure returns it.Restrict
L1Confidence thresholdsRoute low-confidence tests to a test-engineering read before the quality engineer sees them.Require review
Model coreTest drafted — the assertion, the fixture, its trace and confidence
L1 – L2
Test whether a draft may stand
L3
Leaves the merge to a named engineer
L4 – L5
Keep the test and the requirement behind it
L6
Writes no assertion at all when signals degrade

How Nestack evaluates it

Evaluate the whole generation path — not only the suite that comes out.

Coverage runs the whole depth of the workflow, and every layer is cut by slice.

Surface — the suite a release is measured against

  1. E1Final-output evaluationDid the test trace to a requirement, or only back to the code?
  2. E2Step-level evaluationDid the agent read the right requirement, contract and open defects?
  3. E3Tool evaluationDid it read and write the correct module and the correct suite?
  4. E4Confidence calibrationDo low-confidence tests actually attract more engineer amendments?
  5. E5Slice evaluationHow does performance change across specific code classes?
  6. E6Business outcomeHow many tests needed an amendment before the engineer approved?

Floor — the defect the suite was meant to catch

Failure modes

Where each failure originates in the agent

Seven failure modes, each set where the test suite first shows it.

Agent lifecycle

Direction of processing →

  1. 01 · Retrieval

    1 mode

    • QS-03

      Contract inferred, not read

      The interface is taken from the calling code.

    Stage gathers

    The spec, the contract, the suite and the open defects

  2. 02 · Reasoning

    2 modes

    • QS-04

      Assertion froze the defect

      The expected value is the value the bug returns.

    • QS-06

      Fixture masks the case

      A permissive fixture matches whatever it is given.

    Stage proposes

    The assertion, the fixture, its trace and confidence

  3. 03 · Tool / write

    2 modes

    • QS-02

      Merged without a person

      Agent completes a merge that needed an engineer.

    • QS-05

      Test weakened silently

      A failing assertion is rewritten until it passes.

    Stage writes

    Only where write access and approval policy allow it

  4. 04 · Output

    1 mode

    • QS-01

      Approved unread

      The record shows approval but nobody read the suite.

    Stage returns

    The suite the next release is measured against

  5. 05 · Change / Version

    1 mode

    • QS-07

      Silent coverage drift

      A model or rule change widens what the agent will assert.

    Stage tracks

    Model, prompt, generation rules and fixture sources

  • Sev-1 · a suite merged outside the boundary
  • Sev-2 · a defect certified by a passing test
  • Sev-3 · trace degrades, test routes to review
See the complete Engineering failure-mode catalogue

Affected slices

Untraced code absorbs the amendments

A suite-level oracle figure can read clean while a couple of code classes carry nearly all of the amendments. Nestack reports the engineer-amendment rate by code class, not only in total.

Slice performance — reported separately, not only in aggregate

Illustrative example
SliceFailure rateLiftLift vs. thresholdStatus
Code with an open defect13.5%3.7×Review
Untraced legacy modules9.6%2.6×Review
Concurrency and timing paths6.0%1.6×Watch
Stable well-specified modules2.7%0.7×Normal

Bar: engineer-amendment-rate lift vs. stable-module baseline · scale 0–4.0× · tick marks the 2.0× review threshold

2 of 4 slices over threshold

Evidence-linked improvement

What a wildcard costs later

The loop shuts when the assertion that froze a defect is a regression case. That suite is what the next suite generated is measured against.

Improvement cycle · five stages

Switchback — the path turns at Improve and returns at Learn

  1. 01

    Detect

    Amendment rate rises in one code class.

  2. 02

    Diagnose

    The wildcard in the twenty-first field, written because it was convenient, is worked back until one cause is left.

  3. 03

    Improve

    Every change goes out numbered, with the tests that caused it attached.

  4. 04

    Verify

    Nothing releases while one touched assertion case is still red.

  5. 05

    Learn

    The case stays on, and the authoring rules are rewritten alongside it.

  6. Learn → Detect

    The return edge. The next suite generated is measured against a suite one case longer.

Typical build scope

Twelve workstreams across six weeks

The build scope read against the delivery timeline. Week structure follows the six-week plan — discovery, sources, test generation, evaluation, integration, then production validation and handover.

Workstream

Week 1Week 2Week 3Week 4Week 5Week 6

  1. 01Test-workflow discovery and automation-boundary work.Week 1
  2. 02Runner, coverage and forge assessment.Week 1, Week 2
  3. 03Requirement-tracing and fixture-source rule mapping.Week 1, Week 2
  4. 04Suite and runner ingestion.Week 2
  5. 05Generation logic and trace binding.Week 2, Week 3
  6. 06Confidence scoring and flag routing.Week 3
  7. 07Quality approval workflow.Week 3, Week 4
  8. 08Repository and runner integration.Week 5
  9. 09Assertion and traceability cases.Week 4, Week 5
  10. 10Guardrails and generation controls.Week 4
  11. 11Assertion-trail instrumentation.Week 5, Week 6
  12. 12Deployment, documentation and Agent Care handover.Week 6

12 workstreams · 6 weeks · bar shows the weeks a workstream is active — several run in parallel

Final scope and sequence confirmed in discovery

Engagement tiers

What each tier includes

Rows are the capabilities named in each tier's scope. Higher tiers include everything below them.

Capability✓ in scope · — not at this tierPilotOne repository, one suiteProductionProduction repositoriesAdvancedMultiple repositories / teams
Introduced at Pilot
Generation to your requirements✓✓✓
Named quality approval✓✓✓
Suite-inventory baseline✓✓✓
Introduced at Production
Reporting by assertion class—✓✓
Approval workflow in your systems—✓✓
Approved write-back—✓✓
Runner-and-repository integration—✓✓
Introduced at Advanced
Multi-language generation rules——✓
Multi-stage quality approvals——✓
Large suite volume——✓
Multi-suite generation controls——✓
Build priceFrom $5,000From $8,000Custom quote
Final build price
Confirmed after discovery based on integrations, workflow complexity, suite volume, approval controls and deployment requirements.
Separate from build
Build pricing is separate from recurring Agent Care, which covers managed monitoring, evaluations, incidents and verified improvements after launch.

What we need from you

What you bring, and what we build with it

Each input maps to a piece of build scope and a week in the delivery timeline.

  1. 01Your suites and the requirements they trace to

    Suite ingestion and trace mapping

    Week 1

  2. 02Representative modules and their contracts

    Generation baseline, contract binding and the trace matrix

    Week 2

  3. 03Your assertion and fixture policy

    Requirement-tracing and fixture-source rule mapping

    Week 1

  4. 04Access to relevant APIs, feeds or exports

    Runner, coverage and repository assessment, then integration setup

    Week 2

  5. 05Assertions you would not want submitted

    Traceability cases and the evaluation round

    Week 4

  6. 06What no test may certify

    Confidence scoring, flag routing, guardrails and approval controls

    Week 3

  7. 07A named quality engineer to approve tests

    Quality approval workflow, then pilot and production validation

    Weeks 5–6

Nothing else is required

Deployment, documentation and Agent Care handover are ours.

Delivery timeline

Four phases across six weeks

Widths follow the work, not the grid, which is why one band is drawn beneath another in the fifth week.

Phase

W1

W2

W3

W4

W5

W6

  1. DiscoveryW1
  2. BuildW2 – W3
  3. EvaluateW4 – W5
  4. Pilot & LaunchW5 – W6

Week focus

W1Test workflow discovery, traceability mapping and the automation boundary

W2Source integration and the generation baseline

W3Generation workflow, confidence logic and approval controls

W4Evaluation suite, traceability cases and failure-mode testing

W5Repository integration, pilot suites and targeted corrections

W6One verification cycle run under the quality owner, then Agent Care handover

Reading the band
Each bar covers only the weeks its own work is named for, and week five is shared by design.
At the end of W6
Once the assertion record validates, Agent Care assumes the agent.
Duration
Six-week plan shown · typical delivery 4–6 weeks depending on scope confirmed in discovery.

Next step · Engineering AI agent

Build a test-writing agent around the requirement your last suite never traced to.

Show us one suite, the requirements behind it and who signs the test report. If your suite was written from the code, then it can tell you the behaviour changed and it cannot tell you the behaviour was ever right.

Typical delivery
4–6 weeks
Deployment
Your repository
Named signer
Trace first
After launch
Agent Care available

Nestack Agents · Test writing

AGT-ENG-02 · Agent Care available after launch

Build a test-writing agent

Four questions so the call starts from your actual setup — then pick a time that suits you.

About you
Your suites

No obligation. We’ll map the workflow, name the automation boundary and tell you if an agent is the wrong answer.