Quality AssuranceManaged AI Agents

Quality Assurance AI Agents,
Monitored for Rigor

Nestack Agent Care helps QA teams monitor, evaluate, and optimize AI agents used for test generation, defect detection, release checks, and reporting — before small AI errors become quality or escaped-defect issues.

93failure modes
35SEV-1 failure modes
2020+baseline eval cases
24/7Agent Monitoring

Scope

Quality Assurance AI agents we build & manage

Observability

What we make observable

Every QA agent session is traced across ten layers — what we capture and the evidence we keep.

  1. Evidence we keep
    Goalconstraintsapproval requirement
  2. Evidence we keep
    Sourceversiontimestamprelevancecitation
  3. Evidence we keep
    Planned sequenceactual sequenceworkflow status
  4. Evidence we keep
    Task statusresultretryfailure reason
  5. Evidence we keep
    Tool nameversioninputoutputpermissionresult
  6. Evidence we keep
    Model/versioninput/outputtoken usagelatencycost
  7. Evidence we keep
    Evaluation typemetricthresholdresult
  8. Evidence we keep
    Guardrail targettriggeractionenforcement result
  9. Evidence we keep
    Reviewerdecisioncorrectionreason
  10. Evidence we keep
    Outcome statusbusiness resultlinked trace

Catalog

Failure modes

Filter failure modes by where they occur in the agent lifecycle—from goals and retrieval to tools, evaluations, guardrails and outcomes.

Nothing on this page matches that. Try a layer (tool, retrieval, guardrail), a code, or a severity.

0 documented↑↓ move · ↵ open · ⌘↵ filter matrix
93 documented · select a cell to filter
17
25
22
30
7
29
66
28
17
39

Core catalog

QAD-01False-pass — defective builds or units clearedSEV-1
QAD-02Coverage hallucination — reported coverage that doesn’t existSEV-2
QAD-03Requirement-trace gaps — tests not linked to requirementsSEV-2
QAD-04Defect-triage misclassification — criticals marked minorSEV-2
QAD-05Release-gate bypass under pressureSEV-1
QAD-06Stale test data vs. current specSEV-2
QAD-07Injection via bug reports and test artifactsSEV-2
QAD-08Flaky-test misdiagnosis — real defects retried to greenSEV-2
QAD-09Oracle mirroring — generated tests assert the bug, not the specSEV-2
QAD-10Defect-report fabrication — invented repro steps and environmentsSEV-3
QAD-11Environment conflation — staging results certified as production-equivalentSEV-2
QAD-12Regression-suite pruning errors — safety-relevant tests dropped as redundantSEV-1
QAD-13Inspection-sampling bias — non-random selection passing skewed batchesSEV-2

Agent & harness integrity

QAD-14Harness tampering — runner configs or graders edited to force greenSEV-1
QAD-15Phantom test execution — tests reported as run that never executedSEV-1
QAD-16Hardcoded-pass stubs — code memorizes fixtures to satisfy the visible suiteSEV-2
QAD-17Scope bleed — test-scoped agent silently edits product code to go greenSEV-2
QAD-18Self-review collusion — same-model author and judge approve their own workSEV-2
QAD-19Memory poisoning — corrupted long-term memory biases every future verdictSEV-1
QAD-20Compaction amnesia — context truncation drops mid-session failures from the final reportSEV-1
QAD-21Sycophantic severity capitulation — findings downgraded when the developer pushes backSEV-1

Generated-suite quality

QAD-22Assertion emptiness — high coverage, near-zero fault detectionSEV-1
QAD-23Happy-path bias — negative and boundary cases systematically missingSEV-2
QAD-24Over-mocking — suites validate mock wiring, not the systemSEV-2
QAD-25Flakiness injection — generated tests embed nondeterminismSEV-2
QAD-26Suite bloat — near-duplicate tests and smells inflating CI without powerSEV-3

Triage & defect-knowledge integrity

QAD-27False duplicate-merge — distinct defects closed as duplicatesSEV-2
QAD-28Stale-bot auto-close — valid reports closed for inactivitySEV-3
QAD-29Stale-index closures — regressions dismissed as “already fixed”SEV-2
QAD-30Root-cause misattribution — AI incident summaries blame the wrong subsystemSEV-2
QAD-31Repro-context evaporation — summaries drop environment and repro stepsSEV-3
QAD-32Slop flooding — hallucinated AI reports drown real triage capacitySEV-2
QAD-33Confidentiality leakage — restricted data surfaces in triage digestsSEV-1

Physical inspection & vision QA

QAD-34Scene drift — lighting, camera or material-lot shift silently degrades the modelSEV-1
QAD-35Novel-defect blindness — defect classes outside training data pass as normalSEV-1
QAD-36Silent input degradation — fogged optics still yield confident passesSEV-1
QAD-37False-reject storm — false calls flood verifiers into bulk-passingSEV-2
QAD-38Minority-class suppression — rare critical defects under-detectedSEV-2
QAD-39Label-noise propagation — annotation errors baked into the modelSEV-2
QAD-40Golden-sample overfitting — tuned to pristine references, rejects normal variationSEV-3

Fleet & oversight operations

QAD-41Cross-agent verdict divergence — QA agents disagree; orchestrator picks silentlySEV-2
QAD-42Model-update regression — provider drift shifts verdicts without revalidationSEV-2
QAD-43Verdict rubber-stamping — human sign-off stops verifying agent outputSEV-2
QAD-44Runaway loop cost — retry spirals burn CI budget without progressSEV-3

A · Test-case generation agents

QS-03Contract inferred, not read01 · RetrievalSEV-3
QS-04Assertion froze the defect02 · ReasoningSEV-1
QS-06Fixture masks the case02 · ReasoningSEV-1
QS-02Merged without a person03 · Tool / writeSEV-1
QS-05Test weakened silently03 · Tool / writeSEV-2
QS-01Approved unread04 · OutputSEV-2
QS-07Silent coverage drift05 · Change / VersionSEV-2

B · Defect-triage assistants

VE-03The report that never reached the tracker01 · RetrievalSEV-3
VE-04Merged away with the injury line02 · ReasoningSEV-1
VE-06Two regimes, one due date02 · ReasoningSEV-1
VE-02Severity gates the reportability read03 · Tool / writeSEV-1
VE-05Cluster seen, nobody told03 · Tool / writeSEV-2
VE-01Closed, the deliberation unrecorded04 · OutputSEV-2
VE-07Quiet accuracy drift05 · Change / VersionSEV-2

C · Release-gate copilots

VF-02Skipped suites read as green01 · RetrievalSEV-3
VF-04Coverage read as correctness02 · ReasoningSEV-1
VF-05Anomaly listed, never evaluated02 · ReasoningSEV-1
VF-03Override left off the record03 · Tool / writeSEV-1
VF-07Approval logged without attribution03 · Tool / writeSEV-2
VF-06Criteria aimed at the wrong object04 · OutputSEV-2
VF-01Superseded rule read as live05 · Change / VersionSEV-2

D · Inspection-support agents

QI-01Superseded drawing revision01 · RetrievalSEV-1
QI-02Angle the station cannot see01 · RetrievalSEV-1
QI-03Rare defect class missed02 · ReasoningSEV-1
QI-04Record ties to no part03 · Record / fileSEV-2
QI-05Reading without gauge capability04 · OutputSEV-2
QI-06False rejects train waving through04 · OutputSEV-3
QI-07Drift after a rebuild05 · Change / VersionSEV-1

E · Requirements-trace agents

VH-03Live view read as the record01 · RetrievalSEV-3
VH-04Link asserted on wording alone02 · ReasoningSEV-1
VH-06Orphan closed by nearest match02 · ReasoningSEV-1
VH-02Reverse direction never drawn03 · Tool / writeSEV-1
VH-05Derived requirement given a parent03 · Tool / writeSEV-2
VH-01Signed over links nobody read04 · OutputSEV-2
VH-07A change leaves the links behind05 · Change / VersionSEV-2

F · Visual-inspection models (AOI / vision)

VI-03Truth set by the same eyes01 · RetrievalSEV-3
VI-04Accuracy read off a thin class02 · ReasoningSEV-1
VI-06Reserved section worked as live02 · ReasoningSEV-1
VI-02Offline score not seen on the line03 · Tool / writeSEV-1
VI-05Unlabelled defect class passes03 · Tool / writeSEV-2
VI-01A light moved, nobody looked04 · OutputSEV-2
VI-07Silent revalidation trigger05 · Change / VersionSEV-2

G · Incident & RCA summarizers

RW-03Stale evidence read01 · RetrievalSEV-3
RW-04Cause asserted, not evidenced02 · ReasoningSEV-1
RW-06Retired review rule read as live02 · ReasoningSEV-1
RW-02Thin review passed forward03 · Tool / writeSEV-1
RW-05Review bound to wrong incident03 · Tool / writeSEV-2
RW-01Signed, evidence unrecorded04 · OutputSEV-2
RW-07Silent conclusion drift05 · Change / VersionSEV-2

Guardrails

Critical guardrails for QA agents

Ten controls that hold regardless of prompt, plan or pressure. Open one to see what it protects, what trips it, what the agent is forced to do, who may release it, and what is written to the record.

GR-01No release certification without independent gate approvalOverride defined
  1. Target

    Release sign-off endpoints, build promotion pipelines and quality-gate records feeding production deploys

  2. Trigger

    Any attempt to mark a build releasable while gate criteria are unmet or unevaluated

  3. Action — enforced

    Platform blocks the promotion and holds the build in quarantine; agent may assemble evidence and request human gate review.

  4. Human override

    QA release manager via signed gate-waiver ticket recorded against the build id

Logged evidencebuild id · gate criteria snapshot · unmet-criteria list · approver identity · waiver ticket id · UTC timestamp

GR-02No instruction embedded in bug reports ever executedNo override
  1. Target

    Inbound bug reports, repro attachments, CI logs, fixtures and test artifacts parsed by QA agents

  2. Trigger

    Imperative or tool-directing text detected inside any submitted report, log, fixture or attachment

  3. Action — enforced

    Platform strips the content to inert data and refuses execution; agent may quote it verbatim and continue triage.

  4. Human override

    None — cannot be overridden in session

Logged evidenceartifact hash · detected instruction span · source reporter id · sanitized copy hash · detection timestamp

GR-03No restricted data in triage digests or reportsNo override
  1. Target

    Triage digests, defect summaries and cross-team reports assembled from restricted repositories and customer data

  2. Trigger

    Classified-restricted records, secrets or customer identifiers detected in any outbound digest draft

  3. Action — enforced

    Platform blocks distribution and redacts the digest; agent may cite record ids and route readers to source systems.

  4. Human override

    None — cannot be overridden in session

Logged evidencedigest id · classification labels hit · redaction diff hash · intended recipients · block timestamp

GR-04No agent edits to runner configs or gradersOverride defined
  1. Target

    CI runner configurations, grader definitions, scoring scripts and harness manifests across all pipelines

  2. Trigger

    Any write targeting harness, runner or grader files from a test-generation or triage session

  3. Action — enforced

    Platform denies the write and freezes the pipeline run; agent may file a harness-change request for human review.

  4. Human override

    Build-infrastructure lead via signed pull request reviewed outside the agent session

Logged evidenceattempted diff hash · target file path · pipeline run id · agent identity · denial timestamp

GR-05No pass verdict without executed-run evidenceOverride defined
  1. Target

    Pass/fail verdicts, coverage figures and final reports published to dashboards and release records

  2. Trigger

    Reported result lacking a matching runner ledger entry, or coverage exceeding instrumented measurement

  3. Action — enforced

    Platform withholds the verdict and flags the report unverified; agent may re-run tests and attach ledger references.

  4. Human override

    QA lead via evidence-reconciliation review attaching the missing runner ledger extract

Logged evidenceverdict id · runner ledger hash · coverage instrument version · mismatch list · reconciliation timestamp

GR-06No pruning of safety-relevant regression testsOverride defined
  1. Target

    Regression suites, mandatory-test registries and their mapping to safety and compliance requirements

  2. Trigger

    Deletion or skip-tagging of any test linked to a safety or regulatory requirement

  3. Action — enforced

    Platform blocks the removal and keeps the test mandatory; agent may propose retirement with impact analysis for review.

  4. Human override

    Head of QA via documented retirement review with traceability sign-off

Logged evidencetest id · linked requirement ids · proposed-change diff · requester identity · decision timestamp

GR-07No product-code edits by test-scoped agentsOverride defined
  1. Target

    Product source repositories reachable from test-generation, triage and verification agent credentials

  2. Trigger

    Any commit or write outside declared test directories from a QA-scoped session

  3. Action — enforced

    Platform denies the commit and reverts staged changes; agent may raise a defect describing the failing behavior instead.

  4. Human override

    Engineering manager via scope-extension grant issued through the change-control system

Logged evidenceblocked diff hash · repository path · session scope manifest · agent identity · denial timestamp

GR-08No downgrade of critical defect severity on pushbackOverride defined
  1. Target

    Severity fields on open defects classified critical or safety-relevant in the tracker

  2. Trigger

    Severity reduction requested in-conversation without new reproduction evidence contradicting the original classification

  3. Action — enforced

    Platform holds the original severity and escalates to a human triager; agent may record the dissenting rationale.

  4. Human override

    Triage board chair via two-person severity review logged on the defect

Logged evidencedefect id · original and requested severity · requester identity · escalation ticket · decision timestamp

GR-09No staging results certified as production-equivalentOverride defined
  1. Target

    Certification records asserting environment equivalence for builds, models and inspection lines

  2. Trigger

    Verdict citing a non-production environment while claiming production validity or omitting environment labels

  3. Action — enforced

    Platform blocks the certification and labels results by environment; agent may schedule a production-parity run.

  4. Human override

    QA release manager via documented environment-parity assessment attached to the record

Logged evidencerun id · environment fingerprint · config diff vs production · certifier identity · block timestamp

GR-10No retry-to-green on failing testsOverride defined
  1. Target

    CI retry policies and re-run controls across build, integration and inspection pipelines

  2. Trigger

    Repeated re-execution of a failing test until a passing sample is obtained

  3. Action — enforced

    Platform caps retries and records every attempt as failed; agent may open a flakiness investigation with full run history.

  4. Human override

    QA lead via quarantine-list entry that tracks the test as known-flaky

Logged evidencetest id · attempt count · per-attempt outcomes · seed and config hashes · cap timestamp

Oversight

Human review — triggers, decisions and evidence

When a defined risk trigger fires, the affected action is routed to a named reviewer. Every decision is recorded with its correction, escalation and final outcome for full traceability.

  • ConfidenceLow-confidence defect verdict
  • Financial impactScrap or rework decision
  • Identity / change riskTest-harness or config change
  • Irreversible actionRelease-gate sign-off
  • Policy riskCoverage or trace-gap flag
  • Safety controlGuardrail override
  • Quality failureFailed critical evaluation
Humanreviewnamed reviewer
  • Revieweridentity + role
  • Decisionapprove / reject / amend
  • Correctionwhat changed
  • Escalationwho, why and severity
  • Final outcomereleased / blocked / returned for rework

7 triggers · any one halts the agent

1 record · 5 fields, every time

Compliance

Regulatory mapping

  • 02 Retrieval
  • 03 Workflow
  • 04 Task
  • 05 Tool
  • 06 LLM
  • 07 Evaluation
  • 08 Guardrail
  • 09 Human review

Release integrity

Gate bypasses under deadline pressure defeat the department’s purpose.

Lifecycle layer:03 Workflow · 08 Guardrail · 09 Human review

  • QAD-05

Evaluations

Baseline evaluation suite — in detail

Baseline evaluations are completed during onboarding and repeated based on the selected plan. Agents that fail critical checks remain restricted until they pass re-testing.

Detailed case sets
32
Failure modes covered
44
Retired & rotated / quarter
10%
Audit-ready scorecard
Monthly

Output evaluation4 suites · 260 cases

100 casesFalse-pass controlcatches QAD-01
80 casesTriage recallcatches QAD-04
40 casesRepro-validity setcatches QAD-10
40 casesPrune-safety setcatches QAD-12

Use your work email to unlock the full grid. We use it to send the requested Agent Care material and qualify business inquiries. We do not sell or share your email.

  1. Department lead review

    For applicable high-risk agents, the client’s designated department leader reviews the evaluation criteria and pass thresholds before baseline approval.

  2. Test-case rotation

    Evaluation cases are refreshed regularly to reduce memorisation and maintain reliable performance measurement.

  3. Scorecard integration

    Scorecards track results against the approved baseline and flag material declines for review and escalation.

  4. Department-specific extensions

    Where included in scope, evaluations may be expanded using approved workflows, tools, templates, policies, and incident history.

Monitoring

Change-aware monitoring

When agent performance changes, Nestack correlates the shift with changes to the agent, prompt, model, tools, knowledge base, guardrails and evaluation suite.

Version changes by layer
01AgentAgent changed in week 3
02PromptPrompt changed in week 5Prompt changed in week 10
03ModelModel changed in week 4
04ToolTool changed in week 6Tool changed in week 12
05Knowledge-baseKnowledge-base changed in week 10
06GuardrailGuardrail changed in week 3
07Eval-suiteEval-suite changed in week 8Eval-suite changed in week 10
Defect-detection rate92–100%
  • Week 1 · 98.3%
  • Week 2 · 98.2%
  • Week 3 · 98.4%
  • Week 4 · 98.3%
  • Week 5 · 98.5%
  • Week 6 · 98.3%
  • Week 7 · 98.4%
  • Week 8 · 94.7%
  • Week 9 · 94.5%
  • Week 10 · 98.3%
  • Week 11 · 98.4%
  • Week 12 · 98.5%

Week readouthover or select

Week 8of 12

  • 07Eval-suiteevals 2026.07

94.7%Defect-detection rate

  • 7 layers stamped on every run · 12-week window
  • Catches QAD-12 · regression-suite pruning errors

Something missing?

Don’t see your agent’s issue here?

Every AI environment is different. Share what you’re seeing, and we’ll review the behaviour, assess the risk and recommend the evaluations or controls that may help.

Report an agent issue

Tell us what you’re seeing

The more specific, the faster we can reproduce it. Playbook: Quality Assurance

We reply within one business day.

See pricing

No commitment. Even if you never become a client, we’ll tell you what we think is happening.

Process

Universal incident runbook

Severity is assigned based on business impact, customer harm, data exposure, operational disruption and overall scope.

  • SEV-1 Critical
  • SEV-2 Major
  • SEV-3 Moderate
  • SEV-4 Minor
STEP 01

Automated monitoring or human review identifies unusual behaviour. Alerts are recorded and routed according to severity.

Cost control

Keep quality assurance AI agent costs under control

Token spend is monitored, optimised and reported as part of Agent Care — and savings never come at the expense of quality, because every change is verified against your evaluation baseline.

We review token spend by agent, workflow, model, and session so you can understand where AI costs are coming from.

Running quality assurance AI agents in production?

Get a free assessment of one agent. We’ll review its behaviour, run a baseline evaluation and highlight potential risks and performance gaps.

Loading open times…

Open times

Select an available date above to view open 45-minute times from our calendar.