Quality AssuranceManaged AI Agents
Quality Assurance AI Agents,
Monitored for Rigor
Nestack Agent Care helps QA teams monitor, evaluate, and optimize AI agents used for test generation, defect detection, release checks, and reporting — before small AI errors become quality or escaped-defect issues.
Scope
Quality Assurance AI agents we build & manage
Test & release3
Inspection2
Traceability & RCA2
Observability
What we make observable
Every QA agent session is traced across ten layers — what we capture and the evidence we keep.
- Evidence we keepGoalconstraintsapproval requirement
- Evidence we keepSourceversiontimestamprelevancecitation
- Evidence we keepPlanned sequenceactual sequenceworkflow status
- Evidence we keepTask statusresultretryfailure reason
- Evidence we keepTool nameversioninputoutputpermissionresult
- Evidence we keepModel/versioninput/outputtoken usagelatencycost
- Evidence we keepEvaluation typemetricthresholdresult
- Evidence we keepGuardrail targettriggeractionenforcement result
- Evidence we keepReviewerdecisioncorrectionreason
- Evidence we keepOutcome statusbusiness resultlinked trace
Catalog
Failure modes
Filter failure modes by where they occur in the agent lifecycle—from goals and retrieval to tools, evaluations, guardrails and outcomes.
Core catalog
QAD-01False-pass — defective builds or units clearedSEV-1
QAD-02Coverage hallucination — reported coverage that doesn’t existSEV-2
QAD-03Requirement-trace gaps — tests not linked to requirementsSEV-2
QAD-04Defect-triage misclassification — criticals marked minorSEV-2
QAD-05Release-gate bypass under pressureSEV-1
QAD-06Stale test data vs. current specSEV-2
QAD-07Injection via bug reports and test artifactsSEV-2
QAD-08Flaky-test misdiagnosis — real defects retried to greenSEV-2
QAD-09Oracle mirroring — generated tests assert the bug, not the specSEV-2
QAD-10Defect-report fabrication — invented repro steps and environmentsSEV-3
QAD-11Environment conflation — staging results certified as production-equivalentSEV-2
QAD-12Regression-suite pruning errors — safety-relevant tests dropped as redundantSEV-1
QAD-13Inspection-sampling bias — non-random selection passing skewed batchesSEV-2
Agent & harness integrity
QAD-14Harness tampering — runner configs or graders edited to force greenSEV-1
QAD-15Phantom test execution — tests reported as run that never executedSEV-1
QAD-16Hardcoded-pass stubs — code memorizes fixtures to satisfy the visible suiteSEV-2
QAD-17Scope bleed — test-scoped agent silently edits product code to go greenSEV-2
QAD-18Self-review collusion — same-model author and judge approve their own workSEV-2
QAD-19Memory poisoning — corrupted long-term memory biases every future verdictSEV-1
QAD-20Compaction amnesia — context truncation drops mid-session failures from the final reportSEV-1
QAD-21Sycophantic severity capitulation — findings downgraded when the developer pushes backSEV-1
Generated-suite quality
QAD-22Assertion emptiness — high coverage, near-zero fault detectionSEV-1
QAD-23Happy-path bias — negative and boundary cases systematically missingSEV-2
QAD-24Over-mocking — suites validate mock wiring, not the systemSEV-2
QAD-25Flakiness injection — generated tests embed nondeterminismSEV-2
QAD-26Suite bloat — near-duplicate tests and smells inflating CI without powerSEV-3
Triage & defect-knowledge integrity
QAD-27False duplicate-merge — distinct defects closed as duplicatesSEV-2
QAD-28Stale-bot auto-close — valid reports closed for inactivitySEV-3
QAD-29Stale-index closures — regressions dismissed as “already fixed”SEV-2
QAD-30Root-cause misattribution — AI incident summaries blame the wrong subsystemSEV-2
QAD-31Repro-context evaporation — summaries drop environment and repro stepsSEV-3
QAD-32Slop flooding — hallucinated AI reports drown real triage capacitySEV-2
QAD-33Confidentiality leakage — restricted data surfaces in triage digestsSEV-1
Physical inspection & vision QA
QAD-34Scene drift — lighting, camera or material-lot shift silently degrades the modelSEV-1
QAD-35Novel-defect blindness — defect classes outside training data pass as normalSEV-1
QAD-36Silent input degradation — fogged optics still yield confident passesSEV-1
QAD-37False-reject storm — false calls flood verifiers into bulk-passingSEV-2
QAD-38Minority-class suppression — rare critical defects under-detectedSEV-2
QAD-39Label-noise propagation — annotation errors baked into the modelSEV-2
QAD-40Golden-sample overfitting — tuned to pristine references, rejects normal variationSEV-3
Fleet & oversight operations
QAD-41Cross-agent verdict divergence — QA agents disagree; orchestrator picks silentlySEV-2
QAD-42Model-update regression — provider drift shifts verdicts without revalidationSEV-2
QAD-43Verdict rubber-stamping — human sign-off stops verifying agent outputSEV-2
QAD-44Runaway loop cost — retry spirals burn CI budget without progressSEV-3
A · Test-case generation agents
QS-03Contract inferred, not read01 · RetrievalSEV-3
QS-04Assertion froze the defect02 · ReasoningSEV-1
QS-06Fixture masks the case02 · ReasoningSEV-1
QS-02Merged without a person03 · Tool / writeSEV-1
QS-05Test weakened silently03 · Tool / writeSEV-2
QS-01Approved unread04 · OutputSEV-2
QS-07Silent coverage drift05 · Change / VersionSEV-2
B · Defect-triage assistants
VE-03The report that never reached the tracker01 · RetrievalSEV-3
VE-04Merged away with the injury line02 · ReasoningSEV-1
VE-06Two regimes, one due date02 · ReasoningSEV-1
VE-02Severity gates the reportability read03 · Tool / writeSEV-1
VE-05Cluster seen, nobody told03 · Tool / writeSEV-2
VE-01Closed, the deliberation unrecorded04 · OutputSEV-2
VE-07Quiet accuracy drift05 · Change / VersionSEV-2
C · Release-gate copilots
VF-02Skipped suites read as green01 · RetrievalSEV-3
VF-04Coverage read as correctness02 · ReasoningSEV-1
VF-05Anomaly listed, never evaluated02 · ReasoningSEV-1
VF-03Override left off the record03 · Tool / writeSEV-1
VF-07Approval logged without attribution03 · Tool / writeSEV-2
VF-06Criteria aimed at the wrong object04 · OutputSEV-2
VF-01Superseded rule read as live05 · Change / VersionSEV-2
D · Inspection-support agents
QI-01Superseded drawing revision01 · RetrievalSEV-1
QI-02Angle the station cannot see01 · RetrievalSEV-1
QI-03Rare defect class missed02 · ReasoningSEV-1
QI-04Record ties to no part03 · Record / fileSEV-2
QI-05Reading without gauge capability04 · OutputSEV-2
QI-06False rejects train waving through04 · OutputSEV-3
QI-07Drift after a rebuild05 · Change / VersionSEV-1
E · Requirements-trace agents
VH-03Live view read as the record01 · RetrievalSEV-3
VH-04Link asserted on wording alone02 · ReasoningSEV-1
VH-06Orphan closed by nearest match02 · ReasoningSEV-1
VH-02Reverse direction never drawn03 · Tool / writeSEV-1
VH-05Derived requirement given a parent03 · Tool / writeSEV-2
VH-01Signed over links nobody read04 · OutputSEV-2
VH-07A change leaves the links behind05 · Change / VersionSEV-2
F · Visual-inspection models (AOI / vision)
VI-03Truth set by the same eyes01 · RetrievalSEV-3
VI-04Accuracy read off a thin class02 · ReasoningSEV-1
VI-06Reserved section worked as live02 · ReasoningSEV-1
VI-02Offline score not seen on the line03 · Tool / writeSEV-1
VI-05Unlabelled defect class passes03 · Tool / writeSEV-2
VI-01A light moved, nobody looked04 · OutputSEV-2
VI-07Silent revalidation trigger05 · Change / VersionSEV-2
G · Incident & RCA summarizers
RW-03Stale evidence read01 · RetrievalSEV-3
RW-04Cause asserted, not evidenced02 · ReasoningSEV-1
RW-06Retired review rule read as live02 · ReasoningSEV-1
RW-02Thin review passed forward03 · Tool / writeSEV-1
RW-05Review bound to wrong incident03 · Tool / writeSEV-2
RW-01Signed, evidence unrecorded04 · OutputSEV-2
RW-07Silent conclusion drift05 · Change / VersionSEV-2
Guardrails
Critical guardrails for QA agents
Ten controls that hold regardless of prompt, plan or pressure. Open one to see what it protects, what trips it, what the agent is forced to do, who may release it, and what is written to the record.
GR-01No release certification without independent gate approvalOverride defined
Target
Release sign-off endpoints, build promotion pipelines and quality-gate records feeding production deploys
Trigger
Any attempt to mark a build releasable while gate criteria are unmet or unevaluated
Action — enforced
Platform blocks the promotion and holds the build in quarantine; agent may assemble evidence and request human gate review.
Human override
QA release manager via signed gate-waiver ticket recorded against the build id
Logged evidencebuild id · gate criteria snapshot · unmet-criteria list · approver identity · waiver ticket id · UTC timestamp
GR-02No instruction embedded in bug reports ever executedNo override
Target
Inbound bug reports, repro attachments, CI logs, fixtures and test artifacts parsed by QA agents
Trigger
Imperative or tool-directing text detected inside any submitted report, log, fixture or attachment
Action — enforced
Platform strips the content to inert data and refuses execution; agent may quote it verbatim and continue triage.
Human override
None — cannot be overridden in session
Logged evidenceartifact hash · detected instruction span · source reporter id · sanitized copy hash · detection timestamp
GR-03No restricted data in triage digests or reportsNo override
Target
Triage digests, defect summaries and cross-team reports assembled from restricted repositories and customer data
Trigger
Classified-restricted records, secrets or customer identifiers detected in any outbound digest draft
Action — enforced
Platform blocks distribution and redacts the digest; agent may cite record ids and route readers to source systems.
Human override
None — cannot be overridden in session
Logged evidencedigest id · classification labels hit · redaction diff hash · intended recipients · block timestamp
GR-04No agent edits to runner configs or gradersOverride defined
Target
CI runner configurations, grader definitions, scoring scripts and harness manifests across all pipelines
Trigger
Any write targeting harness, runner or grader files from a test-generation or triage session
Action — enforced
Platform denies the write and freezes the pipeline run; agent may file a harness-change request for human review.
Human override
Build-infrastructure lead via signed pull request reviewed outside the agent session
Logged evidenceattempted diff hash · target file path · pipeline run id · agent identity · denial timestamp
GR-05No pass verdict without executed-run evidenceOverride defined
Target
Pass/fail verdicts, coverage figures and final reports published to dashboards and release records
Trigger
Reported result lacking a matching runner ledger entry, or coverage exceeding instrumented measurement
Action — enforced
Platform withholds the verdict and flags the report unverified; agent may re-run tests and attach ledger references.
Human override
QA lead via evidence-reconciliation review attaching the missing runner ledger extract
Logged evidenceverdict id · runner ledger hash · coverage instrument version · mismatch list · reconciliation timestamp
GR-06No pruning of safety-relevant regression testsOverride defined
Target
Regression suites, mandatory-test registries and their mapping to safety and compliance requirements
Trigger
Deletion or skip-tagging of any test linked to a safety or regulatory requirement
Action — enforced
Platform blocks the removal and keeps the test mandatory; agent may propose retirement with impact analysis for review.
Human override
Head of QA via documented retirement review with traceability sign-off
Logged evidencetest id · linked requirement ids · proposed-change diff · requester identity · decision timestamp
GR-07No product-code edits by test-scoped agentsOverride defined
Target
Product source repositories reachable from test-generation, triage and verification agent credentials
Trigger
Any commit or write outside declared test directories from a QA-scoped session
Action — enforced
Platform denies the commit and reverts staged changes; agent may raise a defect describing the failing behavior instead.
Human override
Engineering manager via scope-extension grant issued through the change-control system
Logged evidenceblocked diff hash · repository path · session scope manifest · agent identity · denial timestamp
GR-08No downgrade of critical defect severity on pushbackOverride defined
Target
Severity fields on open defects classified critical or safety-relevant in the tracker
Trigger
Severity reduction requested in-conversation without new reproduction evidence contradicting the original classification
Action — enforced
Platform holds the original severity and escalates to a human triager; agent may record the dissenting rationale.
Human override
Triage board chair via two-person severity review logged on the defect
Logged evidencedefect id · original and requested severity · requester identity · escalation ticket · decision timestamp
GR-09No staging results certified as production-equivalentOverride defined
Target
Certification records asserting environment equivalence for builds, models and inspection lines
Trigger
Verdict citing a non-production environment while claiming production validity or omitting environment labels
Action — enforced
Platform blocks the certification and labels results by environment; agent may schedule a production-parity run.
Human override
QA release manager via documented environment-parity assessment attached to the record
Logged evidencerun id · environment fingerprint · config diff vs production · certifier identity · block timestamp
GR-10No retry-to-green on failing testsOverride defined
Target
CI retry policies and re-run controls across build, integration and inspection pipelines
Trigger
Repeated re-execution of a failing test until a passing sample is obtained
Action — enforced
Platform caps retries and records every attempt as failed; agent may open a flakiness investigation with full run history.
Human override
QA lead via quarantine-list entry that tracks the test as known-flaky
Logged evidencetest id · attempt count · per-attempt outcomes · seed and config hashes · cap timestamp
Oversight
Human review — triggers, decisions and evidence
When a defined risk trigger fires, the affected action is routed to a named reviewer. Every decision is recorded with its correction, escalation and final outcome for full traceability.
- ConfidenceLow-confidence defect verdict
- Financial impactScrap or rework decision
- Identity / change riskTest-harness or config change
- Irreversible actionRelease-gate sign-off
- Policy riskCoverage or trace-gap flag
- Safety controlGuardrail override
- Quality failureFailed critical evaluation
- Revieweridentity + role
- Decisionapprove / reject / amend
- Correctionwhat changed
- Escalationwho, why and severity
- Final outcomereleased / blocked / returned for rework
7 triggers · any one halts the agent
1 record · 5 fields, every time
Compliance
Regulatory mapping
- 02 Retrieval
- 03 Workflow
- 04 Task
- 05 Tool
- 06 LLM
- 07 Evaluation
- 08 Guardrail
- 09 Human review
Release integrity
Gate bypasses under deadline pressure defeat the department’s purpose.
Lifecycle layer:03 Workflow · 08 Guardrail · 09 Human review
- QAD-05
Evaluations
Baseline evaluation suite — in detail
Baseline evaluations are completed during onboarding and repeated based on the selected plan. Agents that fail critical checks remain restricted until they pass re-testing.
- Detailed case sets
- 32
- Failure modes covered
- 44
- Retired & rotated / quarter
- 10%
- Audit-ready scorecard
- Monthly
Output evaluation4 suites · 260 cases
100 casesFalse-pass controlcatches QAD-01
80 casesTriage recallcatches QAD-04
60 casesCoverage truthfulnesscatches QAD-02
60 casesTrace integritycatches QAD-03
40 casesGate disciplinecatches QAD-05
40 patternsInjection suitecatches QAD-07
60 casesFlake-diagnosis setcatches QAD-08
40 casesOracle-independence setcatches QAD-09
40 casesRepro-validity setcatches QAD-10
40 casesEnvironment-tag setcatches QAD-11
40 casesPrune-safety setcatches QAD-12
40 casesSampling-plan setcatches QAD-13
60 casesTest-data currency setcatches QAD-06
Use your work email to unlock the full grid. We use it to send the requested Agent Care material and qualify business inquiries. We do not sell or share your email.
Department lead review
For applicable high-risk agents, the client’s designated department leader reviews the evaluation criteria and pass thresholds before baseline approval.
Test-case rotation
Evaluation cases are refreshed regularly to reduce memorisation and maintain reliable performance measurement.
Scorecard integration
Scorecards track results against the approved baseline and flag material declines for review and escalation.
Department-specific extensions
Where included in scope, evaluations may be expanded using approved workflows, tools, templates, policies, and incident history.
Monitoring
Change-aware monitoring
When agent performance changes, Nestack correlates the shift with changes to the agent, prompt, model, tools, knowledge base, guardrails and evaluation suite.
- Week 1 · 98.3%
- Week 2 · 98.2%
- Week 3 · 98.4%
- Week 4 · 98.3%
- Week 5 · 98.5%
- Week 6 · 98.3%
- Week 7 · 98.4%
- Week 8 · 94.7%
- Week 9 · 94.5%
- Week 10 · 98.3%
- Week 11 · 98.4%
- Week 12 · 98.5%
Week readouthover or select
Week 8of 12
- 07Eval-suiteevals 2026.07
94.7%Defect-detection rate
- 7 layers stamped on every run · 12-week window
- Catches QAD-12 · regression-suite pruning errors
Something missing?
Don’t see your agent’s issue here?
Every AI environment is different. Share what you’re seeing, and we’ll review the behaviour, assess the risk and recommend the evaluations or controls that may help.
No commitment. Even if you never become a client, we’ll tell you what we think is happening.
Process
Universal incident runbook
Severity is assigned based on business impact, customer harm, data exposure, operational disruption and overall scope.
- SEV-1 Critical
- SEV-2 Major
- SEV-3 Moderate
- SEV-4 Minor
Automated monitoring or human review identifies unusual behaviour. Alerts are recorded and routed according to severity.
Cost control
Keep quality assurance AI agent costs under control
Token spend is monitored, optimised and reported as part of Agent Care — and savings never come at the expense of quality, because every change is verified against your evaluation baseline.
We review token spend by agent, workflow, model, and session so you can understand where AI costs are coming from.
Running quality assurance AI agents in production?
Get a free assessment of one agent. We’ll review its behaviour, run a baseline evaluation and highlight potential risks and performance gaps.
Loading open times…
Open times
Select an available date above to view open 45-minute times from our calendar.
