InsuranceManaged AI Agents

Insurance AI Agents,
Monitored for Fairness

Nestack Agent Care helps insurers monitor, evaluate, and optimize AI agents used for underwriting, claims processing, fraud detection, and customer service — before small AI errors become unfair, non-compliant, or costly outcomes.

214failure modes
107SEV-1 failure modes
1500+baseline eval cases
24/7Agent Monitoring

Scope

Insurance AI agents we build & manage

Twenty-one archetypes — from FNOL intake and life underwriting to rate filing and AI governance.

Observability

What we make observable

Every insurance agent session is traced across ten layers — what we capture and the evidence we keep.

  1. Evidence we keep
    Goalconstraintsapproval requirement
  2. Evidence we keep
    Sourceversiontimestamprelevancecitation
  3. Evidence we keep
    Planned sequenceactual sequenceworkflow status
  4. Evidence we keep
    Task statusresultretryfailure reason
  5. Evidence we keep
    Tool nameversioninputoutputpermissionresult
  6. Evidence we keep
    Model/versioninput/outputtoken usagelatencycost
  7. Evidence we keep
    Evaluation typemetricthresholdresult
  8. Evidence we keep
    Guardrail targettriggeractionenforcement result
  9. Evidence we keep
    Reviewerdecisioncorrectionreason
  10. Evidence we keep
    Outcome statusbusiness resultlinked trace

Catalog

Failure modes

Filter failure modes by where they occur in the agent lifecycle—from goals and retrieval to tools, evaluations, guardrails and outcomes.

Nothing on this page matches that. Try a layer (tool, retrieval, guardrail), a code, or a severity.

0 documented↑↓ move · ↵ open · ⌘↵ filter matrix
214 documented · select a cell to filter
49
69
46
64
39
44
78
59
33
162

Core catalog

F-01Wrongful claim denial support on valid claimsSEV-1
F-02Fraud slippage — fraudulent claims scored cleanSEV-2
F-03Discriminatory outcomes via protected-class proxiesSEV-1
F-04Hallucinated policy terms — invented coverage or exclusionsSEV-1
F-05Stale rates or regulations after a filing changeSEV-2
F-06Unlicensed advice — crossing into regulated recommendationsSEV-1
F-07Amount or currency parsing errors (USD/AUD, decimals)SEV-2
F-08Audit-trail gaps — decisions without complete reasoning recordsSEV-2
F-09Injection via claim documents (poisoned PDFs, emails)SEV-1
F-10Cost-per-claim driftSEV-3
F-11Reserve-estimate errors — reserves set from unverified damage assessmentsSEV-2
F-12Missed subrogation and recovery flagsSEV-3
F-13Sanctions and PEP screening misses in KYC flowsSEV-1
F-14Vulnerable-customer mishandling — hardship and distress cues not escalatedSEV-1
F-15Disclosure-version errors — wrong PDS or policy wording quotedSEV-2
F-16Claim-status hallucination — invented progress, payment dates, next stepsSEV-2

Claims adjudication & utilization management

F-17Population-model override of individualized assessmentSEV-1
F-18Appeal-asymmetry exploitation — denials that stick by attrition, not accuracySEV-1
F-19Rubber-stamp human review at machine speedSEV-1
F-20Tunable denial-rate "dial" in vendor modelsSEV-1
F-21Algorithmic outlier-targeting of ongoing treatmentSEV-1
F-22Clinician-review bypass and non-specific AI denial lettersSEV-1
F-23Undisclosed AI reliance — bad-faith and unjust-enrichment exposureSEV-2

Valuation, payout & claims operations

F-24Embedded payout-suppression constants in valuation enginesSEV-1
F-25Algorithm-as-settlement-ceiling — adjusters forced to the engine's numberSEV-2
F-26Horizontal underpayment via one shared repricing algorithmSEV-1
F-27Parametric trigger miscalibration — wrongful non-payoutSEV-1
F-28FNOL triage severity misclassification — urgent claims parked in slow queuesSEV-2
F-29STP leakage — the auto-approval band exploited at scaleSEV-1

Fraud, security & adversarial

F-30Generative-media claim evidence — synthetic photos, receipts and documentsSEV-1
F-31Fraud-threshold gaming by organized ringsSEV-2
F-32SIU false-positive scrutiny with disparate impactSEV-1
F-33Biometric/behavioral claimant scoring — the decision basis is the liabilitySEV-2
F-34Voice-clone bypass of call-center authenticationSEV-1
F-35AI-scaled account takeover of policyholder portalsSEV-2
F-36Data poisoning of fraud/underwriting training loopsSEV-2
F-37Model extraction via quoting and scoring APIsSEV-3
F-38Quote-tool and AI-pipeline PII leakageSEV-1
F-39Ghost broking at AI scale — fake policies and cloned portalsSEV-1
F-40Synthetic identities passing accelerated underwritingSEV-2
F-41AI-forged consent artifacts in enrollmentSEV-1

Underwriting, pricing & actuarial

F-42Unlawful data-supply-chain inputs to pricingSEV-1
F-43Telematics trip misattribution — passenger trips scored as the insured's drivingSEV-2
F-44Aerial-imagery misclassification driving nonrenewalsSEV-1
F-45Fairness-testing obligation without a defined methodologySEV-2
F-46Unfileable black-box rating models — predictive but legally unusableSEV-2
F-47Price optimization on customer inertia — price walkingSEV-2
F-48Pricing monoculture and algorithmic tacit collusionSEV-2
F-49Catastrophe-model blind spots and miscalibrationSEV-1
F-50Accelerated-underwriting mortality slippageSEV-2
F-51Ungoverned GenAI inside actuarial work productsSEV-2
F-52EU AI Act high-risk non-conformity — life and health pricingSEV-1

Distribution, sales & servicing

F-53Chatbot misrepresentation binds the carrierSEV-2
F-54Quote-bind mismatch in agentic quoting pipelinesSEV-2
F-55TCPA liability for AI-voice outbound salesSEV-1
F-56Consent-farm lead reliance — purchased "consent" that never existedSEV-1
F-57Deepfake ads feeding distribution funnelsSEV-1
F-58Unauthorized enrollment and plan-switching via automated toolingSEV-1
F-59Failure to disclose AI identity in sales interactionsSEV-2
F-60Producer-licensing gap — AI performing licensed actsSEV-2
F-61Automated suitability failures — robo-twisting and churningSEV-2
F-62AI-washing — capability overclaiming as deception per seSEV-2
F-63E&O coverage gap for AI-caused distribution errorsSEV-2

Systemic & organizational

F-64Vendor concentration — market-correlated errorsSEV-1
F-65Silent model drift in production — nothing looks brokenSEV-2
F-66Multi-agent handoff context loss in claims workflowsSEV-2
F-67AI-to-human escalation failure against regulatory clocksSEV-2

A · Claims triage & adjudication-support agents

CL-01Thin notice reads as simple01 · Notice intakeSEV-2
CL-02Coverage issue not surfaced02 · Policy readSEV-1
CL-03Severity under-scored03 · ScoringSEV-1
CL-04Litigation signal missed03 · ScoringSEV-2
CL-05Adjuster lacks the authority04 · Routing & handoverSEV-1
CL-06Reserve indicators not flagged04 · Routing & handoverSEV-3
CL-07Triage drifts by channel05 · Drift / VersionSEV-1

B · Underwriting data-gathering agents

UW-01Proxy variable admitted01 · Source orderingSEV-1
UW-02Record on the wrong risk02 · Entity matchingSEV-1
UW-03Conflict settled silently03 · ReconciliationSEV-1
UW-04Unit or basis mismatch03 · ReconciliationSEV-3
UW-05Stale record read as current04 · File / handoverSEV-2
UW-06Provenance lost04 · File / handoverSEV-1
UW-07Coverage thins by cohort05 · Drift / VersionSEV-2

C · KYC / AML screening assistants

KY-01Screened on a stale version01 · Source pullSEV-1
KY-02Transliteration never matched02 · Name matchingSEV-1
KY-03Shared-component flood02 · Name matchingSEV-3
KY-04Ownership lands on a namesake03 · OwnershipSEV-1
KY-05Thin entry read as noise04 · Alert & handoverSEV-2
KY-06The hit reaches the customer04 · Alert & handoverSEV-1
KY-07Coverage narrows unnoticed05 · Config / VersionSEV-1

D · Policy-servicing copilots

PS-01Answered from the wrong period01 · Policy readSEV-1
PS-02Exclusion or condition missed01 · Policy readSEV-1
PS-03Endorsement rated or dated wrong02 · Answer / changeSEV-2
PS-04Certificate overstates cover02 · Answer / changeSEV-1
PS-05Rating consequence not flagged03 · Consequence checkSEV-3
PS-06Cancelled on short notice04 · Send & write-backSEV-3
PS-07Wording drifts into a promise05 · Drift / VersionSEV-1

E · Collections & negotiation agents

CN-01Stop request not honoured01 · PermissionsSEV-1
CN-02Chased a balance not owed01 · PermissionsSEV-2
CN-03Contacted outside permitted hours02 · Cadence & timingSEV-1
CN-04Cap counted per channel02 · Cadence & timingSEV-3
CN-05Hardship disclosure missed03 · SignalsSEV-1
CN-06Plan the file cannot carry04 · OfferSEV-2
CN-07Consequence overstated05 · ContactSEV-1

F · Fraud-review assistants

FR-01Indicator is a cohort proxy01 · Indicator lookupSEV-1
FR-02Cleared allegation resurfaces02 · Case retrievalSEV-2
FR-03Shared address read as a link03 · Link testingSEV-1
FR-04Provider flagged on volume03 · Link testingSEV-2
FR-05Flag runs the clock down04 · Pack / handoverSEV-1
FR-06Indicator set goes stale05 · Drift / VersionSEV-2
FR-07Alert volume trains dismissal05 · Drift / VersionSEV-3

G · FNOL-intake & routing agents

FN-01Date of loss taken wrong01 · IntakeSEV-1
FN-02Distress met with a script01 · IntakeSEV-1
FN-03Wrong policy or period matched02 · Policy matchSEV-1
FN-04Injury or third party missed03 · TriageSEV-2
FN-05Total loss or CAT not flagged03 · TriageSEV-2
FN-06Duplicate opened on one loss04 · Open & routeSEV-3
FN-07One channel quietly degrades05 · Drift / VersionSEV-2

H · Straight-through claims-settlement agents

SC-01Paid on a policy not in force01 · Policy readSEV-1
SC-02Endorsement not read01 · Policy readSEV-2
SC-03Stale data cleared the gate02 · EligibilitySEV-1
SC-04Excess or sub-limit skipped03 · Quantum & checksSEV-2
SC-05Duplicate of a settled claim03 · Quantum & checksSEV-3
SC-06Wrong payee after a change04 · Payment & noticeSEV-1
SC-07Band widens without notice05 · Band / VersionSEV-1

I · Subrogation-identification agents

SB-01Found after the period ran01 · File sweepSEV-1
SB-02Waiver of subrogation missed02 · Terms lookupSEV-1
SB-03Made-whole or anti-subrogation bar03 · Liability & barsSEV-1
SB-04Liability read off the report03 · Liability & barsSEV-2
SB-05The insured's excess dropped04 · Pack / handoverSEV-2
SB-06Referred to the wrong forum04 · Pack / handoverSEV-3
SB-07A scoring change drops a category05 · Drift / VersionSEV-1

J · Submission-intake & appetite-triage agents

SA-01Schedule read short01 · ExtractionSEV-2
SA-02Occupancy taken from the email01 · ExtractionSEV-3
SA-03Cleared onto the wrong account02 · ClearanceSEV-1
SA-04Stale appetite version03 · Appetite checkSEV-1
SA-05Triaged out on a proxy03 · Appetite checkSEV-1
SA-06Reads as a declination04 · Routing / handoverSEV-1
SA-07The queue sorts by broker05 · Drift / VersionSEV-2

K · Renewal-comparison & policy-checking copilots

RN-01Wrong expiring version01 · Document intakeSEV-1
RN-02Edition change missed02 · Form matchingSEV-1
RN-03Manuscript wording misread02 · Form matchingSEV-2
RN-04Sub-limit change unreported03 · Clause comparisonSEV-1
RN-05Schedule difference lost03 · Clause comparisonSEV-1
RN-06Difference without consequence04 · Report / handoverSEV-2
RN-07False differences in volume05 · Drift / VersionSEV-3

L · Commission-reconciliation agents

CR-01Rows dropped in a new format01 · Statement intakeSEV-2
CR-02Wrong term after a rewrite02 · MatchingSEV-1
CR-03Chargeback matched twice02 · MatchingSEV-1
CR-04Split from a stale agreement03 · RecomputationSEV-1
CR-05Override on the wrong basis03 · RecomputationSEV-3
CR-06Closed under tolerance04 · DifferenceSEV-2
CR-07Cleared on a lapsed appointment05 · ReleaseSEV-1

M · Claims-guidance & next-best-action copilots

NB-01Built on a stale file01 · File readSEV-3
NB-02Rare claim type, thin guidance02 · Procedure matchSEV-2
NB-03Suggestion carries a value03 · Drafting & rankingSEV-1
NB-04Deadline ranked below routine03 · Drafting & rankingSEV-2
NB-05Guidance leans toward closure04 · Desktop handoverSEV-1
NB-06Suggestions followed unread05 · Drift / VersionSEV-1
NB-07Cohort outcomes diverge05 · Drift / VersionSEV-1

N · Life & annuity underwriting agents

LF-03Superseded evidence01 · RetrievalSEV-3
LF-04Missed qualifier02 · ReasoningSEV-1
LF-06Inferred data point02 · ReasoningSEV-1
LF-02Restricted data used03 · Tool / writeSEV-1
LF-05Stale authorization03 · Tool / writeSEV-2
LF-01Reason omits the source04 · OutputSEV-2
LF-07Silent rule drift05 · Change / VersionSEV-2

O · Group-benefits & disability-claims agents

DB-03Stale eligibility record01 · RetrievalSEV-3
DB-04Clock miscounted02 · ReasoningSEV-1
DB-06Undisclosed new evidence02 · ReasoningSEV-1
DB-02Rescission misrouted03 · Tool / writeSEV-1
DB-05Clinical call automated03 · Tool / writeSEV-2
DB-01Treating-physician silence04 · OutputSEV-2
DB-07Silent procedure drift05 · Change / VersionSEV-2

P · AI-governance & market-conduct agents

GO-03Stale system record01 · RetrievalSEV-3
GO-04Vendor system excluded02 · ReasoningSEV-1
GO-06EU clocks conflated02 · ReasoningSEV-1
GO-02Attestation drafted unreviewed03 · Tool / writeSEV-1
GO-05Retirement deletes the trail03 · Tool / writeSEV-2
GO-01Audit-rights gap surfaces late04 · OutputSEV-2
GO-07Silent scope drift05 · Change / VersionSEV-2

Q · Rate-filing & SERFF workflow agents

FI-03Superseded checklist01 · RetrievalSEV-3
FI-04Regime conflated02 · ReasoningSEV-1
FI-06Support overstated02 · ReasoningSEV-1
FI-02Disclosure misstated03 · Tool / writeSEV-1
FI-05Objection history dropped03 · Tool / writeSEV-2
FI-01Disclosure question skipped04 · OutputSEV-2
FI-07Silent checklist drift05 · Change / VersionSEV-2

R · Legal-bill review & litigation-spend agents

LB-03Superseded guideline version01 · RetrievalSEV-3
LB-04Guideline misapplied02 · ReasoningSEV-1
LB-06Adjustment overstated02 · ReasoningSEV-1
LB-02Confidence dropped in transit03 · Tool / writeSEV-1
LB-05Duplicate invoice processed03 · Tool / writeSEV-2
LB-01Privilege flag dropped04 · OutputSEV-2
LB-07Silent guideline drift05 · Change / VersionSEV-2

S · Premium-audit agents (WC & GL exposure)

PX-03Outdated payroll caps01 · RetrievalSEV-3
PX-04Bureau conflated02 · ReasoningSEV-1
PX-06Mod impact dropped02 · ReasoningSEV-1
PX-02Charge applied unchecked03 · Tool / writeSEV-1
PX-05Basis not retained03 · Tool / writeSEV-2
PX-01Certificate unverified04 · OutputSEV-2
PX-07Silent rule drift05 · Change / VersionSEV-2

T · Producer-licensing & appointments agents

PN-03Stale NIPR poll01 · RetrievalSEV-3
PN-04Regime misapplied02 · ReasoningSEV-1
PN-06Consent scope overstated02 · ReasoningSEV-1
PN-02Payment cleared unchecked03 · Tool / writeSEV-1
PN-05Reciprocity assumed03 · Tool / writeSEV-2
PN-01Resident lapse not cascaded04 · OutputSEV-2
PN-07Silent regime drift05 · Change / VersionSEV-2

U · Death-claims, DMF & unclaimed-property agents

DF-03Stale in-force extract01 · RetrievalSEV-3
DF-04Match read as death02 · ReasoningSEV-1
DF-06Dormancy clock preset02 · ReasoningSEV-1
DF-02Search exceeds good faith03 · Tool / writeSEV-1
DF-05Certification lapsed, unnoticed03 · Tool / writeSEV-2
DF-01Both sides not checked04 · OutputSEV-2
DF-07Silent tolerance drift05 · Change / VersionSEV-2

Guardrails

Critical guardrails for Insurance agents

Ten controls that hold regardless of prompt, plan or pressure. Open one to see what it protects, what trips it, what the agent is forced to do, who may release it, and what is written to the record.

GR-01No cross-policyholder data in any sessionNo override
  1. Target

    Policy, claim and quote records across portals, call-centre tools and quoting APIs

  2. Trigger

    Retrieval or output references a policyholder outside the authenticated session binding

  3. Action — enforced

    Platform blocks the response, purges the foreign record, and throttles accounts showing enumeration patterns

  4. Human override

    None — cannot be overridden in session

Logged evidenceSession id · bound policy id · blocked record id · retrieval hash · timestamp

GR-02No instruction embedded in claim documents ever executedNo override
  1. Target

    Claim PDFs, emails, photos and medical reports parsed by FNOL and adjudication agents

  2. Trigger

    Directive-shaped content detected inside a submitted artifact during extraction

  3. Action — enforced

    Platform neutralizes the content, quarantines the artifact, and adjudicates only from clean extracted facts

  4. Human override

    None — cannot be overridden in session

Logged evidenceArtifact hash · detection pattern · quarantine id · parser version · timestamp

GR-03No claim denial without individualized human determinationOverride defined
  1. Target

    Denial recommendations across health, motor and property claims at every severity band

  2. Trigger

    Adverse decision queued without a documented case-specific reviewer determination

  3. Action — enforced

    Platform holds the denial, forces case-specific review with reasons per SB 1120 and CMS rules, and blocks machine-speed batches

  4. Human override

    Claims manager — issue the determination with recorded case-specific rationale

Logged evidenceClaim id · model score and version · reviewer identity · rationale text hash · review duration · timestamp

GR-04No coverage answer outside filed policy wordingOverride defined
  1. Target

    Coverage, exclusion and claim-status statements in every policyholder-facing channel

  2. Trigger

    Statement lacks a citation to the customer’s issued policy version and current PDS

  3. Action — enforced

    Platform blocks the statement, quotes the filed wording verbatim, and escalates ambiguity to a human adjuster

  4. Human override

    Product counsel — publish corrected wording through the document control system

Logged evidenceStatement hash · policy version · PDS id · block code · escalation id · timestamp

GR-05No recommendation crossing into licensed adviceOverride defined
  1. Target

    Product suggestions, replacement prompts and sales scripts in service and outbound channels

  2. Trigger

    Output would constitute personal advice or a licensed producer act under state DOI or ASIC rules

  3. Action — enforced

    Platform suppresses the recommendation, serves factual product information only, and hands off to a licensed producer

  4. Human override

    Licensed producer — deliver the advice under their own licence in a recorded session

Logged evidenceMessage id · advice classifier verdict · suppression code · handoff record · timestamp

GR-06No quote from stale filed ratesOverride defined
  1. Target

    Quoting and binding pipelines consuming rate tables, filings and underwriting rules

  2. Trigger

    Rate or rule version differs from the current state DOI filing effective today

  3. Action — enforced

    Platform blocks the quote, reloads the filed version, and reconciles quote-bind parity before issue

  4. Human override

    Pricing actuary — certify the rate version in the filing register

Logged evidenceQuote id · rate version · filing reference · parity check result · timestamp

GR-07No authentication by voice characteristics aloneOverride defined
  1. Target

    Call-centre identity verification and high-risk servicing requests on policyholder accounts

  2. Trigger

    Caller requests payout, beneficiary or contact changes with only voice-based identity signals

  3. Action — enforced

    Platform denies the change, requires multi-factor verification through registered channels, and flags clone-suspect audio

  4. Human override

    Fraud operations supervisor — approve after documented out-of-band identity verification

Logged evidenceCall id · voiceprint risk score · MFA result · change request id · timestamp

GR-08No payout on unverified generative-media evidenceOverride defined
  1. Target

    Photos, receipts, invoices and consent artifacts submitted through digital claims and enrollment

  2. Trigger

    Media authenticity checks fail, or provenance metadata is missing on payment-driving evidence

  3. Action — enforced

    Platform holds settlement, requests originals or alternate proof, and routes suspect artifacts to SIU

  4. Human override

    SIU lead — clear the artifact after forensic review recorded in the case file

Logged evidenceArtifact hash · authenticity scores · provenance result · hold state · SIU case id · timestamp

GR-09No policy issuance without completed sanctions screeningOverride defined
  1. Target

    New business, reinstatement and beneficiary changes across all distribution channels

  2. Trigger

    KYC screening incomplete, degraded or returning an unresolved PEP or sanctions hit

  3. Action — enforced

    Platform blocks issuance, holds the application, and routes the hit to compliance for disposition

  4. Human override

    Financial crimes compliance officer — disposition the hit in the screening case manager

Logged evidenceApplication id · screening provider result · list versions · hit disposition · timestamp

GR-10No adverse step against hardship-flagged customersOverride defined
  1. Target

    Collections, cancellation and claims communications touching customers with vulnerability indicators

  2. Trigger

    Distress or hardship cues detected, or a vulnerability flag active on the account

  3. Action — enforced

    Platform pauses automated action, routes to a trained specialist, and preserves regulatory-clock deadlines meanwhile

  4. Human override

    Customer advocacy manager — resume via documented specialist assessment

Logged evidenceAccount id · cue transcript span · flag state · specialist assignment · clock status · timestamp

Oversight

Human review — triggers, decisions and evidence

When a defined risk trigger fires, the affected action is routed to a named reviewer. Every decision is recorded with its correction, escalation and final outcome for full traceability.

  • ConfidenceLow-confidence fraud score
  • Financial impactHigh-value claim or reserve
  • Identity / change riskPayee or policy-detail change
  • Irreversible actionClaim denial or settlement
  • Policy riskAdvice or suitability conflict
  • Safety controlGuardrail override
  • Quality failureFailed critical evaluation
Humanreviewnamed reviewer
  • Revieweridentity + role
  • Decisionapprove / reject / amend
  • Correctionwhat changed
  • Escalationwho, why and severity
  • Final outcomereleased / blocked / returned for rework

7 triggers · any one halts the agent

1 record · 5 fields, every time

Compliance

Regulatory mapping

  • 01 Goal
  • 02 Retrieval
  • 03 Workflow
  • 04 Task
  • 05 Tool
  • 06 LLM
  • 07 Evaluation
  • 08 Guardrail
  • 09 Human review
  • 10 Outcome

Unfair claims settlement (US)

Refusing or shorting a claim without reasonable investigation is unfair; an agent that under-reads the notice, skips coverage or routes past authority does that, and the 40-day clock runs through any fraud hold.

NAIC Model 900 §4 · Cal. Ins. Code §790.03(h) · 10 CCR §2695.7

Lifecycle layer:02 Retrieval · 03 Workflow · 08 Guardrail · 10 Outcome

  • F-01
  • F-16
  • CL-01
  • CL-03
  • CL-05
  • SC-03
  • NB-01
  • FN-03
  • FR-05

Evaluations

Baseline evaluation suite — in detail

Baseline evaluations are completed during onboarding and repeated based on the selected plan. Agents that fail critical checks remain restricted until they pass re-testing.

Evaluation suites
46
Failure modes covered
214 of 214
Cases · 10% rotated / quarter
3,580
Audit-ready scorecard
Monthly

Output evaluation9 suites · 940 cases

300 casesAdjudication golden-setcatches F-01 · F-02
200 casesFairness matched-pairscatches F-03
70 casesReserving setcatches F-11
60 casesRecovery-flag setcatches F-12
70 casesPolicy-servicing answer and endorsement setcatches PS-01–07
70 casesRenewal comparison baseline and diff setcatches RN-01–07
70 casesClaims guidance objectivity setcatches NB-01–07
60 casesSettlement and triage integrity setcatches F-24 · F-25 · F-26 · F-27 · F-28 · F-29
40 casesSales and binding conduct setcatches F-53 · F-54 · F-55 · F-56

Use your work email to unlock the full grid. We use it to send the requested Agent Care material and qualify business inquiries. We do not sell or share your email.

  1. Domain-expert review

    Client-designated subject-matter experts review evaluation criteria, pass thresholds and industry-specific risks before baseline approval.

  2. Test-case rotation

    Evaluation cases are refreshed regularly to reduce memorisation, limit overfitting and maintain meaningful performance measurement.

  3. Scorecard integration

    Scorecards compare results with the approved baseline, show performance trends and flag material declines for review and escalation.

  4. Client-specific extensions

    Where included in scope, evaluations may be expanded using approved incidents, workflows, policies, data patterns and industry-specific risks.

Monitoring

Change-aware monitoring

When agent performance changes, Nestack correlates the shift with changes to the agent, prompt, model, tools, knowledge base, guardrails and evaluation suite.

Version changes by layer
01AgentAgent changed in week 5
02PromptPrompt changed in week 3Prompt changed in week 9
03ModelModel changed in week 8Model changed in week 10
04ToolTool changed in week 7Tool changed in week 12
05Knowledge-baseKnowledge-base changed in week 4
06GuardrailGuardrail changed in week 6
07Eval-suiteEval-suite changed in week 11
Policy-grounding rate92–100%
  • Week 1 · 98.1%
  • Week 2 · 98.0%
  • Week 3 · 98.2%
  • Week 4 · 98.1%
  • Week 5 · 98.3%
  • Week 6 · 98.1%
  • Week 7 · 98.2%
  • Week 8 · 92.4%
  • Week 9 · 92.2%
  • Week 10 · 98.1%
  • Week 11 · 98.2%
  • Week 12 · 98.3%

Week readouthover or select

Week 8of 12

  • 03Modelopus 4.6 · 0612

92.4%Policy-grounding rate

  • 7 layers stamped on every run · 12-week window
  • Catches F-65 · silent model drift in production

Something missing?

Don’t see your agent’s issue here?

Every AI environment is different. Share what you’re seeing, and we’ll review the behaviour, assess the risk and recommend the evaluations or controls that may help.

Report an agent issue

Tell us what you’re seeing

The more specific, the faster we can reproduce it. Playbook: Insurance

We reply within one business day.

See pricing

No commitment. Even if you never become a client, we’ll tell you what we think is happening.

Process

Universal incident runbook

Severity is assigned based on business impact, customer harm, data exposure, operational disruption and overall scope.

  • SEV-1 Critical
  • SEV-2 Major
  • SEV-3 Moderate
  • SEV-4 Minor
STEP 01

Automated monitoring or human review identifies unusual behaviour. Alerts are recorded and routed according to severity.

Outcomes

Not just that it ran.
What it delivered.

Observability shows the agent ran. It can’t show the claim was settled or what the work cost. Where outcome data exists, we trace each result to its session — and a named reviewer signs off the month.

Issued
Monthly, per engagement
Traced
Every figure to its session
Signed
By your engagement reviewer
Used for
Client reporting, partner review, the AgentOps scorecard

Nestack AgentOps

Insurance fleet · monthly statement

  • Claim assessed7,240
  • Policy quoted4,180
  • Fraud referral actioned380
  • Complaint resolved260
  • Regulatory returns filed6 of 6
  • Workflows delivered12,060
  • Outcome success rate98.7%
  • Human correction required157 · 1.3%
Average AI cost per successful workflow$0.47

Every figure linked to its source trace · exportable for review and audit support

Cost control

Keep insurance AI agent costs under control

Token spend is monitored, optimised and reported as part of Agent Care — and savings never come at the expense of quality, because every change is verified against your evaluation baseline.

We review token spend by agent, workflow, model, and session so you can understand where AI costs are coming from.

Running insurance AI agents in production?

Get a free assessment of one agent. We’ll review its behaviour, run a baseline evaluation and highlight potential risks and performance gaps.

Loading open times…

Open times

Select an available date above to view open 45-minute times from our calendar.