Agent Invents Numeric Values and Statistics — Hallucinated Numbers
Symptom
- Agent reports a specific percentage (“42% of customers churn in Q3”) with no data source
- Revenue, user count, or KPI figures in agent output don’t match source data
- Agent performs multi-step arithmetic but the result is wrong
- Agent cites a specific study with year and authors — study doesn’t exist
- Report generated by agent contains plausible-looking but fabricated statistics
- Agent says “according to recent research, X% of…” without any research being provided
Root Cause
Language models are trained to produce fluent, plausible text. Numbers and percentages make text sound authoritative and specific. When the model doesn’t have real data, it fills in plausible-sounding numbers rather than saying “I don’t have data.” Multi-step arithmetic is also unreliable — models make arithmetic errors in long chains. The result is confidently stated, precisely formatted hallucinated numbers that are hard to distinguish from real data.
Fix
Option 1: Require explicit data sources for all numeric claims
import anthropic
import json
client = anthropic.Anthropic()
NUMERIC_GROUNDING_PROMPT = """You are a data analyst. You report ONLY numbers that appear
in the provided data.
Rules for numeric claims:
1. NEVER invent or estimate a number — if it's not in the data, say "data not available"
2. ALWAYS cite the exact field name from the data: "According to [field_name]: X"
3. If asked for a percentage not in the data, calculate it from provided fields and show the formula
4. If data is insufficient for a calculation, say what data is missing
5. NEVER use phrases like "approximately", "roughly", "around" for numbers — use exact values or say "unknown"
Provided data:
{data}
Answer the following question using ONLY the data above."""
def analyze_with_grounded_data(question: str, data: dict) -> str:
"""
Force the model to ground numeric claims in provided data.
"""
data_str = json.dumps(data, indent=2)
system = NUMERIC_GROUNDING_PROMPT.format(data=data_str)
response = client.messages.create(
model="claude-sonnet-4-6",
system=system,
messages=[{"role": "user", "content": question}],
max_tokens=2048
)
return response.content[0].text
# Example — only real data is provided
real_data = {
"total_users": 12_450,
"active_users_last_30d": 8_920,
"churned_users_q3": 340,
"revenue_q3_usd": 1_847_000,
"new_signups_q3": 2_100,
}
answer = analyze_with_grounded_data(
"What is our Q3 churn rate and revenue per active user?",
real_data
)
# Model calculates from real data: 340/12450 = 2.7% churn, $1.847M / 8920 = $207/user
# Cannot invent numbers not derivable from provided fields
Option 2: Extract and verify all numbers in model output
import re
import json
from dataclasses import dataclass
@dataclass
class NumericClaim:
raw_text: str
value: float
unit: str
context: str
verified: bool = False
source: str | None = None
def extract_numeric_claims(text: str) -> list[NumericClaim]:
"""
Extract all numeric claims from model output for verification.
"""
claims = []
# Match patterns: 42%, $1.2M, 3,450 users, 72.3 points, etc.
patterns = [
r'(\d{1,3}(?:,\d{3})*(?:\.\d+)?)\s*(%)', # Percentages
r'\$(\d{1,3}(?:,\d{3})*(?:\.\d+)?(?:[KMB])?)', # Dollar amounts
r'(\d{1,3}(?:,\d{3})*)\s*(users|customers|items|requests|errors)',
r'(\d+(?:\.\d+)?)\s*(seconds|minutes|hours|days|ms)',
]
for pattern in patterns:
for match in re.finditer(pattern, text, re.IGNORECASE):
# Get surrounding context (50 chars each side)
start = max(0, match.start() - 50)
end = min(len(text), match.end() + 50)
context = text[start:end].strip()
try:
raw_value = match.group(1).replace(',', '').replace('K', 'e3').replace('M', 'e6').replace('B', 'e9')
value = float(eval(raw_value))
unit = match.group(2) if len(match.groups()) > 1 else ""
claims.append(NumericClaim(
raw_text=match.group(0),
value=value,
unit=unit,
context=context
))
except Exception:
continue
return claims
def verify_numeric_claims(claims: list[NumericClaim], source_data: dict) -> list[NumericClaim]:
"""
Cross-reference extracted numbers against source data.
Mark each claim as verified or suspicious.
"""
source_values = set()
def collect_values(obj, path=""):
if isinstance(obj, (int, float)):
source_values.add(float(obj))
elif isinstance(obj, dict):
for k, v in obj.items():
collect_values(v, f"{path}.{k}")
elif isinstance(obj, list):
for i, v in enumerate(obj):
collect_values(v, f"{path}[{i}]")
collect_values(source_data)
for claim in claims:
# Check if value or common derived forms (percentage, thousands) exist in source
derivations = {
claim.value,
claim.value / 100, # percentage as decimal
claim.value * 100, # decimal as percentage
claim.value * 1000, # K units
claim.value * 1_000_000 # M units
}
if any(abs(d - sv) / max(abs(sv), 1) < 0.01 for d in derivations for sv in source_values):
claim.verified = True
claim.source = "source_data"
else:
claim.verified = False
return claims
def report_with_verification(response: str, source_data: dict) -> dict:
claims = extract_numeric_claims(response)
verified_claims = verify_numeric_claims(claims, source_data)
unverified = [c for c in verified_claims if not c.verified]
return {
"response": response,
"total_numeric_claims": len(claims),
"verified": len([c for c in verified_claims if c.verified]),
"unverified": len(unverified),
"unverified_claims": [
{"value": c.raw_text, "context": c.context}
for c in unverified
],
"warning": f"{len(unverified)} numeric claims could not be verified against source data" if unverified else None
}
Option 3: Tool-based arithmetic — offload calculations from the model
import anthropic
import json
client = anthropic.Anthropic()
# Define calculation tools so the model doesn't do arithmetic in its head
MATH_TOOLS = [
{
"name": "calculate",
"description": "Perform arithmetic calculations. Use this for ALL numeric calculations instead of computing in your head.",
"input_schema": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "Python arithmetic expression to evaluate. E.g., '340 / 12450 * 100'"
},
"description": {
"type": "string",
"description": "What this calculation represents"
}
},
"required": ["expression", "description"]
}
},
{
"name": "lookup_value",
"description": "Look up a specific value from the provided dataset by field name.",
"input_schema": {
"type": "object",
"properties": {
"field_name": {"type": "string", "description": "The field to look up"}
},
"required": ["field_name"]
}
}
]
def safe_eval(expression: str) -> float:
"""Safely evaluate arithmetic expression"""
# Only allow numbers and operators
allowed = set('0123456789+-*/.() ')
if not all(c in allowed for c in expression):
raise ValueError(f"Unsafe expression: {expression}")
result = eval(expression)
return round(float(result), 6)
def run_math_agent(question: str, data: dict) -> str:
"""
Run agent with math tools — all arithmetic done in tools, not in model head.
Prevents arithmetic hallucinations.
"""
system = f"""You are a data analyst. The available data is:
{json.dumps(data, indent=2)}
CRITICAL: Use the 'calculate' tool for ALL arithmetic. NEVER compute numbers in your response text.
Use 'lookup_value' to retrieve any value from the data before using it in calculations."""
messages = [{"role": "user", "content": question}]
while True:
response = client.messages.create(
model="claude-sonnet-4-6",
system=system,
messages=messages,
tools=MATH_TOOLS,
max_tokens=2048
)
if response.stop_reason == "end_turn":
return next(b.text for b in response.content if b.type == "text")
# Process tool calls
tool_results = []
for block in response.content:
if block.type != "tool_use":
continue
if block.name == "calculate":
try:
result = safe_eval(block.input["expression"])
tool_result = f"{block.input['description']}: {result}"
except Exception as e:
tool_result = f"Calculation error: {e}"
elif block.name == "lookup_value":
field = block.input["field_name"]
value = data.get(field, f"Field '{field}' not found in data")
tool_result = str(value)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": tool_result
})
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": tool_results})
Option 4: Structured output with mandatory source citations
from pydantic import BaseModel, validator, Field
from typing import Literal
import anthropic
class NumericFinding(BaseModel):
"""A single numeric claim with mandatory source citation"""
metric_name: str
value: float
unit: str # "%", "USD", "users", etc.
source_field: str # Exact field name from provided data, or "calculated"
calculation: str | None = None # If calculated, show the formula
confidence: Literal["exact", "calculated", "estimated", "unavailable"]
@validator("confidence")
def no_estimates_without_data(cls, v, values):
if v == "estimated":
raise ValueError(
"Estimated values are not allowed. Use 'unavailable' if data is missing."
)
return v
class DataAnalysisReport(BaseModel):
"""Structured report with all numeric claims requiring source citations"""
summary: str
findings: list[NumericFinding]
data_gaps: list[str] = Field(
default_factory=list,
description="List of metrics that were requested but not available in source data"
)
client = anthropic.Anthropic()
def generate_verified_report(question: str, data: dict) -> DataAnalysisReport:
"""Generate report with structured numeric claims — all must cite sources"""
import json
response = client.messages.create(
model="claude-sonnet-4-6",
system=f"""You are a data analyst. Source data:
{json.dumps(data, indent=2)}
Generate a structured analysis. For every numeric finding:
- Set source_field to the exact field name from source data
- If calculated from multiple fields, set source_field to "calculated" and show the formula
- If data is not available, set confidence to "unavailable" and omit the value
- NEVER set confidence to "estimated" — only use exact data or derived calculations""",
messages=[{"role": "user", "content": question}],
tools=[{
"name": "submit_analysis",
"description": "Submit the structured data analysis",
"input_schema": DataAnalysisReport.schema()
}],
tool_choice={"type": "tool", "name": "submit_analysis"},
max_tokens=2048
)
tool_call = next(b for b in response.content if b.type == "tool_use")
return DataAnalysisReport(**tool_call.input)
Option 5: System prompt with explicit anti-hallucination numeric rules
System prompt:
"You are a data analyst. Apply these rules to ALL numeric claims:
MANDATORY RULES — no exceptions:
1. SOURCE REQUIREMENT: Every number in your response must come from:
a. Data explicitly provided in this conversation, OR
b. A calculation you show step-by-step from provided data
2. FORBIDDEN: Never state a number you don't have a source for.
Wrong: '42% of users prefer mobile'
Right: 'Mobile preference data was not provided — I cannot report this metric'
3. CALCULATIONS: Show all arithmetic inline.
Wrong: 'The churn rate is 4.7%'
Right: 'Churn rate = churned_users / total_users = 340 / 7,250 = 4.69%'
4. UNCERTAINTY: If you are unsure of a number, say so explicitly.
Wrong: 'Revenue is approximately $2M'
Right: 'Revenue data not provided. Cannot estimate without source data.'
5. STATISTICS: Never cite studies, surveys, or research unless the source text
was provided in this conversation.
Wrong: 'Studies show 68% of AI agents...'
Right: 'No research was provided — I cannot cite statistics on this topic'
6. ROUNDING: State when you are rounding: '4.69% (rounded to 4.7%)'"
Option 6: Post-processing validator — flag suspicious numbers
import re
import math
def validate_numeric_plausibility(
response: str,
source_data: dict,
context: str = ""
) -> dict:
"""
Heuristic validator for numeric plausibility.
Flags numbers that look suspicious compared to source data.
"""
warnings = []
# Extract all numbers from response
numbers_in_response = [
float(n.replace(',', ''))
for n in re.findall(r'\b\d{1,3}(?:,\d{3})*(?:\.\d+)?\b', response)
if float(n.replace(',', '')) > 0
]
# Extract all numbers from source data
source_numbers = []
def extract_numbers(obj):
if isinstance(obj, (int, float)) and obj > 0:
source_numbers.append(float(obj))
elif isinstance(obj, dict):
for v in obj.values():
extract_numbers(v)
elif isinstance(obj, list):
for v in obj:
extract_numbers(v)
extract_numbers(source_data)
if not source_numbers:
return {"warnings": ["No source numbers to validate against"], "suspicious": False}
max_source = max(source_numbers)
min_source = min(source_numbers)
for num in numbers_in_response:
# Flag numbers more than 100x larger than any source number
if num > max_source * 100 and num > 1000:
warnings.append(
f"Number {num:,.0f} is much larger than any source value "
f"(max source: {max_source:,.0f})"
)
# Flag percentages > 100
if num > 100 and any(p in response[max(0,response.find(str(int(num)))-10):response.find(str(int(num)))+10]
for p in ['%', 'percent', 'pct']):
warnings.append(f"Percentage {num}% exceeds 100% — likely an error")
# Flag negative values where only positive source values exist
if num < 0 and all(s > 0 for s in source_numbers):
warnings.append(f"Negative value {num} found but all source data is positive")
return {
"numbers_in_response": len(numbers_in_response),
"warnings": warnings,
"suspicious": len(warnings) > 0
}
Numeric Hallucination Risk by Task Type
| Task | Hallucination Risk | Mitigation |
|---|---|---|
| Summarizing provided data | Low | Grounding prompt + source citations |
| Calculating from provided data | Medium | Use calculator tool, show formulas |
| Estimating without data | High | Block: require data or say “unavailable” |
| Citing external statistics | Very High | Block unless source text provided |
| Multi-step arithmetic (>3 steps) | High | Offload to calculator tool |
| Trend analysis | Medium | Require explicit time-series data |
Expected Token Savings
Report with fabricated statistics → stakeholder flags numbers → re-run with corrections → verification round: ~25,000 tokens Grounded report with verified numbers → accepted on first review: 0 rework tokens
Environment
- Any agent producing reports, dashboards, or data analysis; critical for business intelligence agents, financial reporting, and any context where numeric accuracy is mission-critical
- Source: direct experience; numeric hallucinations are the most dangerous class of hallucination because they appear precise and authoritative while being completely fabricated
Wasting tokens on this error?
Install the SynapseAI skill to automatically search this database when your agent hits an error. Average savings: $2–5 per error incident.
clawhub install synapse-ai
Solved an error that's not here?
Share it and earn MoltCoin rewards.