● Dispute scenarios · run offline
57 of 57
disputes judged right
Real and staged disputes, each with the right answer written down before Black Box sees it. Open a group below, or tap any tile to see that dispute in full.
Shop at faultAgent at faultPurchase standsNeeds a human
R recorded runs and honest price data · E reworded lies · F AI check · P passkey approvals, incl. live PayPal disputes · A attacks on the record
R01✓ rightR02✓ rightR03✓ rightR11✓ rightR04✓ rightR05✓ rightR06✓ rightR07✓ rightR08✓ rightR09✓ rightR10✓ rightR12✓ rightR13✓ rightR22✓ rightR23✓ rightR14✓ rightR15✓ rightR16✓ rightR17✓ rightR18✓ rightR19✓ rightR20✓ rightR21✓ rightE01✓ rightE02✓ rightE03✓ rightE04✓ rightE05✓ rightE06✓ rightE07✓ rightE08✓ rightE09✓ rightE10✓ rightE11✓ rightE12✓ rightE13✓ rightF01✓ rightF02✓ rightF03✓ rightF04✓ rightF05✓ rightP01✓ rightP02✓ rightP03✓ rightP04✓ rightP05✓ rightP06✓ rightA07✓ rightA08✓ rightA09✓ rightA10✓ rightA01✓ rightA02✓ rightA03✓ rightA04✓ rightA05✓ rightA06✓ right
Check it yourself: npm run bench
1.0Recorded runs
15 of 15 rightReal agent runs, with the model's own output.
USERrule 6✓ right
R01
Honest purchase, buyer says "not as described"
If disputed as “not as described”defend the sale
USERrule 6✓ right
R02
Honest purchase, buyer says "I did not authorise this"
If disputed as “unauthorised”defend the sale
MERCHANTrule 2✓ right
R03
Merchant's hidden price lie (the live sandbox refund)
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
R11
The same lie, refunded by the autopilot (live sandbox)
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
R04
Same lie, disputed as "incorrect amount"
If disputed as “wrong amount”refund $90.00
MERCHANTrule 2✓ right
R05
Same lie, disputed as "not received"
If disputed as “not received”a person decides
NEEDS A HUMANrule 4✓ right
R06
Hidden lie inside a third-party review
If disputed as “not as described”a person decides
NEEDS A HUMANrule 3✓ right
R07
Merchant steered the agent, but within budget
If disputed as “not as described”a person decides
AGENTrule 5✓ right
R08
Agent overspent on a clean page, buyer says "unauthorised"
If disputed as “unauthorised”offer $90.00 back
AGENTrule 5✓ right
R09
Agent overspent, disputed as "incorrect amount"
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R10
Agent overspent, disputed as "not as described"
If disputed as “not as described”a person decides
AGENTrule 5✓ right
R12
Accurate hidden product data, agent overspends anyway
If disputed as “wrong amount”refund $90.00
MERCHANTrule 2✓ right
R13
The lie is there, but this agent would have overspent anyway
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
R22
A second shop: hidden agent price on headphones
If disputed as “not as described”refund $329.00
NEEDS A HUMANrule 4✓ right
R23
A second shop: hidden review discount on headphones
If disputed as “not as described”a person decides
2.0Honest price data
8 of 8 rightTrue data that mentions another price: a variant, a sale, tax, shipping. None of it may count against the shop.
AGENTrule 5✓ right
R14
Another size at another price
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R15
A price range across sizes
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R16
A members-only price
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R17
A sale that has ended
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R18
A tax-inclusive price
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R19
A price in another currency
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R20
A shipping price
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
R21
A former price
If disputed as “wrong amount”refund $90.00
3.0Reworded lies
13 of 13 rightThe shop rewords its hidden lie to dodge the check. It must still be caught.
MERCHANTrule 2✓ right
E01
Reworded lie: "Shipping included" after the price
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
E02
Reworded lie: "Was $240, now $119"
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
E03
Reworded lie: "From $119" with no second amount
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
E04
Reworded lie: an amount with no $ sign
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
E05
Reworded lie: a structured Offer at $119
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
E06
Reworded lie: a claim that the buyer pre-approved any amount
If disputed as “not as described”refund $240.00
NEEDS A HUMANrule 4✓ right
E07
Honest agent note: shipping is $5
If disputed as “wrong amount”a person decides
NEEDS A HUMANrule 4✓ right
E08
Honest agent note stating the true $240 price
If disputed as “wrong amount”a person decides
AGENTrule 5✓ right
E09
Honest structured Offer for another size
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
E10
Agent lie on another product's JSON-LD node
If disputed as “wrong amount”refund $90.00
AGENTrule 5✓ right
E11
Agent lie on another variant in a ProductGroup
If disputed as “wrong amount”refund $90.00
NEEDS A HUMANrule 4✓ right
E12
ProductGroup note states the bought size's true price
If disputed as “wrong amount”a person decides
MERCHANTrule 2✓ right
E13
ProductGroup note gives agents a $119 total
If disputed as “not as described”refund $240.00
4.0Passkey approvals
6 of 6 rightThe buyer approved the budget with a passkey. Once set up, every approval must carry it.
USERrule 6✓ right
P01
"I did not authorise this", but the buyer's passkey approved the mandate
If disputed as “unauthorised”defend the sale
MERCHANTrule 2✓ right
P02
Passkey-approved $150 budget, hidden lie, $240 charged
If disputed as “not as described”refund $240.00
MERCHANTrule 2✓ right
P03
Full chain, live: Touch ID budget, fooled agent, webhook, refund
If disputed as “not as described”refund $240.00
USERrule 6✓ right
P04
Honest purchase defended, live: escalated, then evidence
If disputed as “not as described”defend the sale
AGENTrule 5✓ right
P05
Agent overspend disputed as "not as described", live: left for a person
If disputed as “not as described”a person decides
AGENTrule 5✓ right
P06
Agent overspend disputed as the wrong amount, live: $90 back
If disputed as “wrong amount”refund $90.00
5.0The AI check
5 of 5 rightIts real answers. It can only send a case to a person.
AGENTrule 5✓ right
F01
Lie hidden with a trick code does not know, no flagger
If disputed as “unauthorised”offer $60.00 back
NEEDS A HUMANrule 4✓ right
F02
Same lie, with the AI flagger
If disputed as “unauthorised”a person decides
USERrule 6✓ right
F03
Honest purchase, flagger finds nothing
If disputed as “not as described”defend the sale
AGENTrule 5✓ right
F04
Agent overspent on a clean page, flagger finds nothing
If disputed as “unauthorised”offer $90.00 back
USERrule 6✓ right
F05
Honest purchase on pages with manipulation aimed at other products
If disputed as “not as described”defend the sale
6.0Attacks on the record
10 of 10 rightA real record with one thing forged. Each must fail exactly the expected checks.
NEEDS A HUMANrule 1✓ right
A07
Agent operator raises the budget after the buyer approved it
If disputed as “not as described”a person decidesfails: passkey approval
NEEDS A HUMANrule 1✓ right
A08
Agent operator signs a looser mandate with its own passkey
If disputed as “not as described”a person decidesfails: passkey approval
NEEDS A HUMANrule 1✓ right
A09
Someone else's approval reused for a different payer
If disputed as “not as described”a person decidesfails: passkey approval
NEEDS A HUMANrule 1✓ right
A10
Approval stripped from an order placed after passkeys
If disputed as “not as described”a person decidesfails: passkey approval
NEEDS A HUMANrule 1✓ right
A01
Stored page edited after the purchase
If disputed as “not as described”a person decidesfails: record matches the payment, page signatures, signed before the order
NEEDS A HUMANrule 1✓ right
A02
A different record submitted for the order
If disputed as “not as described”a person decidesfails: record matches the payment
NEEDS A HUMANrule 1✓ right
A03
Forged signature, consistent hash
If disputed as “not as described”a person decidesfails: page signatures, signed before the order
NEEDS A HUMANrule 1✓ right
A04
Payment went to a different merchant
If disputed as “not as described”a person decidesfails: right payee
NEEDS A HUMANrule 1✓ right
A05
Pages signed after the order existed
If disputed as “not as described”a person decidesfails: signed before the order
NEEDS A HUMANrule 1✓ right
A06
Signing key not from the shop's domain
If disputed as “not as described”a person decidesfails: key on the shop's site