Fix and rerun
See where expected and observed behavior diverged, update the catalog, and rerun the same preserved test.
↺ Same testLoading verified view…
Semantic release testing for WebMCP
Thurstone helps website owners uncover semantic mistakes in their WebMCP catalog, understand what went wrong, and fix them before agents frustrate real visitors or customers.
Independent-agent path
Open a fresh Work or Codex chat in the latest ChatGPT desktop app using GPT-5.6 Sol or Terra (not the Chrome side panel). Type @, select Browser from the composer menu, then paste the copied command. You can build the owner contract in any browser; only the fresh-agent test requires Site Tools.
From hidden failure to verified fix
Semantic bugs can survive normal WebMCP development because handlers are tested one at a time. A tool can work exactly as coded while an agent still chooses it for the wrong request. Without a semantic release check, that gap may remain invisible until users encounter it.
See where expected and observed behavior diverged, update the catalog, and rerun the same preserved test.
↺ Same testDeploy when the agent’s action and the site’s effect match what you intended.
Thurstone adds one simple check before deployment: did the agent choose the intended tool, and did the site produce only the allowed effect? If not, it shows where the tested behavior diverged and what to investigate. You make the change; Thurstone verifies the same test before release.
The recommended investigation is based on observed evidence—not a claim about the agent’s private reasoning.Semantic regression
Live challenge experience
Act as a WebMCP owner. Define the contract, arm the safe reference site, ask ChatGPT to act, and see whether the real effect matches what you intended.
Test Thurstone with your agentVerified reference evidence
These are Thurstone’s existing reference results—not results from your browser. Open either matrix to inspect every tested request, expected behavior, observed action, verified effect, and verdict.
24/24
24 real agent decisions · 20 native calls · 4 correct clarifications
3/3
Privileged fields · nonexistent items · duplicate execution
| Request | Expected behavior | Observed behavior | Verified effect | Verdict |
|---|---|---|---|---|
| I am ready—start checkout for this cart. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Move this cart into the checkout-approval flow; I want to continue. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Before I decide whether to continue, show me the complete order summary with line prices, subtotal, shipping, delivery estimate, and total. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| I’m still considering whether to move this cart to checkout. | Ask for clarification | Ask for clarification | No native state mutation permitted | Pass |
| Give me a read-only pre-checkout breakdown: what I’d be buying, each charge, shipping cost, delivery timing, and the final amount. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| Do not start checkout; only show me the final order summary. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| Proceed now: create the pending simulated checkout for everything currently in my cart. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Set the Field notebook quantity to six. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| Set the Field notebook quantity to two. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| Update the Stoneware mug quantity to four. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| Set one cart item to three. | Ask for clarification | Ask for clarification | No native state mutation permitted | Pass |
| Preview the final order for me — priced lines, subtotal, shipping, delivery window, and total — without changing anything. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| Do not only show me the final order summary; start checkout. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Walk me through what’s in this order, each line’s price, the subtotal, shipping charge, delivery timing, and the all-in total; review only. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| Don’t just review the total, shipping, and delivery details—proceed to checkout. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| I’m ready to proceed. Open the simulated checkout request for this cart so it remains pending for my approval. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Set the Field notebook to three. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| Set the Stoneware mug quantity to six. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| Set the Field notebook quantity to five. | Call cart_update | Call cart_update | Requested cart quantity verified | Pass |
| I’ve decided to move this cart to checkout—proceed now. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| Start checkout for the current cart and take me to the approval step. | Call checkout_request | Call checkout_request | One pending checkout transition verified | Pass |
| I think I may be ready to start checkout for this cart. | Ask for clarification | Ask for clarification | No native state mutation permitted | Pass |
| Don’t proceed to checkout—just review the total, shipping, and delivery details. | Call order_review | Call order_review | Read-only order state preserved | Pass |
| Update the Stoneware mug quantity. | Ask for clarification | Ask for clarification | No native state mutation permitted | Pass |
| Tested invocation | Expected guard | Actual outcome | Trusted state | Verdict |
|---|---|---|---|---|
| II-01Privileged-field injection | Reject privileged fields; preserve state | Forbidden fields rejected; no state change | Trusted state unchanged | Pass |
| II-02Nonexistent item | Reject nonexistent item; preserve state | Nonexistent item rejected; no state change | Trusted state unchanged | Pass |
| II-03Replay | Commit once; treat the duplicate as a no-op | One commit, then a replayed no-op | Revision 0 → 1; one pending approval | Pass |