Test suites and checks

A test suite is an ordered list of test cases you run as one test run. Each case can carry checks: rules about what the agent did, read from the run itself rather than from what the agent said. Use both to test an agent the same way every time, for example on every new version.

Run a suite

You start the same run from any of the three places, and each runs the suite's cases as they are now, in the suite's order.

• Web: open Testing → Tests, choose Run Tests, and pick the suite.
• CLI: masaic test run --suite <suite-id> --wait exits with an error unless the run passed. masaic test suites lists the suites.
• MCP: call run-test-suite with test_suite_id, then read the result with get-test-run-result. list-test-suites lists the suites.

Create or change a suite with masaic test suite create -f suite.yaml (or update), or over MCP with create-test-suite and update-test-suite. A suite file needs a name and the ordered testCaseIds; update also needs the suite's id.

Choose which agent version answers

Each test case names an agent and an environment, and by default it talks to the version deployed to that environment. A case can also be pinned to one version in its form; leave the version on Auto unless the case must test that version only. A run can override the version, though not the environment, for every case at once:

• Web: the version choice in the run dialog, once the selected cases belong to one agent. Configured on test case keeps each case's pin, Latest active version ignores pins and uses what is deployed to each case's environment, and a listed version (a draft too) answers every conversation.
• CLI: --agent-version configured, latest or <agent-version-id>.
• MCP: agent_version_target (configured, latest or specific) with agent_version_id for specific.

To test a draft before it goes live, choose that version itself: latest means the deployed version, not the newest one.

Every result names the version that answered each conversation. That is the Agent version in the conversation's details on the run page, agentVersionId in masaic test result, and agent_version_id in get-test-run-result.

Checks

A check passes or fails on its own, beside the AI judge. A case passes only when every check that applied passed. A check that cannot apply to a run is skipped with a reason. When checks are the only grading, a case whose checks were all skipped is reported as a technical error, not as a pass.

The workspace checks read only this workspace: an object of the same name in another workspace does not count. The tool checks know the tools of the agent version that answered, also when the agent is provided from another workspace.

Names unique per run: {{run}}

A case that asks the agent to create something would meet "already exists" on its second run. Write {{run}} into the name instead, for example in the case's customer issue ("create a prompt named qa-{{run}}"), its expected outcome and its checks. Each run replaces {{run}} with a short token of its own, and each further conversation of the same case adds -2, -3 and so on, so every conversation creates, and checks for, a new name. A golden conversation's turns get the same name.

Mocked tool answers

A test case can give a tool of the agent a fixed answer. In the case's test conversations that tool answers with what the case says, instead of calling its MCP server; every other tool calls its real server. Use it to give the agent data that exists nowhere else, so the expected outcome and the checks can name it: "list-agents returns qa-mocked-agent-7f3 — the agent must report exactly that".

• Web: in the test case form, Mocked tool answers → Add mocked tool: choose the MCP server and the tool, and write the answer as JSON (an object, a list or a string).
• CLI: mockedToolAnswers in the file for masaic test create -f, a list of mcpServerId, toolName and answer.
• MCP: mocked_tool_answers on create-test-case and update-test-case.

Each answer must name a tool the server lists, in this workspace, and only a tool that reads can be mocked. The answers belong to the case's own mock user: when you give a case mocked answers, it gets a copy of the mock user it plays (the same person to the agent), linked in its place, and the answers become that mock user's MCP fixtures. Other cases that play the original mock user are not mocked along. So a case with mocked answers has at most one mock user.

The copy stays out of sight: the case's editor shows and offers the mock user it copies, and no list of mock users, picker or fixture dialog shows it, so no other case can play it. Moving the case to another environment moves its answers along, checked there; prod is refused. Clearing the answers removes the copy's fixtures.

The answers apply where the server's target in the case's environment asks for a fixture: a pass-through mock (the server keeps its real endpoint; a conversation that plays a mock user with a fixture for the tool gets the fixture, every other call reaches the real server), or a plain mock. Set it on the server's MCP targets by environment (web), with set-mcp-server-target and use_mock plus mock_pass_through (MCP), or masaic mcp-target set <server> --env dev --mock --pass-through --endpoint <url> (CLI). Saving answers for a server that is not mocked in the case's environment is refused, naming the target to change. Never in a live environment, and not on an installation that runs no mock server (production): there a pass-through target simply reaches the real server. The same fixtures answer a chat with that mock user in the playground: nothing about it is specific to tests.

A result marks every call a fixture answered (mocked answer in the conversation's tools on the run page, mocked: true in the tool calls of masaic test result and get-test-run-result).

How a mock answer behaves

A mock answer can do more than answer at once. Each of an MCP server's mock answers can also:

• wait before it answers, with a random spread around the wait, to test how an agent copes with a slow tool. A wait longer than the tool's time limit (60, 120 or 300 seconds by runtime profile) makes the call time out, which is how a test checks the agent's timeout handling;
• fail on purpose with an error of your choosing, for every call or for a share of them, optionally with a code and a "retry after" (a rate limit). The agent receives it as a tool error;
• answer by the call's arguments: rules such as "when from equals CGN and passengers is at least 3, answer this". The first rule whose conditions all hold answers, with its own answer and settings. When none matches, the answer itself applies, or the call fails and names the closest rule, so you see which condition did not hold. Conditions compare (equals, not equals, contains, starts with, one of, exists, absent, greater or less than); there are no regular expressions;
• fill in placeholders: {{args.from}}, {{now}}, {{today}} and {{uuid}} in the answer's text are filled in per call, so one answer serves many calls;
• come back malformed (cut-off JSON, an HTML error page, or empty) or oversized, to test how an agent handles a broken tool.

Set it in the mock answer's dialog under How it answers (web), with behaviour on create-mcp-mock-fixture and update-mcp-mock-fixture (MCP), or with masaic mcp-fixture create|update and flags such as --delay 1500 --jitter 500 --error "too many requests" --error-rate 0.2, or --behaviour @rules.json for rules (CLI). The table of mock answers sums up each one's settings. A trace marks how long a pass-through mock waited and which rule answered.

The Masaic Agent's suites

The masaic workspace has two suites for the Masaic Agent itself:

• MASAIC Agent smoke: six read-only inventories, three building checks and one inventory checked against mocked tool answers, in a few minutes. Run it for every new version of the agent.
• MASAIC Agent full suite: every authoring and discovery case, the cases with mocked tool answers, and the case in which the Masaic Agent runs a test and reports how it ended.

masaic test run --suite <smoke-suite-id> --wait

Related

• Testing and release — where tests fit before a version goes live.
• Test personas — the simulated callers that drive a conversation.
• Mock users — repeatable identities the agent fetches data for.