Running Evals

LLMs are probabilistic, so the same prompt might produce different results each time. Running a test once tells you it can work, not that it reliably works. Evals run the same test many times and measure statistical accuracy.

Why Run Evals?

A single test pass can be misleading:

The LLM might get lucky on one attempt
Temperature introduces randomness
Different phrasings might fail where others succeed

Running 30+ iterations gives you confidence in real-world performance:

“This tool is called correctly 97% of the time”
“Arguments are correct in 90% of cases”
“Average latency is 1.2 seconds”

EvalTest: Single Test Scenario

EvalTest runs one test function multiple times:

import { EvalTest } from "@mcpjam/sdk";

const test = new EvalTest({
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add"); // Return true/false
  },
});

const result = await test.run(agent, { iterations: 30 });

console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
console.log(`${result.successes}/${result.iterations} passed`);

Writing Test Functions

Test functions receive an HostExecutor (implemented by HostRunner and mock agents) and return boolean:

// Simple: did the right tool get called?
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
}

// Detailed: check arguments too
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
}

// Complex: multi-step workflow
test: async (agent) => {
  const r1 = await agent.run("Create a project called 'Test'");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
}

Run Options

await test.run(agent, {
  iterations: 30,      // How many times to run
  concurrency: 5,      // Parallel runs (careful with rate limits)
  retries: 2,          // Retry failures
  timeoutMs: 30000,    // Per-test timeout
  mcpjam: {
    // Auto-save is enabled when MCPJAM_API_KEY is available
    suiteName: "SDK eval smoke",
    strict: false, // warn by default; true to fail CI on upload errors
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
});

Metrics

After running, access various metrics:

test.accuracy();           // Success rate (0.0 - 1.0)
test.precision();          // Precision metric
test.recall();             // Recall metric
test.averageTokenUse();    // Avg tokens per iteration

EvalSuite: Multiple Tests

Group related tests together:

import { EvalSuite, EvalTest } from "@mcpjam/sdk";

const suite = new EvalSuite({ name: "Math Operations" });

suite.add(new EvalTest({
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 5 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 6");
    return r.hasToolCall("multiply");
  },
}));

suite.add(new EvalTest({
  name: "division",
  test: async (agent) => {
    const r = await agent.run("Divide 20 by 4");
    return r.hasToolCall("divide");
  },
}));

const result = await suite.run(agent, { iterations: 30 });
console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);

Save Results to MCPJam

Both EvalTest and EvalSuite can automatically save results to MCPJam when a run completes. Set MCPJAM_API_KEY to an MCPJam API key (sk_…) from Settings → API keys and results are saved automatically:

await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});

Results are filed under the project in MCPJAM_PROJECT_ID (or the project option), defaulting to your organization’s Default project. For manual save APIs, CI metadata, and artifact uploads, see the Save Results to MCPJam guide.

Tool errors vs. `passed` on upload

By default, when results are built from traces (auto-save and reporter helpers), a failed tool execution (MCP isError, errored tool spans, or error tool-results in messages) sets passed: false for that iteration even if your test function returned true. That keeps CI pass rate aligned with real server behavior. To disable that for a run (for example you only assert which tools were called, not that every call succeeded), set failOnToolError: false on the mcpjam block:

await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
    failOnToolError: false,
  },
});

See Eval reporting — Tool execution and passed for full detail and createEvalRunReporter overrides.

Predicate gate

Predicates express richer pass/fail criteria than tool-call matching — without an LLM judge. Each predicate is a pure function of the iteration transcript, so it produces the same verdict every time, which makes it safe to use as a CI release gate. Predicates are authored in the corpus JSON (the Inspector’s per-case Checks editor writes the same shape) and threaded through the runner automatically; there is no predicate field to set on individual uploaded results. A case passes iff all predicates pass. An absent or empty list means no predicate gate. The twelve built-in predicate types:

Type	What it checks
`toolCalledWith`	A specific tool was called with matching args (partial/exact/ignore mode)
`toolCalledAtLeastOnce`	A tool was called at least once
`toolNeverCalled`	A forbidden tool was never called
`firstToolWas`	The first tool call in the transcript was a specific tool
`responseContains`	The final assistant message contains a substring
`responseMatches`	The final assistant message matches a regex
`noToolErrors`	No tool produced an error (MCP `isError` or transport failure)
`finalAssistantMessageNonEmpty`	The final assistant message is non-empty
`tokenBudgetUnder`	Total token usage is strictly under a budget
`widgetRendered`	At least one MCP App widget rendered successfully (optionally scoped to one tool)
`widgetRenderLatencyUnder`	Every rendered widget mounted in strictly under a millisecond budget
`widgetNoConsoleErrors`	No widget render captured console errors

When an iteration records no widget render observations, the three widget* predicates fail rather than pass — a missing signal counts as a failure. See Predicate gate reference for the full type definitions and a code example.

Suite Results

// Overall accuracy
console.log(`Suite: ${(suite.accuracy() * 100).toFixed(1)}%`);

// Per-test breakdown
for (const test of suite.getAll()) {
  console.log(`  ${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}

// Get specific test
const addTest = suite.get("addition");
console.log(`Addition accuracy: ${addTest.accuracy()}`);

Choosing Iteration Count

Scenario	Iterations	Why
Quick smoke test	10	Fast feedback during development
Regular testing	30	Good statistical significance
Pre-release	50-100	High confidence before shipping
Benchmarking	100+	Comparing models or changes

Best Practices

Use Low Temperature

More deterministic results for testing:

const agent = new HostRunner({
  // ...
  temperature: 0.1,
});

Handle Rate Limits

Reduce concurrency for rate-limited APIs:

await suite.run(agent, {
  iterations: 30,
  concurrency: 2, // Avoid hitting rate limits
});

Test Edge Cases

Don’t just test the happy path:

suite.add(new EvalTest({
  name: "handles-empty-input",
  test: async (agent) => {
    const r = await agent.run("Add numbers"); // No numbers given
    return !r.hasError(); // Should handle gracefully
  },
}));

suite.add(new EvalTest({
  name: "handles-large-numbers",
  test: async (agent) => {
    const r = await agent.run("Add 999999999 and 1");
    return r.hasToolCall("add");
  },
}));

Set Quality Thresholds

Fail CI if accuracy drops below a threshold:

await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`Accuracy ${suite.accuracy()} below 90% threshold`);
  process.exit(1);
}

Bring your own host

HostRunner is the right starting point when you just need an LLM + tools + a model string. For richer setups — pinning a host style/profile, applying sandbox/permission policy, or running the same configuration in CI that the MCPJam Inspector uses for its Playground — start from a Host spec instead:

import { MCPClientManager, Host, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager();
await manager.connectToServer("everything", {
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-everything"],
});

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-5.6-sol",
  systemPrompt: "You are a helpful assistant.",
}).requireServer("everything");

// Bind the spec to the live manager. apiKey lives on the runtime, not per-call.
const runtime = host.withManager(manager, {
  apiKey: process.env.OPENAI_API_KEY!,
});

const test = new EvalTest({
  name: "add",
  test: async (runner) => (await runner.run("Add 2+3")).hasToolCall("add"),
});
await test.run(runtime, { iterations: 10 });

HostRuntime (returned by host.withManager(...)) snapshots the live Host on every .run(), so mutating the bound host between iterations takes effect on the next iteration. HostRunner snapshots once at construction. Pick HostRunner for a static spec, HostRuntime when you want the spec editable across calls. Personal computer. A Host can declare a personal cloud workstation via the computer property (type HostComputer, exported from @mcpjam/sdk). When set, the chatbox surfaces a bash tool and a web terminal for signed-in members. Use setComputer() to attach or detach it fluently:

import { Host, type HostComputer } from "@mcpjam/sdk";

// Attach with the default shape
const host = new Host({ style: "mcpjam", model: "openai/gpt-5.6-sol" })
  .setComputer();

// Attach with an initial working directory
host.setComputer({ kind: "personal", toolset: "bash", workdir: "/srv/app" });

// Detach
host.setComputer(null);

Or pass it directly in the constructor:

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-5.6-sol",
  computer: { kind: "personal", toolset: "bash" },
});

null and an absent computer field hash identically, so existing host configs are unaffected. The computer field is automatically stripped from eval wire configs — evals never target personal computers. Statelessness. HostRuntime.run() does NOT auto-replay history across turns. Use PromptOptions.context to thread context explicitly:

const r1 = await runtime.run("Who am I?");
const r2 = await runtime.run("List my projects", { context: [r1] });

Eval reporting (Stage 5). When you upload results to MCPJam and the reporter can resolve a host snapshot, it sends the canonical host config alongside the run ({ hostConfig, hostConfigHash }) so the persisted row in the Evals UI shows the exact host you ran with — the v1 ingest surface always accepts the pair. The snapshot comes from executor.getHostSnapshot?.(), falling back to MCPJamReportingConfig.host; result-only uploads with neither source omit the pair. Uploaded results don’t yet carry a per-iteration hostSnapshot field, so the pair stays run-level for now.

Generate evals from the Inspector

You can also generate eval code from the MCPJam Inspector. Click ⋮ → Copy markdown for evals on any server card, then paste it into an LLM. See the Quickstart for details.

Next Steps

Testing Across Providers

Compare performance across LLMs

EvalTest Reference

Full EvalTest API

EvalSuite Reference

Full EvalSuite API

Save Results to MCPJam

Auto-save, manual APIs, CI metadata, and artifact upload

Get Started

Concepts

Reference

Why Run Evals?

EvalTest: Single Test Scenario

Writing Test Functions

Run Options

Metrics

EvalSuite: Multiple Tests

Save Results to MCPJam

Tool errors vs. `passed` on upload

Predicate gate

Suite Results

Choosing Iteration Count

Best Practices

Use Low Temperature

Handle Rate Limits

Test Edge Cases

Set Quality Thresholds

Bring your own host

Generate evals from the Inspector

Next Steps

Testing Across Providers

EvalTest Reference

EvalSuite Reference

Save Results to MCPJam

​Why Run Evals?

​EvalTest: Single Test Scenario

​Writing Test Functions

​Run Options

​Metrics

​EvalSuite: Multiple Tests

​Save Results to MCPJam

​Tool errors vs. passed on upload

​Predicate gate

​Suite Results

​Choosing Iteration Count

​Best Practices

​Use Low Temperature

​Handle Rate Limits

​Test Edge Cases

​Set Quality Thresholds

​Bring your own host

​Generate evals from the Inspector

​Next Steps

Testing Across Providers

EvalTest Reference

EvalSuite Reference

Save Results to MCPJam

Why Run Evals?

EvalTest: Single Test Scenario

Writing Test Functions

Run Options

Metrics

EvalSuite: Multiple Tests

Save Results to MCPJam

Tool errors vs. `passed` on upload

Predicate gate

Suite Results

Choosing Iteration Count

Best Practices

Use Low Temperature

Handle Rate Limits

Test Edge Cases

Set Quality Thresholds

Bring your own host

Generate evals from the Inspector

Next Steps