Library

Testing Utilities

Test AI agents like regular code. Record responses, mock LLM calls, and run deterministic tests in CI/CD.

The Challenge

AI systems are non-deterministic. The same prompt returns different responses each time. Your tests pass locally and fail in CI. You can't write reliable assertions when the output keeps changing.

Running actual LLM calls in tests is slow and expensive. But without them, you're not really testing the integration. You need a way to capture real LLM behavior and replay it consistently.

The FireFoundry Way

Record actual LLM responses during development, then replay them in tests. Get deterministic tests that reflect real model behavior. When you want to evaluate actual outputs, use AI-powered assertions that judge semantic correctness instead of exact matches.

Record and replay LLM responses:

import { test, createRecorder } from '@firefoundry/agent-sdk/testing';
import { SupportBot } from './support-bot';

// Record mode: captures actual LLM responses
test('support bot handles refund requests', async () => {
  const recorder = createRecorder('support-bot-refunds');

  const bot = new SupportBot({ recorder });
  const response = await bot.respond('I want a refund for order #123');

  expect(response).toContain('refund');
  expect(response).toContain('123');
});

// Replay mode: uses recorded responses (deterministic)
test('support bot handles refund requests', async () => {
  const recorder = createRecorder('support-bot-refunds', { mode: 'replay' });

  const bot = new SupportBot({ recorder });
  const response = await bot.respond('I want a refund for order #123');

  // Same response every time - safe for CI
  expect(response).toMatchSnapshot();
});

AI-powered semantic assertions:

import { test, expectSemantic, expectJson } from '@firefoundry/agent-sdk/testing';

test('research bot provides accurate citations', async () => {
  const bot = new ResearchBot();
  const response = await bot.research('What causes climate change?');

  // Semantic assertion - checks meaning, not exact words
  await expectSemantic(response)
    .toMention('greenhouse gases')
    .toMention('carbon dioxide')
    .toBeFactuallyAccurate()
    .toIncludeSources();

  // Structured data assertion
  await expectJson(response.metadata)
    .toHaveProperty('sources')
    .where('sources', sources => sources.length >= 2);
});

Mock entire model groups for unit tests:

import { test, mockModelGroup } from '@firefoundry/agent-sdk/testing';

test('bot handles API errors gracefully', async () => {
  // Mock the model group to simulate failures
  mockModelGroup('fast-chat', {
    responses: [
      { error: 'rate_limit_exceeded' },
      { error: 'rate_limit_exceeded' },
      { content: 'Here is your answer...' }  // Third retry succeeds
    ]
  });

  const bot = new SupportBot();
  const response = await bot.respond('Help me');

  // Bot should have retried and eventually succeeded
  expect(response).toBeDefined();
  expect(bot.metrics.retries).toBe(2);
});

Record & Replay

Capture LLM responses during development, replay them in CI. Get deterministic tests without mocking everything.

Semantic Assertions

Assert on meaning, not exact text. Check if responses are accurate, relevant, and well-structured using AI evaluation.

Model Mocking

Mock entire model groups to test error handling, retries, and edge cases without real API calls.

Fast Execution

Replayed tests run in milliseconds. No API calls, no rate limits, no flaky tests from network issues.

Testing Workflow

Develop

Record real LLM responses

Test

Replay for determinism

Evaluate

AI-powered assertions

Deploy

Confidence in CI/CD

Why It Matters

100%

Deterministic tests with replay mode

100x

Faster than live API tests

API cost for replayed tests

Learn More

Testing API Reference AgentSDK Overview Back to Developer Ecosystem