Building a Document Processing Agent

Document processing is one of the most requested enterprise AI use cases. Invoices, contracts, reports, emails -- organizations are drowning in unstructured documents that need to be read, understood, classified, and routed to the right people. Manual processing is slow and error-prone. Traditional automation handles only rigid, templated formats. AI agents can do better.

In this tutorial, we will build a document processing agent on FireFoundry that accepts uploaded documents, extracts text using the Document Processing Service, classifies the document type with a structured-output bot, extracts relevant data fields, and routes to the appropriate downstream workflow. By the end, you will have a production-ready pipeline for intelligent document handling.

What We Are Building

Our document processing agent will handle the complete lifecycle from upload to routing:

• Accept uploaded documents -- PDFs, Word files, and images (scanned documents, photos of receipts)
• Extract text using the Document Processing Service, which handles OCR for images and structured text extraction for PDFs
• Classify the document type -- invoice, contract, report, or correspondence -- using a bot with validated structured output
• Extract structured data based on the classification (e.g., vendor name, amount, and due date for invoices; parties and terms for contracts)
• Route to the appropriate workflow based on classification -- invoices go to accounting, contracts go to legal, and so on

This pattern is different from building a full document-to-report pipeline (which you can read about in our Document to Report tutorial). Here we focus specifically on the extraction-classification-routing pattern that forms the backbone of most enterprise document workflows.

Step 1: The Document Entity

Every FireFoundry agent starts with the Entity Graph -- the persistent domain model that tracks the lifecycle of your business objects. For document processing, we need an entity that represents a document as it moves through our pipeline.

import { EntityMixin, RunnableEntity } from '@firebrandanalytics/ff-agent-sdk';

interface DocumentDTOData {
  filename: string;
  mimeType: string;
  classification: string;
  extractedData: Record<string, any>;
  status: 'uploaded' | 'processing' | 'classified' | 'extracted' | 'routed';
  confidence: number;
  routedTo: string;
  extractedText: string;
}

@EntityMixin({ specificType: 'DocumentEntity', generalType: 'DocumentEntity', allowedConnections: {} })
class DocumentEntity extends RunnableEntity {
  // Properties are defined in the DTO data interface above
}

Code examples are simplified for clarity. See the full SDK tutorials for production-ready patterns.

The status field tracks where the document is in our pipeline. Each stage updates the status, giving us full visibility into processing progress. Properties are defined in a DTO data interface and persisted automatically -- they survive restarts, they are queryable, and they show up in the management console's entity explorer.

This is one of the strengths of the Entity Graph approach. Instead of tracking document state in a database table you manage yourself, the entity is the state. The platform handles persistence, indexing, and lifecycle. You write the business logic; FireFoundry handles the plumbing.

Step 2: The Classification Bot

Once we have extracted text from a document, we need to classify it. This is where FireFoundry's structured output capability shines. Instead of parsing free-text LLM responses and hoping for the best, we define a schema using Zod and let the platform validate the model's output against it.

import { RegisterBot, ComposeMixins, MixinBot, StructuredOutputBotMixin } from '@firebrandanalytics/ff-agent-sdk';
import { withSchemaMetadata } from '@firebrandanalytics/ff-agent-sdk';
import { z } from 'zod';

const classificationSchema = withSchemaMetadata(
  z.object({
    documentType: z.enum(['invoice', 'contract', 'report', 'correspondence']),
    confidence: z.number().min(0).max(1),
    reasoning: z.string()
  }),
  'ClassificationResult',
  'Classification output for a document'
);

@RegisterBot('ClassificationBot')
class ClassificationBot extends ComposeMixins(MixinBot, StructuredOutputBotMixin) {
  constructor() {
    super();
    // Configure with prompt groups and classification schema
  }
}

The StructuredOutputBotMixin guarantees that the model's response conforms to the Zod schema. If the model returns an invalid classification (say, a type not in our enum), the SDK automatically retries with corrective feedback. You never get malformed output in production. The confidence score is especially important -- we will use it later to route low-confidence classifications to human review rather than blindly trusting the model.

Step 3: The Extraction Bot

Classification tells us what the document is. Extraction tells us what is in it. Different document types have different fields of interest, so our extraction bot adapts its prompt and schema based on the classification result.

const invoiceSchema = z.object({
  vendorName: z.string(),
  invoiceNumber: z.string(),
  amount: z.number(),
  currency: z.string(),
  dueDate: z.string(),
  lineItems: z.array(z.object({
    description: z.string(),
    quantity: z.number(),
    unitPrice: z.number()
  }))
});

const contractSchema = z.object({
  parties: z.array(z.string()),
  effectiveDate: z.string(),
  expirationDate: z.string().optional(),
  contractType: z.string(),
  keyTerms: z.array(z.string()),
  governingLaw: z.string().optional()
});

const reportSchema = z.object({
  title: z.string(),
  author: z.string().optional(),
  date: z.string().optional(),
  summary: z.string(),
  keyFindings: z.array(z.string())
});

const correspondenceSchema = z.object({
  from: z.string(),
  to: z.array(z.string()),
  date: z.string().optional(),
  subject: z.string(),
  summary: z.string(),
  actionItems: z.array(z.string())
});

function getSchemaForType(docType: string) {
  switch (docType) {
    case 'invoice': return invoiceSchema;
    case 'contract': return contractSchema;
    case 'report': return reportSchema;
    case 'correspondence': return correspondenceSchema;
    default: return reportSchema;
  }
}

function getPromptForType(docType: string): string {
  switch (docType) {
    case 'invoice':
      return `Extract structured invoice data: vendor, invoice number,
amount, currency, due date, and line items.`;
    case 'contract':
      return `Extract contract details: parties involved, effective and
expiration dates, contract type, key terms, and governing law.`;
    case 'report':
      return `Extract report metadata: title, author, date, executive
summary, and key findings.`;
    case 'correspondence':
      return `Extract correspondence details: sender, recipients, date,
subject, summary, and any action items.`;
    default:
      return `Extract the key structured information from this document.`;
  }
}

Each document type gets its own Zod schema with the specific fields that matter. An invoice has line items and amounts; a contract has parties and terms; correspondence has action items. The extraction bot selects the right schema and prompt dynamically based on the classification result. This means every extraction is validated against a precise, type-specific schema -- not a generic "extract whatever you find" approach.

Step 4: Document Processing Service Integration

Before we can classify or extract anything, we need the raw text. FireFoundry's Document Processing Service handles the heavy lifting of getting text out of diverse file formats.

import { WorkingMemoryProvider } from '@firebrandanalytics/ff-agent-sdk';

async function processDocument(file: Buffer, filename: string, mimeType: string) {
  const wmProvider = new WorkingMemoryProvider();

  // Upload the file to Working Memory for processing
  const fileRef = await wmProvider.upload(file, { filename, mimeType });

  // Use the Document Processing Service for text extraction
  // (configured via platform services, not a direct SDK import)
  const extraction = await wmProvider.extractText(fileRef, {
    ocr: true,             // Enable OCR for scanned documents and images
    tableExtraction: true   // Extract structured table data
  });

  return extraction;
}

The Document Processing Service supports multi-format extraction: PDFs (both native text and scanned), Word documents, spreadsheets, HTML, and images. For images and scanned PDFs, the service runs OCR automatically when the ocr flag is enabled. It also handles image preprocessing -- upscaling low-resolution scans and adjusting colorspace for better recognition accuracy.

Table extraction is particularly valuable for invoices and financial documents. Instead of getting a messy blob of text where table rows are jumbled together, the service extracts tables as structured data that the extraction bot can work with directly.

The WorkingMemoryProvider class serves as the interface to temporary file storage for processing. Files uploaded to Working Memory are accessible to all platform services during the processing lifecycle, then cleaned up according to your retention policies. This keeps sensitive document data under platform governance rather than scattered across ad-hoc storage locations.

Step 5: Routing Logic

With the document classified and data extracted, the final step is routing to the appropriate downstream workflow. Each document type maps to a different business process.

async function routeDocument(doc: DocumentEntity) {
  // Low confidence? Route to human review instead of automation
  if (doc.confidence < 0.8) {
    await humanReviewQueue.enqueue({
      documentId: doc.id,
      classification: doc.classification,
      confidence: doc.confidence,
      reason: 'Low confidence classification requires human verification'
    });
    doc.status = 'routed';
    doc.routedTo = 'human-review';
    return;
  }

  switch (doc.classification) {
    case 'invoice':
      await accountingWorkflow.submit({
        documentId: doc.id,
        vendor: doc.extractedData.vendorName,
        amount: doc.extractedData.amount,
        dueDate: doc.extractedData.dueDate
      });
      doc.routedTo = 'accounting-approval';
      break;

    case 'contract':
      await legalReviewWorkflow.submit({
        documentId: doc.id,
        parties: doc.extractedData.parties,
        contractType: doc.extractedData.contractType,
        keyTerms: doc.extractedData.keyTerms
      });
      doc.routedTo = 'legal-review';
      break;

    case 'report':
      await distributionWorkflow.submit({
        documentId: doc.id,
        title: doc.extractedData.title,
        summary: doc.extractedData.summary,
        keyFindings: doc.extractedData.keyFindings
      });
      doc.routedTo = 'summary-distribution';
      break;

    case 'correspondence':
      await correspondenceWorkflow.submit({
        documentId: doc.id,
        from: doc.extractedData.from,
        subject: doc.extractedData.subject,
        actionItems: doc.extractedData.actionItems
      });
      doc.routedTo = 'correspondence-triage';
      break;
  }

  doc.status = 'routed';
}

Notice the confidence threshold check at the top. When the classification bot is less than 80% certain about a document's type, we route to a human review queue rather than letting an uncertain classification cascade into the wrong workflow. This is a critical production pattern -- your agent should know when it does not know.

The routing destinations are FireFoundry workflows -- multi-step orchestrations that can include additional AI processing, human approval gates, external system integrations, and notifications. The invoice path might trigger an approval chain in your accounting system. The legal review path might notify the legal team on Slack and create a task in your project tracker. Each workflow is its own agent component, keeping concerns cleanly separated.

Putting It All Together

Here is the orchestration that ties the pipeline together -- from upload through routing:

async function handleDocumentUpload(file: Buffer, filename: string, mimeType: string) {
  // Create the entity to track this document
  const doc = await DocumentEntity.create({
    filename,
    mimeType,
    status: 'uploaded'
  });

  try {
    // Step 1: Extract text
    doc.status = 'processing';
    const extraction = await processDocument(file, filename, mimeType);
    doc.extractedText = extraction.text;

    // Step 2: Classify
    const classification = await classificationBot.run({
      input: extraction.text
    });
    doc.classification = classification.documentType;
    doc.confidence = classification.confidence;
    doc.status = 'classified';

    // Step 3: Extract structured data
    const schema = getSchemaForType(classification.documentType);
    const extractionPrompt = getPromptForType(classification.documentType);
    const extracted = await extractionBot.run({
      input: extraction.text,
      systemPrompt: extractionPrompt,
      schema
    });
    doc.extractedData = extracted;
    doc.status = 'extracted';

    // Step 4: Route
    await routeDocument(doc);
  } catch (error) {
    doc.status = 'processing';
    throw error;
  }
}

Production Considerations

The pipeline above works, but production deployments need a few additional patterns to be truly robust.

Progress Streaming

Document processing can take time, especially for large PDFs with many pages or images requiring OCR. Use FireFoundry's progress streaming to keep users informed. The platform supports server-sent events that your frontend can consume to show real-time status updates: "Extracting text... Classifying document... Extracting invoice data... Routing to accounting."

Human Review for Low Confidence

We already implemented a confidence threshold, but in production you should also track classification accuracy over time. The management console's telemetry dashboards can show you which document types the model struggles with, which lets you refine your classification prompts or add training examples. If your agent consistently misclassifies a particular vendor's invoice format, that is a signal to adjust the extraction pipeline for that vendor.

Telemetry and Observability

Every step in the pipeline generates telemetry automatically through the FireFoundry platform. You get distributed traces that show the full journey of each document: upload timestamp, extraction duration, classification result and confidence, extraction output, and routing destination. When an invoice gets misrouted, you can trace back through the entire decision chain to understand exactly where things went wrong. The management console provides natural language search across these traces -- ask "show me all invoices classified with less than 70% confidence this week" and get actionable results.

Entity Graph for Document Lifecycle

The DocumentEntity we defined is not just a data container -- it is a node in the Entity Graph that tracks the complete document lifecycle. You can query entities by status to find processing bottlenecks, by classification to understand your document mix, or by confidence to identify candidates for human review. The entity graph also enables relationships: link a document to the workflow instance it was routed to, to the user who uploaded it, or to other related documents. This creates an auditable record that compliance teams require.

Try It Yourself

Document processing is one of the patterns that demonstrates the full power of the FireFoundry platform: entities for state management, bots for AI reasoning with structured output, service integrations for document handling, and workflows for downstream orchestration. Each piece is focused and testable on its own, but together they form a production-grade pipeline.

For more hands-on tutorials, visit the tutorials section in our developer documentation. You will find guides for building other common agent patterns -- from conversational assistants to data analysis workflows.

If you are building document processing into your organization's workflows and want to try this on the FireFoundry platform, request beta access and our team will help you get started. We are particularly interested in teams processing high volumes of diverse document types -- the pattern scales beautifully, and we would love to help you prove it out.