Async Streams & Scheduling

Production-grade scheduling for GenAI applications. Capacity management, priority queues, and backpressure mechanisms—out of the box.

The Challenge: Scaling GenAI Applications

LLM-based applications face unique scaling challenges. Every request can spawn multiple model calls, each consuming expensive GPU capacity. Without proper scheduling, you'll hit rate limits, exhaust resources, and deliver inconsistent experiences to your users.

Building this yourself means solving hard problems: How do you ensure paying customers get priority over free users? How do you prevent one runaway workflow from starving others? How do you gracefully degrade when you're at capacity instead of failing catastrophically?

These aren't trivial problems. We provide battle-tested solutions out of the box.

Example: QoS Tiers for Your Users

Your paying customers expect better service. With our scheduling primitives, implementing differentiated QoS is straightforward:

import { HierarchicalTaskPoolRunner, PriorityCapacitySource } from '@firefoundry/agent-sdk';

// Define capacity with priority tiers
const capacitySource = new PriorityCapacitySource({
  totalCapacity: 100,  // Total concurrent LLM calls
  tiers: {
    enterprise: { reserved: 60, priority: 1 },  // 60 slots guaranteed
    pro: { reserved: 30, priority: 2 },         // 30 slots guaranteed
    free: { reserved: 10, priority: 3 }         // 10 slots, lowest priority
  }
});

// Create the task runner
const runner = new HierarchicalTaskPoolRunner(taskSource, capacitySource);

// Tasks automatically scheduled by tier
// Enterprise tasks start immediately if capacity exists
// Free tier tasks wait if higher-priority work is queued

Example: Backpressure That Actually Works

When your system is overwhelmed, you need controlled degradation—not crashes. Our buffering primitives implement backpressure automatically:

import { PushPullBufferObj } from '@firefoundry/agent-sdk';

// Bounded buffer - producers block when full
const taskQueue = new PushPullBufferObj<Task>({
  maxSize: 1000,  // Queue won't grow unbounded
  onFull: 'block' // Producers wait instead of failing
});

// Ingestion (fast) - automatically slows when queue fills
async function ingestRequests(requests: Request[]) {
  for (const req of requests) {
    await taskQueue.next(createTask(req));  // Blocks if queue full
  }
}

// Processing (slow) - works at its own pace
async function processWithLLM() {
  for await (const task of taskQueue) {
    await llm.complete(task);  // Takes time, that's ok
  }
}

The Architecture

FireFoundry's async streaming is built on two complementary models that can be composed together: