Background Jobs Plugin

A comprehensive CRON-based job scheduling plugin for the Fractal Synapse AI agent system. This plugin enables autonomous execution of AI agent tasks on scheduled intervals with persistent storage and comprehensive job management capabilities.

Features

• CRON-based Scheduling: Full CRON expression support for flexible job timing
• Autonomous Agent Execution: Creates dedicated agents for background task execution
• Persistent Storage: JSON-based job storage with execution history tracking
• Real-time Management: Create, edit, pause, resume, cancel, and delete jobs
• Execution History: Complete tracking of job executions with agent conversation links
• Graceful Resource Management: Proper startup/shutdown with resource cleanup
• Configurable Logging: Integrated logging interface with customizable logger support
• Comprehensive API: Full programmatic access to all job management functions

Installation

cd packages/plugins/background-jobs
npm install
npm run build

Dependencies

• @fractal-synapse/agent-core: Core agent functionality and interfaces
• cron-parser: CRON expression parsing and scheduling
• Requires storage-fs plugin for data persistence
• Requires agent-factory plugin for agent creation

Quick Start

Basic Usage

import { BackgroundJobsPlugin } from 'background-jobs';
import { StorageFsPlugin } from '@fractal-synapse/storage-fs';
import { AgentFactoryPlugin } from '@fractal-synapse/agent-factory';
import { getLogger } from '@fractal-synapse/agent-core';

// Initialize required dependencies
const storagePlugin = new StorageFsPlugin('./data');
const agentFactory = new AgentFactoryPlugin();
const logger = getLogger(); // Optional: provide custom logger

// Create background jobs plugin
const backgroundJobs = new BackgroundJobsPlugin({
  storagePlugin,
  agentFactory,
  // agentManager: undefined, // optional agent manager
  logger, // optional logger
  evaluationIntervalMs: 1000 // check jobs every second
});

// Start background execution
await backgroundJobs.startBackgroundExecution();

// Create a scheduled job
const jobId = await backgroundJobs.createJob(
  'Daily Report',
  '0 9 * * *', // Every day at 9 AM
  'Generate and save a daily status report'
);

console.log(`Created job: ${jobId}`);

Job Management Examples

// Get all jobs
const allJobs = await backgroundJobs.getJobs();

// Filter jobs by status
const runningJobs = await backgroundJobs.getJobs({
  status: [JobStatus.RUNNING, JobStatus.WAITING]
});

// Search jobs by name
const reportJobs = await backgroundJobs.getJobs({
  nameContains: 'report'
});

// Get a specific job
const job = await backgroundJobs.getJob(jobId);

// Pause a job
await backgroundJobs.pauseJob(jobId);

// Resume a job
await backgroundJobs.resumeJob(jobId);

// Cancel a job
await backgroundJobs.cancelJob(jobId);

// Delete a job
await backgroundJobs.deleteJob(jobId);

Advanced Usage

// Get execution statistics
const stats = await backgroundJobs.getStatistics();
console.log(`Total jobs: ${stats.totalJobs}`);
console.log(`Running jobs: ${stats.runningJobs}`);
console.log(`Total executions: ${stats.totalExecutions}`);

// Force job evaluation (useful for testing)
await backgroundJobs.forceEvaluation();

// Update a job's CRON expression
await backgroundJobs.updateJob(jobId, {
  cronExpression: '0 */2 * * *' // Every 2 hours
});

// Clear all jobs (use with caution)
await backgroundJobs.clearAllJobs();

CRON Expression Format

This plugin supports standard CRON expressions with 5 fields:

┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-7, Sunday = 0 or 7)
│ │ │ │ │
* * * * *

Common CRON Examples

• 0 9 * * * - Every day at 9:00 AM
• */15 * * * * - Every 15 minutes
• 0 */2 * * * - Every 2 hours
• 0 9 * * 1 - Every Monday at 9:00 AM
• 0 9 1 * * - First day of every month at 9:00 AM
• 0 9 1 1 * - January 1st at 9:00 AM

Data Structure

Job Data

interface JobData {
  id: string;                    // Unique job identifier
  name: string;                  // Human-readable job name
  cronExpression: string;        // CRON scheduling expression
  prompt: string;                // AI agent prompt to execute
  status: JobStatus;             // Current job status
  created: string;               // ISO creation timestamp
  lastRun?: string;              // ISO timestamp of last execution
  nextRun?: string;              // ISO timestamp of next scheduled run
  executionHistory: JobExecution[]; // Complete execution history
  metadata?: Record<string, any>; // Optional custom metadata
}

Job Execution

interface JobExecution {
  id: string;         // Unique execution identifier
  agentId: string;    // Agent ID that executed the job
  startTime: string;  // ISO start timestamp
  endTime?: string;   // ISO end timestamp
  status: ExecutionStatus; // Execution result status
  prompt: string;     // The prompt that was executed
  result?: string;    // Agent response/result
  error?: string;     // Error message if execution failed
}

Job Status

• WAITING - Job is scheduled and waiting for next execution
• RUNNING - Job is currently executing
• PAUSED - Job is temporarily paused
• CANCELED - Job has been canceled
• COMPLETED - Job has finished all scheduled executions

Execution Status

• PENDING - Execution is queued
• RUNNING - Execution is in progress
• SUCCESS - Execution completed successfully
• FAILURE - Execution failed
• INCOMPLETE - Execution was interrupted
• ERROR - Execution encountered an error

API Reference

BackgroundJobsPlugin

Core Methods

// Start/stop background execution
await startBackgroundExecution(): Promise<void>
await stopBackgroundExecution(): Promise<void>

// Job CRUD operations
await createJob(name: string, cronExpression: string, prompt: string): Promise<string>
await getJob(jobId: string): Promise<JobData | null>
await getJobs(filters?: JobFilters): Promise<JobData[]>
await updateJob(jobId: string, updates: Partial<JobData>): Promise<boolean>
await deleteJob(jobId: string): Promise<boolean>

// Job state management
await pauseJob(jobId: string): Promise<boolean>
await resumeJob(jobId: string): Promise<boolean>
await cancelJob(jobId: string): Promise<boolean>

// Administrative functions
await getStatistics(): Promise<JobStatistics>
await forceEvaluation(): Promise<void>
await clearAllJobs(): Promise<void>

Job Filtering

interface JobFilters {
  status?: JobStatus[];           // Filter by job status
  nameContains?: string;          // Filter by name substring
  cronPattern?: string;           // Filter by CRON pattern
  dateRange?: {                   // Filter by creation date
    start: string;
    end: string;
  };
}

Plugin Integration

With Agent Core

import { Agent } from '@fractal-synapse/agent-core';

// Initialize agent with background jobs plugin
const agent = new Agent('my-agent');
await agent.addPlugin(backgroundJobsPlugin);

// Plugin will automatically initialize with the agent

With Storage FS Plugin

import { StorageFsPlugin } from '@fractal-synapse/storage-fs';

// Initialize storage for background jobs
const storage = new StorageFsPlugin('./data/background-jobs');

// Background jobs will automatically create:
// - ./data/background-jobs/jobs.json (job definitions)
// - ./data/background-jobs/executions/ (execution logs)

With Agent Factory Plugin

import { AgentFactoryPlugin } from '@fractal-synapse/agent-factory';

// Initialize agent factory for job execution
const agentFactory = new AgentFactoryPlugin({
  defaultDefinition: 'background-job',
  maxConcurrentAgents: 10
});

// Each job execution creates a dedicated agent
// Agents are automatically cleaned up after execution

Testing

Run Tests

# Run all tests
npm test

# Run tests once (for CI)
npm run test:run

# Run specific test file
npx vitest run tests/unit/job.test.ts

Test Coverage

The plugin includes comprehensive test suites:

• Unit Tests: Core classes (Job, JobManager, utilities)
• Integration Tests: Plugin lifecycle and API
• Performance Tests: Large-scale operations and concurrent execution
• Error Handling Tests: Failure scenarios and recovery

Mock Strategy

For testing without real dependencies:

import { createMockStoragePlugin, createMockAgentFactory } from './tests/mocks';

const mockStorage = createMockStoragePlugin();
const mockAgentFactory = createMockAgentFactory();

const plugin = new BackgroundJobsPlugin({
  storagePlugin: mockStorage,
  agentFactory: mockAgentFactory
});

Architecture

Core Components

1. BackgroundJobsPlugin: Main plugin interface and lifecycle management
2. JobManager: Core scheduling engine with 1-second evaluation loop
3. Job: Individual job data model with CRON scheduling logic
4. CRON Utils: Utilities for CRON expression parsing and validation

Data Flow

1. Jobs are created and stored via the storage plugin
2. JobManager evaluates all jobs every second
3. Jobs ready for execution trigger agent creation via agent factory
4. Agents execute job prompts and return results
5. Execution results are stored in job history
6. Jobs calculate their next run time and return to waiting state

Storage Structure

storage-directory/
├── background-jobs/
│   ├── jobs.json           # All job definitions
│   └── executions/         # Individual execution logs
│       ├── job-1-exec-1.json
│       ├── job-1-exec-2.json
│       └── ...

Configuration

Plugin Options

interface BackgroundJobsConfig {
  storagePlugin: PluginStorageInterface;    // Required: Storage interface
  agentFactory: AgentFactoryPlugin;         // Required: Agent creation
  agentManager?: AgentManagerPlugin;        // Optional: Agent management
  evaluationIntervalMs?: number;            // Optional: Evaluation frequency (default: 1000ms)
}

Performance Tuning

• Evaluation Interval: Adjust evaluationIntervalMs based on precision needs
• Concurrent Jobs: Limited by agent factory's maxConcurrentAgents
• Storage Frequency: Jobs are saved after each modification
• Memory Usage: Job history is kept in memory; consider cleanup for long-running jobs

Error Handling

Common Errors

1. Invalid CRON Expression: Throws error during job creation
2. Storage Failures: Gracefully handled with logging
3. Agent Creation Failures: Jobs marked as failed with retry logic
4. Agent Execution Failures: Captured in execution history

Error Recovery

• Jobs with failed executions remain scheduled for next run
• Storage errors don't affect in-memory job state
• Plugin can be restarted without losing job definitions
• Corrupted job data is logged and skipped

Best Practices

Job Design

• Keep prompts focused: Single-purpose jobs are more reliable
• Use descriptive names: Makes management easier
• Consider execution time: Long-running jobs may impact performance
• Handle failures gracefully: Design prompts to be idempotent

Scheduling

• Avoid overlapping executions: Use appropriate CRON intervals
• Consider system resources: Don't over-schedule intensive tasks
• Test CRON expressions: Use online tools to verify timing
• Monitor execution history: Regular review for optimization

Production Usage

• Enable logging: Monitor job execution for issues
• Regular backups: Backup job definitions and history
• Performance monitoring: Watch memory and CPU usage
• Graceful shutdown: Always call stopBackgroundExecution() on exit

Troubleshooting

Common Issues

Jobs not executing:

• Check CRON expression syntax
• Verify startBackgroundExecution() was called
• Check agent factory is properly configured

Storage errors:

• Verify write permissions on storage directory
• Check disk space availability
• Ensure storage plugin is properly initialized

Performance issues:

• Reduce evaluation frequency (evaluationIntervalMs)
• Limit concurrent job executions
• Clean up old execution history

Agent creation failures:

• Check agent factory configuration
• Verify agent definitions exist
• Monitor agent resource usage

Debug Mode

Enable debug logging:

const plugin = new BackgroundJobsPlugin({
  storagePlugin: storage,
  agentFactory,
  evaluationIntervalMs: 1000,
  // logger: customLogger // Provide custom logger for detailed logging
});

Contributing

This plugin is part of the Fractal Synapse project. Follow the project's contribution guidelines:

1. Development Setup: Follow the TypeScript Node Module Guide
2. Testing: All changes must include comprehensive tests
3. Documentation: Update this README for API changes
4. Git Flow: Follow the project's Git Guide for commits

License

ISC License - Part of the Fractal Synapse project.

Changelog

See CHANGELOG.md for version history and changes.