Checkpointing Specification

Document ID: WF-007
Version: 1.0.0
Status: Draft
Owner: Workflow Engine Team
Last Updated: 2026-06-26

1. Purpose

This document defines the checkpointing architecture used by the Wovyr Workflow Engine.

Checkpointing enables:

Durable workflow execution
Fast recovery after failures
Event replay optimization
Long-running workflows
Distributed execution
Workflow migration
Version-safe recovery

The checkpoint system is the foundation of fault tolerance within the workflow engine.

2. Objectives

The checkpoint subsystem must provide:

Durable persistence
Fast recovery
Deterministic replay
Incremental snapshots
Storage abstraction
Compression support
Encryption support
Horizontal scalability

3. Design Principles

The checkpoint system follows these principles:

Checkpoints are immutable.
Every checkpoint is versioned.
Checkpoints never replace event history.
Events remain the source of truth.
Recovery always starts from the latest valid checkpoint.
Checkpoints are portable across workers.
Storage implementation is pluggable.

4. Architecture

               Workflow Runtime
                      │
                      ▼
             Checkpoint Manager
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
 Snapshot Store  Compression   Encryption
        │
        ▼
 Persistence Adapter
        │
        ▼
 PostgreSQL / S3 / RocksDB / Object Storage

5. Checkpoint Lifecycle

Workflow Running
        │
        ▼
Checkpoint Triggered
        │
        ▼
Collect Runtime State
        │
        ▼
Serialize Snapshot
        │
        ▼
Compress
        │
        ▼
Encrypt
        │
        ▼
Persist
        │
        ▼
Checkpoint Available

6. Checkpoint Triggers

Checkpoints may be created:

After workflow creation
After every completed activity
Before waiting
Before compensation
After compensation
Before workflow completion
Periodically
Before worker migration
Before shutdown

Trigger policies are configurable.

7. Checkpoint Types

Full Checkpoint

Contains the complete workflow state.

Used for:

Initial snapshots
Periodic snapshots
Long-running workflows

Incremental Checkpoint

Contains only changes since the previous checkpoint.

Benefits:

Smaller storage footprint
Faster persistence
Lower bandwidth usage

Manual Checkpoint

Created through API or CLI.

Useful for:

Maintenance
Migration
Debugging

Automatic Checkpoint

Generated by runtime policies.

Default mode for production.

8. Checkpoint Contents

Every checkpoint stores:

checkpointId:
workflowId:
executionId:
workflowVersion:
checkpointVersion:
createdAt:
workerId:
stateVersion:
currentState:
activeNodes:
completedNodes:
variables:
activityResults:
pendingEvents:
leases:
metadata:

9. Snapshot Structure

Checkpoint
│
├── Metadata
├── Workflow State
├── Activity State
├── Variables
├── Active DAG
├── Retry State
├── Compensation State
├── Event Cursor
├── Scheduler State
└── Security Metadata

10. Serialization

Supported serialization formats:

CBOR (default)
MessagePack
JSON (debugging)
Protobuf (future)

Serialization format is configurable.

11. Compression

Supported compression algorithms:

Algorithm	Purpose
Zstd	Default
Gzip	Compatibility
LZ4	High-speed recovery

Compression should occur after serialization.

12. Encryption

Checkpoint encryption supports:

AES-256-GCM
Envelope encryption
Customer-managed keys
Cloud KMS integration

Sensitive workflow data must remain encrypted at rest.

13. Persistence Providers

Supported providers:

Provider	Use Case
PostgreSQL	Default
S3	Enterprise
Azure Blob Storage	Cloud
Google Cloud Storage	Cloud
RocksDB	Embedded
Local Filesystem	Development

Storage providers implement a common interface.

14. Checkpoint Versioning

Each checkpoint maintains:

Workflow Version
Checkpoint Version
Schema Version
Serialization Version
Encryption Version

Older checkpoints remain readable.

15. Recovery Process

Recovery steps:

Worker Starts
      │
      ▼
Locate Latest Checkpoint
      │
      ▼
Load Snapshot
      │
      ▼
Decrypt
      │
      ▼
Decompress
      │
      ▼
Deserialize
      │
      ▼
Replay Events
      │
      ▼
Resume Workflow

Recovery must produce identical workflow state.

16. Event Replay

Replay begins after the checkpoint event cursor.

Example:

Checkpoint -> Event 231

Replay:

232
233
234
235

Resume Execution

Events before the checkpoint are never replayed.

17. Checkpoint Consistency

Checkpoint creation must be atomic.

Required guarantees:

No partial snapshots
No duplicate checkpoints
Consistent DAG state
Consistent variable state
Consistent activity state

18. Worker Migration

Checkpointing enables workflow migration.

Migration steps:

Worker releases lease.
Latest checkpoint persisted.
Scheduler assigns new worker.
New worker restores checkpoint.
Replay remaining events.
Resume execution.

Migration must be transparent.

19. Checkpoint Retention

Retention policies:

Keep latest N checkpoints
Keep daily snapshots
Keep milestone checkpoints
Delete expired snapshots

Retention is configurable per workflow.

20. Garbage Collection

The Checkpoint Manager periodically removes:

Expired checkpoints
Superseded incremental checkpoints
Orphaned snapshots
Failed snapshot attempts

Garbage collection must never remove checkpoints required for recovery.

21. Failure Handling

Checkpoint failures may occur during:

Serialization
Compression
Encryption
Persistence

Policies:

Retry
Fallback storage
Alert operator
Pause workflow
Abort workflow (configurable)

22. Performance Targets

Metric	Target
Checkpoint creation	< 20 ms
Restore time	< 100 ms
Compression overhead	< 10 ms
Encryption overhead	< 5 ms
Recovery startup	< 250 ms

Targets assume medium-sized enterprise workflows.

23. Observability

Metrics:

Checkpoints created
Failed checkpoints
Restore duration
Snapshot size
Compression ratio
Encryption duration
Recovery duration

Logs include:

Workflow ID
Checkpoint ID
Version
Worker ID
Storage provider

24. Security

Checkpoint storage enforces:

Encryption at rest
TLS in transit
Tenant isolation
Access auditing
Integrity validation
Digital signatures (optional)

25. Rust API

pub trait CheckpointStore {
    fn save(
        &self,
        checkpoint: WorkflowCheckpoint,
    ) -> Result<CheckpointId>;

    fn load(
        &self,
        id: CheckpointId,
    ) -> Result<WorkflowCheckpoint>;

    fn latest(
        &self,
        workflow_id: WorkflowId,
    ) -> Result<Option<WorkflowCheckpoint>>;

    fn delete(
        &self,
        id: CheckpointId,
    ) -> Result<()>;
}

26. Crate Organization

engine-workflow/
└── checkpoint/
    ├── manager.rs
    ├── snapshot.rs
    ├── serializer.rs
    ├── compression.rs
    ├── encryption.rs
    ├── recovery.rs
    ├── retention.rs
    ├── gc.rs
    ├── storage.rs
    ├── version.rs
    └── mod.rs

27. Testing Strategy

Unit Tests

Serialization
Compression
Encryption
Version compatibility

Integration Tests

Recovery
Worker migration
Incremental checkpoints
Storage providers

Performance Tests

Large workflows
Massive variables
Thousands of activities
Concurrent checkpoints

Chaos Tests

Crash during checkpoint
Storage outage
Corrupted snapshot
Encryption failure

28. Non-Functional Requirements

Requirement	Target
Durability	No data loss after committed checkpoint
Availability	99.99%
Replay correctness	100%
Restore correctness	100%
Compression support	Yes
Encryption support	Yes

Workflow Overview
Execution Model
Workflow DSL
DAG Engine
Scheduler
State Machine
Retry Engine
Compensation
Distributed Execution
Persistence
Rust Crate Design

30. Revision History

Version	Date	Description
1.0.0	2026-06-26	Initial Checkpointing Specification