Skip to main content

Event Projection System

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

This document is a comprehensive technical documentation for the event projection system, targeting data architects and backend engineers, systematically explains the working principle of projectors, the conversion process from events to read models, index building mechanism, as well as the implementation strategies for full and incremental projections, projection state management and consistency assurance. The document also demonstrates the implementation methods of event filters, projection processors, and index update mechanisms, covering best practices for parallel processing, batch operations and transaction management, and provides recommendations for performance optimization, caching strategies and data synchronization mechanisms.

Project Structure​

The event projection system is located in pkg/projection and pkg/adapter/tasks, pkg/tasks and other directories, focusing on the main chain of "event reading → aggregate root index → projection calculation → view storage"; at the same time, through the task scheduler, it achieves automated execution and concurrency control for full projections.

Core Components​

Interface Layer (projection/core.go)​

  • Aggregate Root Indexer Interface: Provides the ability to get valid aggregate root IDs by aggregate type, used for full/scheduled projections.
  • Event Reader Interface: Reads event collections by aggregate type and event type, used as the event source for full projections.
  • Projector Interface: Replays event streams and projects them into view entities for the storage layer to write.

Full Projector (projection/full.go)​

  • Combines event reader, indexer and projector, reads events in batches by event type, then traverses aggregate root IDs one by one for projection.

JetStream Implementation​

  • JetStream Indexer: Scans the event stream based on JetStream consumers, collecting aggregate root IDs for specified aggregate types.
  • JetStream Event Reader: Scans the event stream based on JetStream consumers, filters by aggregate type and event type, and deduplicates the highest version events.

Task Adapter (adapter/tasks/full_proj.go)​

  • Full Projection Task: Encapsulates the complete process of "index aggregate roots → read events → project → store", supports logging and error handling.

Task Scheduler (tasks/*)​

  • Task and Scheduling Interface: Unified task lifecycle, status and retry strategy.
  • Sequential Scheduler: Strictly serial execution, suitable for scheduled/periodic tasks.
  • Concurrent Scheduler: Controls maximum concurrency, supports task timeout, retry and expiration cleanup.

Application Integration (bootstrap/*)​

  • Register Projector: Registers JetStream event reader and indexer through the container for task invocation.

Repository Interface (usecase/repo.go)​

  • Provides generic repository capabilities, supports single and batch persistence for view writing.

Entity and Event Interface (entity/*)​

  • Defines the minimum contract for domain events and entities, ensuring type safety for projectors and task adapters.

Architecture Overview​

The event projection system adopts an "event-driven + task orchestration" architecture: events are carried by NATS JetStream, indexers and event readers are responsible for extracting aggregate root IDs and events from the stream; projectors replay event streams into view entities; task schedulers are responsible for triggering, concurrency and retry of full projections; finally, views are written to the read database through the repository interface.

Detailed Component Analysis​

Component A: Full Projector (projection/full.go)​

Responsibilities​

  • Reads events in batches by event type dimension;
  • Enumerates aggregate root IDs by aggregate type dimension;
  • Calls projector for each aggregate root ID to generate views and persist them.

Key Points​

  • Event Reading: Filters by event type, internally deduplicates and retains the highest version events;
  • Aggregate Root Enumeration: Filters by aggregate type, collects all valid IDs;
  • Projection Execution: Replays event streams one by one for each aggregate root, generates view entities;
  • Error Propagation: Any step failure terminates immediately, returns error.

Performance Characteristics​

  • Sequential traversal of aggregate root IDs, naturally serial;
  • Event reader internal batch processing and deduplication, reduces impact of duplicate events.

Component B: JetStream Event Reader (projection/nats_events.go)​

Responsibilities​

  • Scans event stream based on JetStream consumer;
  • Filters by aggregate type and event type;
  • Paginates to pull messages, deduplicates and retains the highest version events;
  • Deserializes events and acknowledges messages.

Key Points​

  • Filter Subject: Constructs filter subject by aggregate type;
  • Consumption Strategy: Instant delivery, explicit acknowledgment;
  • Batch Processing: Fixed batch size, paginated consumption;
  • Deduplication: Retains the highest version events by aggregate root ID;
  • Error Handling: Stream does not exist, consumer creation failure, deserialization failure all log errors and return.

Performance Characteristics​

  • Pagination and batch processing reduce memory peaks;
  • Deduplication avoids duplicate projections caused by duplicate events.

Component C: JetStream Indexer (projection/nats_indexer.go)​

Responsibilities​

  • Scans event stream based on JetStream consumer;
  • Filters by aggregate type;
  • Collects all aggregate root IDs that have appeared and deduplicates them.

Key Points​

  • Filter Subject: Constructs filter subject by aggregate type;
  • Consumption Strategy: Instant delivery, explicit acknowledgment;
  • Deduplication: Uses map for deduplication;
  • Error Handling: Stream does not exist, consumer creation failure, deserialization failure all log errors and return.

Performance Characteristics​

  • Consistent batch processing and deduplication strategy with event reader.

Component D: Full Projection Task Adapter (adapter/tasks/full_proj.go)​

Responsibilities​

  • Encapsulates the complete process of "index aggregate roots → read events → project → store";
  • Provides logging and error propagation;
  • Skips processing for aggregate roots without events.

Key Points​

  • Event Reading: Uses message layer StreamReader to read event streams by aggregate root ID;
  • Projection Execution: Calls projector to generate views;
  • View Storage: Uses generic repository interface Save, recommends using UPSERT to ensure idempotency;
  • Error Handling: Any step failure returns immediately, avoids partial updates.

Concurrency and Batch​

  • This adapter itself does not introduce concurrency, can be executed in parallel at the scheduler level with multiple task instances.

Component E: Task Scheduler (tasks/*)​

Task and Scheduling Interface (tasks/core.go)​

  • Unified task lifecycle: schedule, cancel, status query, list, close;
  • Task information: status, schedule time, retry count, error, TTL, etc.;
  • Execution modes: sequential and concurrent two modes.

Sequential Scheduler (sequential_scheduler.go)​

  • Strictly serial execution, suitable for scheduled/periodic tasks;
  • Built-in exponential backoff retry and expiration cleanup;
  • Task queue and pending queue cooperate to ensure orderliness.

Concurrent Scheduler (concurrent_scheduler.go)​

  • Controls maximum concurrency, avoids resource contention;
  • Polls and waits for available slots, timeout control;
  • Supports task timeout, retry and expiration cleanup;
  • Condition variables and mutex coordinate active task count.

Component F: Application Integration (bootstrap/*)​

Application Startup (bootstrap/app.go)​

  • Provides NATS connection, stream publisher, stream reader factory;
  • Provides graceful shutdown and retry startup mechanism.

Projection Registration (bootstrap/opts.go)​

  • WithProjection: Registers full projector to container, injects JetStream event reader and indexer.

Component G: Entity and Event Interface (entity/*)​

Domain Event Interface (entity/domain_event.go)​

  • Provides event ID, type, aggregate ID, aggregate type, version, etc.;
  • Version is used for event sourcing and deduplication.

Entity Interface (entity/entity.go)​

  • Provides entity ID, creation/update time, etc.;
  • View entities follow this interface for unified storage.

Dependency Analysis​

Component Coupling​

  • projection/full.go depends on EventReader, AggregateIndexer, Projector;
  • adapter/tasks/full_proj.go depends on Indexer, StreamReader, Projector, Repository;
  • tasks/* is decoupled from specific implementations, through interfaces and container injection;
  • bootstrap/opts.go registers projectors through container, avoids hard-coded dependencies.

External Dependencies​

  • NATS JetStream: Event stream storage and consumers;
  • Logger: Unified recording of errors and runtime status;
  • Repository Interface: Abstracts view persistence.

Performance Considerations​

Event Reading and Indexing​

  • Pagination and Batch Processing: Both event readers and indexers use fixed batch size for paginated pulling, reducing memory peaks.
  • Deduplication Strategy: Event readers retain the highest version events by aggregate root ID, avoiding duplicate projections.

Concurrency and Throughput​

  • Sequential Scheduler: Suitable for scheduled/periodic tasks, ensures orderliness and stability.
  • Concurrent Scheduler: Controls resource usage through maximum concurrency, combined with timeout and retry to improve reliability.

Batch and Idempotency​

  • Repository interface provides batch capabilities such as SaveBatch, recommends implementing UPSERT at the view storage layer to ensure idempotency.

Caching Strategy​

  • Can cache views of hot aggregate roots at the projector layer, reducing duplicate calculations;
  • Cache recent task status at the task scheduler layer, reducing query overhead.

Data Synchronization​

  • Both event readers and indexers are based on JetStream consumers, with high reliability and order guarantee;
  • Recommend enabling transactions or batch commits at the view storage layer to ensure consistency.

Troubleshooting Guide​

Common Problem Location​

  • NATS Connection Failure: Check NATS URL and authentication configuration, confirm JetStream is enabled;
  • Stream Does Not Exist: Event readers and indexers log warnings and return empty results when stream does not exist, need to create stream first;
  • Deserialization Failure: Event payload format exception, logs error and skips the message;
  • Consumer Creation Failure: Check permissions and stream configuration.

Logging and Monitoring​

  • Both event readers and indexers log error logs for easy problem location;
  • Task adapters log aggregate root processing progress and errors for easy tracking.

Retry and Rollback​

  • Both sequential and concurrent schedulers support retry and expiration cleanup, recommend reasonably setting maximum retry count and TTL;
  • View storage recommends using UPSERT to avoid data inconsistency caused by partial updates.

Conclusion​

The event projection system achieves efficient conversion from event streams to read models through clear interface layering and task orchestration. JetStream, as the event infrastructure, provides reliable event storage and consumer capabilities; task schedulers meet the needs of orderliness and throughput in different execution modes; full projectors and adapters encapsulate complex processes into reusable components. Combined with batch operations, idempotent writes and reasonable caching strategies, the system can maintain stability and high performance in high concurrency scenarios.

Appendix​

Example Test (Event Projection Verification)​

  • Test files demonstrate how to use NATS containers, JetStream publish/subscribe and event readers to verify the correctness of event streams and the connectivity of projection processes.

Best Practices Checklist​

  • Use UPSERT to ensure view storage idempotency;
  • Set reasonable timeout and retry strategies for tasks;
  • Prioritize concurrent scheduler in high concurrency scenarios;
  • Implement local caching for hot aggregate roots;
  • Regularly clean up expired tasks and historical data to maintain system health.