Skip to main content

Project Introduction

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

Sparrow is an enterprise-oriented microservice framework that emphasizes the integration of "Event-Driven + Domain-Driven + Clean Architecture". Its core values are:

  • Event Sourcing and CQRS as the core, providing highly traceable, auditable, and extensible business models;
  • Domain-Driven Design (DDD) as guidance, abstracting business semantics into aggregate roots and domain events, reducing business complexity;
  • Clean Architecture layering for decoupling, clarifying boundaries and dependency directions, improving testability and evolvability;
  • Providing out-of-the-box infrastructure: event bus, task scheduling, authentication/authorization, multi-storage adapters, projections, and indexes.

Target Users:

  • Teams building highly reliable, evolvable enterprise-level microservice systems;
  • Medium to large projects preferring event-driven and domain modeling;
  • Engineering teams with high requirements for observability, testability, and maintainability.

Core Problems Solved:

  • Business-data coupling caused by traditional REST direct database connections, making auditing and replay difficult;
  • Complex consistency and cross-service transaction challenges brought by synchronous calls between microservices;
  • Lack of unified event carrying and delivery mechanisms, causing business events to be scattered;
  • Repository layer reinventing the wheel, chaotic interfaces, inconsistent error handling.

Unique Advantages:

  • Unified abstraction of event bus and event store, supporting multiple backends (NATS, RabbitMQ, Redis, etc.);
  • Generic repositories and query DSL, balancing ease of use and flexibility;
  • Aggregate root templates and event sourcing support for rapid DDD implementation;
  • Projection and index interfaces, supporting CQRS query-side optimization;
  • Unified error and logging encapsulation, improving observability and consistency.

Development History and Version Evolution:

  • The project is in continuous evolution, with recent focus on event bus implementation, repository layer refactoring, event store consistency, and error handling standardization;
  • The refactoring checklist shows phased achievements in event processing security, repository interface consistency, error wrapping, and logging unification;
  • Current maturity and stability: Has production-ready basic capabilities, recommended for pilot testing in controlled scenarios before gradually expanding application scope.

Project Structure​

Overall adopts "layering + domain module" organization:

  • bootstrap: Application startup, container, HTTP engine, event bus integration, graceful shutdown, and retry mechanisms;
  • config: Centralized configuration loading and default value settings;
  • entity: Domain models and event base classes, aggregate root interfaces and templates;
  • usecase: Use case layer interfaces and generic repository base classes, query DSL;
  • persistence: Multi-storage adapters (SQL, Redis, Badger, Mongo, etc.);
  • messaging: Event bus abstraction, JetStream encapsulation, subscriptions, and stream Hub;
  • projection: Projectors, event readers, aggregate indexers;
  • tasks: Task scheduler interfaces and task models;
  • auth: Token structures and authentication-related tools;
  • logger: Logging encapsulation;
  • ws: WebSocket support;
  • .qoder/.trae: Development process and documentation accumulation.
  • Application Bootstrap and Container: Responsible for configuration loading, HTTP engine, event bus integration, subprocess startup and graceful shutdown, exponential backoff retry;
  • Domain Models: Aggregate root interfaces, domain event base classes and templates, supporting event sourcing and snapshots;
  • Use Case Layer: Generic repository base classes and query DSL, shielding underlying differences;
  • Persistence: Multi-storage adapters (SQL, Redis, Badger, Mongo), providing generic repository implementations;
  • Messaging Layer: Event bus abstraction and JetStream encapsulation, supporting stream Hub and subscriptions;
  • Projection Layer: Projectors, event readers, aggregate indexers, supporting CQRS query side;
  • Task Layer: Task scheduler interfaces and task models, supporting concurrent/sequential execution and retry;
  • Authentication Layer: Token structures and authentication tools;
  • Logging Layer: Unified logging encapsulation.

Sparrow's overall architecture centers on "Event-Driven + Domain-Driven + Clean Architecture" as its core design philosophy:

  • Event-Driven: Event bus carries cross-service communication, aggregate roots generate domain events, events are persisted and propagated;
  • Domain-Driven: Aggregate roots as the center, encapsulating business invariants and state changes, event sourcing ensures traceability;
  • Clean Architecture: Clear layering, dependencies pointing downward, use case layer coordinates business, infrastructure layer provides storage and messaging capabilities.

Application Bootstrap and Startup Process​

  • Configuration Loading: Load configuration through Viper and set default values;
  • Logger Initialization: Unified logging encapsulation;
  • HTTP Engine: Based on Gin, default middleware and routing;
  • Event Bus: Resolve NATS connection through container, provide JetStream publish/subscribe;
  • Subprocess Management: Supports Startable/GracefulClose interfaces, with exponential backoff retry and graceful shutdown;
  • Task Scheduling: Provides scheduler wrapper for easy integration of task execution.
  • Aggregate Root Interface: Defines aggregate identity, version control, command handling, event application, snapshots, and validation;
  • Domain Events: Unified event ID, type, aggregate ID/type, version, and timestamp;
  • Event Base Class: Provides common fields and interface implementations, ensuring event transmission consistency.
  • Generic Repository Base Class: Provides unified error wrapping, timestamp handling, and query method skeleton;
  • Generic SQL Repository: Auto-infers table names, soft delete support, batch operations, pagination, and conditional queries;
  • Query DSL: Supports multiple conditions, sorting, pagination, and aggregation, shielding SQL differences.
  • Stream Hub: Encapsulates JetStream subscriptions, supporting handler registration by aggregate type and event type;
  • Subscription Registration: AddSub method uniformly registers domain event handlers;
  • Graceful Shutdown: Implements Startable/GracefulClose, supporting timeout shutdown and resource cleanup.
  • Aggregate Indexer: Provides ability to get valid ID sets by aggregate type;
  • Event Reader: Reads event streams by aggregate type and event type;
  • Projector: Replays event streams and projects them into view entities.
  • Task Interface: Contains ID, type, scheduled time, handler function, callbacks, and periodicity configuration;
  • Scheduler Interface: Supports scheduling, cancellation, status query, concurrency control, and graceful shutdown;
  • Execution Modes: Concurrent/sequential modes, meeting different business scenarios.

  • Token Structure: Contains access token, refresh token, scope, and expiration;

  • Application Integration: Works with middleware and session services for authentication and authorization.

  • External Dependencies: NATS, RabbitMQ, Redis, MongoDB, PostgreSQL, Zap, Gin, Viper, etc.;

  • Cohesion and Coupling: bootstrap and messaging have higher coupling, but isolated through interface abstraction; usecase and persistence decoupled through interfaces;

  • Circular Dependencies: No obvious circular dependency patterns observed;

  • External Integration Points: Event bus, task scheduling, logging, configuration loading.

  • Event Bus: JetStream publish/subscribe has high throughput and persistence capabilities, recommend reasonable stream and consumer strategy settings;
  • Repository Layer: SQL repository supports batch operations and soft delete, recommend enabling batch and transactions for high-frequency write scenarios;
  • Projection: When replaying historical events, recommend combining snapshots to reduce replay costs;
  • Task Scheduling: Concurrent mode suitable for CPU-intensive or I/O parallel scenarios, sequential mode suitable for strong consistency requirements;
  • Logging: Unified logging encapsulation, avoid frequent instance creation, recommend output pruning by level and field.

Troubleshooting Guide​

  • Startup Failure: Check configuration loading and default value settings, confirm logger initialization and HTTP listening address;
  • Event Subscription Anomalies: Confirm NATS connection resolution and stream Hub registration, review subscription startup and graceful shutdown logs;
  • Repository Errors: Focus on RepositoryError wrapping, locate specific operations and entity types;
  • Event Store: Unified error handling and logging, use error wrapping and context information to locate root causes;
  • Retry Mechanism: Exponential backoff and maximum retry count, avoid cascading failures.

Sparrow centers on event-driven and domain modeling, combined with clean architecture and multi-storage adapters, providing an implementable engineering solution for enterprise-level microservices. Through event sourcing and CQRS, the system has good traceability and scalability; through unified infrastructure and interface abstraction, it reduces the complexity of cross-service collaboration. It is recommended to pilot in controlled scenarios, gradually improve testing and monitoring systems, and form reusable best practices.

Appendix​

  • Use Cases and Examples (Conceptual Description):

    • Order System: Aggregate root is Order, events include OrderPlaced, PaymentReceived, Shipped, Completed, etc., driven by event bus for asynchronous processing and projection;
    • User Center: Aggregate root is User, events include Registered, LoggedIn, PermissionChanged, etc., combined with Casbin for permission verification;
    • Data Platform: Project event streams into wide tables through projectors, supporting reports and analysis;
    • Task Orchestration: Use task scheduler to execute periodic or one-time tasks, combined with retry and dead letter strategies for reliability.

  • Positioning for Different Technical Levels:

    • Beginners: Start with aggregate root templates and basic repositories, understand event sourcing and CQRS;
    • Intermediate Developers: Deeply master event bus, projection, and task scheduling, improve system observability;
    • Senior Architects: Extend multi-storage and event store based on framework, formulate evolution roadmap and governance strategies.