Skip to main content

Project Overview

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

Sparrow is an enterprise-oriented microservice framework that emphasizes event-driven architecture, Domain-Driven Design (DDD), multi-datasource support, and observability. Its design philosophy centers around the "events as facts" event sourcing paradigm, combining clean architecture layering with use case layer abstraction, providing full-chain capabilities from aggregate roots, event stores, event buses to task scheduling and authentication/authorization. Through unified configuration center, containerized startup process, and pluggable infrastructure (NATS/JetStream, RabbitMQ, Kafka, SQL/Redis/Badger, etc.), Sparrow aims to reduce microservice development complexity and improve system maintainability, scalability, and evolvability.

Project Structure​

The repository adopts a modular organization divided by responsibility domains. The core directories and responsibilities are outlined below:

  • pkg/bootstrap: Application startup, container, configuration loading, HTTP service, and graceful shutdown
  • pkg/config: Centralized configuration model and default values
  • pkg/entity: Domain model and DDD aggregate root interfaces
  • pkg/usecase: Use case layer abstraction (repositories, event stores, sessions, Sagas, executors, etc.)
  • pkg/persistence/repo: Multi-datasource repository implementations (SQL, Redis, Badger, Mongo, etc.)
  • pkg/messaging: Event bus and stream processing (NATS JetStream, subscribe/publish, stream Hub)
  • pkg/tasks: Task scheduler and task lifecycle management
  • pkg/auth: Token model and authentication-related capabilities
  • pkg/adapter/http: HTTP adapters (routing and handler placeholders)
  • pkg/logger, pkg/ws, pkg/codes, pkg/errs, etc.: Supporting modules
  • Application Launcher (App): Responsible for configuration loading, logger initialization, HTTP service startup, subprocess management, and graceful shutdown; built-in exponential backoff retry mechanism ensures subsystem stability.
  • Domain Model (AggregateRoot): Unified aggregate root interface supporting event sourcing, snapshots, version control, and business validation.
  • Use Case Layer Abstraction: Defines core use case contracts such as repositories, event stores, sessions, and Sagas through interfaces, decoupling business logic from implementation details.
  • Multi-datasource Repositories: Provides generic CRUD and advanced query capabilities for SQL, Redis, Badger, Mongo, etc., supporting soft delete, pagination, conditional queries, and batch operations.
  • Event Bus and Stream Processing: Event publish/subscribe, stream Hub, and event store integration based on NATS JetStream, supporting domain event to generic event conversion.
  • Task Scheduling: Unified task interface and scheduler abstraction, supporting immediate, scheduled, and periodic tasks, providing concurrent/sequential execution modes and retry strategies.
  • Authentication and Authorization: Token model and middleware integration points, facilitating integration with JWT, RSA permission verification, and other capabilities.
  • HTTP Adapter: Routing and handler placeholders based on Gin, facilitating quick integration of REST APIs.

Sparrow adopts an event-driven clean architecture layering, combining event sourcing and CQRS principles to form a "command-event-query" closed loop. After application startup, subsystems are injected through containers and loaded from the configuration center; events are generated from aggregate roots, propagated through event stores and event buses, and projection/view layers are updated in real-time; task schedulers are responsible for asynchronous job orchestration; authentication and authorization run throughout the request chain.

Application Launcher (App) and Container​

  • Responsibilities: Load configuration, initialize logger, build HTTP engine, register containers, manage subprocesses, and gracefully shut down.
  • Key Capabilities:
    • Subprocess Exponential Backoff Retry: Automatic retry on failure, up to 10 times, with a maximum 1-minute backoff.
    • Graceful Shutdown: Signal listening, timeout control, resource cleanup.
    • NATS/JetStream Integration: Provides event publish/subscribe and stream Hub.
  • Design Points: Achieves component decoupling and replaceability through container registration and resolution.
  • Aggregate Root Interface: Unified definition of aggregate type, ID, version control, event handling and application, uncommitted event management, snapshot loading, and business validation.
  • Templates and Generation: Provides aggregate root templates and YAML configuration-driven code generation to accelerate DDD practice implementation.
  • Event Store Contract: Distinguishes between database-type and event stream-type event stores, supporting queries, snapshots, and batch operations.
  • Use Case Layer Interfaces: Core abstractions such as repositories, event stores, sessions, Sagas, and executors, shielding underlying implementation differences.
  • Executor: Unified command/query executor interface, supporting Exec(action) pattern, facilitating business intent extension.
  • SQL Repository: Supports soft delete, transactions, batch operations, pagination, conditional queries, random sampling, etc.; unified timestamp handling and error wrapping.
  • Redis/Badger/Mongo: Provides generic CRUD and query capabilities for key-value/document stores, adapting to different scenario performance and consistency requirements.
  • Design Points: Combines generics and reflection to reduce repetitive code; unified error types and logging.
  • Stream Hub: Encapsulates NATS connection, providing subscription registration, startup, and graceful shutdown capabilities.
  • Event Publisher: Reads events from event store and publishes to event bus, supporting domain event to generic event conversion and topic naming constraints.
  • Design Points: Event store and event bus decoupling, event stream as the "single source of truth".
  • Scheduler Interface: Unified Schedule, Cancel, status query, concurrency settings, start/stop, and graceful shutdown.
  • Task Model: Task ID, type, scheduled time, handler function, callback, periodicity, and timeout control.
  • Execution Modes: Concurrent and sequential modes, supporting immediate, scheduled, and periodic tasks.

  • Token Structure: Contains access token, refresh token, scope, and expiration time, facilitating integration with JWT, RSA, and other authentication schemes.

  • Design Points: Works with HTTP adapter and middleware to provide unified authentication entry point.

  • Routing and handlers are currently in placeholder state, facilitating future REST API extensions as needed.

  • Design Points: Decoupled from use case layer, accessing business services through dependency injection.

  • Technology Stack Choices:

    • Web: Gin (HTTP routing and middleware)
    • Messaging: NATS (JetStream), RabbitMQ, Kafka
    • Databases: SQL (PostgreSQL/SQLite), Redis, Badger, MongoDB
    • Configuration: Viper
    • Logging: zap (wrapped as project Logger)
    • Security: JWT, RSA, Casbin

  • Dependency Characteristics: High cohesion, low coupling; replaceability achieved through interface abstraction and container injection; event bus and event store separated, supporting multiple implementations.

  • Event Store and Event Bus Decoupling: Event stream as the "single source of truth", avoiding write concurrency conflicts and improving throughput.
  • SQL Repository Transaction Batching: Batch save and update operations execute within transactions, reducing round-trip overhead.
  • Task Scheduling Concurrency Control: Balances throughput and consistency through maximum concurrency settings and execution modes (concurrent/sequential).
  • Logging and Error Wrapping: Unified logging and error types, facilitating performance bottleneck and anomaly identification.

Troubleshooting Guide​

  • Startup Failure: Check configuration loading and logger initialization; review App startup logs and error stack traces.
  • Event Publishing Failure: Confirm event store connection, event bus availability, and topic naming; verify event conversion and serialization.
  • Task Execution Anomalies: Review task status, retry count, and callback logs; check execution mode and timeout settings.
  • Repository Operation Anomalies: Verify entity ID, soft delete fields, and transaction commits; review unified error types and context information.

Sparrow provides evolvable, scalable, and observable infrastructure for enterprise-level microservices through event-driven architecture and domain-driven design. Its multi-datasource support, event bus and stream processing, task scheduling, and authentication/authorization capabilities can effectively address challenges in complex business scenarios. It is recommended to combine DDD aggregate root templates and event store contracts in actual projects, gradually improving the use case layer and adapter layer to achieve higher development efficiency and system stability.

Appendix​

  • Refactoring and Optimization: The project undergoes continuous refactoring, covering event bus implementation, repository layer, event store, error handling, and test coverage, significantly improving code quality and consistency.
  • Version and Evolution: Through unified configuration model and containerized startup process, Sparrow supports smooth upgrades and canary releases, facilitating long-term evolution.