Skip to main content

Development History

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

This document systematically outlines the evolution process of the Sparrow microservice framework from initial design to current version, focusing on the following aspects:

  • Key Version Milestones: Locating framework version evolution clues through application version numbers and configuration default values marked in the code.
  • Major Feature Additions and Architecture Adjustments: Tracing the introduction and optimization of functional modules with event bus, task scheduling, repository abstraction, aggregate root templates as the main threads.
  • Technical Decisions and Design Trade-offs: Summarizing framework technology choices from aspects of dependency library selection, logging and configuration systems, error handling, and testing strategies.
  • Community and Open Source Collaboration: Demonstrating community contribution and continuous improvement activity through refactoring checklists, documentation, and toolchains.
  • Future Planning and Roadmap: Proposing expected functional development directions and practice suggestions based on existing capabilities and refactoring plans.
  • Version Compatibility and Migration Guide: Providing compatibility judgments and migration points based on existing code to assist technology selection.

Project Structure​

Sparrow adopts a modular organization divided by responsibility, with core directories and responsibilities outlined below:

  • pkg/bootstrap: Application bootstrap and container assembly, responsible for configuration loading, HTTP engine initialization, subprocess startup, and graceful shutdown.
  • pkg/config: Centralized configuration model and default value settings, covering app, server, logging, NATS, SQL, Redis, Casbin, etc.
  • pkg/logger: Logging encapsulation and production/development mode switching, providing unified logging interface.
  • pkg/messaging: Event stream bus and subscription mechanism, supporting JetStream, RabbitMQ, Redis, and other backend adapters.
  • pkg/tasks: Task scheduling abstraction and multiple scheduler implementations, supporting concurrent/sequential, scheduled/periodic tasks.
  • pkg/entity: Aggregate root templates and event sourcing support, providing code generation and usage examples.
  • pkg/persistence/repo: Repository layer implementation and abstraction, covering in-memory, Badger, PostgreSQL, Redis, MongoDB, etc.
  • pkg/usecase: Use case layer abstraction and event store, session services, etc.
  • pkg/adapter: Adapter layer, including HTTP handlers, middleware, projection, and Saga coordinators, etc.
  • .trae/documents: Documentation and iteration plans, reflecting architecture evolution and practice accumulation.
  • Application Bootstrap and Lifecycle
    • Application entry point responsible for loading configuration, initializing logger, building Gin engine, registering service container, and starting subprocesses and HTTP server; supports exponential backoff retry and graceful shutdown.
    • Key Points: Subprocesses uniformly implement GracefulClose interface, ensuring timeout control and resource recovery during shutdown.
  • Configuration System
    • Provides centralized configuration model through Viper, covering default values and runtime parameters for app, server, logging, NATS, SQL, Redis, Casbin, and other modules.
  • Logging System
    • Logger wraps zap, supports development and production mode switching, production mode enables rotation logging and level control.
  • Event Bus and Stream Processing
    • StreamHub encapsulates JetStream/other backend subscribers, providing subscription registration and start/close capabilities.
  • Task Scheduling
    • TaskScheduler abstraction defines task scheduling, cancellation, status query, and concurrency control; supports immediate, scheduled, and periodic tasks.
  • Aggregate Root Templates
    • Provides YAML-based aggregate root generation templates, supporting event sourcing, snapshots, version control, and validation rules.

Sparrow's overall architecture centers around "bootstrap layer - configuration layer - communication layer - task layer - domain layer - persistence layer - adapter layer", forming clear layering and decoupling. Bootstrap layer is responsible for application lifecycle and resource management; configuration layer provides unified parameters; communication layer connects services with event streams as the core; task layer supports asynchronous and scheduled jobs; domain layer implements business modeling through aggregate root templates and event sourcing; persistence layer provides multi-backend repository implementations; adapter layer interfaces with HTTP, projection, and Saga.

Application Bootstrap and Lifecycle (App)​

  • Version Identification: Application version number declared in bootstrap layer, reflecting version evolution.
  • Startup Process: Load configuration and logging → Build Gin engine → Register service container → Start subprocesses (with exponential backoff retry) → Start HTTP server → Signal listening → Graceful shutdown.
  • Key Points: Subprocesses uniformly implement Startable/GracefulClose interfaces, ensuring retryability and controllable shutdown; HTTP server configuration includes timeout parameters, ensuring stability.
  • Configuration Model: Modular configuration for App, Server, CORS, Log, NATS, SQL, Redis, Badger, RabbitMQ, Kafka, Casbin, etc.
  • Default Values: Set reasonable default values for each module through SetDefaults, facilitating local development and minimal configuration.
  • Runtime Parameters: Supports key parameters such as host, port, log level, NATS connection, SQL/Redis/Badger paths.
  • Design Goals: Unified logging interface, supports development and production modes; production mode enables rotation logging and level control.
  • Key Points: Choose different logging implementations based on configuration mode, provides common methods such as Info/Error/Fatal/Warn.
  • StreamHub: Encapsulates subscribers, providing subscription registration, startup, and shutdown capabilities; internally uses JetStream/other backend implementations.
  • Key Points: Subscribers need to implement Startable/GracefulClose interfaces, ensuring controllable lifecycle.
  • Abstraction Definition: TaskScheduler interface defines scheduling, cancellation, status query, concurrency control, and lifecycle management.
  • Task Model: Task/TaskInfo/TaskSchedule/TaskStatus and other types constitute the task domain model.
  • Execution Modes: Supports concurrent and sequential execution, meeting throughput and consistency needs of different business scenarios.
  • Template Features: Supports interface implementation, event sourcing, snapshots, version control, and configuration-driven code generation.
  • Usage Method: Configure aggregate root fields and validation rules through YAML, use scripts to generate aggregate root code, and supplement event handling logic.
  • Integration Examples: Works with event store and snapshots to implement state recovery and event replay.
  • Language and Toolchain: Go 1.25, Makefile provides build, test, format, lint, Docker-related commands.
  • External Dependencies: NATS, RabbitMQ, Redis, MongoDB, PostgreSQL, Casbin, Gin, Zap, Viper, Testify, OpenTelemetry, etc.
  • Dependency Relationships: Framework manages dependencies through go.mod, core modules are decoupled through interfaces, avoiding strong coupling.
  • Event Bus and Stream Processing: Choice of JetStream/RabbitMQ/Redis backends should combine throughput and consistency requirements; production mode recommends enabling log rotation and reasonable log levels.
  • Task Scheduling: Choose concurrent/sequential execution mode based on business load; set reasonable timeout and retry strategies for tasks to avoid resource exhaustion.
  • Repository Layer: Choose appropriate index and query strategies for different backends (in-memory/Badger/Redis/PG/Mongo) to reduce unnecessary full table scans.
  • Logging and Monitoring: Use Logger interface uniformly to output key metrics and error stack traces, combine with OpenTelemetry for link tracing and metrics collection.

Troubleshooting Guide​

  • Startup Failure and Retry: App enters exponential backoff retry process when subprocess startup fails, can view retry count and final failure reason through logs.
  • Graceful Shutdown: HTTP server and subprocesses both support timeout-controlled graceful shutdown, if shutdown times out, check subprocess Close implementation and resource occupation.
  • Configuration Issues: Confirm if configuration items are correctly loaded and default values meet expectations; print configuration structure if necessary to locate issues.
  • Log Location: Enable rotation logging in production mode, focus on error level logs and key fields, combine context information for quick problem location.

Sparrow framework has formed relatively complete infrastructure in event-driven, task scheduling, and domain modeling, achieving good scalability and maintainability through modularization and interface abstraction. The current version has production-ready basic capabilities in configuration, logging, event bus, and task scheduling; combined with refactoring checklists and documentation, further improvements can be made in error handling consistency, test coverage, and template generation in the future. For users and contributors, the framework provides clear development directions and participation opportunities.

Appendix​

Version Compatibility and Migration Guide​

  • Version Identification: Application version number declared in bootstrap layer, can be used to distinguish deployments and upgrades of different versions.

  • Migration Suggestions:

    • Configuration Migration: If new configuration items are added, recommend providing default values through SetDefaults and gradually introducing new parameters during upgrades.
    • Logging Migration: Use Logger interface uniformly, avoid directly creating log instances; enable rotation logging in production mode.
    • Event Bus Migration: Subscribers need to implement Startable/GracefulClose interfaces, ensuring controllable lifecycle.
    • Task Migration: Adjust execution mode and concurrency strategy based on business load; set reasonable timeout and retry limits for tasks.
    • Aggregate Root Templates: When upgrading templates, note compatibility of event handling logic and validation rules.

  • Error Handling and Testing: Refer to refactoring checklist, unify error handling patterns and test coverage, improve code quality and maintainability.

  • Event Bus Optimization: Improve JetStream event bus implementation, reduce reflection and generic map parsing, improve runtime safety and performance.

  • Repository Layer Abstraction: Continue to simplify BaseRepository interface, use composition rather than inheritance, reduce interface complexity and implementation cost.

  • Event Store Tools: Reuse generic event serialization/deserialization tools, reduce repetitive code, improve consistency.

  • Code Standards and Naming: Fix spelling errors and unify struct export strategies, improve error types and factory functions.

  • Test-Driven: Add unit tests for key components following TDD principles, covering event bus, repository layer, and utility functions.

  • Documentation and Iteration: Accumulate best practices and evolution experience through documents and iteration plans in .trae/documents.

  • Toolchain: Makefile provides standardized development and operations processes, facilitating community members to get started quickly.

  • Contribution Suggestions: Follow existing code style and testing strategies, prioritize starting with issues in refactoring checklist, gradually improve framework quality and coverage.