Skip to main content

Redis Repository

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

This document systematically explains Redis-based repository implementation in Sparrow, covers design principles, data structure selection, key space design, serialization strategy, expiration strategy, Redis-specific operation support for sets/sorted sets/hashes, pipeline execution optimization, transaction processing, connection configuration, cluster and sentinel mode configuration, performance tuning, memory monitoring and failure recovery mechanisms. The goal is to provide operations and development teams with a directly actionable Redis repository usage guide.

Project Structure​

Key modules around Redis repository are distributed as follows:

  • Repository Implementation: pkg/persistence/repo/redis.go
  • Event Bus (Redis Pub/Sub): pkg/eventbus/redis/redis.go
  • Redis Configuration Model: pkg/config/redis.go
  • Application Layer Database Aggregation: pkg/bootstrap/db.go
  • Application Startup and Container: pkg/bootstrap/app.go, pkg/bootstrap/container.go
  • Test Cases: pkg/persistence/repo/redis_test.go

Core Components​

RedisRepository[T]​

  • Generic repository implementation, encapsulates entity CRUD, batch operations, pagination, conditional query, random sampling, TTL management, etc.

redisBus​

  • Event bus based on Redis Pub/Sub, responsible for event publishing and subscription.

RedisConfig​

  • Redis connection configuration model (host, port, password, DB, event store dedicated DB).

Database​

  • Application layer aggregation database client (Redis, SQL, Badger).

Container​

  • Simple IoC container, supports named and anonymous dependency resolution.

Architecture Overview​

Redis repository injects Redis client through configuration and container during application startup phase, repository instance holds client and organizes data with "key prefix:entity ID" key space; event bus also uses Redis client for publish/subscribe. Both repository and event bus depend on unified Redis configuration model.

Detailed Component Analysis​

RedisRepository[T] Design and Implementation​

  • Generic and Interface Compliance: Implements usecase.Repository[T], extends capabilities through BaseRepository[T].
  • Key Space Design: Key format is "prefix:id", prefix is passed by constructor, ensures different entity types isolation.
  • Serialization Strategy: Unified JSON serialization/deserialization, convenient for cross-language and human-readable.
  • Expiration Strategy:
    • Global TTL can be set during construction; if >0, Set operation carries TTL.
    • Supports setting/querying TTL for specific keys (SetTTLForKey, GetTTLForKey), as well as global TTL query and setting.
  • Data Structure Selection:
    • String Storage: Key-value pair stores complete entity JSON.
    • Batch Read/Write: MGET, DEL, EXISTS, KEYS and other command combinations achieve efficient batch operations.
  • Pipeline Execution Optimization: SaveBatch internally uses Pipeline, reduces RTT, improves throughput.
  • Conditional Query and Pagination: FindWithConditions/FindByFieldWithPagination/CountWithConditions based on memory filtering and sorting, suitable for small to medium scale datasets.
  • Random Sampling: Random achieves approximate random sampling through Keys + random shuffle + MGET.
  • Error Handling: Unified wrapping RepositoryError, clarifies entity type, operation, ID and error information.

Redis Event Bus (redisBus)​

  • Publish: Serializes event JSON and publishes to channel (event type as subject).
  • Subscribe: Maintains a PubSub list for each event type, starts goroutine to asynchronously process messages.
  • Lifecycle: Closes all subscriptions and Redis connections during Close, ensures graceful shutdown.
  • Error Handling: Subscription failure, message parsing failure, handler exceptions all have log recording and error return.

Key Design and Serialization​

  • Key Space: prefix:id, prefix is passed by constructor, ensures different entity types isolation; internally automatically completes trailing colon.
  • Serialization: Unified JSON; deserialization failure returns error, avoids dirty data.
  • TTL: Supports global TTL and single key TTL; GetTTLForKey returns -2 indicates key does not exist (Go-Redis behavior).

Batch Operations and Transactions​

  • Pipeline Execution: SaveBatch uses Pipeline, submits multiple Set instructions at once, significantly reduces network round trips.
  • Transaction Processing: Current repository does not explicitly use Multi/Exec; event bus also does not use transactions. For strong consistency scenarios, can encapsulate Multi/Exec in upper business layer.

Conditional Query and Pagination Implementation​

Repository implements "full table scan + memory filtering/sorting/pagination" on Redis, suitable for small to medium scale datasets. Process is as follows:

Cluster and Sentinel Configuration​

  • Current Implementation: Uses redis.NewClient and basic options (address, password, DB) to create client.
  • Cluster/Sentinel: Code does not explicitly use go-redis's ClusterOptions/SentinelOptions. If cluster/sentinel is needed, please replace client creation logic in application initialization, pass corresponding Options.

Dependency Analysis​

Application Layer Integration​

  • Database aggregates RedisClient, for repository and event bus use.
  • App creates Database during startup and injects into container, repository resolves to get RedisClient through container.
  • Container supports named and anonymous dependency resolution, convenient for multi-repository/multi-client scenarios.

Module Dependency Relationships​

  • Repository depends on Redis client and configuration model, provides unified entity operation interface.
  • Event bus depends on Redis client and configuration model, provides publish/subscribe capability.
  • Application layer injects Database through container, repository and event bus share same RedisClient.

Performance Considerations​

Optimization Recommendations​

  • Pipeline Optimization: SaveBatch uses Pipeline, reduces RTT, suitable for batch writes.
  • Batch Read: FindAll/FindByIDs uses MGET, reduces multiple network round trips.
  • Key Space Isolation: Isolates different entity types through prefix, avoids key conflicts.
  • TTL Strategy: Reasonably set global TTL and single key TTL, combines business lifecycle to control memory usage.
  • Conditional Query: Currently memory filtering, recommend introducing index or external search engine for large scale data scenarios.
  • Connection Reuse: Repository and event bus share same RedisClient, avoids duplicate connections.

Troubleshooting Guide​

Common Issues​

  • Connection Failure: Check RedisConfig's Host/Port/Password/DB; event bus will Ping to validate connection during creation.
  • Key Not Exists: FindByID returns RepositoryError, message indicates entity not found; GetTTLForKey returns -2 indicates key does not exist.
  • Batch Operation Exception: SaveBatch/SaveBatch/DeleteBatch will return error when any entity fails; check entity ID and JSON serialization.
  • Conditional Query No Result: Confirm field name matches JSON tag; Operator supports EQ/NEQ/GT/GTE/LT/LTE/LIKE/IN/IS_NULL/IS_NOT_NULL.
  • Random Sampling: When entity count is less than sample count, returns all entities; note Keys/random shuffle overhead.

Conclusion​

Sparrow's Redis repository uses concise key space design and JSON serialization as core, combines pipeline optimization and batch read/write, meets common entity persistence requirements. Event bus provides lightweight decoupled communication based on Redis Pub/Sub. For large scale data and strong consistency scenarios, recommend introducing index/search engine and transaction encapsulation; for high availability and elastic scaling, recommend replacing with cluster/sentinel client in application layer.

Appendix​

Redis-Specific Operation Support Status​

  • Sets/Sorted Sets/Hashes: Current repository does not directly use these data structures; if needed, can encapsulate corresponding commands in upper business layer.
  • Pipeline Execution: Already uses Pipeline to optimize batch writes.
  • Transaction Processing: Does not explicitly use Multi/Exec; if strong consistency is needed, can encapsulate in business layer.

Cluster and Sentinel Configuration Recommendations​

  • Current configuration model only includes basic fields; if cluster/sentinel is needed, should create corresponding Options in application initialization and inject into repository and event bus.
  • Recommend enabling connection pool parameters in production environment (such as minimum idle connections, maximum idle connections, connection timeout, etc.) to improve stability.

Testing Strategy​

  • Uses testcontainers to start Redis container, ensures test environment consistency.
  • Covers Save/Find/Delete/batch operations/pagination/conditional query/TTL/random sampling and other critical paths.
  • Validates error scenarios (empty ID, non-existent key, deserialization failure, etc.).