Skip to main content

Memory Repository

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendix

Introduction​

This document is a comprehensive technical document for Sparrow memory repository implementation, focusing on memory-based generic repository design and implementation. This implementation adopts generic entity model, provides complete CRUD operations, batch operation optimization, pagination query, conditional query, random retrieval and other functions, and ensures concurrent safety through read-write locks. The document also covers reflection operation usage scenarios, error handling strategies, testing strategies and examples, as well as performance characteristics and optimization recommendations, helping developers efficiently use memory repository in test and development environments.

Project Structure​

Memory repository is located in the repository module of persistence layer, follows Clean Architecture layered organization:

  • Repository Interface and Basic Capabilities: Located in usecase layer, defines generic repository contracts and query options.
  • Entity Model: Located in entity layer, provides unified entity interface and base entity structure.
  • Memory Implementation: Located under persistence/repo, implements memory-based repository logic.
  • Error Handling: Located in errs layer, provides unified repository error types.
  • Utilities and Helpers: Located in utils layer, provides sorting and random utility methods.
  • Application Integration: Located in bootstrap layer, provides application container and repository resolution capabilities.

Core Components​

Generic Memory Repository MemoryRepository[T]​

  • Based on generic entity type, implements complete CRUD, batch operations, pagination, conditional query, random retrieval, etc.
  • Uses read-write locks to ensure concurrent safety, internally stores entities with map[string]T.

Base Repository BaseRepository[T]​

  • Defines repository contracts and default behaviors, provides query options, conditions and sorting structures.

Entity Model​

  • Entity interface and BaseEntity structure, unified entity identification and timestamp management.
  • Task entity as example entity, demonstrates how to embed BaseEntity and implement Entity interface.

Error Handling​

  • RepositoryError provides unified repository error type, contains entity type, operation, ID, message and cause.

Utilities and Helpers​

  • SortEntities provides entity sorting utility, minimizes reflection usage; Random method for random entity retrieval.

Architecture Overview​

Memory repository adopts "generic + read-write lock" architecture design, combines reflection and type assertion, ensures type safety while improving performance and flexibility. Overall interaction is as follows:

Detailed Component Analysis​

Memory Repository Class Diagram​

Concurrency Control​

  • Write Operations (Save, Update, Delete, SaveBatch, DeleteBatch, Clear) use mutex lock (write lock), ensures only one write operation executes at the same time, avoids race conditions.
  • Read Operations (FindByID, FindAll, FindByIDs, FindWithPagination, Count, FindByField, Exists, FindWithConditions, CountWithConditions, Random) use read lock, allows multiple read operations to execute concurrently, improves read performance.
  • In Save, first checks if exists, then decides to insert or update, internally still uses mutex lock to ensure idempotency and consistency.

Generic Implementation​

  • MemoryRepository[T] and BaseRepository[T] are both generic implementations, requires T must implement Entity interface.
  • Through type assertion and reflection combination, achieves automatic setting and reading of entity fields (such as timestamps), ensures type safety and extensibility.
  • NewMemoryRepository[T] returns initialized memory repository instance, internally records entity type string, convenient for error message output.

Reflection Usage​

  • During insert and update, sets entity's CreatedAt/UpdatedAt fields through reflection, if entity does not declare corresponding fields then ignores.
  • In conditional query and field query, uses reflection to get field values and compare, supports comparison of multiple basic types and time.Time.
  • In Random method, gets entity ID list through reflection, then randomly shuffles and selects specified number of entities.

Basic CRUD Operations​

  • Save: Validates entity ID, if exists then updates, otherwise inserts; sets timestamp during insert.
  • Update: Validates ID and existence, sets update time then updates.
  • FindByID: Finds entity by ID, returns repository error if not exists.
  • FindAll: Returns all entity copies.
  • Delete: Deletes entity by ID.
  • Exists: Checks if entity exists.
  • Count: Returns entity count.

Batch Operations​

  • SaveBatch: Batch saves entities, acquires write lock once, sets timestamps one by one and writes, avoids multiple locking overhead.
  • FindByIDs: Batch finds, only one read lock, traverses ID list and filters non-existent items.
  • DeleteBatch: Batch deletes, one-time write lock, deletes one by one.
  • Above operations all quickly return for empty input, reduces meaningless processing.

Query and Pagination​

  • Pagination Query FindWithPagination: Prepares all entity slices, handles limit/offset boundaries, returns specified range.
  • Conditional Query FindWithConditions: Traverses entities, matches conditions one by one, supports EQ, NE, GT, LT, GTE, LTE and other operators.
  • Field Query FindByField: Matches by field name and value, supports type conversion and pointer dereference.
  • Random Retrieval Random: Randomly shuffles entity ID list, selects specified number of entities as needed.
  • Statistics CountWithConditions: Similar to conditional query, but only counts.

Error Handling​

  • RepositoryError: Unified repository error type, contains entity type, operation, ID, message and cause, convenient for locating problems.
  • For empty ID, entity not found, duplicate insert and other situations, returns clear error type and message.
  • Base repository provides default unimplemented error, ensures upper implementation covers key methods.

Testing Strategy​

  • Unit tests cover CRUD, batch operations, pagination, conditional query, random retrieval, concurrent insert and other scenarios.
  • Concurrent tests verify correctness and performance of read-write locks under high concurrency.
  • Uses Task entity as example, ensures tests are consistent with actual business.

Dependency Analysis​

Memory repository's dependency relationships are clear, responsibility separation is clear:

  • MemoryRepository[T] depends on Entity interface and BaseEntity structure, ensures entities have unified identification and timestamps.
  • Base repository BaseRepository[T] provides query options and default unimplemented methods, reduces concrete implementation burden.
  • Reflection and type assertion are used cautiously in memory repository, prefers type assertion to reduce reflection cost.
  • Utility layer SortEntities and Random methods provide auxiliary capabilities for query and sorting.

Performance Considerations​

Concurrency Model​

  • In read-heavy-write-light scenarios, read-write locks significantly improve throughput; write operations use mutex locks, avoids competition.

Reflection and Type Assertion​

  • During insert/update, sets timestamps through reflection, field existence check and type judgment ensure safety; uses reflection in conditional query and field query, but only when necessary.

Memory Usage​

  • Stores entities with map[string]T, suitable for small to medium scale data; for large scale data, recommend considering external storage or cache layer.

Pagination and Conditional Query​

  • FindWithPagination and FindWithConditions will traverse full data, suitable for small to medium scale data; for large scale data, recommend introducing index or external storage.

Random Retrieval​

  • Random method randomly shuffles entity IDs, time complexity O(n), suitable for scenarios with moderate entity count.

Troubleshooting Guide​

Common Error Types​

  • RepositoryError: Contains entity type, operation, ID, message and cause, convenient for quickly locating problems.

Common Problems and Solutions​

  • Empty ID Error: Ensure entity ID is not empty before calling Save/Update/Delete/Exists.
  • Entity Not Found: Check if entity has been saved, or confirm if ID is correct.
  • Concurrent Conflict: Ensure using read-write locks in high concurrency scenarios, avoids external race conditions.

Debugging Recommendations​

  • Use unit tests to cover critical paths, focus on concurrent tests and boundary conditions.
  • For conditional query and field query, check if field names and types match.

Conclusion​

Memory repository achieves high-performance, easy-to-extend generic repository capabilities through generics and read-write locks, suitable for test environments and scenarios that do not require persistence. Its design balances type safety, reflection and type assertion, provides complete CRUD, batch operations and query functions. In actual use, should choose appropriate deployment strategy according to data scale and concurrency characteristics, and ensure stability through sufficient unit tests and concurrent tests.

Appendix​

Usage Guide​

  • Create Memory Repository Instance: Call NewMemoryRepository[T].
  • Based on Entity Interface: Ensure entity implements Entity interface, for automatic timestamp setting.
  • Query and Pagination: Use QueryOptions and pagination methods for conditional query and pagination.
  • Concurrent Access: In high concurrency scenarios, read-write locks can effectively ensure consistency and performance.

Performance Optimization Recommendations​

  • Control Entity Count: Memory repository is suitable for small to medium scale data, avoid overly large maps affecting performance.
  • Reduce Reflection Usage: In entity design, try to expose directly accessible timestamp fields, reduce reflection overhead.
  • Batch Operations: Prefer using SaveBatch, FindByIDs, DeleteBatch and other batch methods, reduce lock competition.
  • Conditional Query: For large scale data, recommend introducing external storage or index to improve query efficiency.