Skip to main content

Database Configuration

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion

Introduction​

This document provides comprehensive explanation of database configuration, covering configuration points, connection parameters, driver selection and connection pool settings for relational databases (PostgreSQL, MySQL, SQLite), Redis and BadgerDB, and provides performance optimization suggestions and troubleshooting guide. The document is organized based on configuration models and startup processes in the repository, helping developers quickly understand and deploy.

Project Structure​

Database configuration is mainly distributed in the following modules:

  • Configuration Layer: Defines various database configuration structures and default value loading
  • Startup Layer: Creates database connections and repository instances based on configuration
  • Repository Layer: Provides generic CRUD and query capability encapsulation

Core Components​

Configuration Structures​

  • SQLConfig: Driver, host, port, username, password, database name, event store dedicated database name
  • RedisConfig: Host, port, password, database number, event store dedicated database number
  • BadgerConfig: Data directory, event store directory, value threshold, compactor, compactor count

Default Values and Loading​

  • SetDefaults: Sets default values for each configuration item (such as default host, port, path for Redis/SQL/Badger, etc.)
  • Load: Loads configuration file, environment variables, validates configuration validity

Startup and Connection​

  • Database: Uniformly holds Redis, SQL, Badger clients
  • Startup Options: Registers SQL/Redis/Badger instances, builds repositories

Architecture Overview​

The overall process of database configuration and startup is as follows:

Detailed Component Analysis​

Relational Database Configuration (PostgreSQL, MySQL, SQLite)​

Driver and Connection Parameters​

  • Driver: Specified through SQLConfig.Driver (default postgres)
  • Connection Parameters: Host, Port, User, Password, Dbname, ESDbname (event store dedicated database name)
  • DSN Construction: SQLConfig.Dsn()/ESDsn() provides standard DSN format

Connection Pool Settings​

  • Code does not directly expose connection pool parameters (such as maximum connection count, idle count, lifetime), but can be configured during startup phase through database driver extension interface
  • Example: SQLiteDB startup option performed PRAGMA configuration for sqlite3 (WAL, sync level, cache size, busy_timeout, foreign key constraints)

Event Store Dedicated Database​

  • ESDsn and ESDbname used for independent database in event store scenario

Redis Configuration​

Connection Information and Authentication​

  • Host and Port: Host, Port
  • Password Authentication: Password
  • Database Selection: DB (default 0)
  • Event Store Dedicated Database: ESDB (default 1)

Client Creation​

  • newRedis constructs redis.Client based on configuration, includes Addr, Password, DB

BadgerDB Configuration​

Data Directory and Event Store Directory​

  • DataDir: Normal view storage directory
  • EventStoreDir: Event store directory

Performance Parameters​

  • ValueThreshold: Threshold, determines if small values enter LSM tree
  • NumCompactors: Compactor count
  • Compactor: Default ZSTD (can be set in startup options)

Memory and File Size​

  • Startup options set memtable size, value log file size and other parameters, used to balance throughput and resource occupation

Repository Implementation​

SQL Repository​

  • Generic implementation, supports transactions, soft delete detection, pagination, conditional query, batch operations
  • Executes SQL operations through underlying sql.DB

Redis Repository​

  • Based on key prefix namespace, supports TTL, batch Pipeline, full table scan conditional query

Badger Repository​

  • Based on key prefix and iterator, supports transaction read/write, batch update, pagination and conditional filtering

Dependency Analysis​

Configuration to Startup Dependency​

  • Config aggregates SQL/Redis/Badger configurations
  • Load responsible for default value injection and validation
  • Startup options create specific clients based on configuration and register to container

Repository to Database Dependency​

  • Repository executes operations through underlying database client
  • Different database features (such as soft delete, transactions) reflected in repository layer

Performance Considerations​

Relational Database (PostgreSQL/MySQL/SQLite)​

  • Connection pool parameters not directly exposed in configuration, can be set during startup phase through driver extension interface (such as sqlite3 PRAGMA already exemplified)
  • Recommend adjusting maximum connection count, idle connection count, connection lifetime combined with business concurrency and query characteristics

Redis​

  • Use Pipeline batch operations to reduce RTT
  • Reasonably set TTL and key space prefix, avoid full table scan
  • Choose appropriate database number based on data scale, isolate event store

BadgerDB​

  • Set memtable size, value log file size and compactor through startup options, balance throughput and resources
  • ValueThreshold controls storage strategy for small and large values, affects LSM tree structure and read/write performance
  • NumCompactors affects background compaction load, needs tuning according to data growth rate

Troubleshooting Guide​

Configuration Loading Failure​

  • Check configuration file format and path, confirm default values are correctly set
  • Confirm environment variable key name replacement rule matches actual environment variables

Connection Failure​

  • Validate host, port, password, database name are correct
  • For Redis: confirm password and database number; for SQL: confirm DSN format and driver availability

Runtime Errors​

  • SQL: Pay attention to transaction open, rollback and commit timing, ensure entity ID and soft delete field exist
  • Redis: Pay attention to key prefix and TTL settings, avoid data loss caused by mistaken deletion or expiration
  • Badger: Pay attention to key prefix and iterator usage, avoid concurrent read/write conflicts

Conclusion​

This project provides clear database configuration model and startup process, supports basic configuration and runtime access for relational databases, Redis and BadgerDB. For connection pool and more fine-grained performance parameters, supplementary configuration can be made during startup phase through driver extension interface. Recommend targeted optimization of connection pool, compaction strategy and key space design combined with business scenarios, and strengthen configuration validation and monitoring alerts in production environment.