Skip to main content

Message Queue Configuration

Table of Contents​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion

Introduction​

This document systematically organizes and explains message queue and event bus configuration in the project, covering the following:

  • Configuration parameters and connection methods for message middleware such as NATS, RabbitMQ, Kafka
  • Event bus configuration, including JetStream stream configuration, consumer group settings, message retention policy, etc.
  • Performance tuning configuration, including batch size, retry strategy, dead letter queue and other advanced features
  • Message reliability guarantee and failure recovery configuration

Project Structure​

Related modules around message queue and event bus are distributed as follows:

  • Configuration Layer: Defines configuration structures and default values for each middleware
  • Event Bus Layer: Event bus implementations for NATS/RabbitMQ
  • Event Stream Layer: Publish, subscribe and reader implementations based on NATS JetStream
  • Core Interface Layer: Unified publish/subscribe/read interface definitions

Core Components​

  • NATS Configuration Structure: Contains connection URL, event stream name, message maximum retention time and other fields
  • RabbitMQ Configuration Structure: Contains host, port, username, password, virtual host, exchange and other fields
  • Kafka Configuration Structure: Contains Broker list, consumer group ID, topics and other fields
  • Event Bus Interface: Unified publish/subscribe/unsubscribe/close interface
  • JetStream Publisher/Subscriber/Reader: Event stream implementation based on NATS JetStream

Architecture Overview​

The overall architecture is divided into three layers:

  • Configuration Layer: Centrally defines configuration items and default values for each middleware
  • Event Bus Layer: Provides application-oriented event publish/subscribe capabilities (NATS/RabbitMQ)
  • Event Stream Layer: Provides JetStream-based event stream capabilities (publish/subscribe/read/replay)

Detailed Component Analysis​

NATS Configuration and Event Bus​

Configuration Parameters​

  • Connection String: Used to connect to NATS server
  • Event Stream Name: Stream name used for event bus (compatible with standard event stream)
  • Maximum Retention Time: Event stream maximum retention time (unit: days, default permanent retention)

Event Bus Implementation Points​

  • Uses standard NATS connection, supports infinite reconnection and reconnection interval configuration
  • Publish/subscribe based on topic (event type), does not use JetStream persistence
  • Provides Pub/Sub/Unsub/Close capabilities

RabbitMQ Configuration and Event Bus​

Configuration Parameters​

  • Host, port, username, password, virtual host, exchange name

Event Bus Implementation Points​

  • Uses AMQP protocol connection, declares topic exchange
  • Publishes with event type and timestamp header information
  • Creates temporary queue for each topic during subscription and automatically acknowledges messages
  • Supports multiple subscribers, each subscription consumes independently

Kafka Configuration​

Configuration Parameters​

  • Broker List: Kafka cluster address list
  • Consumer Group ID: Used for partition rebalancing and offset management
  • Topics: Topic name for event publish/subscribe

Notes​

  • This repository does not provide specific Kafka implementation code, only defines configuration structure; actual use requires introducing corresponding dependencies and implementing publish/subscribe logic according to business scenarios.

JetStream Publisher Configuration​

Design Points​

  • Uses service name as stream name, aggregate root type as topic prefix, forms {aggregate type}.{aggregate ID}.{event type} topic pattern
  • Default uses LimitsPolicy and unlimited message/stream size, can set maximum message age, stream size, message size through options
  • Supports memory storage debug mode

Key Process​

JetStream Subscriber Configuration​

Design Points​

  • Uses durable consumer (Durable), explicit acknowledgment (AckExplicit) ensures reliable processing
  • Consumer filters topic as {aggregate type}.{event type}, supports delivery from earliest message and immediate replay
  • Supports graceful shutdown: waits for processing completion through context cancellation, WaitGroup and other mechanisms

Key Process​

JetStream Reader Configuration​

Design Points​

  • Reader uses ephemeral consumer (non-durable), avoids control plane pressure from frequent creation/destruction
  • Supports filtering by aggregate ID, replay by version range, replay by physical offset
  • Sorts by version ascending after reading and applies to aggregate root

Key Process​

Event Stream Bus Hub Configuration​

Design Points​

  • Uniformly encapsulates subscriber collection, provides AddSub, Start, Close and other capabilities
  • Uses service name as event stream name, local name for consumer unique identification

Key Process​

Dependency Analysis​

Configuration Layer Dependencies​

  • NATsConfig/RabbitMQConfig/KafkaConfig carried by Config uniformly
  • Default values injected through SetDefaults

Event Bus Dependencies​

  • NATS event bus depends on NATsConfig
  • RabbitMQ event bus depends on RabbitMQConfig

Event Stream Dependencies​

  • JetStream publisher/subscriber/reader depend on NATS connection and JetStream client
  • Event stream bus Hub depends on subscriber collection

Performance Considerations​

NATS​

  • Connection Retry: Supports infinite reconnection and fixed reconnection interval, improves availability
  • Topic Naming: Recommend dividing topics by event type, avoid excessive wildcards causing performance degradation

RabbitMQ​

  • Exchange Type: Uses topic exchange to support flexible routing
  • Auto Acknowledge: Subscription side uses auto acknowledge, reduces latency; if strong consistency needed, can change to manual acknowledge and combine with prefetch count
  • Queue Declaration: Temporary queue suitable for one-time subscription, long-term subscription recommend using persistent queue

Kafka​

  • Partition and Consumer Group: Reasonably set partition count and consumer group count, avoid excessive competition
  • Batch Commit: Adjust batch size and commit interval according to throughput requirements

JetStream​

  • Stream Policy: Default uses LimitsPolicy, suitable for event storage scenarios
  • Consumer Strategy: Explicit acknowledgment (AckExplicit) ensures reliability; choose DeliverAllPolicy or offset-based delivery according to scenario
  • Debug Mode: Can switch to memory storage to reduce disk IO impact on testing

Retry Strategy​

  • Maximum retry count, initial/maximum backoff time, backoff multiplier, whether to enable retry
  • Recommend combining with idempotent processing and dead letter queue, avoid repeated processing

Troubleshooting Guide​

NATS Event Bus​

  • Connection Failure: Check URL and network connectivity; confirm infinite reconnection configuration takes effect
  • Publish Failure: Check topic and event serialization; view log output
  • Subscription Exception: Confirm event type matches topic; check handler return errors

RabbitMQ Event Bus​

  • Connection Failure: Verify host, port, username, password, virtual host
  • Exchange Declaration Failure: Confirm permissions and exchange name; manually create if necessary
  • Consumption Exception: Check queue binding and auto acknowledge settings; pay attention to handler error logs

JetStream Event Stream​

  • Publish Failure: Confirm stream exists and topic matches; check message size limit
  • Subscribe Failure: Confirm consumer creation and filter topic; check Ack/Nak behavior
  • Read Failure: Confirm if stream exists; check ephemeral consumer creation and message fetching

General Recommendations​

  • Log Level: Production environment recommend info or higher; can temporarily elevate during problem diagnosis
  • Timeout Control: Set reasonable context timeout for publish/subscribe/read
  • Resource Cleanup: Ensure WaitGroup and other synchronization primitives wait correctly during graceful shutdown

Conclusion​

This project provides complete middleware configuration and event bus implementation, covering key capabilities of NATS, RabbitMQ and JetStream. Through unified interfaces and clear configuration structures, can meet publish/subscribe, event stream storage and replay and other requirements under event-driven architecture. Recommend further optimizing retry strategy, batch size and consumer group configuration combined with business characteristics in production environment, and supporting monitoring and alerting system to ensure stability and observability.