Hybrid Scheduler

Table of Contents

1. Introduction
2. Project Structure
3. Core Components
4. Architecture Overview
5. Detailed Component Analysis
6. Dependency Analysis
7. Performance Considerations
8. Troubleshooting Guide
9. Conclusion
10. Appendix

Introduction

This document is technical documentation for the hybrid scheduler, systematically describing its architecture design and implementation principles, focusing on the following aspects:

• Coordination of multiple scheduling modes: unified orchestration of concurrent scheduling and sequential scheduling
• Dynamic task allocation: intelligent routing based on task type and scheduling configuration
• Intelligent scheduling decisions: differentiated processing of timed tasks, periodic tasks, and immediate tasks
• Load balancing and resource control: concurrency slot control, task expired cleanup, retry and dead letter handling
• Configuration methods and usage scenarios: flexible configuration through wrapper and application launcher
• Performance optimization strategies and tuning suggestions: concurrency limits, TTL, retry strategies and monitoring
• Practical application examples of complex scheduling scenarios: concurrent/sequential hybrid, timed/periodic tasks, retry and dead letter

Project Structure

The hybrid scheduler is located in the task module (pkg/tasks), centered around the unified interface TaskScheduler, containing three core implementations:

• Hybrid Scheduler (HybridScheduler): Composes concurrent and sequential schedulers, routes by task type and configuration
• Concurrent Scheduler (ConcurrentScheduler): Supports multi-task concurrent execution, with memory management, timeout and retry
• Sequential Scheduler (SequentialScheduler): Serial execution, suitable for timed and periodic tasks

• Task Interface and State
  • TaskScheduler: Unified scheduler interface, defining capabilities such as scheduling, cancellation, status query, listing, concurrency limit setting, startup/stop/shutdown
  • Task/TaskInfo/TaskInfoProvider: Task abstraction, task information and information provider interfaces
  • TaskStatus/ExecutionMode: Task status and execution mode enumerations
• Scheduler Implementation
  • HybridScheduler: Composes concurrent and sequential schedulers, routes by task type and configuration
  • ConcurrentScheduler: Concurrent execution, supports concurrency slot control, timed/retry task processing, expired cleanup
  • SequentialScheduler: Sequential execution, supports timed task triggering, periodic task rescheduling, expired cleanup
• Task Build and Wrap
  • TaskBuilder: Task builder, supports ID, type, handler, callbacks, scheduling time, timeout, retry, priority, etc.
  • SchedulerWrapper: Scheduler wrapper, provides convenient submission interface, default retry, concurrency limit setting, task type mode mapping
• Application Integration
  • App: Application instance, holds SchedulerWrapper
  • Tasks(): Launcher options, injects SchedulerWrapper into App

The hybrid scheduler provides capabilities externally through the unified interface TaskScheduler, internally composing schedulers with two execution modes: concurrent and sequential. Its core features include:

• Task Routing: Decides to use concurrent or sequential execution based on task type and configuration; timed and periodic tasks default to sequential execution
• Concurrency Control: Concurrent scheduler maintains active task count and condition variables, limiting maximum concurrency
• Sequential Control: Sequential scheduler maintains pending execution queue, executor and monitor, ensuring serial execution
• State Management: Unified task information (TaskInfo) and state (TaskStatus), supporting retry, dead letter, expired cleanup
• Wrapper Integration: SchedulerWrapper provides convenient API and default configuration, App integrates through Tasks() option

Hybrid Scheduler (HybridScheduler)

• Design Points
  • Composition Pattern: Holds concurrent and sequential two scheduler instances
  • Execution Mode Mapping: Maps task types to execution modes through executionModes, supports concurrent/sequential
  • Routing Strategy: Priority check of task type mapping; if timed or periodic task, force sequential execution; otherwise concurrent execution
  • Unified Lifecycle: Start/Stop/Close delegate to two sub-schedulers respectively
• Key Methods
  • WithConcurrent/WithSequential: Configure task type to execution mode mapping
  • Schedule: Select scheduler and execute according to rules
  • Cancel/GetTaskStatus/ListTasks: Find/operate in two schedulers
  • SetMaxConcurrentTasks: Only effective on concurrent scheduler
• Thread Safety: Execution mode mapping uses read-write lock protection

• Design Points
  • Concurrent Execution: Multiple tasks can run simultaneously, controlling concurrency slots through active task count and condition variables
  • Task Monitoring: Timer periodically scans timed tasks and retry tasks, triggers execution
  • Memory Management: Limits maximum task count, cleans up oldest completed/cancelled/dead letter tasks by update time
  • Timeout and Retry: Supports task timeout, retries by maximum retry count and linear backoff strategy when failed, enters dead letter when exceeding threshold
• Key Methods
  • Start: Start task monitor goroutine
  • Schedule: Store task, execute immediately when necessary
  • Cancel: Update status and trigger cancellation callback
  • Stop/Close: Wait for active tasks to complete or graceful shutdown
  • SetMaxConcurrentTasks: Set maximum concurrency
  • processScheduledTasks/cleanupExpiredTasks: Timed processing and cleanup
  • executeTask: Execute single task, handle timeout, retry and completion callback

• Design Points
  • Serial Execution: Only one task executes at the same time, ensuring order consistency
  • Pending Execution Queue: pendingTasks maintains pending task IDs, taskQueue provides buffering
  • Task Executor: Takes out tasks in order for execution, supports timeout and retry
  • Monitor: Periodically checks if timed tasks have reached execution time, cleans up expired tasks
  • Periodic Tasks: Recalculates next execution time and backfills queue after completion
• Key Methods
  • runTaskExecutor: Loop to find waiting timed/expired tasks and execute
  • runTaskMonitor: Periodically cleanup expired tasks and trigger timed tasks
  • executeTask: Execute single task, handle timeout, retry and completion callback
  • cleanupExpiredTasks/cleanupOldTasks: Expired and old task cleanup

• Design Points
  • SchedulerWrapperOptions: Default retry count, scheduler instance, task type mode mapping, maximum concurrency
  • NewSchedulerWrapper: Creates default hybrid scheduler according to options, applies concurrency limit and task type mapping
  • RunTask series: Encapsulates TaskBuilder, supports immediate, timed, periodic, delayed execution
  • SetTaskTypeMode/SetMaxConcurrentTasks: Dynamically adjust hybrid scheduler's mode and concurrency limit
• Application Integration
  • Tasks() option: Injects SchedulerWrapper into App, implements task system startup and cleanup

• Design Points

  • Supports ID, type, handler, completion/cancellation callbacks, scheduling time, timeout, retry count, priority
  • Build: Validates parameter legality, generates builtTask and fills TaskInfo
  • builtTask: Implements Task and TaskInfoProvider interfaces, provides task information access

• Cooperation with Scheduler

  • SchedulerWrapper builds tasks through TaskBuilder, uniformly injects default retry count

• Interface and Implementation

  • TaskScheduler is unified interface, HybridScheduler/ConcurrentScheduler/SequentialScheduler all implement this interface
  • Task and TaskInfoProvider are contracts for task and task information

• Wrapper and Scheduler

  • SchedulerWrapper holds TaskScheduler instance, provides high-level API
  • HybridScheduler composes ConcurrentScheduler and SequentialScheduler

• Application Integration

  • App holds SchedulerWrapper, Tasks() option injects it into application lifecycle

• Concurrency Limit Control
  • ConcurrentScheduler controls concurrency through active task count and condition variables, avoiding resource contention
  • SchedulerWrapper.SetMaxConcurrentTasks sets maximum concurrency, HybridScheduler only affects concurrent part
• Memory and Expired Cleanup
  • When task count reaches limit, removes oldest completed/cancelled/dead letter tasks
  • Expired tasks cleaned up by TTL, avoiding memory bloat
• Retry and Backoff
  • Concurrent scheduler uses linear backoff, sequential scheduler uses exponential backoff, avoiding avalanche effects
• Monitoring and Observability
  • Obtain task status through ListTasks and GetTaskStatus, facilitating monitoring and alerting
• Tuning Suggestions
  • Reasonably set maximum concurrency based on business peak
  • Use concurrent mode for high throughput tasks, sequential mode for strong consistency requirements
  • Reasonably configure retry count and backoff strategy, avoiding frequent retries causing jitter
  • Regularly cleanup expired tasks, maintaining system health

[This section is general performance guidance, no specific file source needed]

Troubleshooting Guide

• Task State Anomaly
  • Use GetTaskStatus to confirm task status, combine ListTasks to locate problems
• Concurrency Blocking
  • Check active task count and maximum concurrency setting, confirm if there are long-running tasks
• Task Not Executing
  • Check task type mapping and scheduling time, confirm if routed to sequential mode
• Retry Failure
  • View retry count and backoff strategy, confirm if maximum retry reached
• Resource Leak
  • Confirm Stop/Close call, ensure scheduler graceful shutdown

The hybrid scheduler achieves seamless connection of concurrent and sequential scheduling through unified interface and composition pattern. Its intelligent routing, concurrency control, sequential assurance and comprehensive lifecycle management make it suitable for diverse task scheduling scenarios. Cooperating with SchedulerWrapper and application launcher, it can be quickly implemented and flexibly tuned in actual business.

[This section is summary content, no specific file source needed]

Appendix

Configuration Methods and Usage Scenarios

• Configuration Methods

  • Set default retry count, maximum concurrency, task type mode mapping through SchedulerWrapperOption
  • Inject SchedulerWrapper into App through Tasks() option, implement task system startup and cleanup

• Usage Scenarios

  • High throughput asynchronous tasks: concurrent mode
  • Strong consistency timed tasks: sequential mode
  • Mixed scenarios: hybrid scheduler routes by type

• Concurrent/Sequential Hybrid Execution

  • Submit different types of concurrent/sequential tasks through RunTypedTaskWithID, verify execution mode

• Timed/Periodic Tasks

  • Submit timed/periodic tasks through RunTaskAt/RunTaskRecurring, sequential scheduler ensures strict order

• Retry and Dead Letter

  • Observe retry and dead letter behavior through default retry configuration and status check