Enhanced Early Stopping with min_epochs Parameter

Overview

The Trixy ML training framework now includes enhanced early stopping functionality that supports a min_epochs parameter. This ensures that training continues for at least a specified number of epochs before early stopping can take effect, giving models sufficient time to learn initial patterns.

Why min_epochs is Important

Problem with Standard Early Stopping

Standard early stopping can terminate training too early, especially for:

Complex models that need time to converge
Deep learning models with slow initial learning phases
Audio models that need to learn spectral patterns
Speaker recognition where embeddings need time to stabilize

Benefits of min_epochs

Prevents premature termination during initial learning phases
Ensures model stability before considering stopping
Improves final model quality by allowing sufficient training time
Maintains backward compatibility with existing configurations

Configuration

TrainerConfig Parameters

from trainer.base import TrainerConfig

config = TrainerConfig(
    num_epochs=100,           # Maximum number of epochs
    min_epochs=20,            # Minimum epochs before early stopping can trigger
    early_stopping_patience=10, # Epochs to wait for improvement after min_epochs
    # ... other parameters
)

Parameter Relationships

min_epochs: Minimum number of epochs that must complete before early stopping is considered
num_epochs: Maximum number of epochs (training stops here regardless)
early_stopping_patience: Number of epochs to wait for improvement after min_epochs
max_epochs: Hard upper limit (for safety)

Validation Rules

The configuration system validates these relationships:

min_epochs >= 0
min_epochs <= num_epochs
min_epochs <= max_epochs
num_epochs <= max_epochs
early_stopping_patience > 0

How Enhanced Early Stopping Works

Training Flow

Initial Phase (epochs 0 to min_epochs-1):
- Training proceeds normally
- Validation loss is tracked
- Best model state is saved when validation improves
- Early stopping counter is NOT incremented (even if validation worsens)
- Early stopping counter is reset to 0 if still below min_epochs
Early Stopping Phase (epochs min_epochs and beyond):
- Early stopping counter increments when validation doesn't improve
- Training stops when counter reaches early_stopping_patience
- Best model state from entire training is restored

Code Example

# Enhanced early stopping logic
if val_loss is not None:
    if val_loss < self.metrics.best_val_loss:
        self.best_model_state = self.model.state_dict().copy()
        self.early_stopping_counter = 0
    else:
        # Only increment counter after min_epochs
        if epoch >= self.config.min_epochs:
            self.early_stopping_counter += 1
            
            if self.early_stopping_counter >= self.config.early_stopping_patience:
                self.logger.info(f"Early stopping triggered after {epoch + 1} epochs "
                               f"(minimum {self.config.min_epochs} epochs completed)")
                break
        else:
            # Reset counter if we're still below min_epochs
            self.early_stopping_counter = 0

Examples

Wakeword Detection

config = TrainerConfig(
    trainer_name='wakeword_trainer',
    num_epochs=100,
    min_epochs=20,              # Allow 20 epochs to learn wakeword patterns
    early_stopping_patience=15, # Then wait 15 epochs for improvement
    # With this config, earliest stopping is at epoch 35 (20 + 15)
)

Voice Recognition

config = TrainerConfig(
    trainer_name='voice_recognition_trainer', 
    num_epochs=200,
    min_epochs=30,              # Allow 30 epochs for embeddings to stabilize
    early_stopping_patience=20, # Then wait 20 epochs for improvement
    # With this config, earliest stopping is at epoch 50 (30 + 20)
)

Conservative Training

config = TrainerConfig(
    num_epochs=500,
    min_epochs=100,             # Ensure substantial training
    early_stopping_patience=50, # Very patient after min_epochs
    # With this config, earliest stopping is at epoch 150 (100 + 50)
)

Training Scenarios

Scenario 1: Normal Early Stopping

Epochs: 1-19   | min_epochs phase, validation worsening, counter stays 0
Epoch 20:      | min_epochs reached, validation still bad, counter = 1
Epochs 21-24:  | validation continues worsening, counter = 2,3,4,5
Epoch 25:      | counter = 6, reaches patience (5), training stops
Result:        | Training stops at epoch 25 (min_epochs + patience)

Scenario 2: Improvement During min_epochs

Epochs: 1-10   | min_epochs phase, validation worsening, counter stays 0
Epoch 11:      | validation improves! counter reset to 0
Epochs 12-19:  | validation worsens again, but still in min_epochs, counter stays 0
Epoch 20:      | min_epochs reached, validation bad, counter = 1
Epochs 21-24:  | validation continues worsening, counter = 2,3,4,5
Epoch 25:      | counter = 6, reaches patience, training stops
Result:        | Training stops at epoch 25, but best model is from epoch 11

Scenario 3: No Early Stopping

Epochs: 1-100  | Validation improves periodically, counter never reaches patience
Result:        | Training completes all 100 epochs normally

Backward Compatibility

Default Values

Existing configurations without min_epochs get default value of 10
This provides reasonable protection without changing behavior dramatically
All validation still works as expected

Migration

No code changes required for existing trainers:

# Old code still works
config = TrainerConfig(
    num_epochs=100,
    early_stopping_patience=10
    # min_epochs automatically set to 10
)

Best Practices

Choosing min_epochs

Model Complexity: More complex models need higher min_epochs
- Simple models: 5-15 epochs
- Medium models: 15-30 epochs
- Complex models: 30+ epochs
Dataset Size: Smaller datasets may need more epochs to stabilize
- Large datasets: Lower min_epochs acceptable
- Small datasets: Higher min_epochs recommended
Architecture Type:
- CNN models: 10-20 epochs for feature learning
- RNN/LSTM: 20-40 epochs for sequence learning
- Transformer: 30+ epochs for attention learning
- Speaker embeddings: 25-50 epochs for embedding space
Learning Rate: Higher learning rates may converge faster
- High LR: Lower min_epochs
- Low LR: Higher min_epochs

Recommended Configurations

# Rapid prototyping
TrainerConfig(num_epochs=50, min_epochs=5, early_stopping_patience=5)

# Balanced training  
TrainerConfig(num_epochs=100, min_epochs=15, early_stopping_patience=10)

# Production model
TrainerConfig(num_epochs=200, min_epochs=30, early_stopping_patience=20)

# Research/experimental
TrainerConfig(num_epochs=500, min_epochs=50, early_stopping_patience=30)

Implementation Details

Files Modified

trainer/base.py: Enhanced early stopping logic in training loop
trainer/config.py: Added validation rules and default values
trainer/wakeword/trainer.py: Updated default configurations
trainer/voice_recognition/trainer.py: Updated default configurations

Validation Added

Range validation: min_epochs must be >= 0
Relationship validation: min_epochs <= num_epochs <= max_epochs
Cross-trainer validation in config manager

Logging Enhanced

Early stopping messages now include min_epochs information:

Early stopping triggered after 35 epochs (minimum 20 epochs completed)

Testing

The enhanced functionality has been tested for:

✅ Correct early stopping behavior after min_epochs
✅ Counter reset during min_epochs phase
✅ Configuration validation edge cases
✅ Backward compatibility with existing configs
✅ Integration with both wakeword and voice recognition trainers

Summary

The min_epochs parameter provides fine-grained control over early stopping behavior, ensuring models receive adequate training time while still preventing overfitting. This enhancement maintains full backward compatibility while providing powerful new training control capabilities.

enhanced_early_stopping.md 8.2 KB History Raw