8.12.3.9. Fan Blade Anomaly Detection — On-Device Training Example

8.12.3.9.1. Table of Contents

• 1. Overview

• 2. Project Structure

• 3. Application Flow

  • Stage 1: Initialization

  • Stage 2: Golden Vector Validation

  • Stage 3: Pre-Training Inference

  • Stage 4: On-Device Training

  • Stage 5: Threshold Recalculation

  • Stage 6: Post-Training Inference

• 4. Key Code Patterns

• 5. Hyperparameter Reference

• 6. How to Adapt for Your Own Data

• 7. Further Reading

---

8.12.3.9.2. 1. Overview

This is a complete working example that demonstrates on-device training for anomaly detection on the F29H85x microcontroller. The application trains an autoencoder model directly on the device using embedded data, recalculates the anomaly detection threshold, and evaluates the model — all without any PC involvement after deployment.

What this example demonstrates:

1. Golden vector validation (verify artifacts are loaded correctly)

2. Pre-training inference (baseline — shows the model hasn’t learned yet despite coincidental accuracy)

3. On-device training with early stopping (model learns from scratch or fine-tunes)

4. Automatic threshold recalculation using validation data

5. Post-training inference

Result from the example run: The model trains from completely untrained weights (0 PC-side epochs). While pre-training accuracy happened to be 100% on this small test set by coincidence (the initial threshold landed in the right spot), the model had not actually learned anything — normal reconstruction errors were ~5800. After on-device training, normal errors dropped to ~10 and the separation between normal and anomaly improved from 1.18× to 8.6×, producing a genuinely robust detector.

8.12.3.9.2.1. Prerequisites

• Hardware: F29H85x LaunchPad

• Software: F29H85x SDK, Code Composer Studio (CCS), C29 CGT Tools

• Artifacts generated via ModelZoo with ondevice_training: True (see Running ModelZoo for ODT)

• Conceptual background: Overview

8.12.3.9.2.2. Project location

tinyml-sdk/c29/ai/examples/fan_blade_anomalydetection_ondevice_learning/

---

8.12.3.9.3. 2. Project Structure

8.12.3.9.3.1. SDK layout (before CCS import)

fan_blade_anomalydetection_ondevice_learning/
├── application_main.c                      # Main application code
└── f29h85x/
    ├── artifacts/                          # All auto-generated model files
    └── CCS/
        └── *.projectspec                   # CCS project definition

8.12.3.9.3.2. CCS project layout (after import)

After importing the project into CCS, the project links in shared SDK libraries and organizes files into these key directories:

fan_blade_anomalydetection_ondevice_learning_f29h85x/
│
├── artifacts/                              # Auto-generated by ModelZoo pipeline
│   ├── mod.a                               # Compiled frozen model (static library)
│   ├── tvmgen_default.h                    # Frozen model inference API
│   ├── trainable_model_config.h/.c         # Trainable layer architecture + weights
│   ├── ondevice_training_data.h/.c         # Embedded train/val/test datasets
│   ├── user_input_config.h                 # Feature extraction config + initial threshold
│   └── test_vector.c                       # Golden vectors for validation
│
├── feature_extract/                        # Linked from SDK 
│   ├── feature_extract.h/.c                # Feature extraction implementation
│   └── feature_extract_c29.c               # C29-specific optimizations
│
├── ondevice_training/                      # Linked from SDK
│   ├── anomaly_detection_odt.h/.c          # Task library (state machine, training loop)
│   └── ondevice_training_lib.h/.c          # Core library (forward, backward, SGD)
│
├── libs/                                   # Linked from SDK — DSP libraries
│   └── cfft_f32_*                          # FFT library for feature extraction
│
├── application_main.c                      # Main application (this walkthrough)
└── *.cmd / *.syscfg                        # Linker scripts + SysConfig

8.12.3.9.3.3. Key files by role

Role	Files	Source
Application	application_main.c	You write/modify this
Model artifacts	artifacts/*	Auto-generated by ModelZoo
ODT libraries	ondevice_training/*	Linked from SDK (shared)
Feature extraction	feature_extract/*	Linked from SDK (shared)
DSP libraries	libs/cfft_f32_*	Linked from SDK (shared)

---

8.12.3.9.4. 3. Application Flow

The application runs through six stages sequentially:

Stage 1: Initialization
     │
     ▼
Stage 2: Golden Vector Validation
     │
     ▼
Stage 3: Pre-Training Inference (baseline)
     │
     ▼
Stage 4: On-Device Training (debugger-triggered)
     │
     ▼
Stage 5: Threshold Recalculation
     │
     ▼
Stage 6: Post-Training Inference (evaluation)

---

8.12.3.9.4.1. Stage 1: Initialization

The application initializes the hardware, dataset iterators, hyperparameters, the anomaly detection model, and the feature extraction pipeline.

Board and data setup:

Device_init();
Device_initGPIO();
Board_init();
init_dataset_iterators();

Hyperparameter configuration:

ad_model_config.learning_rate = 0.0001;
ad_model_config.patience = 10;
ad_model_config.min_improvement_delta = 0.5;
ad_model_config.ema_alpha = 0.5;
ad_model_config.batches_per_epoch = 10;
ad_model_config.initial_threshold = RECONSTRUCTION_ERROR_THRESHOLD;
ad_model_config.threshold_samples = 500;
ad_model_config.threshold_percentile = 0.99;
ad_model_config.gaussian_std_multiplier = 4;
ad_model_config.use_percentile_threshold = 1;
ad_model_config.inference_window_size = 1;
ad_model_config.stats_window_size = 100;

Model initialization:

AnomalyDetection_Init(&ad_model_ctx, &ad_model_config);

This validates all hyperparameters, initializes the core training library (ODT_Init), sets up layer pointers and buffers, and starts in MODE_INFERENCE.

Console output:

========================================
ANOMALY DETECTION MODEL INITIALIZATION
========================================
Learning rate(multiplied by 10^5): 10
Batches per epoch: 10, total samples per epoch 10
Train batch size: 1
Val batch size: 1
Patience: 10
Threshold samples: 500

========================================
ODT INITIALIZATION
========================================
  Layer 0: Linear(48→24)
  Layer 1: ReLU(24)
  Layer 2: Linear(24→12)
  ...
  Layer 10: Linear(24→48)

---

8.12.3.9.4.2. Stage 2: Golden Vector Validation

A known test input is processed through the complete pipeline (feature extraction → frozen model → trainable model) and the output is compared against a pre-computed golden reference.

// Feature extraction
FE_initFeatureExtract(fe_handle);
FE_runFeatureExtract(fe_handle);

// Frozen model
tvmgen_default_run(&inputs, &outputs);

// Trainable model forward pass
AnomalyDetection_ForwardPass(&ad_model_ctx, frozen_output, model_input, all_model_output, &reconstruction_error);

// Compare against golden output
for (output_index = 0; output_index < FE_NN_OUT_SIZE; output_index++) {
    float difference = fabs(golden_output[output_index] - all_model_output[output_index]);
    if (difference > 1e-3) error++;
}

Console output:

Loss for test vector: 5820.863770
Golden vectors matched: 48 not matched: 0

All 48 output elements match within tolerance, confirming that mod.a, trainable_model_config, and feature extraction are all loaded and working correctly.

---

8.12.3.9.4.3. Stage 3: Pre-Training Inference

All 12 test samples are run through the full pipeline using the initial threshold from PC-side training. This establishes a baseline before on-device training.

test_iter.current_index = 0;
for (i = 0; i < NUM_TEST_SAMPLES; i++) {
    process_one_sample(&test_iter);
    AnomalyDetection_ProcessInferenceFrame(&ad_model_ctx, frozen_output, model_input, &result);
    // Log result
}

Console output:

=== Pre-training Inference on Test Data ===
Sample   0: error=7471.3379 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample   1: error=11115.2373 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample   2: error=5879.2905 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Sample   3: error=6949.8081 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample   4: error=5703.4355 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Sample   5: error=5691.7251 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Sample   6: error=10856.5381 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample   7: error=5873.1714 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Sample   8: error=8936.2441 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample   9: error=5829.2065 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Sample  10: error=8369.9053 threshold=6430.9824 predicted=ANOMALY label=ANOMALY OK
Sample  11: error=5727.6235 threshold=6430.9824 predicted=NORMAL label=NORMAL OK
Pre-training accuracy: 12/12 = 100.0%

Note: The 100% pre-training accuracy is misleading — the model has not been trained at all (0 PC-side epochs). The autoencoder has not learned to reconstruct normal data, which is evident from the extremely high reconstruction errors: normal samples produce errors around 5700–5900, meaning the reconstruction is essentially garbage. The 100% accuracy is purely coincidental — the initial threshold (6430.98, exported from the PC-side pipeline) happens to land between the normal and anomaly error ranges on this small test set. The tiny margin between normal errors (~5900) and the threshold (~6430) means even slight data variation could flip predictions. This is not a usable model — on-device training is needed to actually learn meaningful reconstructions.

---

8.12.3.9.4.4. Stage 4: On-Device Training

The application waits for the user to trigger training via the CCS debugger.

printf("Set start_training=1 in debugger to begin training...\n");
while (!start_training) {}

How to trigger: In CCS, open the Variables view , add start_training, and change its value from 0 to 1.

Once triggered, the training loop runs:

AnomalyDetection_StartTraining(&ad_model_ctx);

while (current_mode == MODE_TRAINING) {
    if (ad_model_ctx.current_phase == PHASE_TRAIN) {
        process_one_sample(&train_iter);     // Feed training data
    } else {
        process_one_sample(&val_iter);       // Feed validation data
    }
    AnomalyDetection_ProcessTrainingFrame(&ad_model_ctx, frozen_output, model_input);
    AnomalyDetection_GetMode(&ad_model_ctx, &current_mode);
}

The library automatically:

• Alternates between training and validation phases each epoch

• Applies EMA smoothing to validation loss

• Saves best weights when validation improves

• Triggers early stopping after patience epochs without improvement

Console output (abbreviated):

=== Training Started ===

========================================
STARTING ON-DEVICE TRAINING
========================================

Epoch 0: Train Loss = 5571
Epoch 0: Val Loss = 6109 (raw), 6109 (smoothed)  Best model, weights saved!
Epoch 1: Train Loss = 5547
Epoch 1: Val Loss = 5818 (raw), 5963 (smoothed)  Best model, weights saved!
...
Epoch 93: Train Loss = 3132
Epoch 93: Val Loss = 1827 (raw), 3192 (smoothed)  Best model, weights saved!
Epoch 94: Train Loss = 765
Epoch 94: Val Loss = 126 (raw), 1659 (smoothed)  Best model, weights saved!
Epoch 95: Train Loss = 68
Epoch 95: Val Loss = 43 (raw), 851 (smoothed)  Best model, weights saved!
...
Epoch 111: Val Loss = 7 (raw), 10 (smoothed)  Best model, weights saved!
...
Epoch 121: Val Loss = 13 (raw), 16 (smoothed)  No improvement (10/10)

========================================
EARLY STOPPING TRIGGERED
========================================
Best val loss: 10 (epoch 111)

Training Summary:
  epochs trained: 121
  Best val loss: 10
  Quit mode: Early Stopped

---

8.12.3.9.4.5. Stage 5: Threshold Recalculation

After training, the model’s reconstruction errors are very different, so the threshold must be recalculated. The library automatically transitioned to MODE_THRESHOLD_CALC after training stopped.

val_iter.current_index = 0;
while (current_mode == MODE_THRESHOLD_CALC) {
    process_one_sample(&val_iter);
    AnomalyDetection_ProcessThresholdFrame(&ad_model_ctx, frozen_output, model_input);
    AnomalyDetection_GetMode(&ad_model_ctx, &current_mode);
}

The 6 validation samples are cycled as we can store limited data in flash. In real world this comes from a sensor collecting live data.

Console output:

=== Threshold Calculation ===
Threshold calculation progress: 200/500
Threshold calculation progress: 400/500

========================================
THRESHOLD CALCULATION
========================================
Samples collected: 500

Threshold Calculation Results:
  Mean error: 14
  Std error: 6
  Gaussian (mean + 4*std): 39
  95th percentile: 25
  99th percentile: 25
  Configured percentile (0): 25
  Method used: Percentile
  FINAL THRESHOLD: 25

The threshold dropped from 6430.98 (pre-training) to 25.77 (post-training) — a ~250× reduction, reflecting the dramatically lower reconstruction errors on normal data.

---

8.12.3.9.4.6. Stage 6: Post-Training Inference

The same 12 test samples are evaluated with the trained model and new threshold.

Console output:

=== Post-training Inference on Test Data ===
Sample   0: error=692.4103 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample   1: error=1001.3439 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample   2: error=10.0213 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Sample   3: error=131.5526 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample   4: error=15.1960 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Sample   5: error=7.4827 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Sample   6: error=922.0417 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample   7: error=9.4103 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Sample   8: error=404.9972 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample   9: error=12.5238 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Sample  10: error=308.8339 threshold=25.7727 predicted=ANOMALY label=ANOMALY OK
Sample  11: error=11.2795 threshold=25.7727 predicted=NORMAL label=NORMAL OK
Post-training accuracy: 12/12 = 100.0%

8.12.3.9.4.7. Before vs. After Training Comparison

Metric	Pre-Training	Post-Training	Change
Accuracy	12/12 (100%)	12/12 (100%)	Same
Threshold	6430.98	25.77	~250× lower
Normal error range	5691 – 5879	7.5 – 15.2	~400× lower
Anomaly error range	6949 – 11115	131 – 1001	~10× lower
Separation ratio (min anomaly / max normal)	6949 / 5879 = 1.18×	131 / 15.2 = 8.6×	Much clearer separation

While both pre- and post-training accuracy are 100%, the quality of the model is fundamentally different. Before training, the autoencoder hadn’t learned anything — normal reconstruction errors (~5800) were nearly as high as anomaly errors (~7000–11000), and the 100% accuracy was coincidental. After training, the model has genuinely learned what “normal” looks like: normal errors dropped to ~10, while anomaly errors remain high (~130–1000). The separation ratio improved from 1.18× to 8.6×, meaning the detector is now robust to noise and threshold sensitivity rather than relying on a lucky threshold placement.

---

8.12.3.9.5. 4. Key Code Patterns

8.12.3.9.5.1. 4.1 process_one_sample()

This function processes one raw sample from the dataset through feature extraction and the frozen model, producing frozen_output ready for the trainable model.

void process_one_sample(DatasetIterator_t* iter) {
    // For each frame in num_frame_concat:
    //   1. Extract raw sensor data for all channels from the sample
    //   2. Shift feature history (for temporal context)
    //   3. Run feature extraction (FE_runFeatureExtract)
    
    // After all frames processed:
    //   4. Run frozen model (tvmgen_default_run)
    //   5. Advance iterator (wraps around at end)
}

Why this complexity? Raw sensor data has multiple channels (X, Y, Z) and potentially multiple frames for temporal context. The feature extraction pipeline processes one frame at a time, maintaining a history buffer for concatenation. This function encapsulates that multi-frame, multi-channel processing.

8.12.3.9.5.2. 4.2 DatasetIterator_t

A simple struct for cycling through embedded datasets:

typedef struct {
    const float (*inputs)[RAW_INPUT_SIZE];   // Pointer to 2D array
    const int16_t* labels;                    // Pointer to labels
    uint16_t num_samples;                     // Total samples
    uint16_t current_index;                   // Current position (wraps)
} DatasetIterator_t;

Three iterators are used — train_iter, val_iter, test_iter — each pointing to the corresponding arrays in ondevice_training_data.c.

8.12.3.9.5.3. 4.3 Training Loop Pattern

The training loop checks the library’s internal phase to decide which data to feed:

while (current_mode == MODE_TRAINING) {
    if (ad_model_ctx.current_phase == PHASE_TRAIN) {
        process_one_sample(&train_iter);      // Training data
    } else {
        process_one_sample(&val_iter);        // Validation data
    }
    AnomalyDetection_ProcessTrainingFrame(&ad_model_ctx, frozen_output, model_input);
    AnomalyDetection_GetMode(&ad_model_ctx, &current_mode);
}

The library handles everything internally — epoch counting, phase switching, batch management, early stopping. The application just feeds the right data and checks when training is done.

8.12.3.9.5.4. 4.4 Debugger Variable Trigger

volatile uint16_t start_training = 0;
// ...
while (!start_training) {}

The volatile keyword prevents the compiler from optimizing away the loop. In CCS, use Variables view to change start_training to 1 while the application is running.

---

8.12.3.9.6. 5. Hyperparameter Reference

Parameter	Value	Set In	What It Controls
learning_rate	0.0001	main()	Weight update step size
patience	10	main()	Epochs without improvement before early stop
min_improvement_delta	0.5	main()	Minimum loss decrease to count as improvement
ema_alpha	0.5	main()	Validation loss smoothing (higher = more reactive)
batches_per_epoch	10	main()	Training samples per epoch (× batch_size)
threshold_samples	500	main()	Normal samples for threshold calculation
threshold_percentile	0.99	main()	Percentile for threshold (if percentile method)
gaussian_std_multiplier	4	main()	Std devs above mean (if Gaussian method)
use_percentile_threshold	1	main()	1 = percentile, 0 = Gaussian
inference_window_size	1	main()	Majority voting window (1 = no voting)
stats_window_size	100	main()	Frames per statistics report
initial_threshold	6430.98	user_input_config.h	Pre-training threshold from ModelMaker
TRAIN_BATCH_SIZE	1	trainable_model_config.h	Samples per weight update (compile-time)
VAL_BATCH_SIZE	1	trainable_model_config.h	Samples per validation evaluation (compile-time)

---

8.12.3.9.7. 6. How to Adapt for Your Own Data

To use this example with a different dataset or sensor:

1. Generate new artifacts via ModelZoo with your dataset and YAML configuration (see Running ModelZoo for ODT)

2. Replace artifact files in the artifacts/ directory:

   • mod.a — new frozen model

   • tvmgen_default.h — new frozen model API

   • trainable_model_config.h/.c — new trainable architecture and weights

   • ondevice_training_data.h/.c — new embedded dataset

   • user_input_config.h — new feature extraction config

   • test_vector.c — new golden vectors

3. Adjust hyperparameters in main() if needed:

   • Different data may need different learning_rate (start with 0.0001, adjust if training diverges or converges too slowly)

   • Adjust patience based on expected convergence time

   • Adjust threshold_samples based on available validation data

4. Modify process_one_sample() if your sensor pipeline differs (different number of channels, different frame structure, etc.)

5. Rebuild the CCS project and flash the device

---

8.12.3.9.8. 7. Further Reading

Document	Description
Overview	What is on-device training, architecture, features
Running ModelZoo for ODT	Generate artifacts for this example
Generated Artifacts Overview	What each artifact file contains
Anomaly Detection ODT Library	API reference for the library used in this example
On-Device Training Library	Core training primitives