8. Advanced Features

Tiny ML Tensorlab includes several advanced features to help you build more accurate and efficient models. This section covers these capabilities in detail.

Contents

8.10. Feature Overview

Neural Architecture Search (NAS)

Automatically discover optimal neural network architectures for your dataset. NAS can optimize for memory usage or computational efficiency.

  • Preset sizes: s, m, l, xl, xxl

  • Optimization modes: Memory or Compute

  • GPU recommended for practical use

Quantization

Reduce model size and improve inference speed through quantization:

  • QAT (Quantization-Aware Training) - Best accuracy

  • PTQ (Post-Training Quantization) - Faster, no retraining

  • Weight bit-widths: 2-bit, 4-bit, 8-bit

Automatic Mixed Precision Quantization

Fully automatic, Hessian-aware per-layer bit width assignment using a greedy algorithm. Enabled by setting auto_quantization: True:

  • Estimates per-layer sensitivity via Hessian eigenvalue (power iteration)

  • Greedy assignment from {2, 4, 8, 32} bit widths maximising accuracy per bit

  • Automatic average bit width selection via binary search calibration

  • Fixes regression tasks where uniform 8-bit QAT fails

Standalone Quantization Examples

Runnable Python examples demonstrating direct use of quantization wrappers:

  • FMNIST, Audio KWS, Motor Fault, MNIST, Torque Regression

  • QAT and PTQ workflows with 2/4/8-bit quantization

  • ONNX export and inference validation

Feature Extraction

Transform raw time-series data into meaningful features:

  • FFT (Fast Fourier Transform)

  • Binning and normalization

  • Haar and Hadamard wavelets

  • Logarithmic scaling

Goodness of Fit Test

Evaluate whether your dataset is suitable for classification before training. Uses PCA and t-SNE visualization to assess class separability.

Post-Training Analysis

Understand model performance with:

  • ROC curves for classification

  • Confusion matrices

  • FPR/TPR threshold analysis

  • PCA visualization of feature-extracted data

On-Device Training (ODT)

Enable models to continue training directly on microcontrollers:

  • Deploy frozen backbone + trainable head

  • Adapt to local data and environment drift

  • Reduce re-deployment costs

  • Support for classification, regression, anomaly detection tasks