5. Bring Your Own Data

This section explains how to format your datasets for use with Tiny ML Tensorlab. The toolchain supports various data formats and automatically handles preprocessing.

Contents

• 5.1. Classification Dataset Format
  • 5.1.1. Directory Structure
  • 5.1.2. Data File Format
  • 5.1.3. Supported File Types
  • 5.1.4. Annotations (Optional)
  • 5.1.5. Configuration
  • 5.1.6. Dataset Splitting Modes
  • 5.1.7. Example: 3-Class Vibration Data
  • 5.1.8. Class Balancing
  • 5.1.9. Common Issues
  • 5.1.10. Key Differences: Classification vs Regression vs Forecasting
• 5.2. Regression Dataset Format
  • 5.2.1. Directory Structure
  • 5.2.2. Data File Format
  • 5.2.3. Time Column Handling
  • 5.2.4. Annotation Files (Required)
  • 5.2.5. Configuration
  • 5.2.6. Target Processing
  • 5.2.7. Complete Example
  • 5.2.8. Troubleshooting
  • 5.2.9. Best Practices
  • 5.2.10. Common Issues
• 5.3. Forecasting Dataset Format
  • 5.3.1. Directory Structure
  • 5.3.2. Data File Format
  • 5.3.3. Key Difference from Regression
  • 5.3.4. Configuration
  • 5.3.5. Variable Specification Options
  • 5.3.6. Windowing Behavior
  • 5.3.7. Complete Example
  • 5.3.8. Important Notes
  • 5.3.9. Minimum Data Requirements
  • 5.3.10. Common Issues
• 5.4. Anomaly Detection Dataset Format
  • 5.4.1. Folder Structure
  • 5.4.2. Concrete Example
  • 5.4.3. Data Splitting Strategy
  • 5.4.4. What If You Don’t Have Anomaly Data?
  • 5.4.5. Datafile Format (CSV)
  • 5.4.6. See Also
• 5.5. Data Splitting
  • 5.5.1. Split Methods
  • 5.5.2. Configuration
  • 5.5.3. Annotation File Format
  • 5.5.4. Split Examples
  • 5.5.5. When to Use Each Method
  • 5.5.6. Best Practices
  • 5.5.7. Creating Annotation Files
  • 5.5.8. Verifying Splits

5.6. Dataset Format Overview

Tiny ML Tensorlab uses different folder structures depending on the task type:

Classification Tasks

dataset_name/
├── classes/
│   ├── class1/
│   │   ├── file1.csv
│   │   └── file2.csv
│   ├── class2/
│   │   └── file1.csv
│   └── classN/
└── annotations/          # Optional - auto-generated if missing
    ├── instances_train_list.txt
    └── instances_val_list.txt

Anomaly Detection Tasks

dataset_name/
├── classes/
│   ├── Normal/           # Training data (normal operation only)
│   │   ├── file1.csv
│   │   └── file2.csv
│   └── Anomaly/          # Test-only data (fault/anomaly samples)
│       ├── file1.csv
│       └── file2.csv
└── annotations/          # Optional - auto-generated if missing

See Anomaly Detection Dataset Format for full details.

Regression & Forecasting Tasks

dataset_name/
├── files/               # MUST be named "files"
│   ├── datafile1.csv
│   └── datafileN.csv
└── annotations/         # Required for these tasks
    ├── instances_train_list.txt
    └── instances_val_list.txt

5.7. Supported File Formats

• CSV files (.csv) - Most common, human-readable

• Text files (.txt) - Same format as CSV

• NumPy arrays (.npy) - Binary format, faster loading

• Pickle files (.pkl) - Python serialized pandas DataFrames

5.8. Data Sources

You can provide your dataset as:

• A local directory path

• A local ZIP file path

• A remote URL to a ZIP file (automatically downloaded)

Example:

dataset:
  dataset_name: my_dataset
  input_data_path: '/path/to/dataset'  # or URL