5. Bring Your Own Data

This section explains how to format your datasets for use with Tiny ML Tensorlab. The toolchain supports various data formats and automatically handles preprocessing.

Contents

• 5.1. Classification Dataset Format
  • 5.1.1. Directory Structure
  • 5.1.2. Data File Format
  • 5.1.3. Supported File Types
  • 5.1.4. Annotations (Optional)
  • 5.1.5. Configuration
  • 5.1.6. Example: 3-Class Vibration Data
  • 5.1.7. Class Balancing
  • 5.1.8. Common Issues
• 5.2. Regression Dataset Format
  • 5.2.1. Directory Structure
  • 5.2.2. Data File Format
  • 5.2.3. Time Column Handling
  • 5.2.4. Annotation Files (Required)
  • 5.2.5. Configuration
  • 5.2.6. Target Processing
  • 5.2.7. Complete Example
  • 5.2.8. Common Issues
• 5.3. Forecasting Dataset Format
  • 5.3.1. Directory Structure
  • 5.3.2. Data File Format
  • 5.3.3. Key Difference from Regression
  • 5.3.4. Configuration
  • 5.3.5. Variable Specification Options
  • 5.3.6. Windowing Behavior
  • 5.3.7. Complete Example
  • 5.3.8. Important Notes
  • 5.3.9. Minimum Data Requirements
  • 5.3.10. Common Issues
• 5.4. Data Splitting
  • 5.4.1. Split Methods
  • 5.4.2. Configuration
  • 5.4.3. Annotation File Format
  • 5.4.4. Split Examples
  • 5.4.5. When to Use Each Method
  • 5.4.6. Best Practices
  • 5.4.7. Creating Annotation Files
  • 5.4.8. Verifying Splits

5.5. Dataset Format Overview

Tiny ML Tensorlab uses different folder structures depending on the task type:

Classification Tasks

dataset_name/
├── classes/
│   ├── class1/
│   │   ├── file1.csv
│   │   └── file2.csv
│   ├── class2/
│   │   └── file1.csv
│   └── classN/
└── annotations/          # Optional - auto-generated if missing
    ├── instances_train_list.txt
    └── instances_val_list.txt

Regression & Forecasting Tasks

dataset_name/
├── files/               # MUST be named "files"
│   ├── datafile1.csv
│   └── datafileN.csv
└── annotations/         # Required for these tasks
    ├── instances_train_list.txt
    └── instances_val_list.txt

5.6. Supported File Formats

• CSV files (.csv) - Most common, human-readable

• Text files (.txt) - Same format as CSV

• NumPy arrays (.npy) - Binary format, faster loading

• Pickle files (.pkl) - Python serialized pandas DataFrames

5.7. Data Sources

You can provide your dataset as:

• A local directory path

• A local ZIP file path

• A remote URL to a ZIP file (automatically downloaded)

Example:

dataset:
  dataset_name: my_dataset
  input_data_path: '/path/to/dataset'  # or URL