3.8. Machine Learning

Sitara Machine Learning toolkit brings machine learning to the edge by enabling machine learning inference on all Sitara devices (Arm-only, Arm + GPU, and Arm + specialized hardware accelerators). It is provided as part of TI’s Processor SDK Linux, free to download and use. Sitara machine learning today consists of ONNX Runtime, TensorFlow Lite, Arm NN, NNStreamer, and RNN library.

Fig. 3.3 Sitara Machine Learnining Offering

TensorFlow Lite

• Open source deep learning runtime for on-device inference.

• Runs on all Cortex-A ARM cores (AM3x, AM4x, AM5x, AM6x Sitara devices).

• Imports Tensorflow Lite models.

• Uses TIDL import tool to create TIDL offloadable Tensorflow Lite models, which can be executed via Tensorflow Lite runtime with TIDL acceleration.

ONNX Runtime

• Open source inference engine available from Arm.

• Runs on all Cortex-A ARM cores (AM3x, AM4x, AM5x, AM6x Sitara devices).

Arm NN

• Open source inference engine available from Arm.

• Runs on all Cortex-A ARM cores (AM3x, AM4x, AM5x, AM6x Sitara devices).

• Imports ONNX and TensorFlow Lite models.

• Provides TensorFlow Lite delegate.

RNN Library

• Provides Long Short-Term Memory (LSTM) and fully connected layers in a standalone library to allow for rapid prototyping of inference applications that require Recurrent Neural Networks.

• Runs on all Cortex-A ARM cores (AM3x, AM4x, AM5x, AM6x Sitara devices).

• Integrated into TI’s Processor SDK Linux in an OOB demo for Predictive Maintenance.

TI Deep Learning (TIDL)

• Accelerates deep learning inference on C66x DSP cores and/or on Embedded Vision Engine (EVE) subsystems.

• Available on AM57x device only.

• Supports CNN at the moment, and imports Caffe, ONNX, and Tensorflow models.

ML inference Library	Version	Delegate / Execution provider	Python API	C/C++ API
TensorFlow Lite	2.18.0	CPU, XNNPACK, ARMNN	YES	YES
ONNX Runtime	1.20.1	CPU, ACL	YES	YES
Arm NN	1.20.1	ACL	YES	YES

• 3.8.1. TensorFlow Lite (LiteRT)
  • 3.8.1.1. Introduction
  • 3.8.1.2. Features
  • 3.8.1.3. Inference backends and delegates
  • 3.8.1.4. Benchmark Tool for TFLite Model
  • 3.8.1.5. Performance Numbers of Benchmark Tool
  • 3.8.1.6. Example Applications
• 3.8.2. ONNX Runtime
  • 3.8.2.1. Introduction
  • 3.8.2.2. Supported version
  • 3.8.2.3. ONNX Runtime test applications
• 3.8.3. Arm NN
  • 3.8.3.1. Introduction
  • 3.8.3.2. Supported versions
• 3.8.4. Arm Compute Library
  • 3.8.4.1. Introduction
  • 3.8.4.2. Supported versions
  • 3.8.4.3. Arm Compute Library Testing
• 3.8.5. NNStreamer
  • 3.8.5.1. Introduction
  • 3.8.5.2. Supported versions
  • 3.8.5.3. Testing NNStreamer