TI Deep Learning Library User Guide
TFLite Runtime + TIDL Heterogeneous Execution

Table of Contents

Introduction

The Processor SDK implements TIDL offload support using the TFlite Delegates TFLite Delgate runtime

This heterogeneous execution enables:

  1. TFlite runtime as the top level inference API for user applications
  2. Offloading subgraphs to C7x/MMA for accelerated execution with TIDL
  3. Runs optimized code on ARM core for layers that are not supported by TIDL

TFLite runtime based user work flow

Find Below picture for TFLite based work flow. User needs to run the model complitation (Sub graphs creation and quantization) on PC and the generated artifacts can be used for running inference on the device.

tflrt_work_flow.png
TFLite runtime based user work flow

Model Compilation on PC

osrt_compile_steps.png
OSRT Compile Steps

The Processor SDK package includes all the required python packages for runtime support.

Pre-requisite : PSDK RA should be installed on the Host Ubuntu 18.04 machine and able to run pre-built demos on EVM.

Following steps need to be followed : (Note - All below scripts to be run from ${PSDKRA_PATH}/tidl_xx_xx_xx_xx/ti_dl/test/tflrt/ folder)

  1. Prepare the Environment for the Model compilation
    source prepare_model_compliation_env.sh
    This script needs to be executed only once when user opens a new terminal. It performs the below operations. User also can perform these steps manually by following the scripts.
    • Installs all the python dependent packages like tensorflow lite runtime, numPy, Python image library (Pillow) etc. If user has a conflicting package version because of other installations, we recommend to create conda environment with python 3.6.x and run these scripts.
    • Download the models used by the OOB scripts if not available in the file system
    • Sets the environment variables required by the script e.g. path to tools and shared libraries
    • Checks if all the TIDL required tools are available in tools path

      Note
      If you observe any issue in pip. Run below command to update pip

      python -m pip install --upgrade pip
  2. Run for model compilation – This step generates artifacts needed for inference in the tflrt-artifacts folder. Each subgraph is identified in the artifacts using the tensor index of its output in the model
    python3 tflrt_delegate.py -c
  3. Run Inference on PC - Optionally user can test the inference in PC emulation mode and check the output; the output images will be saved in the corresponding specified artifacts folder
    python3 tflrt_delegate.py
  4. Run Inference on PC without offload - Optionally user can test the inference in PC emulation mode without using any delegation to TI Delegate
    python3 tflrt_delegate.py -d

Run Model on EVM

osrt_run_steps.png
OSRT Run Steps
  1. Copy the “${PSDKR_PATH}/tidl_xx_xx_xx_xx/ti_dl/test/” folder to the file system where the EVM is running Linux (SD card or NFS mount). This has all the OOB scripts and artifacts.
  2. "LD_LIBRARY_PATH=/usr/lib python3 tflrt_delegate.py" - Run the inference on the EVM and check the results, performance etc.

Note : These scripts are only for basic functionally testing and performance check. For accuracy benchmarking, we will be releasing more tutorials in upcoming release

User options

Call to interpreter from the python interface using delegate mechanism: 

interpreter = tflite.Interpreter(model_path='path_to_model', \
                    experimental_delegates=[tflite.load_delegate('libtidl_tfl_delegate.so.1.0', delegate_options)])

'delegate_options' in the interpreter call comprise of the below options (required and optional):

Required

The following options need to be specified by user while creating TFLite interpreter:

Name Value
tidl_tools_path to be set to ${PSDKRA_PATH}/tidl_xx_xx_xx_xx/tidl_tools/ - Path from where to pick TIDL related tools
artifacts_folder folder where user intends to store all the compilation artifacts
import 'yes' - if model import needed, 'no' if only inference

Optional

The following options are set to default values, to be specified if modification needed by user:

Name Description Default values
tidl_platform "J7" "J7"
tidl_version TIDL version - supported from version 7.2 onwards (7,2)
tidl_tensor_bits Number of bits for TIDL tensor and weights - 8/16 8
tidl_calibration_method "simple" (optimized for compilation time)/"advanced" (optimized for accuracy) "advanced"
tidl_calibration_options:num_frames_calibration number of frames to be used for calibration - min 10 frames recommended 20
tidl_calibration_options:bias_calibration_iterations Advanced calibration option for bias calibration iterations [^1] 50
tidl_denylist force disable offload of a particular operator to TIDL [^2] "" - Empty list
debug_level 0 - no debug, 1 - tflite rt debug prints, 2 - dump layer level traces and debug prints 0
num_tidl_subgraphs offload up to <num> tidl subgraphs [^3] 16

[^1]: Advanced calibration can help improve 8-bit quantization. Please see TIDL Quantization for details. [^2]: Denylist is a string of numbers which represent the operators as identified in tflite builtin ops. Please refer Tflite builtin ops , e.g. deny_list = "1, 2" to deny offloading 'AveragePool2d' and 'Concatenation' operators to TIDL. [^3]: Will be supported in next release

C API based demo

Pre-requisite: Compiled artifacts stored in artifacts folder as specified in step 2 of 'Model Compilation on PC' above

Following steps are needed to run C API based demo for tflite runtime: 

cd ${PSDKRA_PATH}
git clone --single-branch -b r2.4 https://github.com/tensorflow/tensorflow.git

cd tensorflow
git checkout 582c8d236cb079023657287c318ff26adb239002
git am ../tidl_j7_01_04_00_03/ti_dl/tfl_delegate/0001-tflite-interpreter-add-support-for-custom-data.patch
./tensorflow/lite/tools/make/download_dependencies.sh

cd ../tidl_j7_01_04_00_03
make demos

# To run on EVM:

Mount ${PSDKR_PATH} on EVM
export LD_LIBRARY_PATH=/usr/lib
cd ${PSDKR_PATH}/tidl_j7_01_04_00_03/ti_dl/demos/out/J7/A72/LINUX/release/

./tidl_tfl_classification.out -m ../../../../../../test/testvecs/models/public/tflite/mobilenet_v1_1.0_224.tflite -l ../../../../../../test/testvecs/input/labels.txt -i ../../../../../../test/testvecs/input/airshow.bmp -a 1

Known Issues/Limitations

  • Some amount of accuracy degradation is seen in quant models on multiple subgraph creation, this will be fixed in the next release.
  • tidl_denylist feature is not extensivey tested for object detection networks in this release.
  • Some networks with odd input resolutions e.g. 513X513 are seen to get stuck on target with the use of 8 bit mode, please use 16 bit mode for these networks. This issue will be fixed in next release.