TIDL-RT: Model Translation / Import Tool

Table of Contents

• Introduction
  • Purpose
  • Input and Output format
• Directory Structure
• Diagrams
  • Main flow

Introduction

TIDL-RT release include import tool source code. Users can modify the code and recompile it to implement their own design.

Purpose

Different deep learning framework will generate model format to describe their models. TIDL-RT inference engine understands TI's internal model exchange format (net bin file) to describe the deep learning models. The import tool is to translate/convert/import different file type to a TI's model exchange format. Import tool utilize understanding of underneath device knowledge to optimize the users model for best inference time on the device. Import tool also calls the Quantizer and Graph compiler tools to do range collection and device specific optimization.

TIDL-RT inference engine supports many of the popular layers/operations involved in deep neural networks. For new layer/operators which can be converted to supported layers/operators, users can modify the source code of this tool to enable them. Users can easily adapt this tool to customize add support for any un-supported model exchange format (see here for supported model format).

Input and Output format

• This application reads import config file to collect necessary information listed as mentioned here .
• Import tool generates following files for TIDL-RT inference engine:
  • Net description file
  • IO buf description file.
• Apart from the above it also generates few files which provides some useful information about the network:
  • < network_bin >layer_info.txt : Provides mapping of tensor in original network and the same in TIDL imported network
  • < network_bin >netLog.txt : Provides information about layer type and input/output dimensions for each layer
  • < network_bin >.svg : Model Visualization output. Refer here fore details.

Directory Structure

TIDL import tool is at below path in TIDL release package::

├── ti_dl                                           # TIDL release package
│   ├── utils                                       # Utilities
│   │   ├── tidlModelImport                         # Import tool
│   │   │   └── ti_dl.h                             # Import tool intermediate structure
│   │   │   └── tidl_caffeImport.cpp                # Old caffe import code, default unused, keep for fail backup.
│   │   │   └── tidl_caffeImport_param.cpp          # New caffe import code for map functions.
│   │   │   └── tidl_caffeImport_param.h
│   │   │   └── tidl_caffeImport_v2.cpp             # New caffe import code for net searching.
│   │   │   └── tidl_import_common.cpp              # Common code for layer merging, reshape functions, utilities.
│   │   │   └── tidl_import_common.h
│   │   │   └── tidl_import_common_model_check.cpp  # Common code for model sanity check.
│   │   │   └── tidl_import_common_model_check.h
│   │   │   └── tidl_import_config.cpp              # Common code for parsing import config file.
│   │   │   └── tidl_import_config.h
│   │   │   └── tidl_import_include.h
│   │   │   └── tidl_import_main.cpp                # Main function of Import tool
│   │   │   └── tidl_onnxImport.cpp                 # ONNX import code
│   │   │   └── tidl_tfImport.cpp                   # Tensorflow import code
│   │   │   └── tidl_tfLiteImport.cpp               # tfLite import code
│   │   │   └── tidl_tfMetaArchImport.cpp           # MetaArch code

Diagrams

Main flow

Following flow shows the main execution flow of TIDL import tool(tidl_import_main.cpp).

Inside import process

Following flow shows the internal execution flow of import process(same import flow for different framework).

Implement details

Reverse search mechanism

• Import tool uses a reverse search mechanism to add all related layers into internal list.

Map functions

• Map functions aim to translate layer/operator string name into TIDL layer name. Import tool match each string to corresponding map functions.

  sTIDL_caffeLayerParamMap_t tidl_caffeLayerParamMapTable[] =
  {
    { "Concat",                          TIDL_caffeMapConcatParams },
    { "Convolution",                     TIDL_caffeMapConvParams },
    { "ConvolutionDepthwise",            TIDL_caffeMapConvParams },
    { "Pooling",                         TIDL_caffeMapPoolingParams },
    { "ReLU",                            TIDL_caffeMapReluParams },
    { "PReLU",                           TIDL_caffeMapPReLUParams },
    { "Dropout",                         TIDL_caffeMapDropoutParams },
    { "Softmax",                         TIDL_caffeMapSoftmaxParams },
    { "softmax",                         TIDL_caffeMapSoftmaxParams },
    { "Deconvolution",                   TIDL_caffeMapDeconvParams },
    { "Argmax",                          TIDL_caffeMapArgmaxParams },
    { "ArgMax",                          TIDL_caffeMapArgmaxParams },
    { "Bias",                            TIDL_caffeMapBiasParams },
    { "ShuffleChannel",                  TIDL_caffeMapShuffleParams },
    { "Eltwise",                         TIDL_caffeMapEltwiseParams },
    { "BatchNorm",                       TIDL_caffeMapBatchNormParams },
    { "Scale",                           TIDL_caffeMapScaleParams },
    { "InnerProduct",                    TIDL_caffeMapInnerProductParams },
    { "Split",                           TIDL_caffeMapSplitParams },
    { "Slice",                           TIDL_caffeMapSliceParams },
    { "Crop",                            TIDL_caffeMapCropParams },
    { "Flatten",                         TIDL_caffeMapFlattenParams },
    { "Permute",                         TIDL_caffeMapPermuteParams },
    { "PriorBox",                        TIDL_caffeMapPriorBoxParams },
    { "Reshape",                         TIDL_caffeMapReshapeParams },
    { "DetectionOutput",                 TIDL_caffeMapDetectionOutputParams }
  };

• Map function is used to translate layer parameters into internal structure.

  int32_t TIDL_caffeMapEltwiseParams(sTIDL_OrgNetwork_t   *pOrgTIDLNetStructure,
    int32_t              i,
    int32_t&             layerIndex,
    int32_t&             dataIndex,
    NetParameter&        netStructure,
    NetParameter&        netParams)
  {
    sTIDL_LayerPC_t &TIDLPCLayers = pOrgTIDLNetStructure->TIDLPCLayers[layerIndex];
  
    TIDLPCLayers.layerType = TIDL_EltWiseLayer;
    TIDLPCLayers.numOutBufs = 1;
    TIDLPCLayers.numInBufs = netStructure.layer(i).bottom_size();
    TIDLPCLayers.outData[0].dataId = dataIndex++;
  
    if(netStructure.layer(i).eltwise_param().has_operation())
    {
      TIDLPCLayers.layerParams.eltWiseParams.eltWiseType = netStructure.layer(i).eltwise_param().operation();
    }
    else
    {
      TIDLPCLayers.layerParams.eltWiseParams.eltWiseType = TIDL_EltWiseSum;
    }
  
    return 0;
  }

• This function can only handle information inside current layer.

Reshape functions

• Reshape functions aims to calculate the size, the element type, MACs.

  int32_t TIDL_tfOutReshapeSoftmax(sTIDL_OrgNetwork_t   *pOrgTIDLNetStructure, int32_t layerIndex)
  {
    sTIDL_LayerPC_t &TIDLPCLayers = pOrgTIDLNetStructure->TIDLPCLayers[layerIndex];
    TIDLPCLayers.outData[0].elementType = TIDL_SinglePrecFloat;
    TIDLPCLayers.outData[0].numDim = TIDLPCLayers.inData[0].numDim;
    TIDLPCLayers.outData[0].dimValues[0] = TIDLPCLayers.inData[0].dimValues[0];
    TIDLPCLayers.outData[0].dimValues[1] = 1;
    TIDLPCLayers.outData[0].dimValues[2] = 1;
    TIDLPCLayers.outData[0].dimValues[3] = TIDLPCLayers.inData[0].dimValues[3] * TIDLPCLayers.inData[0].dimValues[1] * TIDLPCLayers.inData[0].dimValues[2];
    TIDLPCLayers.numMacs =
      (int64_t)((int64_t)TIDLPCLayers.outData[0].dimValues[0] * TIDLPCLayers.outData[0].dimValues[1] *
        TIDLPCLayers.outData[0].dimValues[2] * TIDLPCLayers.outData[0].dimValues[3]);
  
    return 0;
  }

• For layers has the same output as input, use "TIDL_tfOutReshapeIdentity"

  sTIDL_tfOutRehapeMap_t sTIDL_caffeOutRehapeTable[] =
  {
    { TIDL_DataLayer                     ,  TIDL_tfOutReshapeDataLayer },
    { TIDL_ConvolutionLayer              ,  TIDL_tfOutReshapeConvLayer },
    { TIDL_PoolingLayer                  ,  TIDL_tfOutReshapePoolingLayer },
    { TIDL_ReLULayer                     ,  TIDL_tfOutReshapeRelu },
    { TIDL_PReLULayer                    ,  TIDL_tfOutReshapePRelu },
    { TIDL_EltWiseLayer                  ,  TIDL_tfOutReshapeIdentity },
    { TIDL_InnerProductLayer             ,  TIDL_tfOutReshapeIPLayer },
    { TIDL_SoftMaxLayer                  ,  TIDL_tfOutReshapeSoftmax },
    { TIDL_BatchNormLayer                ,  TIDL_tfOutReshapeBN },
    { TIDL_BiasLayer                     ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ScaleLayer                    ,  TIDL_tfOutReshapeIdentity },
    { TIDL_Deconv2DLayer                 ,  TIDL_tfOutReshapeDeConvLayer },
    { TIDL_ConcatLayer                   ,  TIDL_tfOutReshapeConcatLayer },
    { TIDL_SplitLayer                    ,  TIDL_tfOutReshapeSliceLayer },
    { TIDL_SliceLayer                    ,  TIDL_tfOutReshapeSliceLayer },
    { TIDL_CropLayer                     ,  TIDL_tfOutReshapeCropLayer },
    { TIDL_FlattenLayer                  ,  TIDL_tfOutReshapeFlattenLayer },
    { TIDL_DropOutLayer                  ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ArgMaxLayer                   ,  TIDL_tfOutReshapeArgmaxLayer },
    { TIDL_DetectionOutputLayer          ,  TIDL_tfOutReshapeDetOutLayer },
    { TIDL_ShuffleChannelLayer           ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ResizeLayer                   ,  TIDL_tfOutReshapeResize },
    { TIDL_RoiPoolingLayer               ,  TIDL_tfOutReshapeRoiPoolingLayer},
    { TIDL_OdPostProcessingLayer         ,  TIDL_tfOutReshapeOdPostProcessingLayer},
    { TIDL_CustomLayer                   ,  TIDL_tfOutReshapeIdentity },
    { TIDL_UnsupportedLayer              ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ConstDataLayer                ,  TIDL_tfOutReshapeIdentity },
    { TIDL_PriorBoxLayer                 ,  TIDL_tfOutReshapeIdentity },
    { TIDL_PermuteLayer                  ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ReshapeLayer                  ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ShapeLayer                    ,  TIDL_tfOutReshapeIdentity },
    { TIDL_SqueezeLayer                  ,  TIDL_tfOutReshapeIdentity },
    { TIDL_PadLayer                      ,  TIDL_tfOutReshapePadLayer },
    { TIDL_TransposeLayer                ,  TIDL_tfOutReshapeIdentity },
    { TIDL_ClipLayer                     ,  TIDL_tfOutReshapeClip }
  };

• It can utilize previous layer’s size and current layer’s parameters from internal structure.

Merge/convert layers

• These functions are used by the import process.
• These functions can search, remove, add, modify all the layers/operators imported.
• These functions locates in tidl_import_common.cpp, so it cannot access prototxt/ONNX/PB.

  int32_t tidl_mergePoolingLayer(sTIDL_OrgNetwork_t  &pOrgTIDLNetStructure, int32_t layerIndex)
  {
    int32_t i1, i2, i3, i4;
    int32_t status = 0;
    int32_t merged;
  
    for (i1 = 0; i1 < layerIndex; i1++)
    {
      if (pOrgTIDLNetStructure.TIDLPCLayers[i1].layerType == TIDL_PoolingLayer &&
          pOrgTIDLNetStructure.TIDLPCLayers[i1].layerParams.poolParams.kernelH != 0 &&
          pOrgTIDLNetStructure.TIDLPCLayers[i1].layerParams.poolParams.kernelW != 0)
      {
        merged = 0;
        int32_t  idx = tidl_getInLayer(pOrgTIDLNetStructure, layerIndex, pOrgTIDLNetStructure.TIDLPCLayers[i1].inData[0].dataId);
        if (idx == -1)
        {
          continue;
        }
        sTIDL_LayerPC_t &TIDLPCLayers = pOrgTIDLNetStructure.TIDLPCLayers[idx];
        if ((TIDLPCLayers.layerType == TIDL_ConvolutionLayer) &&
          (TIDLPCLayers.outConsumerCnt[0] == 1) &&
          (pOrgTIDLNetStructure.TIDLPCLayers[i1].layerParams.poolParams.kernelH == 2) &&
          (pOrgTIDLNetStructure.TIDLPCLayers[i1].layerParams.poolParams.kernelW == 2))
        {
          merged = 1;
        }
  
        if (merged == 1)
        {
          int32_t  idx = tidl_getInLayer(pOrgTIDLNetStructure, layerIndex, pOrgTIDLNetStructure.TIDLPCLayers[i1].inData[0].dataId);
          if (idx == -1)
          {
            return -1;
          }
          sTIDL_LayerPC_t &TIDLPCLayers = pOrgTIDLNetStructure.TIDLPCLayers[idx];
          TIDLPCLayers.numMacs += pOrgTIDLNetStructure.TIDLPCLayers[i1].numMacs;
          TIDLPCLayers.outData[0] = pOrgTIDLNetStructure.TIDLPCLayers[i1].outData[0];
          strcpy((char*)TIDLPCLayers.outDataNames[0], (char*)pOrgTIDLNetStructure.TIDLPCLayers[i1].outDataNames[0]);
          TIDLPCLayers.outConsumerCnt[0] = pOrgTIDLNetStructure.TIDLPCLayers[i1].outConsumerCnt[0];
          TIDLPCLayers.layerParams.convParams.poolParams = pOrgTIDLNetStructure.TIDLPCLayers[i1].layerParams.poolParams;
          TIDLPCLayers.layerParams.convParams.enablePooling = 1;
          pOrgTIDLNetStructure.TIDLPCLayers[i1].numInBufs = -1;
          pOrgTIDLNetStructure.TIDLPCLayers[i1].numOutBufs = -1;
        }
      }
    }
  
    return 0;
  }