TI Deep Learning Product User Guide
TIDL-RT: Input and Output Tensors Format

Table of Contents

Introduction

This section describes the following:

  • Input and output tensor formats for TIDL-RT inference
  • Input and output file formats of TIDL-RT inference test application and import tool

Input and Output Tensor Formats for TIDL-RT inference

  • The input and output tensor format of a network to perform inference are described by the sTIDL_IOBufDesc_t.
  • Information for the structure mentioned in above statement is generated in a file by TIDL-RT import tool. The name of file can be provided by user in TIDL-RT import configuration file by setting parameter "outputParamsFile".
  • This information is expected to be used for the following purposes:
    • To allocate the input and output buffers supplied to TIDL-RT library from higher level application.
    • To identify the order in which user is expected to provide the input to TIDL-RT inference engine in cases where network has more than one input. sTIDL_IOBufDesc_t.inDataName parameter contains the tensor names as in the original network which shall be used to provide the input in right order.
    • To read/de-quantize the final output of the network. A sample code is given as an example to understand how the final output can be read/de-quantize. In this sample code the data layout is assumed to be in NCHW format.
      /* Lets assume the outPtr is the output pointer for one of the output buffer whose index is outIdx for a given network.
      OutElemTemplateType is the type of the output and it can be read from sTIDL_IOBufDesc_t.outElementType, e.g. uint8_t, int16_t etc.
      All parameter are for a given output and can be accessed by sTIDL_IOBufDesc_t.<parameter>[outIdx]
      Following example is for the output but same format is also applicable for input
      */
      OutElemTemplateType outPtrStart = &outPtr[outPadT[outIdx] + outPadL[outIdx]];
      int32_t outLinePitch = outWidth[outIdx] + outPadL[outIdx] + outPadR[outIdx];
      int32_t outBatchPitch = outChannelPitch[outIdx] * outNumChannels[outIdx];
      float32_tidl outputVal;
      for ( batchIdx = 0; batchIdx < outNumBatches[outIdx];batchIdx++)
      {
      for ( channelIdx = 0; channelIdx < outNumChannels[outIdx];channelIdx++)
      {
      for ( heightIdx = 0; heightIdx < outHeight[outIdx];heightIdx++)
      {
      for ( widthIdx = 0; widthIdx < outWidth[outIdx];widthIdx++)
      {
      int32_t outIdx = widthIdx +
      (heightIdx * outLinePitch) +
      (channelIdx * outChannelPitch[outIdx]) +
      (batchIdx * outBatchPitch);
      /* Note that this is the final de-quantized output */
      outputVal = outPtrStart[outIdx] / outTensorScale[outIdx];
      printf("Output Element %f \n", outputVal);
      }
      }
      }
      }
  • Below image describes properties of one channel in a tensor (This format is applicable for both input and output tensor of the network).
IO_Tensor_channel_1.PNG
TIDL Input Tensor - One 2D Channel
  • If the input/output has more than one channel then these channels are stacked continuously in the memory.
IO_Tensor_channel_2.PNG
TIDL Input Tensor - 3 Channels
  • The Padded region in the input buffer has to be filled with zero by application. This can be done once by application during buffer allocation during system boot time.

Supported data types for the input tensor

  • TIDL_UnsignedChar
  • TIDL_SignedChar
  • TIDL_UnsignedShort
  • TIDL_SignedShort
  • TIDL_SinglePrecFloat (Only when data convert layer is added on input side and the same can be done using TIDL-RT import config parameter "addDataConvertToNet")

Supported data types for the output tensor

  • TIDL_UnsignedChar
  • TIDL_SignedChar
  • TIDL_UnsignedShort
  • TIDL_SignedShort
  • TIDL_SinglePrecFloat (Only for Softmax and SSD Post Processing Layer)

Points to be noted

  • All the Input and output tensor follow the above described buffer format, including the SSD post processing layer. Additionally TIDL_ODLayerHeaderInfo and TIDL_ODLayerObjInfo can be used to access the detected objects.
  • Refer the gParams.postProcType == 2 in ti_dl/test/src/tidl_image_postproc.c for more information OD-SSD output

IO File Formats of TIDL-RT Test Application and Import tool

  • TIDL-RT test application accepts input to be fed in a certain way and it is controlled by "inFileFormat" parameter as described here
  • TIDL-RT inference and import uses the same test application.
  • JPEG and PNG files are supported when running the import tool for host emulation mode. For target (EVM) execution, only BMP file or raw inFileFormat is supported as input.
  • Output is always stored in RAW format to file along with any post processing output as requested by the user. User can refer the TIDL: Output Post Processing in Test Application page to understand the various post processing options available.
  • In case of multiple input images, set inFileFormat = 2 in the import config file. Provide "inData" as a txt file which specifies the input file names separated by space e.g. "testvecs/inputs/input1.bmp testvecs/inputs/input2.bmp". The order of the inputs can be specified by providing the input tensor names using the "inDataNamesList" option in import config file as described here.
  • If the input is not an image, then array of size "numBatches x numChannels x inHeight x inWidth x elementSize" can be provided as a raw binary file to the TIDL-RT test application and import tool. In this case user should set TIDL-RT import/infer config parameter as inFileFormat = 1.
    • If more than one input tensor is used by network, then all the input array are sequentially read from the same input binary file.
    • Order in which the multiple tensors are sequentially provided in the input binary file can be specified using the "inDataNamesList" option in import config file as described here.
    • If TIDL-RT inference or import is executed for more than one frame for a given network then all the input tensors can be appended to the same input raw binary file