5. Troubleshooting

This chapter covers common errors and their solutions.

5.1. Environment Setup Errors

5.1.1. TIDL_TOOLS_PATH not set

Error:

Environment variable TIDL_TOOLS_PATH is not set!

Solution:

Set the environment variable to point to your TIDL tools installation:

export TIDL_TOOLS_PATH=/path/to/tidl_tools

Or run the setup script:

source /path/to/edgeai-tidl-tools/scripts/setup/setup_env.sh

5.1.2. ARM64_GCC_PATH not set

Error:

Environment variable ARM64_GCC_PATH is not set!

Solution:

This is required when compiling for device (compile_for_device=1). Set it to point to your ARM64 GCC toolchain:

export ARM64_GCC_PATH=/path/to/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu

5.1.3. CGT7X_ROOT not set

Error:

Set environment variable CGT7X_ROOT to location of C7000 Code Generation Tools

Solution:

Required when c7x_codegen=1. Set it to point to your C7000 CGT installation:

export CGT7X_ROOT=/path/to/ti-cgt-c7000_x.x.x

5.2. Compilation Errors

5.2.1. artifacts_folder is not empty

Error:

'artifacts_folder' is not empty - please clear the folder and re-run!

Solution:

Clear the artifacts folder before recompiling:

rm -rf ./artifacts/*

5.2.2. Dynamic shapes not supported

Error:

Dynamic shape/network not supported by TVM+TIDL yet!!!

Solution:

TVM+TIDL requires static shapes. Ensure your model has fixed input dimensions. If the model has dynamic axes, fix them before compilation using the ONNX opset tools or by re-exporting from the training framework with explicit input shapes.

5.2.3. TIDL import failed

Error:

TIDL import of Relay IR graph failed.

Possible causes:

  1. Unsupported operator encountered

  2. Operator parameter constraints violated

  3. Calibration data issue

Solution:

  1. Set TIDL_RELAY_IMPORT_DEBUG=1 to see detailed import logs

  2. Check relay_graph.import.txt in artifacts folder

  3. Verify calibration data matches model input requirements

5.2.4. Unsupported operator

Error in debug output:

Layer type not supported by TIDL --- layer type - X, Node name - Y

Solution:

The operator will fall back to Arm/C7x execution. Options:

  1. Use c7x_codegen=1 to run unsupported ops on C7™ NPU

  2. Check if an equivalent supported operator exists

  3. Use the deny_list option to explicitly exclude problematic layers

5.2.5. Only ONNX models supported

Error:

ERROR: Only ONNX models can be converted to Relay IR internally.

Solution:

Convert your model to ONNX format first, or convert to Relay IR manually:

# For PyTorch models
torch.onnx.export(model, dummy_input, "model.onnx")

5.3. Inference Errors

5.3.1. Vision apps not initialized

Symptom: Inference hangs or fails silently

Solution:

Ensure vision_apps is initialized on the EVM:

cd /opt/vision_apps
source ./vision_apps_init.sh

5.3.2. Trace file shows slow performance

Symptom: tvm_arm.trace shows unexpectedly high execution times

Possible causes:

  1. First inference includes initialization overhead

  2. TIDL subgraphs not being used

  3. Too many Arm fallback operations

Solution:

  1. Run multiple inferences and ignore the first one

  2. Check that TIDL artifacts are being loaded

  3. Review relay_graph.partitioned.txt to see TIDL offload coverage

5.4. Debugging Tips

For detailed debug output during compilation, see Debugging Compilation in the Compiling Models chapter.

For inference debugging using TIDL_RT_DEBUG and TVM_RT_DEBUG, see the Debugging Inference section in the Running Inference chapter.

5.5. FAQ

Q: Why are some operators not offloaded to TIDL?

A: An operator may not be offloaded if:

  • It’s not in the supported operators list (see Supported Operators)

  • Its parameters violate TIDL constraints (e.g., kernel size, stride)

  • It’s explicitly in the deny_list

Q: Should I use c7x_codegen=0 or c7x_codegen=1?

A: Use c7x_codegen=1 for production deployments — it runs TIDL-unsupported layers on the C7™ NPU and eliminates Arm-C7™ NPU data transfer overhead. Use c7x_codegen=0 only if the C7000 CGT toolchain is not available or as a temporary fallback during initial bring-up.

Q: How do I know which layers are running on TIDL vs Arm/C7x?

A: Check relay_graph.partitioned.txt - nodes prefixed with tidl_ run on TIDL.

Q: My model accuracy is different from the original framework?

A: This is usually due to quantization. Try:

  • Increase calibration_frames and calibration_iterations

  • Use tensor_bits=16 for higher precision

  • Use accuracy_level=9 with custom calibration options

Q: How many TIDL subgraphs should my model have?

A: Fewer subgraphs generally means better performance due to reduced data transfer overhead. If you see many small subgraphs, check if unsupported operators are fragmenting the graph. Consider using c7x_codegen=1 to consolidate execution on the C7™ NPU.

Q: What calibration data should I use?

A: Use representative data from your actual use case. The calibration data helps TIDL determine the dynamic range for quantization. Using non-representative data (e.g., all zeros or random noise) can lead to poor accuracy.