NEOX® | AI SDK
Software development kit for offline graph optimization and AI inference
The NEOX® AI SDK is a collection of open source and proprietary tools for converting, optimizing, analyzing and deploying pre-trained and post-trained neural network models on the newly released NEOX® | GA100.
The SDK can perform offline graph optimization and AI inference based on TensorFlow Lite for MCU. It includes various open source, proprietary tools, and algorithms for analyzing, visualizing, converting, compressing, and deploying Deep Neural Networks (DNNs) on NEOX® architecture.
It allows to perform various iterative steps in model compression and model analysis, until the desired balance between “accuracy-performance-memory” is achieved.
NEOX® AI SDK is based on TFLite for MCUs, but the user can also start with PyTorch or Keras formats which are converted to the AI SDK internal format.
After converting the model, the NEOX® AI SDK offline tool can validate the model prediction accuracy. It then performs graph-level optimizations through our two proprietary optimization techniques (SVD and FBP). The model is further optimized for ΝΕΟΧ® architecture (e.g. kernel fusions, operators emplacement) and quantized to int8.
The optimized model can then be validated/checked that all operations (e.g., convolution, average pooling) and datatypes (e.g., int8, float32) of the converted model are supported on the underlying HW.
The graph analyzer generates performance (inference time), memory and power statistics. The graph invoker invokes the TFLite graph either on the host CPU (x86), or on the NEOX® HW simulator platform, taking the model data as input and outputting various runtime statistics.
The user can run their own TFLM test code in C++ using the NEOX® HW simulation platform. After the test execution, the user gathers insightful information, such as number of simulated instructions, memory transactions, or cache statistics.
The AI SDK package is delivered in a docker image that contains all the above tools as executable files.
Democratizing ML on NEOX® for wearables and AIoT
NEOX® | AI SDK is also integrated into Edge Impulse platform, enabling ML developers to easily target RISC-V GPGPU, NEOX® | GA100 for edge applications on wearable and AIoT devices (sports, wellness, audio, gesture and vision-based AI applications etc.).
Integrating the NEOX® AI SDK into Edge Impulse allows embedded ML developers to access a complete set of ML tools to collect and shape data, perform model training and easily assess how much inference time, power consumption and RAM/Flash will be required for their specific AI use case on the NEOX® | GA100.
You can explore Edge Impulse and start building your edge AI solutions at edgeimpulse.com.