NPU Model API Deployment Process*
In GX8010, both the CPU and MCU have an NPU. The NPU in the CPU is the main NPU, while the one in the MCU is the SNPU. The NPU is more powerful but consumes more power compared to the SNPU. The CPU can control both NPU and SNPU, while the MCU can only control the SNPU. Due to the different characteristics of the CPU and MCU, the NPU API usage is different on each.
1. Using NPU or SNPU in the CPU*
To generate a model file that can run on the CPU, you need to specify OUTPUT_TYPE: raw in the model compilation configuration file. The CPU has relatively abundant memory resources, and model instructions are loaded as files, which increases flexibility. NPU processor's input and output data types are both float16, and float16 to float32 conversion is handled internally by the API, so the application does not need to worry about it.
1.1 API Calling Process*
- Open the NPU device.
- Pass the model file to get the model task.
- Obtain input and output information of the task.
- Copy input data to the model memory.
- Run the model to get the output data.
- Release the model task.
- Close the NPU device.
gxdnn.h
/* GXDNN
* Copyright (C) 1991-2017 NationalChip Co., Ltd
*
* gxdnn.h NPU Task loader and executor
*
*/
#ifndef __GXDNN_H__
#define __GXDNN_H__
#ifdef __cplusplus
extern "C" {
#endif
/*===============================================================================================*/
typedef void* GxDnnDevice;
typedef void* GxDnnTask;
typedef enum {
GXDNN_RESULT_SUCCESS = 0,
GXDNN_RESULT_WRONG_PARAMETER,
GXDNN_RESULT_MEMORY_NOT_ENOUGH,
GXDNN_RESULT_DEVICE_ERROR,
GXDNN_RESULT_FILE_NOT_FOUND,
GXDNN_RESULT_UNSUPPORT,
GXDNN_RESULT_UNKNOWN_ERROR,
GXDNN_RESULT_OVERTIME,
} GxDnnResult;
/*===============================================================================================*/
/**
* @brief Open NPU device
* @param [in] devicePath the path to device
* [out] device a handle to the openned device
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* GXDNN_RESULT_DEVICE_ERROR device error
* @remark if devicePath is "/dev/gxnpu", open npu device
* devicePath is "/dev/gxsnpu", open snpu device
*/
GxDnnResult GxDnnOpenDevice(const char *devicePath,
GxDnnDevice *device);
/*===============================================================================================*/
/**
* @brief Close NPU device
* @param [in] device the handle to openned device
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* GXDNN_RESULT_DEVICE_ERROR device error
* @remark
*/
GxDnnResult GxDnnCloseDevice(GxDnnDevice device);
/*===============================================================================================*/
/**
* @brief Load NPU Task from file (in Linux/MacOSX)
* @param [in] device the device handle
* [in] taskPath the path to NPU task file
* [out] task a handle to the loaded task
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* GXDNN_RESULT_MEMORY_NOT_ENOUGH no enough memory
* GXDNN_RESULT_DEVICE_ERROR device error
* GXDNN_RESULT_FILE_NOT_FOUND file not found
* GXDNN_RESULT_UNSUPPORT unsupport NPU type or npu task version
* @remark
*/
GxDnnResult GxDnnCreateTaskFromFile(GxDnnDevice device,
const char *taskPath,
GxDnnTask *task);
/*===============================================================================================*/
/**
* @brief Load NPU task from memory
* @param [in] device the device handle
* [in] taskBuffer the pointer to NPU task buffer
* [in] bufferSize the buffer size of the NPU task buffer
* [out] task a handle to the loaded task
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* GXDNN_RESULT_MEMORY_NOT_ENOUGH no enough memory
* GXDNN_RESULT_DEVICE_ERROR device error
* GXDNN_RESULT_UNSUPPORT unsupport NPU type or npu task version
* @remark
*/
GxDnnResult GxDnnCreateTaskFromBuffer(GxDnnDevice device,
const unsigned char *taskBuffer,
const int bufferSize,
GxDnnTask *task);
/*===============================================================================================*/
/**
* @brief Release NPU task
* @param [in] task the loaded task handle
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* @remark
*/
GxDnnResult GxDnnReleaseTask(GxDnnTask task);
/*===============================================================================================*/
#define MAX_NAME_SIZE 30
#define MAX_SHAPE_SIZE 10
typedef struct NpuIOInfo {
int direction; /* 0: Input; 1: Output */
char name[MAX_NAME_SIZE]; /* name of the IO */
int shape[MAX_SHAPE_SIZE]; /* the shape of the IO */
unsigned int dimension; /* the dimension of the IO */
void *dataBuffer; /* the data buffer */
int bufferSize; /* the data buffer size */
} GxDnnIOInfo;
/**
* @brief Get the IO Num of the loaded task
* @param [in] task the loaded task
* [out] inputNum Input number
* [out] outputNum Output Number
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* @remark
*/
GxDnnResult GxDnnGetTaskIONum(GxDnnTask task,
int *inputNum,
int *outputNum);
/*===============================================================================================*/
/**
* @brief Get the IO Info of the loaded task
* @param [in] task the loaded task
* [out] inputInfo the input information List
* [in] inputInfoSize the size of the output info list buffer
* [out] outputInfo the output information list
* [in] outputInfoSize the size of the output info list buffer
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_ERR_BAD_PARAMETER wrong parameter
* @remark
*/
GxDnnResult GxDnn
GetTaskIOInfo(GxDnnTask task,
GxDnnIOInfo *inputInfo,
int inputInfoSize,
GxDnnIOInfo *outputInfo,
int outputInfoSize);
/*===============================================================================================*/
typedef enum {
GXDNN_EVENT_FINISH,
GXDNN_EVENT_ABORT,
GXDNN_EVENT_FAILURE
} GxDnnEvent;
/**
* @brief The event handler
* @param [in] task the running task
* [in] event the event type
* [in] userData the userData passed by GxDnnRunTask
* @return int 0 break the task
* not 0 continue the task
*/
typedef int (*GxDnnEventHandler)(GxDnnTask task, GxDnnEvent event, void *userData);
/*===============================================================================================*/
/**
* @brief Run task
* @param [in] task the loaded task
* [in] priority the task priority
* [in] eventHandler the event callback (see remark)
* [in] userData a void data will be passed to event handler
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* @remark if eventHandler == NULL, the function will not return until finish or error happens;
* if the task is running, the task will stop first;
*/
GxDnnResult GxDnnRunTask(GxDnnTask task,
int priority,
GxDnnEventHandler eventHandler,
void *userData);
/*===============================================================================================*/
/**
* @brief Stop task
* @param [in] task the loaded task
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* @remark if the task is running, the event handler will be invoked
*/
GxDnnResult GxDnnStopTask(GxDnnTask task);
/*===============================================================================================*/
typedef struct NpuDevUtilInfo {
float ratio;
} GxDnnDevUtilInfo;
/**
* @brief Get device utilization information
* @param [in] GxDnnDevice device
* [out] GxDnnDevUtilInfo info
* @return GxDnnResult GXDNN_RESULT_SUCCESS succeed without error
* GXDNN_RESULT_WRONG_PARAMETER wrong parameter
* GXDNN_RESULT_DEVICE_ERROR device error
*/
GxDnnResult GxDnnGetDeviceUtil(GxDnnDevice device, GxDnnDevUtilInfo *info);
/*===============================================================================================*/
#ifdef __cplusplus
} /* extern C */
#endif
#endif
2. Using SNPU in the MCU*
To generate a model file that can run on the MCU, you need to specify OUTPUT_TYPE: c_code in the model compilation configuration file. The MCU has limited memory resources, so the generated model file is in the form of a C file, which can be directly involved in the compilation. The advantage is that it does not require parsing the model file, but the disadvantage is that when switching models, the code needs to be recompiled. Additionally, due to the MCU's lower efficiency, float16 to float32 conversion cannot be performed on the MCU and must be done by DSP or ARM.
2.1 API Calling Process*
- Open the SNPU.
- Copy input data to the model memory.
- Run the model to get the output data.
- Close the SNPU.
snpu.h
/* Voice Signal Preprocess
* Copyright (C) 1991-2017 Nationalchip Co., Ltd
* All Rights Reserved!
*
* snpu.h: Device Driver for SNPU
*
*/
#ifndef __SNPU_H__
#define __SNPU_H__
int SnpuInit(void);
int SnpuLoadFirmware(void);
int SnpuDone(void);
#ifdef CONFIG_GX8010NRE
int SnpuFloat32To16(unsigned int *in_data, unsigned short *out_data, int num, int exponent_width);
int SnpuFloat16To32(unsigned short *in_data, unsigned int *out_data, int num, int exponent_width);
#endif
typedef enum {
SNPU_IDLE,
SNPU_BUSY,
SNPU_STALL,
} SNPU_STATE;
typedef int (*SNPU_CALLBACK)(SNPU_STATE state, void *private_data);
typedef struct {
const char *version; // version in model.c
void *ops; // ops_content in model.c
void *data; // cpu_content in model.c
void *input; // input in model.c
void *output; // output in model.c
void *cmd; // npu_content in model.c
void *tmp_mem; // tmp_content in model.c
} SNPU_TASK;
int SnpuRunTask(SNPU_TASK *task, SNPU_CALLBACK callback, void *private_data);
SNPU_STATE SnpuGetState(void);
#endif // __SNPU_H__
These API details illustrate how to utilize the NPU and SNPU in both the CPU and MCU to run NPU models effectively. It outlines the necessary steps and configurations to use the NPU and SNPU for different devices.