Skip to content

Quick Development Guide*

1. Introduction*

This document aims to help developers quickly develop applications based on the existing framework. It only covers essential information for application development and does not explain the specific processes of the framework.

2. Multi-core Heterogeneous Architecture and Inter-core Communication*

  • VSP is a software framework for audio signal processing running on MCU, DSP, CPU, and NPU.
  • Inter-core communication between VSP cores is done through interrupt messages and shared memory.

  • There are 5 inter-core communication messages, and each interrupt can carry 32 bytes of data.

    // MCU -> DSP
int DspPostVspMessage(const uint32_t *message, const uint32_t size);
int DspSendVspMessage(const uint32_t *request, const uint32_t size,
                    DSP_VSP_CALLBACK callback, void *priv, unsigned int timeout_ms);

// DSP -> MCU
int McuPostVspMessage(const unsigned int *response, const unsigned int size);

// MCU -> CPU
int CpuPostVspMessage(const uint32_t *message, const uint32_t size);

// CPU -> MCU
static int _VspSendMcuRequest(VSP_CORE *vsp_core, VSP_MSG_CPU_MCU *request, VSP_MSG_MCU_CPU *response);
static int _VspPostMcuRequest(VSP_CORE *vsp_core, VSP_MSG_CPU_MCU *request);

// MCU -> NPU
int SnpuRunTask(SNPU_TASK *task, SNPU_CALLBACK callback, void *private_data);

GX8008 and GX8008C have 1.5MB of on-chip SRAM, which is accessible by MCU, DSP, and other modules. The DSP has a reserved portion of SRAM configured in compilation settings, which leaves that much dedicated memory at the end of SRAM. Other parts are shared among modules.

MCU uses the MCU address space, while other cores and peripheral modules use the DEV address space. You can switch between these address spaces using the following macro functions:

#define MCU_TO_DEV(x) ((unsigned int)x >= 0x40000000 ? \
                    (void *)((unsigned int)x - 0x40000000) : (void *)((unsigned int)x - 0x20000000))
#define DEV_TO_MCU(x) ((void *)((unsigned int)x + 0x40000000))

GX8008 and GX8008C support eXecute In Place (XIP). To enable XIP for MCU and DSP, configure the Enable XIP settings for each of them in the compilation settings. After enabling XIP, you can place variables or functions in XIP using the following two macros:

#define XIP_TEXT_ATTR      __attribute__((section(".xip.text")))
#define XIP_RODATA_ATTR    __attribute__((section(".xip.rodata")))

3. Compilation Configuration System*

  • Source files are various Kconfig files in vsp_sdk.
  • Execute the following command to open the compilation configuration interface: $ make menuconfig
  • Use the arrow keys to select items, left and right keys to choose bottom menus, space key to check or uncheck the current item, and enter key to confirm and enter sub-menus.
  • The configuration system is complex and requires an understanding of the overall software and hardware framework. This document will only cover necessary parts.

4. Board-Level Explanation*

  • Board-level configuration is related to hardware-specific initialization configuration, most of which are determined by the hardware.

4.1 Board Selection*

  • The board-level code is located in mcu/boards/, with many different options. The mcu/boards/nationalchip folder contains the Nationalchip's public version.

  • Execute the following command to select the board-level configuration: $ make menuconfig

4.2 Board Configuration*

  • If you select the GX8008C Wukong Prime Device Board V1.4, you can enter the Board Options: for related configurations. The pin default reuse is GPIO.

4.3 Channel Configuration Example*

  • The following examples are based on the GX8008C Wukong Prime Device Board V1.4 board-level. The order of the channels can be adjusted according to your needs, and the number of valid channels is related to the configuration of VSP I/O Buffer settings. See 5. Data Format and Data Flow for the explanation of channel number configuration.
  • Note: If the hardware development board does not support the following examples, please contact our sales or hardware engineers.

4.3.1 Mic Channel Hardware and Software Mapping*

  • Hardware Mic Input Channel

Figure 1. Chip Pins

Figure 4-1
  • Dmic Pin Multiplexing

Figure 2. Dmic Pin Multiplexing

Figure 4-2
  • Dmic Schematic

Figure 3. Dmic Schematic

Figure 4-3
  • The configuration 0 of Amic in the software corresponds to AIN0 in the hardware as shown in Figure 4-1, and so on. The number 3 corresponds to AIN3.
  • The configuration 0 of Dmic in the software corresponds to 0-0 in the hardware as shown in Figure 4-3. The preceding 0 represents being connected to the PDM_DATA0 line. Configuration 1 corresponds to hardware 0-1, configuration 2 corresponds to 1-0, and configuration 3 corresponds to 1-1.

4.3.2 Example 1:4Amic+2Inter_Ref*

  • Example 1 Firmware package download address
  • In this configuration example, the configuration corresponding to mic channel 0 is 0, so the hardware corresponding to mic channel 0 is AIN0. If 1 is configured, the hardware corresponding to mic channel 0 is AIN1

4.3.3 Example 2:4Dmic+2Amic_Ref*

  • Example 2 Firmware package download address
  • In this configuration example, the configuration corresponding to mic channel 0 is 0, so the hardware corresponding to mic channel 0 is DMic0-0. If 1 is configured, then the hardware corresponding to mic channel 0 is DMic0-1

4.3.4 Example 3:2Amic+2Amic_Ref*

  • Example 3 Firmware package download address
  • In this configuration example, the configuration corresponding to mic channel 0 is 0, so the hardware corresponding to mic channel 0 is AIN0. If 1 is configured, the hardware corresponding to mic channel 0 is AIN1

4.4 Gain configuration*

4.4.1 Mic gain*

  • Mic gain according to the type of the Mic will have different step, Amic range is 0~98, the step is 2 db. Dmic is 0~54, and the step is 6dB.

4.4.2 Ref gain*

  • The Ref gain will have different steps depending on the source of the Ref. The Amic range is 0~98 and the step is 2dB. Dmic/IIS/internal is 0~54, and the step is 6dB.

4.5 Precautions*

  • Pin Multiplexing Conflict: For example, there is a conflict between the DSP UART, I2S Out, and SPI1 pins, and these functionalities cannot be used simultaneously. There are also other pin conflicts, which need attention in the file mcu/boards/nationalchip/leo_mini_gx8008c_wukong_prime_1_4v/misc_board.c. This file provides pin descriptions.

  • If you choose a different board level, you need to modify the mic reference and gain configurations in the corresponding audio_board.c, and modify the pin multiplexing in misc_board.c. Refer to mcu/boards/nationalchip/leo_mini_gx8008c_wukong_prime_1_4v/ for guidance. However, if the board level's original configuration meets your requirements, no additional modifications are needed.

5. Data Formats and Data Flow*

  • VSP audio processing adopts a pipeline structure, and data is transmitted between cores using the context: 
    [Audio In] --interrupt-> [MCU] --interrupt-> [DSP] --interrupt-> [MCU] --interrupt-> [CPU]
                                                                    `-- Application Framework-> [APP]

  • The context data format is defined by the structure VSP_CONTEXT:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
    typedef struct {
    VSP_CONTEXT_HEADER *ctx_header; // DEVICE ADDRESS
    unsigned mic_mask:16;
    unsigned ref_mask:16;
    unsigned int frame_index;       // FRAME index of the first frame in CONTEXT
    unsigned int ctx_index;         // CONTEXT index from 0 - (2^32 - 1)
    unsigned int vad:8;
    unsigned int kws:8;             // Keyword value, filled with a value greater than or equal to 100 when KWS is triggered
    unsigned int mic_gain:8;
    unsigned int ref_gain:8;
    int direction;
    SAMPLE *out_buffer;             // DSP output buffer, usually divided into multiple channels when outputting audio, pay attention to whether interleaving is required
    void *features;                 // DEVICE ADDRESS
    void *snpu_buffer;              // DEVICE ADDRESS
    void *ext_buffer;               // DEVICE ADDRESS
} VSP_CONTEXT;

    The VSP_CONTEXT combined with the buffers it manages forms a new structure:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    typedef struct {
    VSP_CONTEXT context;
#if CONFIG_VSP_OUT_NUM > 0
    SAMPLE out_buffer[VSP_FRAME_SIZE * CONFIG_VSP_OUT_NUM * CONFIG_VSP_FRAME_NUM_PER_CONTEXT]__attribute__ ((aligned (128)));
#endif
#ifdef CONFIG_VSP_VPA_FEATURES_DIM
    float features[CONFIG_VSP_VPA_FEATURES_DIM * CONFIG_VSP_FRAME_NUM_PER_CONTEXT];
#endif
#ifdef CONFIG_VPA_SNPU_BUFFER_SIZE
    SNPU_FLOAT snpu_buffer[CONFIG_VPA_SNPU_BUFFER_SIZE];
#endif
#ifdef CONFIG_VSP_VPA_EXT_BUFFER_SIZE
    unsigned int ext_buffer[(CONFIG_VSP_VPA_EXT_BUFFER_SIZE * 1024) / 4];
#endif
} __attribute__ ((aligned (8))) VSP_CONTEXT_BUFFER;
    The new structure manages cyclic buffers through loops.

    1
    static VSP_CONTEXT_BUFFER s_context_buffer[CONFIG_VSP_SRAM_CONTEXT_NUM] __attribute__((aligned(128)));

  • The cyclic buffer of context is managed through VSP_CONTEXT_HEADER. The VSP_CONTEXT_HEADER also manages mic_buffer and ref_buffer, etc.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
    typedef struct {
    unsigned int version;

    unsigned int mic_num;                // 1 - 8
    unsigned int ref_num;                // 0 - 2
    unsigned int spk_num;                // 0 - 2
    unsigned int out_num;                // 1 - 15
    unsigned int out_interlaced;         // 1: interlace or 0: non-interlace
    unsigned int frame_num_per_context;  // the FRAME num in a CONTEXT
    unsigned int frame_num_per_channel;  // the total FRMAE num in a CHANNEL
    unsigned int frame_length;           // 10ms, 16ms
    unsigned int sample_rate;            // 8000, 16000, 48000

    int ref_offset_to_mic;      // MIC signal offset to REF in unit of context.
                                // If REF is saved before MIC, the offset is > 0;
                                // otherwise, it will be < 0.

    unsigned int features_dim_per_frame; // Features dimension per FRAME

    // CTX buffer for GLOBAL
    void *ctx_buffer;             // Context Buffer header point
    unsigned int ctx_num;                // Context number
    unsigned int ctx_size;               // Context size

    // SNPU, EXTRA and OUT buffer for CONTEXT
    unsigned int snpu_buffer_size;       // Bytes
    unsigned int ext_buffer_size;        // Bytes
    unsigned int out_buffer_size;        // Bytes

    // MIC buffer for GLOBAL
    SAMPLE *mic_buffer;             // DEVICE ADDRESS
    unsigned int mic_buffer_size;        // Bytes
    // REF buffer for GLOBAL
    SAMPLE *ref_buffer;             // DEVICE ADDRESS
    unsigned int ref_buffer_size;        // Bytes
    // SPK buffer for GLOBAL
    SAMPLE *spk_buffer;             // DEVICE ADDRESS
    unsigned int spk_buffer_size;        // Bytes
    // TMP buffer for GLOBAL
    void *tmp_buffer;             // DEVICE ADDRESS
    unsigned int tmp_buffer_size;        // Bytes
} VSP_CONTEXT_HEADER;

  • The general data flow is shown below, with some details omitted (animated gifs can be saved and played frame by frame):

  • The example in the figure sets 4 mic, 2 ref, and 1 spk. The addresses of these buffers are continuous, and the channel length is 12 context periods.

    • AudioIn reports the collected data via an interrupt every context period.
    • The collected audio data will be overwritten after the 12th audio period, and up to 11 context periods of data can be saved for use at any time.
    • 4 contexts are set, and the output addresses in each context are not continuous.
    • The context is cyclically used, and each core does not need to pass it down in a timely manner without affecting the pipeline.

  • These audio acquisition-related parameters need to be configured in make menuconfig under VSP I/O Buffer settings. These configurations also set the corresponding buffer parameters: The buffer management code is in mcu/vsp/vsp_buffer.h and mcu/vsp/vsp_buffer.c.

6. Operation Modes*

  • VSP supports choosing different operation modes, with different focuses.
  • GX8010 and GX8009 support mode switching, while GX8008 and GX8008C generally do not support switching operation modes. Workmode_setting
  • The commonly used operation modes for GX8008C and GX8008 are UAC and PLC.
  • The UAC mode provides all supported UAC functionalities but has weaker support for the application framework: mcu/vsp/vsp_mode_uac.c UAC_mode
  • The PLC mode supports some UAC functionalities and has better support for the application framework: mcu/vsp/vsp_mode_plc.c PLC_mode

7. UAC*

  • UAC core functions are downlink playback and uplink recording.

  • The UAC function needs to be configured during compilation configuration. The UAC mode is recommended.

  • UAC 1.0 compatibility is better, UAC 2.0 can support more paths of data.

  • Downlink broadcast need enable Enable UAC Playback and then set the parameter if you need to get downstream data further processing can make the Enable get playback data at APP or other, please. If Play by Audio out is enabled, it can be played directly through the UAC framework.

  • The data source of the uplink recording is output_buffer data Uplink recording data must be Interlaced and the Number of channels is specified by the OUT Channel Interlaced Number. Other options refer to working mode.

  • If raw data of channels such as mic ref needs to be recorded, the bypass Algorithm Voice Process Algorithm select: (Bypass [Source]) in vsp_sdk is recommended.

    This algorithm can interweave the mic ref and other channel raw data into the output buffer for UAC upstream use.

  • If the development board you use is GX8008C Wukong Prime Device Board V1.4, you can use ./configs/nationalchip_public_version/8008c_wukong_v1.4_uac_demo.config compilation to generate the firmware generated by UAC demo burning to the board, connect to the PC, you can find the audio input and output device named Nationalchip, can carry out 4-channel recording, and 2-channel broadcasting.

8. Acquiring audio data*

8.1 MCU side to obtain audio data*

  • Reference mcu/vsp/hook/vsp_hook_codec_v1_0.c

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    int HookEventResponse(PLC_EVENT plc_event)
{

    if (plc_event.event_id == DSP_PROCESS_DONE_EVENT_ID && plc_event.ctx_index > 20) {
        VSP_CONTEXT *ctx_addr;
        unsigned int ctx_size;
        unsigned context_index = plc_event.ctx_index;

        VspGetSRAMContext(context_index, &ctx_addr, &ctx_size);

        VSP_CONTEXT_HEADER *ctx_header = VspGetSRAMContextHeader();
        int frame_length       = ctx_header->frame_length * ctx_header->sample_rate / 1000;
        int context_sample_num = frame_length * ctx_header->frame_num_per_context;
        int context_sample_size = frame_length * ctx_header->frame_num_per_context * sizeof(SAMPLE);
        int context_num_per_channel = ctx_header->frame_num_per_channel / ctx_header->frame_num_per_context;
        int channel_sample_size = frame_length * ctx_header->frame_num_per_channel * sizeof(SAMPLE);
        int current_context_index = ctx_addr->ctx_index % context_num_per_channel;
        void *current_mic_addr = ctx_header->mic_buffer + context_sample_num * current_context_index;
        void *current_ref_addr = ctx_header->ref_buffer + context_sample_num * current_context_index;
        void *current_spk_addr = ctx_header->spk_buffer + context_sample_num * current_context_index;
        void *current_out_addr = ctx_addr->out_buffer;
        int   play_size = (ctx_header->out_buffer_size / ctx_header->out_num) / 8 * 8; // One context period of a channel corresponds to the data length
#ifdef CONFIG_CODEC_CHANNEL0_OUT0
        void *ch0_addr = current_out_addr + play_size * 0;
#endif

...
    }
}

8.2 The audio data is obtained on the DSP side*

9. DSP Development*

10. Front and back*

  • VSP is the front-background system, the foreground is various interrupts, the background is the main loop in main().
  • VSP interrupts are not interrupted, and each interrupt can save one interrupt state; Do not perform operations that take a long time during interruption.
  • The interrupt mainly includes AudioIn acquisition interrupt, DSP return interrupt, CPU interrupt, AudioOut playback interrupt, key interrupt, timer interrupt and so on.
  • AudioIn acquisition interrupt is the starting point of data pipelining. Some important application events, such as KWS events, are triggered in DSP interrupts. UAC mode: _UacAudioInRecordCallback(), _UacDspCallback() PLC mode: _PlcAudioInRecordCallback(), _PlcDspCallback()

  • The main loop of the background has the mode tick() attached to it, the mode tick() has the application frame tick() attached to it, and the application frame tick() has the application tick() attached to it. These tick() are all called in the main loop.

    1
2
3
    while(VspModeTick()) {
    ...
}

11. Application framework*

  • VSP application framework named PLC through event driven, through such a simple process to achieve [Event trigger->enqueue,dequeuing->Event response,dequeuing->Event response].

  • The application framework needs to be initialized in the pattern initialization function before it can be used, which calls the APP's initialization interface

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    int VspInitializePlcEvent(void)
{
    ...

    VspQueueInit(&s_plc_misc_event_queue, s_plc_misc_event_queue_buffer, VSP_PLC_MISC_QUEUE_LEN * sizeof(PLC_EVENT), sizeof(PLC_EVENT));
    HookProcessInit();

    ...
    s_init_flag = 1;
    return 0;
}

  • Event trigger->enqueue Can be performed anywhere after PLC initialization, including interrupt. Different events are marked with event_id and can carry related parameters.

    1
2
3
4
5
6
7
    typedef struct {
    unsigned int event_id;
    union {
        unsigned int    ctx_index;
        int             uac_volume;
    };
} PLC_EVENT;

    event_id less than 100 is a system event. 100-200 is generally used as a KWS event. Events larger than 200 are used as user-defined events system events are triggered only after Enable system event is enabled.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    // Private Event ID [1~99] for PLC
#define VSP_WAKE_UP_EVENT_ID            (90)
#define ESR_GOODBYE_EVENT_ID            (99)  // It defined in the plc_json
#define DSP_PROCESS_DONE_EVENT_ID       (98)  // Used to notify board to send wav to bluetooth
#define AUDIO_IN_RECORD_DONE_EVENT_ID   (97)

#define UAC_DOWN_STREAM_OFF             (80)
#define UAC_DOWN_STREAM_ON              (81)
#define UAC_DOWN_SET_VOLUME             (82)
#define UAC_UP_STREAM_OFF               (83)

  • dequeuing->Event response It is carried out by the frame in the background loop tick, and may not be timely due to the overall calculation load of MCU. Some events can be carried out by the framework without any action by the APP, such as playing voice reply. All events are passed to the application interface

    1
2
3
4
    if (VspQueueGet(&s_plc_misc_event_queue, (unsigned char *)&plc_event))
{
    HookEventResponse(plc_event);
}

  • dequeuing->Event response

    1
2
3
4
5
6
7
8
    void VspPlcEventTick(void)
{
    ...

    HookProcessTick();

    ...
}

12. New user application*

Users can create their own application, on which to do secondary development, HookProcessInit(), HookEventResponse() and HookProcessTick() three interfaces are the user APP needs and only needs to implement three functions.

Please refer to mcu/vsp/hook/vsp_hook_null.c, And modify mcu/vsp/hook/Kconfig to add a newly created APP option in VSP Customize Functions Settings →