Skip to content

Dynamic frequency modulation APP (lvp_app_kws_state_demo)*

General voice control products are voice controlled in the form of wake word + command word, that is, wake up with the wake word first, and then recognize the secondary command word. Since our decoder requires a high MCU computing power to score multiple instruction words, in order to further save power consumption, the SDK introduces two states (VAD state and wake-up state) for dynamic frequency modulation:

  • VAD state: only the main wakeword can be activated, mcu is of low frequency, generally set to 4M, and the main wakeword is activated into the wakeword state.
  • Wake up state: all words can be activated, mcu is high frequency, timeout enters VAD state .

notice

The SDK provides a reference example:lvp_tws/app/lvp_app_kws_state_demo/lvp_app_kws_state_demo.Convenient for developers to refer to.

1. Code interpretation*

lvp_tws/app/lvp_app_kws_state_demo/lvp_app_kws_state_demo.c
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
// #define DO_NOT_SLEEP_WHEN_WAKING_UP     // The mcu will not go to sleep after the wake-up word is defined, but will only go to sleep after the timeout. Some customers have this requirement

#ifdef DO_NOT_SLEEP_WHEN_WAKING_UP
#include <lvp_pmu.h>
#endif

#define __LOW_FREQUENCE CPU_FREQUENCE_8M        // MCU frequency when entering VAD state
#define __TIME_OUT_S    10                      // After activation, no operation times out. The value is such as the VAD status time. unit(s)
#define __WAKE_UP_KWS_1 100                     // Main wakeword event_id
#define __WAKE_UP_KWS_2 100                     // Some items have multiple primary wakewords

typedef struct TIMEOUT_STANDBY{
    unsigned long first_time;                   // Record the first time
    unsigned long next_time;                    // Record the second time
    unsigned char timeout_flag;                 // If the time difference is greater than the timeout period, the timeout flag bit is activated
}TIMEOUT_STANDBY;                               // For timeout detection

static TIMEOUT_STANDBY kws_state_time = {0, 0, 1};   // The initial state is set to timeout
static int kws_state_init_flag = 0;             // Partial initialization is performed only once, initializing the flag bit
#ifdef DO_NOT_SLEEP_WHEN_WAKING_UP
static int lock = 0;                            // The handle used to hold the lock
#endif
//===================================================================================

static int KwsStateDemoAppSuspend(void *priv)   // The App enters the standby callback interface
{
    printf(LOG_TAG" ---- %s ----\n", __func__);
    BoardSetPowerSavePinMux();                  // Set all GPIO ports as input to reduce standby power consumption
    return 0;
}

static int KwsStateDemoAppResume(void *priv)    // App recovery work callback interface
{
    BoardSetUserPinMux();                       // Restore the user's GPIO configuration
    printf(LOG_TAG" ---- %s ----\n", __func__);
    if (kws_state_time.timeout_flag)            // Determine whether the work times out, because the active state can also enter standby
        LvpDynamiciallyAdjustCpuFrequency(__LOW_FREQUENCE); // Timeout set low frequency
    printf("*************** cpu frequency is %dHz *************\n",gx_clock_get_module_frequence(CLOCK_MODULE_SCPU));  // Obtain the MCU clock frequency
    return 0;
}

static int KwsStateDemoAppInit(void)            // App initializes the interface
{
    if (!kws_state_init_flag)                   // The program in this judgment will only be executed once
    {
#ifdef DO_NOT_SLEEP_WHEN_WAKING_UP
        LvpPmuSuspendLockCreate(&lock);         // Creates a pmu lock that returns a handle to the free lock
#endif
        LvpSetVuiKwsStates(VUI_KWS_VAD_STATE);  // The framework is Active by default and needs to be set to VAD in the App
        kws_state_init_flag = 1;                // Mark position 1
        gx_rtc_get_tick(&kws_state_time.first_time);    // Acquisition time
    }
    return 0;
}

// App Event Process
static int KwsStateDemoAppEventResponse(APP_EVENT *app_event)   // App event response
{
    if (!kws_state_time.timeout_flag) {         // Determines the timeout flag bit. If it has timed out, the command will not be executed
        gx_rtc_get_tick(&kws_state_time.next_time);     // next_time is updated without timeout
        if (kws_state_time.next_time - kws_state_time.first_time > __TIME_OUT_S) {
                                                // Determine whether time out
            kws_state_time.timeout_flag = 1;    // Timeout flag position 1
            LvpSetVuiKwsStates(VUI_KWS_VAD_STATE);  // Set the status to VAD
            LvpDynamiciallyAdjustCpuFrequency(__LOW_FREQUENCE); // Timeout sets mcu low frequency
            printf("*****************cpu frequency is %dHz ************\n",gx_clock_get_module_frequence(CLOCK_MODULE_SCPU));
                                                // Obtain the mcu frequency and check whether it is set successfully
            printf("timeout! ---> next_time%ds - first_time%ds = %ds \n", kws_state_time.next_time,\
                   kws_state_time.first_time, kws_state_time.next_time - kws_state_time.first_time);
                                                // Print timeout message
#ifdef DO_NOT_SLEEP_WHEN_WAKING_UP
            LvpPmuSuspendUnlock(lock);          // Unlock after timeout
#endif
        }
    }

    if (app_event->event_id < 100)              // Anything below 100 is a system event, and anything above 100 is a user event
        return 0;

    LVP_CONTEXT *context;
    unsigned int ctx_size;
    LvpGetContext(app_event->ctx_index, &context, &ctx_size);
    if (app_event->event_id == __WAKE_UP_KWS_1 || app_event->event_id == __WAKE_UP_KWS_2) {
                                                // Determine if the main wakeword is active
        if(kws_state_time.timeout_flag){        // No longer switches frequency of wake up without timeout
            LvpSetVuiKwsStates(VUI_KWS_ACTIVE_STATE);   // Set the status to Active
            LvpDynamiciallyAdjustCpuFrequency(CPU_FREQUENCE_DEFAULT);
                                                // Set the mcu frequency to the user's default frequency
#ifdef DO_NOT_SLEEP_WHEN_WAKING_UP
            LvpPmuSuspendLock(lock);            // Wake up and lock
#endif
        }
        printf("*************** cpu frequency is %dHz *************\n",gx_clock_get_module_frequence(CLOCK_MODULE_SCPU));
                                                // Obtain the mcu frequency and check whether it is set successfully
    }

    gx_rtc_get_tick(&kws_state_time.first_time);// Update the time of the first record after activation
    kws_state_time.timeout_flag = 0;            // Set the timeout flag at position 0. After activating, the timeout is recalculated
    printf(LOG_TAG"event_id %d\n", app_event->event_id);// Print time id

    return 0;
}

// APP Main Loop
static int KwsStateDemoAppTaskLoop(void)        // App loop interface
{
    return 0;
}

2. Development specification*

2.1 Validation model*

  • Before App development, verify that the model is OK. Test whether it can switch frequency normally through the APP of kws_state_demo, and then develop your own App
    • Select the Kws State Demo App Application from the LVP Application Settings in the compile configuration.
    • Configure Enable FFT Vad and Enable Standby in VUI Settings of compile configuration.
    • Verify that it can be activated normally. Print log and switch frequency, indicating that there is no problem with the configuration of the model, and subsequent development can be carried out. Please see 2.3 if it cannot work normally.

2.2 Code filling*

  • After verifying that it is OK, develop the user's own App
  • To copy this.c file code into the app file to be developed, you only need to configure the following macros 
    #define __LOW_FREQUENCE CPU_FREQUENCE_8M        // MCU frequency when entering VAD state
#define __TIME_OUT_S    10                      // After activation, no operation times out. The value is such as the VAD status time. unit(s)
#define __WAKE_UP_KWS_1 100                     // Main wakeword event_id
#define __WAKE_UP_KWS_2 100                     // Some items have multiple primary wakewords
  • The original code in the interface does not need to be modified, just need to process the code to fill the corresponding interface.

2.3 FAQ*

  • Wakeword activation failure: The main wakeword major in the kws_list.h file in the model is not configured, so the program cannot identify which is the main wakeword.
    • For example, if you need to configure lvp/vui/kws/models/xxxx/kws_list.h for the main wakeword in major, find the code corresponding to the main wakeword and set the last parameter to 1. 
      {"xiao ai xiao ai", {14, 38, 55, 31, 14, 38, 55, 31}, 8, CONFIG_KEYWORD_XIAO_AI_XIAO_AI_THRESHOLD, CONFIG_KEYWORD_XIAO_AI_XIAO_AI_VALUE, 1},
  • The wake up rate is low and the print rate is abnormally slow
    • The NPU frequency is low, so the algorithm cannot run. Therefore, the NPU frequency should be increased appropriately.
    • Increase NPU frequency in build configuration Frequency settings.
    • There are too many instruction words. Without grouping, the algorithm cannot run. Therefore, grouping instruction words is required.
    • Compile configuration VUI Settings and select Model Param Setting for grouping.