TensorFlow Example 2*

FSMN Structure Model Compilation Flow*

FSMN is a commonly used structure in speech recognition models in recent years. Its hidden layers contain a memory module. The following example illustrates how gxnpuc supports this model structure with memory modules.

1. Generating NPU Files*

1.1 Writing the Inference Model*

Generally, the inference model differs slightly from the training model. Operations used only during training, such as Dropout, are removed from the inference model, and the CKPT file generated during training is loaded to produce the final inference model PB file.

For the FSMN model, since the model contains memory modules and the NPU is stateless internally, when writing the inference model, the model's memory must be treated as input states and output states. During NPU execution, the output states of the current frame serve as the input states for the next frame.

See the code below for details.

    inputs = tf.placeholder(tf.float32, [batch_size, frame_len, feat_dim*win_len], "Feats") # Specify input feats name
    state1_in = tf.placeholder(tf.float32, [batch_size, 3, linear_size[0], name="State_c0") # Specify input state0 name
    state2_in = tf.placeholder(tf.float32, [batch_size, 4, linear_size[1], name="State_c1") # Specify input state1 name
    state3_in = tf.placeholder(tf.float32, [batch_size, 5, linear_size[2], name="State_c2") # Specify input state2 name
    states = (state1_in, state2_in, state3_in)

    ...

    cnn_outputs = cnn_layer(inputs, seq_len, cnn_info, tf.nn.relu, fusedbn)
    outputs, states = fsmn_layer(cnn_outputs, memory_size, linear_size, hidden_size, states, tf.nn.relu, keep_prob)

    ...

    state1_out = tf.identity(states[0], name="State_c0_out") # Specify output state0 name
    state2_out = tf.identity(states[1], name="State_c1_out") # Specify output state1 name
    state3_out = tf.identity(states[2], name="State_c2_out") # Specify output state2 name
    phone_prob = tf.identity(outputs, name="phone_prob") # Specify final model output name

    ...

    with tf.compat.v1.Session() as sess:
        tf.global_variables_initializer().run(session=sess)
        saver = tf.train.Saver()
        saver.restore(sess, "model.ckpt") # Load CKPT file generated during training

        ...

        tf.train.write_graph(sess.graph_def, "./", "model.pb") # Generate inference model PB file

1.2 Generating CKPT and PB Files, and Merging CKPT and PB into a FROZEN_PB File*

Refer to Generating CKPT and PB Files and Merging CKPT and PB into a FROZEN_PB File

1.3 Editing the NPU Configuration File*

Edit the configuration file config.yaml. The meanings are explained in the comments.

config.yaml

CORENAME: APUS # Chip model
PB_FILE: model_with_ckpt.pb # Input PB file
IN_FEATS_FILE: feats.txt # Input feature values, covering model application scenarios as much as possible
QUANT_FILE: quant.yaml # Output quantization file
OUTPUT_FILE: model.h # Output NPU file name
COMPRESS: true # Compress fully connected layer weights
OUTPUT_TYPE: c_code # NPU file type
INPUT_OPS:
    Feats: [1, 15， 40]
    State_c0: [1, 3, 64]
    State_c1: [1, 4, 64]
    State_c2: [1, 5, 64]
OUTPUT_OPS: [State_c0_out,State_c1_out,State_c2_out,phone_prob] # Output node names, state nodes placed first
FUSE_BN: true # Merge BN parameters into convolution layers (if any)
MAX_CACHE_SIZE: 6402 # Allocate CACHE memory for storing weights and data
USE_DATA_CACHE: true # Store data in CACHE

Note

Here, Feats,State_c0,State_c1,State_c2,State_c0_out,State_c1_out,State_c2_out,phone_prob correspond one-to-one with the model's input and output op names.

Note

In the OUTPUT_OPS configuration item, state output nodes must be placed before the model's actual output nodes.

1.4 Compilation*

First, use the gxnpuc tool to compile and generate the quantization file

$ gxnpuc config.yaml -q

Then, use the gxnpuc tool to compile and generate the NPU file model.h

$ gxnpuc config.yaml

The required memory information is printed:

------------------------
Memory allocation info:
Mem1(data): 11432
Mem2(instruction): 3368
Mem3(in): 2736
Mem4(out): 1666
Mem5(cache): 6300
Mem6(weights): 127298
Total NPU Size (Mem0+Mem1+Mem2+Mem5+Mem6): 148398
Total Memory Size: 152800
------------------------
Compile OK.

The memory regions are described as follows:

Memory Region	Description
Mem1(data)	Intermediate data memory
Mem2(instruction)	Instruction memory
Mem3(in)	Input data memory
Mem4(out)	Output data memory
Mem5(cache)	SRAM memory for weights and data
Mem6(weights)	Weight memory

2. Executing the NPU File*

After the NPU file is generated, the model needs to be deployed and run on the GX830X development board.