Build Errors and Improper Inference for Patch-Based Inference

[goats-9]

I am trying to run the inference tutorial on STM32Cube IDE 1.5.0 as documented in the repo. However, on generating code using the script examples/vww_patchbased.py and moving the generated folder codegen to the inference project folder tutorial/TinyEngine_vww_tutorial/Src/TinyEngine, the following errors are seen in the IDE.

Description	Resource	Path	Location	Type
make: *** [makefile:88: TTE_demo_mcunet.elf] Error 1	TTE_demo_mcunet		 	C/C++ Problem
more undefined references to `arm_nn_requantize' follow	depthwise_kernel3x3_stride2_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 139	C/C++ Problem
undefined reference to `arm_nn_mat_mult_kernel_s8_s16_reordered'	convolve_1x1_s8.c	/TTE_demo_mcunet/Src/TinyEngine/src/kernels/int_forward_op	line 69	C/C++ Problem
undefined reference to `arm_nn_requantize'	depthwise_kernel3x3_stride1_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 110	C/C++ Problem
undefined reference to `arm_nn_requantize'	depthwise_kernel3x3_stride1_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 116	C/C++ Problem
undefined reference to `arm_nn_requantize'	depthwise_kernel3x3_stride1_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 139	C/C++ Problem
undefined reference to `arm_nn_requantize'	depthwise_kernel3x3_stride2_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 110	C/C++ Problem
undefined reference to `arm_nn_requantize'	depthwise_kernel3x3_stride2_inplace_CHW.c	/TTE_demo_mcunet/Src/TinyEngine/codegen/Source	line 116	C/C++ Problem

IDE console output is below.

Do let me know if any other data is required.

[goats-9]

There is a workaround to build the patch-based inference codebase.

Note: The following steps have to be done after cloning the repo and installing the python packages necessary.

1. Copy arm_nn_mat_mult_kernel_s8_s16_reordered.c from the CMSIS library to TinyEngine/src. Execute the below command from the root of the repo.

cp TinyEngine/third_party/CMSIS/CMSIS/NN/Source/ConvolutionFunctions/arm_nn_mat_mult_kernel_s8_s16_reordered.c TinyEngine/src/kernels/int_forward_op/

2. Modify tinyengine_function.h as shown below.

diff --git a/TinyEngine/include/tinyengine_function.h b/TinyEngine/include/tinyengine_function.h
index 55f379c..bb4ed0b 100644
--- a/TinyEngine/include/tinyengine_function.h
+++ b/TinyEngine/include/tinyengine_function.h
@@ -17,6 +17,9 @@
  * Target ISA:  ARMv7E-M
  * -------------------------------------------------------------------- */

+#include "arm_nnfunctions.h"
+#include "arm_nnsupportfunctions.h"
+
 #include <math.h>
 #include <stdbool.h>
 #include <stdint.h>

However, on running the code on the board, I am unable to get an inference result (using mcunet-vww1), with the network always predicting No Person even when a person is displayed to the camera. I am unsure if these changes are the cause of wrong inference.

[rahulvigneswaran]

@goats-9

Copy arm_nn_mat_mult_kernel_s8_s16_reordered.c from the CMSIS library to TinyEngine/src. Execute the below command from the root of the repo.

Simply renaming the function arm_nn_mat_mult_kernel_s8_s16_reordered in convolve_1x1_s8.c to arm_nn_mat_mult_kernel_s8_s16_reordered_8mul does the trick.

[goats-9]

@rahulvigneswaran

Thanks for the reply. We tried renaming the function to both _8mul and oddch prefixes already, but inference results did not improve. We believe the issue may be among the following.

1. Code generation for patch-based inference (in examples/vww_patchbased.py)
2. Models are not pretrained (which is very unlikely looking at the results published by the authors).

Nevertheless, I will add these inputs to the workarounds stated above.

[rahulvigneswaran]

@goats-9

Building upon your insights, we've explored additional strategies, yet encountered persistent issues. Here's a summary:

• Experimented with diverse patch grid configurations.
• Set the 'inplace' parameter to False.

While these adjustments occasionally impact prediction outcomes, the issue of classifying 'person' and 'no person' as the same remains consistent.

[Oslomayor]

@meenchen I am trying the tutorial(vww patchbased) and also getting the same build errors.
I do as @goats-9 shown above,

1. Modify tinyengine_function.h

+#include "arm_nnfunctions.h"
+#include "arm_nnsupportfunctions.h"

 #include <math.h>
 #include <stdbool.h>
 #include <stdint.h>

2. Renaming the function arm_nn_mat_mult_kernel_s8_s16_reordered in convolve_1x1_s8.c to arm_nn_mat_mult_kernel_s8_s16_reordered_8mul.

Finally, I build the project successfully but get wrong inference result as shown in the picture below.

[winffke]

@Oslomayor
I would like to know if you have fully resolved the build issue mentioned there. I have encountered the same problem, I hope to get your guidance on this issue. I would appreciate any further details you can provide.

Thank you for your assistance!

Best regards

[Oslomayor]

@winffke I have no solution, good luck!