It is interesting to use pose estimation for sports counting. This project serves as a demo for this goal, which contains a whole pipeline of model conversion and ncnn Android deployment.
Sit up
counting is supported now. It is easy to add other sports support.
Here is a demo result for Sit up counting.
MoveNet
is used in this project. Google provides its pretrained model in tensorflow format. What I did to convert it to ncnn format is recorded as following.
-
Download the tflite model from tensorflow model hub
- the Lightning version of
MoveNet
is used in this project because it is intended for latency-critical applications.
- the Lightning version of
-
Convert tfllite model to ONNX model using tensorflow-onnx
python -m tf2onnx.convert --tflite movenet.tflite --output movenet.onnx
-
Optimize the ONNX model using onnx-simplifier
python -m onnxsim movenet.onnx sim_movenet.onnx
-
Edit the optimized ONNX model
-
This step is necessary because some of the post processing operators are not supported in ncnn.
-
What we need to do are:
- Remove the pre-processing and post-processing operators.
- They will be implemented in raw C++
- tensorflow use
NHWC
format while ONNX useNCHW
format. So theTranspose
operators in pre-processing and post-processing operators should also be removed.
- rename the model input and output.
- Remove the pre-processing and post-processing operators.
-
Then we get
modified_sim_movenet.onnx
⭐⭐ onnx-modifier is highly recommended for efficient and intuitive ONNX editing! ⭐⭐
-
-
Convert the ONNX model to ncnn format
# onnx2ncnn is provided in ncnn onnx2ncnn modified_sim_movenet.onnx movenet.param movenet.bin
-
Optimize the converted ncnn model
# ncnnoptimize is provided in ncnn ncnnoptimize movenet.param movenet.bin movenet_opt.param movenet_opt.bin 0
Then the ncnn model is ready! What I got is saved here
clone this repo and follow the instructions to build and run the Android demo.
https://github.com/Tencent/ncnn/releases
- Download ncnn-YYYYMMDD-android-vulkan.zip or build ncnn for android yourself
- Extract ncnn-YYYYMMDD-android-vulkan.zip into app/src/main/jni and change the ncnn_DIR path to yours in app/src/main/jni/CMakeLists.txt
https://github.com/nihui/opencv-mobile
- Download opencv-mobile-XYZ-android.zip
- Extract opencv-mobile-XYZ-android.zip into app/src/main/jni and change the OpenCV_DIR path to yours in app/src/main/jni/CMakeLists.txt
- Open this project with Android Studio, build it and enjoy!
The prebuilt android apk file can be downloaded here
Take the supported Sit up
for example, The angle between joints are calculated and serves as clues for counting.
/* The map from joint index to joint:
* 0 : neck; 1 & 2 : eyes; 3 & 4 : ears
* 5 & 6 : shoulders; 7 & 8 : elbows; 9 & 10 : hands
* 11 & 12 : hips; 13 & 14 : knees;
* 15 & 16 : feet
*/
float neck_hip_foot_angle = angle(points[0], points[11], points[15]);
if (!count_lock && neck_hip_foot_angle < 120) {
count_lock = true;
count_number += 1;
}
if (count_lock && neck_hip_foot_angle > 150) {
count_lock = false;
}
The
count_lock
is a bool flag to avoid duplicated counting.
- movenet official blog: https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html
- An earlier version of ncnn movenet: https://github.com/FeiGeChuanShu/ncnn_Android_MoveNet
- The sit-up video is picked here: https://www.wikihow.com/Do-Sit-Ups