1、前言:
在上一期中[开源]基于姿态估计的运动计数APP开发(二)中,我们已经完成了仰卧起坐算法的开发和windows的demo开发。本期主要是将该算法一直到android平台上面,实现一个android手机上可以使用的APP。下面的视频是我在西湖边进行的测试,在背景比较干净的情况下,效果还不错哦。【获取APP源码请留言,或者添加我的微信,15158106211,备注“仰卧起坐APP”,让我们一起学习一起进步。】
(CSDN放不了视频,请见谅)
2、模型改进
有的朋友已经发现,上一期的demo中,只有两个关键点,头部和膝盖,而这次的APP有三个关键点,分别是头部,腰部和膝盖。当只有两个关键点的时候,模型很容易出现误识别,会把关键点定位在一些其他东西上面,很少会考虑这是不是属于一个人的特征。而使用了三个关键点之后,一方面更加有利于判断姿势,另一方面也相对于增加了一个监督信号,并且三个关键点都位于人身上,使得模型更容易理解和学习。因此在性能上有一定的提升,主要是降低了误识别率。
3、APP框架
APP主要要Activiry类,两个SurfaceView类,一个Alg类,一个Camera类组成。Alg类主要负责调用算法进行推理,并返回结果。这里我已经将pytorch训练好的模型转换成了NCNN模型,因此实际上是调用的NCNN库的推理功能。Camera类主要负责摄像头的打开和关闭,以及进行预览回调。第一个SurfaceView(DisplayView)主要用于摄像头预览的展示。第二个SurfaceView(CustomView)主要用于绘制一些关键点信息,计数统计信息等。Activity就是最上层的一个管理类,负责管理整个APP,包括创建按钮,创建SurfaceView,创建Alg类,创建Camera类等。
3、主要类源码
3.1、Activity类核心源码
public class MainActivity extends Activity implements Camera.PreviewCallback, AlgCallBack{
private DisplayView mViewDisplay;
private CustomView mViewCustom;
private Button mBtnCameraOp;
private Button mBtnCameraChange;
private CameraUtil mCameraUtil;
private AlgUtil mAlgUtil;
private int mFrameCount = 0;
@Override
protected void onCreate(Bundle savedInstanceState) {
Log.i("zhengxing", "MainActivity::onCreate");
super.onCreate(savedInstanceState);
this.getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN);
this.requestWindowFeature(Window.FEATURE_NO_TITLE);
this.getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON, WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);
setContentView(R.layout.main);
Log.i("zhengxing", "MainActivity::onCreate set basic info finished");
// 创建两个SurfaceView
mViewDisplay = (DisplayView)this.findViewById(R.id.display_view);
mViewCustom = (CustomView)this.findViewById(R.id.custom_view);
Log.i("zhengxing", "MainActivity::onCreate create visual toolkits finished");
//初始化camera类和alg类
mCameraUtil = new CameraUtil(mViewDisplay, this);
mAlgUtil = new AlgUtil(getAssets(), this);
Log.i("zhengxing", "MainActivity::onCreate create camera util and alg util finished");
}
// 开始按钮
public void onBtnStartClick(View view){
Log.i("zhengxing", "MainActivity::onBtnStartClick");
if (mCameraUtil.getCameraState() < 0){
mCameraUtil.openCamera();
Log.i("zhengxing", "MainActivity::onBtnStartClick the camera is closed, open it");
}
}
//停止按钮
public void onBtnStopClick(View view){
Log.i("zhengxing", "MainActivity::onBtnStopClick");
if (mCameraUtil.getCameraState() >= 0){
mCameraUtil.closeCamera();
Log.i("zhengxing", "MainActivity::onBtnStopClick the camera is open, close it");
}
}
//算法回调函数,处理算法返回结果
@Override
public void onAlgRet(float[] ret) {
float nAlgType = ret[0];
float nClasss = ret[1];
Log.i("zhengxing", "MainActivity::onAlgRet ret value:" + ret[0] + ';' + ret[1]);
mViewCustom.drawAlgRet(ret);
}
//预览回调函数,处理每一帧图片
@Override
public void onPreviewFrame(byte[] data, Camera camera) {
mFrameCount ++;
Log.i("zhengxing", "MainActivity::onPreviewFrame");
Camera.Size size = camera.getParameters().getPreviewSize();
mAlgUtil.addDataToQueue(data, size.width, size.height);
}
}
3.2、Camera类核心源码
public class CameraUtil
{
private Camera mCamera;
private final int mCameraID;
private final SurfaceView mViewDisplay;
private final int mOrientation;
private final Camera.PreviewCallback mPreviewCBack;
public CameraUtil(SurfaceView displayView, Camera.PreviewCallback cameraCBack) {
Log.i("zhengxing", "CameraUtil::CameraUtil");
mCamera = null;
mViewDisplay = displayView;
mCameraID = Camera.CameraInfo.CAMERA_FACING_FRONT;
mOrientation = 0;
mPreviewCBack = cameraCBack;
}
//打开摄像头
public void openCamera() {
Log.i("zhengxing", "CameraUtil::openCamera");
if(mCamera == null) {
mCamera = Camera.open(mCameraID);
Camera.Parameters parameters = mCamera.getParameters();
//parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
//mCamera.setParameters(parameters);
mCamera.setDisplayOrientation(mOrientation);
mCamera.setPreviewCallback(mPreviewCBack);
try {
mCamera.setPreviewDisplay(mViewDisplay.getHolder());
} catch (IOException e) {
e.printStackTrace();
}
mCamera.startPreview();
}
}
//关闭摄像头
public void closeCamera() {
Log.i("zhengxing", "CameraUtil::closeCamera");
if (mCamera != null) {
mCamera.setPreviewCallback(null);
mCamera.stopPreview();
mCamera.release();
mCamera = null;
}
}
}
3.3、Alg类核心源码
class AlgUtil implements Runnable {
private boolean mThreadFlag;
private final ArrayBlockingQueue mQueue;
private final Alg mAlg;
private final int mAlgThreads;
private AlgCallBack mAlgCB;
private final Thread mThread;
public AlgUtil(AssetManager assertManager, AlgCallBack algCallBack) {
Log.i("zhengxing", "AlgUtil::AlgUtil");
mAlgCB = algCallBack;
mAlgThreads = 1;
mQueue = new ArrayBlockingQueue(3);
mAlg = new Alg();
mAlg.Init(assertManager);
mThreadFlag = true;
mThread = new Thread(this);
mThread.start();
}
//将预览图片投递到队列中(由于算法处理可能会比较慢,并不是每一帧图片都做算法处理)
public boolean addDataToQueue(byte [] bytes, int width, int height) {
Log.i("zhengxing", "AlgUtil::addDataToQueue");
Bitmap bmp = null;
try {
YuvImage image = new YuvImage(bytes, ImageFormat.NV21, width, height, null);
if (image != null) {
ByteArrayOutputStream stream = new ByteArrayOutputStream();
image.compressToJpeg(new Rect(0, 0, width, height), 100, stream);
bmp = BitmapFactory.decodeByteArray(stream.toByteArray(), 0, stream.size());
stream.close();
}
} catch (Exception ex) {
}
Bitmap rgba = bmp.copy(Bitmap.Config.ARGB_8888, true);
Bitmap imgSelect = Bitmap.createScaledBitmap(rgba, 312, 312, false);
rgba.recycle();
return mQueue.offer(imgSelect);
}
//返回算法推理结果给上层(Activity)
public void setCallBack(AlgCallBack callBack) {
Log.i("zhengxing", "AlgUtil::setCallBack");
this.mAlgCB = callBack;
}
//线程体(所有算法推理都在线程中执行)
@Override
public void run() {
Log.i("zhengxing", "AlgUtil::run");
while (mThreadFlag) {
try {
Bitmap bmp = (Bitmap) mQueue.poll(1000, TimeUnit.MILLISECONDS);
if (bmp != null) {
float[] x = mAlg.Run(bmp);
this.mAlgCB.onAlgRet(x);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
4、总结
这次仰卧起坐APP开发前后经历了大约一个月的时间,都是在晚上或者周末进行的。虽然很累,但是我也学到了很多东西。从原来完全不了解姿态估计这个领域,以及严重低估关键点检测的难度,到后来慢慢学习,整理相关论文,解决一个又一个难点,对我自己来说是一次极大的提升,也是对自己兴趣爱好的执着。如果整个过程也让你也有一点点的收获,那将是对我极大的鼓励。【获取APP源码请留言或者添加我的微信,15158106211,备注“仰卧起坐APP”,让我们一起学习一起进步。】