Motivation:
When you're attempting to get an Image
from ImageReader
or ImageAnalysis.Analyzer
you actually get 3 separate ByteBuffers
which you can't pass to further processing. You have to merge them but that also is not easy because they are full of row and pixel strides.
Solution:
This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (NV21
or I420
) and ByteBuffer
to pass into OpenCV or a neural network framework.
The whole library is a single file, you can just copy Yuv.java into your project.
Usage
private var reuseBuffer: ByteBuffer? = null
fun convert(image: ImageProxy): Pair<Bitmap, Long> {
val converted = Yuv.toBuffer(image)
// OR pass existing DirectBuffer for reuse
val converted = Yuv.toBuffer(image, reuseBuffer)
reuseBuffer = converted.buffer
val format = when (converted.type) {
Yuv.Type.YUV_I420 -> Imgproc.COLOR_YUV2RGBA_I420
Yuv.Type.YUV_NV21 -> Imgproc.COLOR_YUV2RGBA_NV21
}
// process with one of converters
}
Converters
- OpenCVConverter.kt - the fastest. If your goal is to get Mat you may consider this method from OpenCV.
- RenderScriptConverter.kt - built-in, no additional libraries required.
- MNNConverter.kt - if your goal is futher processing with neural network.
Benchmark
Snapdragon 855 (Xiaomi Mi 9T Pro). Image resolution 480x640.
MNN | OpenCV | RenderScript |
---|---|---|
~7ms | ~1ms | ~2ms |
Alternatives
- (For OpenCV users) Copy private method from OpenCV camera implementation: JavaCamera2View, Mat rgba().
- Capture image from camera directly to RenderScript Allocation.getSurface();
- RenderScript implementation in android/camera-samples repo.
- Switch back to CameraApi 1 (some trade offs);
- Manipulate pixels manually as it has done in TFLite demo ImageUtils. However, even with C++ implementation it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;
- make the library;
- write unit tests;
- add RenderScript example;
- add OpenCV example;
- add MNN example;
- publish to GooglePlay;
- publish to jcenter;
- add TFLite example.