Motivation:

When you're attempting to get an Image from ImageReader or ImageAnalysis.Analyzer you actually get 3 separate ByteBuffers which you can't pass to further processing. You have to merge them but that also is not easy because they are full of row and pixel strides.

Solution:

This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (NV21 or I420) and ByteBuffer to pass into OpenCV or a neural network framework.

The whole library is a single file, you can just copy Yuv.java into your project.

Usage

private var reuseBuffer: ByteBuffer? = null

fun convert(image: ImageProxy): Pair<Bitmap, Long> {

    val converted = Yuv.toBuffer(image)
    // OR pass existing DirectBuffer for reuse
    val converted = Yuv.toBuffer(image, reuseBuffer)
    reuseBuffer = converted.buffer

    val format = when (converted.type) {
        Yuv.Type.YUV_I420 -> Imgproc.COLOR_YUV2RGBA_I420
        Yuv.Type.YUV_NV21 -> Imgproc.COLOR_YUV2RGBA_NV21
    }

    // process with one of converters
}

Converters

1. OpenCVRoteterter.kt - preform rotation on yuv image and then converts color. The fastest in total time.
2. OpenCVConverter.kt - the fastest color conversion without rotation. Rotation of a rgb image is slightly harder.
3. RenderScriptConverter.kt - built-in, no additional libraries required. Uses Bitmap for rotation.
4. MNNConverter.kt - if your goal is futher processing with neural network. Conversion and rotation performed in single operation.

PS: If your goal is to get Mat you may consider this method from OpenCV.

Benchmark

Snapdragon 855 (Xiaomi Mi 9T Pro). Image resolution 480x640.

Alternatives

1. (For OpenCV users) Copy private method from OpenCV camera implementation: JavaCamera2View, Mat rgba().
2. Capture image from camera directly to RenderScript Allocation.getSurface();
3. RenderScript implementation in android/camera-samples repo.
4. Switch back to CameraApi 1 (some trade offs);
5. Manipulate pixels manually as it has done in TFLite demo ImageUtils. However, even with C++ implementation it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;

• make the library;
• write unit tests;
• add RenderScript example;
• add OpenCV example;
• add MNN example;
• publish to GooglePlay;
• publish to jcenter;
• add TFLite example.