Skip to content

Commit

Permalink
update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
gordinmitya committed Aug 1, 2020
1 parent 2ecaab9 commit eea391e
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 4 deletions.
37 changes: 33 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,44 @@ When you're attempting to get an `Image` from `ImageReader` or `ImageAnalysis.An

**Solution**:

This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (`NV21` or `YUV_420`) and `ByteBuffer` to pass into OpenCV or a neural network framework.
This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (`NV21` or `I420`) and `ByteBuffer` to pass into OpenCV or a neural network framework.

The whole library is a single file, you can just copy [Yuv.java](yuv2buf/src/main/java/ru/gordinmitya/yuv2buf/Yuv.java) into your project.

**Usage**

```kotlin
private var reuseBuffer: ByteBuffer? = null

fun convert(image: ImageProxy): Pair<Bitmap, Long> {

val converted = Yuv.toBuffer(image)
// OR pass existing DirectBuffer for reuse
val converted = Yuv.toBuffer(image, reuseBuffer)
reuseBuffer = converted.buffer

val format = when (converted.type) {
Yuv.Type.YUV_I420 -> Imgproc.COLOR_YUV2RGBA_I420
Yuv.Type.YUV_NV21 -> Imgproc.COLOR_YUV2RGBA_NV21
}

// process with one of converters
}
```

**Converters**

1. [OpenCVConverter.kt](app/src/main/java/ru/gordinmitya/yuv2buf_demo/OpenCVConverter.kt) - the fastest. If your goal is to get Mat consider this method from [OpenCV](https://github.com/opencv/opencv/blob/master/modules/java/generator/android-21/java/org/opencv/android/JavaCamera2View.java#L344).
2. [RenderScriptConverter.kt](app/src/main/java/ru/gordinmitya/yuv2buf_demo/RenderScriptConverter.kt) - built-in, no additional libraries required.
3. [MNNConverter.kt](app/src/main/java/ru/gordinmitya/yuv2buf_demo/MNNConverter.kt) - if your goal is futher processing with neural network.

**Alternatives**

1. (For OpenCV users) Copy private method from OpenCV camera implementation: [JavaCamera2View, Mat rgba()](https://github.com/opencv/opencv/blob/master/modules/java/generator/android-21/java/org/opencv/android/JavaCamera2View.java#L344).
2. Capture image from camera directly to [RenderScript Allocation.getSurface()](https://developer.android.com/reference/android/renderscript/Allocation#getSurface());
3. Switch back to CameraApi 1 [(some trade offs)](https://github.com/tensorflow/tensorflow/issues/22620);
4. Manipulate pixels manually as it has done in TFLite demo [ImageUtils](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/env/ImageUtils.java#L161).
3. RenderScript implementation in [android/camera-samples](https://github.com/android/camera-samples/blob/3730442b49189f76a1083a98f3acf3f5f09222a3/CameraUtils/lib/src/main/java/com/example/android/camera/utils/YuvToRgbConverter.kt) repo.
4. Switch back to CameraApi 1 [(some trade offs)](https://github.com/tensorflow/tensorflow/issues/22620);
5. Manipulate pixels manually as it has done in TFLite demo [ImageUtils](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/env/ImageUtils.java#L161).
However, even with [C++ implementation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/jni/yuv2rgb.cc#L61)
it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;

Expand All @@ -28,6 +56,7 @@ The whole library is a single file, you can just copy [Yuv.java](yuv2buf/src/mai
- [x] write unit tests;
- [x] add RenderScript example;
- [x] add OpenCV example;
- [x] add MNN example;
- [ ] publish to GooglePlay;
- [ ] publish to jcenter;
- [ ] add MNN example;
- [ ] add TFLite example.
2 changes: 2 additions & 0 deletions app/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ android {
compileSdkVersion 29
buildToolsVersion "29.0.3"

ndkVersion "21.3.6528147"

defaultConfig {
applicationId "ru.gordinmitya.yuv2buf_demo"
minSdkVersion 21
Expand Down

0 comments on commit eea391e

Please sign in to comment.