Multithreading with worklets

[karol-bisztyga]

Description

This idea has been postponed due to low priority.
The solution here is dirty, it's more like a POC for the future if we decide to focus on this topic again.
The goal is to be able to run worklets on multiple threads different than JS and UI.

Use cases

• downloading data from the Internet
• processing video/audio/images
• obtaining a contact list from the device
• in fact, processing any data in the background in order to avoid blocking UI or JS.

Examples

example code

import React, { useState } from 'react';
import {
  View,
  Button,
  Text,
} from 'react-native';
import {
  useSharedValue,
  spawnThread,
} from 'react-native-reanimated';


const Child = () => {
  const sv = useSharedValue(0)

  spawnThread(() => {
    'worklet';
    sv.value = 1.1;
    console.log('here thread #1 begin', sv.value, _WORKLET);
    for (let i = 0; i < 3000000000; ++i) {}
    sv.value = 1.2;
    console.log('here thread #1 end', sv.value, _WORKLET);
  });

  spawnThread(() => {
    'worklet';
    sv.value = 2.1;
    console.log('here thread #2 begin', sv.value, _WORKLET);
    for (let i = 0; i < 3000000000; ++i) {}
    sv.value = 2.2;
    console.log('here thread #2 end', sv.value, _WORKLET);
  });

  return (
    <View>
      <Text>Child</Text>
      <Button title="check responsiveness" onPress={() => {
        console.log('UI responsive', sv.value, Math.random())
      }} />
    </View>
  );
}

export default function App() {
  const [s, setS] = useState(0);

  return (<View>
    <Button title="change state" onPress={()=>{ setS(s+1) }} />

    { s%2 === 1 ? <Child /> : <Text>EMPTY</Text>}
  </View>)
}

[mrousavy]

Hi @karol-bisztyga! Thanks for taking time to look into this! I'm currently writing a Camera TurboModule, and I'm planning on creating an API where the user can process frames using JS.

So for example:

const processFrame = useWorkletCallback((frame: CameraFrame) => {
  // process frame data in this worklet, for example to detect faces, render a square ontop of a QR code, or whatever.
}, []);


return <Camera {...props} processFrame={processFrame} />;

The benefits of this are hopefully immediately clear:

1. The user has full control over the video frames of the Camera and doesn't have to go native for stuff like object detection.
2. The processFrame function does not block the UI or JS thread.
3. There is no bridge-spam.

I'm currently halfway through implementing the Camera base stuff (format selection, capturing videos, capturing photo, ...), but I'm already excited to have something working in the near future.

Just leaving this here to make it clear that multithreaded worklets do indeed have many use cases, and perhaps we can work together to extend the proof of concept to a fully functional use case with the Camera process frame API 👍

I'll be posting status updates on my Twitter, #1, #2

[mrousavy]

To those interested: I've published the react-native-vision-camera library (🎉) and am now working on the Frame Processor thing. All progress is public at: mrousavy/react-native-vision-camera#2

EDIT

I've finished the Frame Processors API for iOS and Android!! 🎉🎉🎉 - This is an API where you can run multithreaded JS code on every camera frame in realtime (endless possibilities for video frame processing such as Face filters, QR code scanners, WebRTC and more). Take a look at the docs: VisionCamera - Frame Processors

Also, I created a library for running JS code on a separate thread, basically exactly what this PR attempted to do (Multithreading with Worklets for running heavy code in there in parallel) - Check it out: mrousavy/react-native-multithreading

[likern]

It's exactly what I'm looking for and dreamed of. It allows not touching native as much as possible.

I need this for video recorder app.
And it would be cool to use another thread to work with sqlite database, not polluting main and UI thread (where reanimated animations might run).

While I can't help, eagerly waiting for this feature.