feat(Adding U-Net API):

[zaidalyafeai]

A U-Net model is added with input size 128x128. I hosted the model in my github account.

[yining1023]

This is amazing, @zaidalyafeai!
We are hosting some models like Draknet on ml5-data-and-models repo. Would you like to submit a PR to ml5-data-and-models repo to add a 'U-Net' folder?

[shiffman]

Woohoo! Noting that rawgit is deprecated but you can serve files via https://www.jsdelivr.com/rawgit!

I think hosting in data-and-models makes sense for now. (We can also provide it locally with the examples?)

[zaidalyafeai]

@yining1023, seems OK by me. I will wait until the PR is revised, maybe we will have second opinions on how to approach that.

[shiffman]

Hi @zaidalyafeai, thanks so much for this! We'd love to get this merged this week! (We are planning a release this Friday if all goes well.) I think everything here looks great, I might suggest the following steps before merge:

1. Move model files to ml5-data-and-models and update with jsdelivr link.
2. Return an object in result with the masked image in a variable called. . .image?
3. Open a corresponding pull request with a "hello world" style simple example in ml5-examples.

@joeyklee @yining1023 Am I missing anything else important?

[joeyklee]

A U-Net model is added with input size 128x128. I hosted the model in my github account.

Hi @zaidalyafeai ! I hope you're well! I've taken a moment to look into your PR and things are working! Hooray! However there's a few things that I wanted to flag before merging.

Merge conflict

I've updated the index.js of ml5 to fix the existing merge conflict. Since we've gotten some PRs for other functionality, there were some isses, but this is resolved by the following:

// Copyright (c) 2018 ml5
//
// This software is released under the MIT License.
// https://opensource.org/licenses/MIT

import * as tf from '@tensorflow/tfjs';
import pitchDetection from './PitchDetection/';
import imageClassifier from './ImageClassifier/';
import KNNClassifier from './KNNClassifier/';
import featureExtractor from './FeatureExtractor/';
import word2vec from './Word2vec/';
import YOLO from './YOLO';
import poseNet from './PoseNet';
import * as imageUtils from './utils/imageUtilities';
import styleTransfer from './StyleTransfer/';
import charRNN from './CharRNN/';
import pix2pix from './Pix2pix/';
import SketchRNN from './SketchRNN';
import preloadRegister from './utils/p5PreloadHelper';
import { version } from '../package.json';
import uNet from './UNET';

const withPreload = {
  imageClassifier,
};

module.exports = Object.assign({}, preloadRegister(withPreload), {
  uNet,
  KNNClassifier,
  featureExtractor,
  pitchDetection,
  YOLO,
  word2vec,
  styleTransfer,
  poseNet,
  charRNN,
  pix2pix,
  SketchRNN,
  ...imageUtils,
  tf,
  version,
});

uNet example - callback concerns + base64 image is returned

I made an example -- https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js#L25 -- that uses your current implementation of uNet and as you can see in the screenshot, uNet returns an image DOM elemement with the src as a base64 image.

In the context of p5.js we have to use the loadImage() function to convert the base64 image reference to a p5.Image so we can render it on the canvas -- this isn't so bad, but I wonder if maybe returning a json object as in the discussion here -- #312 -- might add consistency to our ml5js API and reduce the step for an additional callback here.

video loading issue

Currently I am running segment() on the video in the modelReady() callback function, but I know it is possible to load the video directly when loading the model. Can you maybe show how this works by chance?

see: https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js#L37

Thanks! Looking forward to hearing your thoughts cc/ @shiffman @yining1023

[shiffman]

Oh, this is so exciting! 🎉

I'm looking at @joeyklee's example and I agree (as related to #302) that figuring out how to get the data easily into a p5.Image feels like the key to all of this. Right now we have:

function gotResult(error, result) {
  loadImage(result.src, anotherCallback);
}

The need for a second callback is awkward and will make this unwieldy for beginners. I probably need to dig into p5.Image more, but I wonder if there is a synchronous way to create an image behind the scenes and just point to it. This would be ideal:

function gotResult(error, result) {
  let img = loadImage(result.image);
  image(img, 0, 0);
}

But maybe we have to do the loading behind the scenes before triggering the callback:

function gotResult(error, result) {
  let img = result.image;
  image(img, 0, 0);
}

We could check for the existence of p5 and if so wrap the data into a p5.Image, otherwise the data could be in a variable called blob (for image blob) or bytes for Array of raw data, i.e.:

{
  "image": p5.Image,
  "blob": Blob,
  "bytes": Array
}

If p5 is not present we could skip the image property?

[zaidalyafeai]

@shiffman @joeyklee I am quite lost (not quite familiar with p5js) . Can you tell me the next step ?

[WenheLI]

@zaidalyafeai This is what I was doing for adding p5.Image support
WenheLI@9dc2f8d
Hope that helps you!

[zaidalyafeai]

@WenheLI thanks! @shiffman I followed your suggestion. Let me know if this is the final design and I will commit the changes.

return {
      blob, //blob 
      tensor, //predicted tensor
      raw, //array of bytes
      image, //p5 image
    };

[shiffman]

@yining1023 @WenheLI @viztopia Is everyone happy with the proposed design as discussed here and in #312? Thumbs up this comment or let us know otherwise and we'll move forward!

[shiffman]

@zaidalyafeai let's go ahead and move forward with this plan!

[zaidalyafeai]

I might have missed something up :/ I am not familiar how to resolve conflicts.

[joeyklee]

Hi @zaidalyafeai - apologies for the delays. I will take some time this week to look into all this again! I will reach out when we're ready to merge! Very excited for this! Thank you!

[joeyklee]

@zaidalyafeai - I've tested your latest version and it's brilliant! I've made a matching example - https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js - and it works like a charm.

I will make a PR to fix 2 things:

1. enable preload for uNet
2. since p5 is attached to the window instead of p5.window.loadImage() it will be just window.loadImage().

Just noting one thought: There's room for maybe some optimization perhaps in the example - https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js - it runs best at a lower frame rate. At higher frame rates, uNet seems to hang a bit. But this is not a big issue.

Thanks so much!

Also - will merge this into development