-
Notifications
You must be signed in to change notification settings - Fork 900
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(Adding U-Net API): #294
Conversation
…8 to segment faces and shoulder
This is amazing, @zaidalyafeai! |
Woohoo! Noting that rawgit is deprecated but you can serve files via https://www.jsdelivr.com/rawgit! I think hosting in data-and-models makes sense for now. (We can also provide it locally with the examples?) |
@yining1023, seems OK by me. I will wait until the PR is revised, maybe we will have second opinions on how to approach that. |
Hi @zaidalyafeai, thanks so much for this! We'd love to get this merged this week! (We are planning a release this Friday if all goes well.) I think everything here looks great, I might suggest the following steps before merge:
@joeyklee @yining1023 Am I missing anything else important? |
Hi @zaidalyafeai ! I hope you're well! I've taken a moment to look into your PR and things are working! Hooray! However there's a few things that I wanted to flag before merging. Merge conflictI've updated the index.js of ml5 to fix the existing merge conflict. Since we've gotten some PRs for other functionality, there were some isses, but this is resolved by the following: // Copyright (c) 2018 ml5
//
// This software is released under the MIT License.
// https://opensource.org/licenses/MIT
import * as tf from '@tensorflow/tfjs';
import pitchDetection from './PitchDetection/';
import imageClassifier from './ImageClassifier/';
import KNNClassifier from './KNNClassifier/';
import featureExtractor from './FeatureExtractor/';
import word2vec from './Word2vec/';
import YOLO from './YOLO';
import poseNet from './PoseNet';
import * as imageUtils from './utils/imageUtilities';
import styleTransfer from './StyleTransfer/';
import charRNN from './CharRNN/';
import pix2pix from './Pix2pix/';
import SketchRNN from './SketchRNN';
import preloadRegister from './utils/p5PreloadHelper';
import { version } from '../package.json';
import uNet from './UNET';
const withPreload = {
imageClassifier,
};
module.exports = Object.assign({}, preloadRegister(withPreload), {
uNet,
KNNClassifier,
featureExtractor,
pitchDetection,
YOLO,
word2vec,
styleTransfer,
poseNet,
charRNN,
pix2pix,
SketchRNN,
...imageUtils,
tf,
version,
}); uNet example - callback concerns + base64 image is returnedI made an example -- https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js#L25 -- that uses your current implementation of uNet and as you can see in the screenshot, uNet returns an image DOM elemement with the src as a base64 image. In the context of p5.js we have to use the video loading issueCurrently I am running see: https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js#L37 Thanks! Looking forward to hearing your thoughts cc/ @shiffman @yining1023 |
Oh, this is so exciting! 🎉 I'm looking at @joeyklee's example and I agree (as related to #302) that figuring out how to get the data easily into a function gotResult(error, result) {
loadImage(result.src, anotherCallback);
} The need for a second callback is awkward and will make this unwieldy for beginners. I probably need to dig into function gotResult(error, result) {
let img = loadImage(result.image);
image(img, 0, 0);
} But maybe we have to do the loading behind the scenes before triggering the callback: function gotResult(error, result) {
let img = result.image;
image(img, 0, 0);
} We could check for the existence of p5 and if so wrap the data into a {
"image": p5.Image,
"blob": Blob,
"bytes": Array
} If p5 is not present we could skip the |
@zaidalyafeai This is what I was doing for adding p5.Image support |
@yining1023 @WenheLI @viztopia Is everyone happy with the proposed design as discussed here and in #312? Thumbs up this comment or let us know otherwise and we'll move forward! |
@zaidalyafeai let's go ahead and move forward with this plan! |
# Conflicts: # dist/ml5.min.js # dist/ml5.min.js.map # src/UNET/index.js # src/index.js
I might have missed something up :/ I am not familiar how to resolve conflicts. |
Hi @zaidalyafeai - apologies for the delays. I will take some time this week to look into all this again! I will reach out when we're ready to merge! Very excited for this! Thank you! |
@zaidalyafeai - I've tested your latest version and it's brilliant! I've made a matching example - https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js - and it works like a charm. I will make a PR to fix 2 things:
Just noting one thought: There's room for maybe some optimization perhaps in the example - https://github.com/ml5js/ml5-examples/blob/UNET/p5js/UNET/UNET_webcam/sketch.js - it runs best at a lower frame rate. At higher frame rates, uNet seems to hang a bit. But this is not a big issue. Thanks so much! Also - will merge this into development |
A U-Net model is added with input size 128x128. I hosted the model in my github account.