Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading the Model to my drive FAILED #4

Open
pumpkinband opened this issue Jun 27, 2019 · 38 comments
Open

Downloading the Model to my drive FAILED #4

pumpkinband opened this issue Jun 27, 2019 · 38 comments

Comments

@pumpkinband
Copy link

pumpkinband commented Jun 27, 2019

`!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

model.save('model.h5')
model_file = drive.CreateFile({'title' : 'model.h5'})
model_file.SetContentFile('model.h5')
model_file.Upload()

drive.CreateFile({'id': model_file.get('id')})`

Used this code to Download the model but Failed

ERROR :-

NameError: name 'model' is not defined

@Santosh-Gupta
Copy link
Member

It looks like you used the train your own models notebook, and now you're trying to save the model you trained? Or is it something else?

@pumpkinband
Copy link
Author

yes sir i want to download model.h5 , can you help me please.
@Santosh-Gupta

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

What are you trying to download? The model weights trained by yourself or the weights that we provide?

@pumpkinband
Copy link
Author

pumpkinband commented Jun 27, 2019

Model weight i trained i guess. (i meant running the CELLS by my self) please it would be a great help @JayYip

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

May I know which notebook are you trying to run?

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

Seems I cannot open the link you provide. That might be a copy of the original colab notebook and I cannot access it. Which notebook in this folder?

@pumpkinband
Copy link
Author

DocProductPresentationV6-0.2.0.ipynb

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

Sorry, but I can't find the corresponding code in the notebook.

BTW, can you download the model files from OneDrive? Seems it's broken again.

@pumpkinband
Copy link
Author

Sure sir but how can i download model.h5 of your model from OneDrive, can you please direct me.

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

I'm not sure what is model.h5. As for the model weights that we provide, you can download here.

@pumpkinband
Copy link
Author

ok but how to download the model weight can you please help me step by step

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

The zip file that I linked above contains the model weights. That is the same file as DataAndCheckpoint.zip in DocProductPresentationV6-0.2.0.ipynb notebook.

First, you download the zip file. Then you need to upload the file you downloaded to colab and rename it to DataAndCheckpoint.zip and execute the notebook.

I am still confused though, what is the model in your original post? Where do you get that chunk of code?

@harshit-saraswat
Copy link

Hello sir, actually I had an issue with the model. I've downloaded the weights and embedding file from your drive and when i tried to run it on my machine(windows os) i got an error in predictor.py file as it imports faiss which isn't available on windows yet. So, I was thinking is there a way to run the model on windows? Also I'm trying to deploy the model on an android based application and the video Siraj posted on his channel doesn't actually covers it neither his repository. Mind shedding in some light?

@pumpkinband
Copy link
Author

same issue as @harshit-saraswat

@JayYip
Copy link
Collaborator

JayYip commented Jun 27, 2019

So, I was thinking is there a way to run the model on windows?

Please take a look at this issue. Seems they're working on a Windows version.

Also I'm trying to deploy the model on an android based application and the video Siraj posted on his channel doesn't actually covers it neither his repository

I don't have many experience of developing apps but as far as I can tell, retrieval based model is not very suitable to put the whole model to mobile device because the embedding vector size is huge. I would suggest you to serve the model on a server.

BTW, if you guys have experience developing apps, maybe you can work with us! If you're interested please contact me via email: [email protected] and let us get to know you and your skillset.

@harshit-saraswat
Copy link

@JayYip there's the problem right, for the model to be used as a server I do need it to be run on my machine as for as far as I know, colab doesn't allow webhooks or post requests from other consoles/webservices such as dialogflow. TBH integrating it with Dialogflow as a chatbot would really be a great deal. I'm really trying on howto send my inputted query string to your model online as an server and get the ouput as result. Any ideas about how can I use this functionality? It needn't be necessary to be used as a dialogflow integration/ backend. I just need some guidance about using the model as a server and sending request and retrieving the output back, since as of now colab doesn't serve that type of services/purposes. Thanks!

@pumpkinband
Copy link
Author

on that topic i want to convert it into .tflite can you help me with that @harshit-saraswat

@harshit-saraswat
Copy link

@pumpkinband Follow the following steps:

  1. Convert the model checkpoints which you've downloaded from the one drive link given by the creators here, to a h5/pb file first. These links will help- https://www.tensorflow.org/tutorials/keras/save_and_restore_models,https://stackoverflow.com/questions/44466066/how-can-i-convert-a-trained-tensorflow-model-to-keras, https://blog.metaflow.fr/tensorflow-how-to-freeze-a-model-and-serve-it-with-a-python-api-d4f3596b3adc.
  2. Now use this link to convert the saved graph instances of the model to a tflite file. https://www.tensorflow.org/lite/convert/python_api

Hope it helped.

@pumpkinband
Copy link
Author

thanks alot @harshit-saraswat and sir @JayYip thank you so much

@Santosh-Gupta
Copy link
Member

@jay 'model.h5' is the name of the file, what that person is doing trying to use the keras function model.save('model.h5') to save the weights into Keras's h.5 format.

So the issue is that either Keras is not saving the model correctly, or that Pydrive is not uploading it correctly.

@pumpkinband did you get your issue fixed?

@harshit-saraswat
Copy link

Anyone able to run the model?or deploy it on android yet?

@JayYip
Copy link
Collaborator

JayYip commented Jun 28, 2019

what that person is doing trying to use the keras function model.save('model.h5') to save the weights into Keras's h.5 format

Yeah. I know. I'm just confused what's the value of that file since I cannot find that chunk of code in the notebook. And he did not provide any info about model object.

Anyone able to run the model?

You should be able to run the model like we did in the notebooks once you downloaded the zip file and get the environment right.

@pumpkinband
Copy link
Author

Screenshot (4)
so can anyone tell me the name of model it is asking, i just want to download the Model in keras (h.5) ??

@harshit-saraswat
Copy link

@pumpkinband can you send a link to your code?

@pumpkinband
Copy link
Author

can you tell me where did i go wrong @harshit-saraswat

@harshit-saraswat
Copy link

@pumpkinband allow access for the link please. maybe change the folders sharing to allow everyone to view.

@pumpkinband
Copy link
Author

@JayYip
Copy link
Collaborator

JayYip commented Jun 28, 2019

@pumpkinband Your code doesn't make much sense to me. What do you want to download to your drive exactly? The model object just came out of nowhere and thus it yields NameError. If you want the model weight, it's in the path newFolder/models/bertffn_crossentropy/bertffn.

@harshit-saraswat
Copy link

harshit-saraswat commented Jun 28, 2019

@pumpkinband you don't need to convert the model to a json or h5 format because the model weight and graph are stored in the BioBert folder and also the new folder as .ckpt/.chkp and .meta file along with some .json file. Now use those files and convert them into a .pb and .pbtxt format and then convert it into a .tflite format.
This link clearly explains how you can convert the files into a .pb and .pbtxt file. To convert those into .tflite you can use

`import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)`

@pumpkinband
Copy link
Author

pumpkinband commented Jun 29, 2019

I am trying to convert it into .pb but I just need one thing, what is output_node_names of this project ?? @harshit-saraswat

@Santosh-Gupta
Copy link
Member

I need to brush up machine learning model storage types. What is the difference between how it's currently stored and .pb? Also, what are 'output_node_names '. I'm not familiar with these, but if you can give more details I may be able to figure out what parts of DocProduct these belong in.

@pumpkinband
Copy link
Author

pumpkinband commented Jun 29, 2019

3 different ways to find output_node_names, but I can’t do any of this :

1, If you still have your python file describing the model, you can always add an my_var = tf.indentity(my_var, name="my_new_name") and then load your previous model and save it again with this new added node

2, Also note that the graph_def is holding nodes in order (per connected graph), so output nodes should be close to the end of the array.

3, Finally, you can always use tensorboard to load an existing graph and just look at the name of your “output” nodes.
@Santosh-Gupta

@pumpkinband
Copy link
Author

can anyone please upload graph.pbtxt to tensorboard and tell me the "output_node_names" ??

@Santosh-Gupta
Copy link
Member

What are 'output_node_names '? I'm not sure what this is. Is it the name of the output variables in the Tensorflow graph?

"3 different ways to find output_node_names, but I can’t do any of this :"

What issues have you run into doing these?

@JayYip
Copy link
Collaborator

JayYip commented Jul 3, 2019

Please take a look at this issue. #6

@Chrome3912
Copy link

try the api: https://pocket-derm.firebaseapp.com/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants