Note: port forwarding is adapted from https://github.com/vsoch/forward
- Clone this repository locally. Navigate to a location on your local machine where you want to clone the repository. Use the command
git clone https://github.com/juliaschaepe/wellception.git
. - Navigate to the
params_deepcell.sh
file in the forward folder. Edit the first line of the file to include your suidFORWARD_USERNAME="jschaepe"
. Next, pick a port to use. If someone else is port forwarding using that port already, this script will not work. If you pick a random number in the range 49152-65335, you should be good. Editparams_deepcell.sh
to include your chosen portPORT="59339"
. - Copy the
organoid_processor.ipynb
to your sherlock account with scp using the following commandscp organoid_processor.ipynb [email protected]:~/<location_for_file>
, replacing jschaepe with your suid. You will also need a fileorganoid_resnet50.h5
in order to run the organoid-trained deepcell model. Contact me to have it sent since it is a large file (303.2 MB). Be sure that it is fully downloaded by checking the file size before using scp to copy it up to Sherlock. - You will also need to at the minimum configure your ssh to recognize sherlock as a valid host. There is a script that will generate recommended ssh configuration snippets to put in your
~/.ssh/config
file. Here is how you can generate this configuration for Sherlock:
bash hosts/sherlock_ssh.sh
Host sherlock
User put_your_username_here
Hostname sh-ln01.stanford.edu
GSSAPIDelegateCredentials yes
GSSAPIAuthentication yes
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/%l%r@%h:%p
If you don't have a file in the location ~/.ssh/config
then you can generate it programatically, but do not run this command if there is content in the file that you might overwrite!
bash hosts/sherlock_ssh.sh >> ~/.ssh/config
- Login to your sherlock account. Load python3.6 by running the command
ml python/3.6.1
. Pip install deepcell by runningpip install --user deepcell
. - If you have not set up notebook authentification before, you will need to set a password via
jupyter notebook password
on your sherlock account. Make sure to pick a secure password! If you run into an error of invalid password, navigate to~/.jupyter
on your sherlock account and look at thejupyter_notebook_config.json
file. If your hashed password starts withargon2
, you will need to rehash this password yourself by opening an ipython session in Sherlock by withipython
and then using the following commands:
from notebook.auth import passwd
password=‘your_password’
passwd(password, ‘sha1’)
Copy the output of this last command (e.g. sha1:eb13d589486a:b852385152418fc49e9f2089262d402ce02d1bfe
). Open jupyter_notebook_config.json
, delete only what is in the double parentheses to look like this "password": ""
and then paste your output so it looks like this "password": "sha1:eb13d589486a:b852385152418fc49e9f2089262d402ce02d1bfe"
. Be sure not to edit anything outside of the double parentheses or add any spaces.
- Navigate to the forward folder. Run the command
bash start_deepcell.sh py3-tensorflow
. Enter your sherlock password when prompted. This requests GPU resources on sherlock, runs a tensorflow container, and starts a jupyter notebook which is then port forwarded to your local machine. You can follow the instructions on your local terminal and copy the port link, for examplehttps://localhost:59338/
, and paste it in your browser. This now interfaces like a typical jupyter notebook. - Open
organoid_processor.ipynb
. Run the first cell that loads libraries. If there are any errors loading libraries, you can pip install that library in sherlock like in Setup part 4. You should also see a print output indicating that tensorflow registered a GPU, for example[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
from the first cell.
- If you need to change the parameters of your sherlock job, for example how long you need the GPU for, you can edit the
params_deepcell.sh
file to reflect that. For processing an entire experiment, you will likely need at least a day24:00:00
. The maximum time that you can request on Sherlock is 2 days,48:00:00
.