Hardware Limitations and Scaling #147

evancofer · 2016-11-19T00:46:52Z

I have written up my initial draft of the subsection on hardware limitations and scaling. As is noted in the outline, this might end up in categorize section rather than the discussion section. It may be worth noting that I could not find DOIs for the NIPS proceedings papers (of which there were several).

agitter

Thanks a lot for the contribution. I left scattered thoughts throughout, not all of which warrant changes. I like the overall direction but wonder if it gets too technical when discussing FPGAs. I haven't read many of these papers so I haven't reviewed them or how the citations support the discussion here.

I'll leave this open for more comments.

agitter · 2016-11-20T21:16:46Z

references/tags.tsv

+Caruana2014_need arXiv:1312.6184
+Chen2015_hashing arXiv:1504.04788
+Chen2016_gene_expr doi:10.1093/bioinformatics/btw074
+Coates2013_cots_hpc http:https://www.jmlr.org/proceedings/papers/v28/coates13.html


agitter · 2016-11-20T21:17:02Z

references/tags.tsv

+Hinton2015_dark_knowledge arXiv:1503.02531
+Hinton2015_dk arXiv:1503.02531v1
+Hubara2016_qnn arXiv:1609.07061
+Krizhevsky2013_nips_cnn https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf


agitter · 2016-11-20T21:19:11Z

sections/06_discussion.md

@@ -49,6 +49,86 @@ with only a couple GPUs.*
 *Some of this is also outlined in the Categorize section. We can decide where
 it best fits.*

+Efficiently scaling deep learning is challenging, and there is a high
+computational cost (e.g., time, memory, energy) associated with training neural
+networks and using them for classification. For these reasons, neural networks


Is it training them and using them or primarily training that is resource intensive?

Fair point. Though the accumulated run time cost may eclipse the design/train time (e.g., a popular and widely used tool), the latter is not distributed, and thus poses a more direct challenge for computational biologists.

agitter · 2016-11-20T21:23:08Z

sections/06_discussion.md

+networks and using them for classification. For these reasons, neural networks
+have only recently found widespread use [@tag:Schmidhuber2014_dnn_overview].
+For biologists, such problems are further complicated by the immense size of
+most biological datasets.


I would argue many of the datasets used in our biomedical examples are not that large, especially compared to some image or speech datasets. Should we highlight more specific examples of the problems where the size of the data on disk is a limiting factor (perhaps imaging, applications with (epi)genomic input data, etc.)

The disk space problem is an issue, but not the only one. I agree that its current placement might alienate discussion of other motivating issues (e.g., "wide" datasets). Perhaps I should leave this sentence out entirely?

I think it could be dropped; there are many other interesting points being made here already.

agitter · 2016-11-20T21:25:19Z

sections/06_discussion.md

+
+*TODO: Perhaps a visual illustrating the changes over time in the cost of
+storage
+(i.e. hard drive space), genome sequencing, and processing power. Additionally,


GPU cores and RAM would also be relevant here. I wouldn't want to have to compile that data ourselves, maybe a different deep learning review has recently? Wikipedia has it all but isn't a great source.

I agree, the GPU cores and RAM would definitely be good to have. I will see if I can find a review that includes such information.

I have already compiled a list of ImageNET classification task winners with accuracy, along with citations for the models where available. However, I am still unsure about how to quantify the increasing complexity of the winning networks, as there is clearly a distinction to be made between types of layers, and I don't know how useful parameter counts would be.

The data in [@tag:Stein2010_cloud] is missing recent years. This is not really a problem, and I can find more data regarding storage costs if that ends up being something we want to have.

I agree parameter counts don't tell the whole story. Some important advances worked well because they reduce parameter counts. If it is hard to capture the ImageNET history without going into a lot of detail, should we drop it?

agitter · 2016-11-20T21:29:09Z

sections/06_discussion.md

+
+Steady improvements in GPU hardware may alleviate this issue somewhat, but it
+is not clear whether it can occur quickly enough to keep up with the increasing
+amount of available biological data. Much has been done to minimize the memory


Amount of data and network size?

agitter · 2016-11-20T21:31:56Z

sections/06_discussion.md

+data (e.g., patient electronic health records) in the cloud,
+secure/regulation-compliant cloud services do exist [@tag:RAD2010_view_cc].
+
+*TODO: Write the transition once more of the Discussion section has been


The transition could include some commentary on how this relates to our guiding question. Will hardware issues slow deep learning from making progress on the problems we have discussed? Do requirements for specialized hardware (GPUs, FPGAs, etc.) or costs of using cloud resources create a barrier to entry that will slow progress because fewer groups can participate?

Conversely, if some of these hardware challenges are resolved do we expect an acceleration of biomedical results?

In regards to cloud computing, it think it enables broader use in the short term. This is largely due to the number of small grants out there for cloud credit (e.g., https://aws.amazon.com/grants/), which are relatively simple to apply for. Additionally, there seems to be significant support for it (https://datascience.nih.gov/commons). Perhaps a bigger issue would be the stability of such a model; it may be unwise to predicate the advancement of deep learning on accessibility to cloud computing.

I suspect that problems with interpretation, model sharing, and reproducibility are just as limiting as hardware. Fortunately, these should be things we can fix right now without significant hardware investment. Even without cloud resources, it should be easy for any lab to acquire a GPU (https://developer.nvidia.com/academic_gpu_seeding), and begin working on these issues immediately.

It would seem that solving hardware challenges would lead to an acceleration of research, at least based on what we've seen in the reviewed papers.

I agree with these thoughts. Would you like to write them into the text? One more comment is that the NVIDIA seeding program is great for getting a group started, but in our case (and I suspect others) it is addictive. Once you have methods working on one GPU, you quickly want to buy more or move to the cloud.

agitter · 2016-11-20T21:32:14Z

sections/06_discussion.md

+@tag:Gupta2015_prec @tag:Bengio015_prec @tag:Sa2015_buckwild
+@tag:Chen2015_hashing @tag:Hubara2016_qnn], but there is also growing
+interest in specialized hardware, such as field-programmable gate arrays
+(FPGAs) [@tag:Edwards2015_growing_pains Lacey2016_dl_fpga] and


Missing @tag

agitter · 2016-11-20T21:32:46Z

sections/06_discussion.md

+Distributed computing is a general solution to intense computational
+requirements, and has enabled many large-scale deep learning efforts. Early
+approaches to distributed computation were not suitable for deep learning
+[@tag:Dean2012_nips_downpour], but significant progress has been made. There


I haven't read this. Why is it not suitable?

Was citing the discussion provided in @tag:Dean2012_nips_downpour, rather than Downpour itself. Updated with clarification.

agitter · 2016-11-20T21:38:22Z

sections/06_discussion.md

+handling very large networks, distributed or parallelized approaches offer
+other advantages, such as improved ensembling [@tag:Sun2016_ensemble] or
+accelerated hyperparameter optimization [@tag:Bergstra2011_hyper
+@tag:Bergstra2012_random].


#104 provides a concrete example that could be referenced here:

In addition, we also expect to obtain better generalization by training a larger deep autoencoder with more data. The chemical structures of close to one hundred million chemical compounds are known, and could be used to train a
single unified embedding of known chemistry. Software packages that use multiple graphical processing units are being applied to this task

Their ability to learn a great feature representation was limited by the number of compounds they could train with on a single GPU. Or maybe they did use a few GPUs but need even more to scale.

I am hesitant to include this in the paragraph on distribution as it is unclear to me what hardware they are currently using, and they have not yet shown if a distributed/multi-GPU implementation will influence their performance. That being said, I think this would go well with the previous paragraph on GPUs, as the memory limitations seemed to be an issue for them.

Point taken, we don't need speculate on what they're doing regarding distributed GPUs.

Clarified discussion based on Issue #147 Added network size to discussion of limitations as in Issue #147 Fixed comma issue

cgreene · 2016-12-19T18:53:10Z

@evancofer : one limitation you may want to discuss is in #24. They explicitly had to restructure their approach (dividing genes arbitrarily in half) due to hardware limitations. I quickly scanned for the DOI (doi's turn out to be hard to scan for!) and didn't see it. Otherwise, I agree with @agitter's points.

agitter · 2016-12-20T12:38:55Z

sections/06_discussion.md

+which makes it difficult to train/use networks of significant size and
+complexity on a single GPU or machine [@tag:Raina2009_gpu
+@tag:Krizhevsky2013_nips_cnn]. This restriction has sometimes stymied the use
+of deep learning for computational biology[@tag:Chen2016_gene_expr] or


@cgreene #24 is included here but with the tag Chen2016_gene_expr, which is why you didn't see the DOI. Re-reading this line, perhaps we could be more specific about the limitation faced in this paper, i.e. the need to split the genes into two groups. Then the phrase ", though some have..." could become a new sentence "Some have...".

@evancofer the relevant quote about splitting genes is in #24 if you need it.

Ahh! I see that now. In that case, I might suggest discussing it in just a little bit more detail. Something like:

This restriction has sometimes forced computational biologists to use workarounds or to limit the size of an analysis. For example, Chen et al. [@tag:Chen2016_gene_expr] aimed to infer the expression levels of all genes, but due to memory restrictions they randomly partitioned genes into two halves and analyzed each separately. In other cases, researchers limited the size of their neural network [@tag:Wang2016_protein_contact]. Some have also chosen to use slower CPU implementations rather than sacrifice network size or performance [@tag:Yasushi2016_cgbvs_dnn].

Agreed. I have made the recommended changes.

Added detail to discussion of paper from issue #24 Added detail to the examples of GPU memory limitations. Removed TODO for visual, as it will not be included.

cgreene

LGTM 👍

cgreene · 2016-12-21T14:38:47Z

@agitter : can you take another look at this. If it looks ready to go I can resolve the conflicts and merge.

agitter · 2016-12-21T15:28:25Z

It's ready to merge, and I'll merge now. I resolved the conflicts in the tags file with the GitHub merge tool. I hadn't used it before, but it was a simple merge and I think everything worked okay.

cgreene · 2016-12-21T15:29:00Z

Woo! Go for it :)

On Wed, Dec 21, 2016 at 10:28 AM Anthony Gitter ***@***.***> wrote: It's ready to merge, and I'll merge now. I resolved the conflicts in the tags file with the GitHub merge tool. I hadn't used it before, but it was a simple merge and I think everything worked okay. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#147 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAhHswDY3J8liIaIZHDuCjYeTQJOaeBRks5rKUWZgaJpZM4K3Eho> .

-- Casey S. Greene, Ph.D. Assistant Professor Dept. of Systems Pharmacology and Translational Therapeutics Perelman School of Medicine University of Pennsylvania web: http:https://www.greenelab.com phone: 215-573-2991

…and-scaling Hardware Limitations and Scaling

evancofer added 2 commits November 18, 2016 18:32

Hardware limitations and scaling portion of the discussion

dc386d4

Fixed typos from document conversion process

a52403d

agitter requested changes Nov 20, 2016

View reviewed changes

Initial revisions based on issue #147

df714ce

Clarified discussion based on Issue #147 Added network size to discussion of limitations as in Issue #147 Fixed comma issue

agitter reviewed Dec 20, 2016

View reviewed changes

Second round of revisions based on issue #147

2711701

Added detail to discussion of paper from issue #24 Added detail to the examples of GPU memory limitations. Removed TODO for visual, as it will not be included.

cgreene approved these changes Dec 21, 2016

View reviewed changes

agitter approved these changes Dec 21, 2016

View reviewed changes

Merge branch 'master' into hardware-limitations-and-scaling

8e8fcb4

agitter merged commit 1b4cb77 into greenelab:master Dec 21, 2016

cgreene mentioned this pull request Dec 21, 2016

12/21 Status Report #165

Merged

cgreene mentioned this pull request Jan 7, 2017

Current Section Status #188

Closed

dhimmel pushed a commit to dhimmel/deep-review that referenced this pull request Nov 3, 2017

Merge pull request greenelab#147 from evancofer/hardware-limitations-…

13d1b14

…and-scaling Hardware Limitations and Scaling

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hardware Limitations and Scaling #147

Hardware Limitations and Scaling #147

evancofer commented Nov 19, 2016

agitter left a comment

agitter Nov 20, 2016

agitter Nov 20, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 27, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 27, 2016

agitter Nov 20, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 27, 2016

agitter Nov 20, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 20, 2016

evancofer Nov 26, 2016

agitter Nov 27, 2016

cgreene commented Dec 19, 2016

agitter Dec 20, 2016

cgreene Dec 20, 2016

evancofer Dec 21, 2016

cgreene left a comment

cgreene commented Dec 21, 2016

agitter commented Dec 21, 2016

cgreene commented Dec 21, 2016 via email

Hardware Limitations and Scaling #147

Hardware Limitations and Scaling #147

Conversation

evancofer commented Nov 19, 2016

agitter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgreene commented Dec 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgreene left a comment

Choose a reason for hiding this comment

cgreene commented Dec 21, 2016

agitter commented Dec 21, 2016

cgreene commented Dec 21, 2016 via email