Skip to content

A list of research papers in the domain of machine learning, deep learning and related fields.

Notifications You must be signed in to change notification settings

anubhavshrimal/Machine-Learning-Research-Papers

Repository files navigation

Machine-Learning-Research-Papers

A list of research papers in the domain of machine learning, deep learning and related fields.

I have curated a list of research papers that I come across and read. I'll keep on updating the list of papers and their summary as I read them every week.

How to read a Research Paper

Professor Andrew Ng gave some awesome tips on how to read a research paper. I have summarised the tips in this PDF.

Table of Contents

The list of papers can be viewed based on differentiating criteria's such as (Conference venue, Year Published, Topic Covered, Authors, etc.).

The following filtered formats are available to view paper's list:

All Papers

Paper Name Status Topic Category Year Conference Author Summary Link
0 ZF Net (Visualizing and Understanding Convolutional Networks) Read CNNs, CV , Image Visualization 2014 ECCV Matthew D. Zeiler, Rob Fergus Visualize CNN Filters / Kernels using De-Convolutions on CNN filter activations. link
1 Inception-v1 (Going Deeper With Convolutions) Read CNNs, CV , Image Architecture 2015 CVPR Christian Szegedy, Wei Liu Propose the use of 1x1 conv operations to reduce the number of parameters in a deep and wide CNN link
2 ResNet (Deep Residual Learning for Image Recognition) Read CNNs, CV , Image Architecture 2016 CVPR Kaiming He, Xiangyu Zhang Introduces Residual or Skip Connections to allow increase in the depth of a DNN link
3 MobileNet (Efficient Convolutional Neural Networks for Mobile Vision Applications) Pending CNNs, CV , Image Architecture, Optimization-No. of params 2017 arXiv Andrew G. Howard, Menglong Zhu link
4 Evaluation of neural network architectures for embedded systems Read CNNs, CV , Image Comparison 2017 IEEE ISCAS Adam Paszke, Alfredo Canziani, Eugenio Culurciello Compare CNN classification architectures on accuracy, memory footprint, parameters, operations count, inference time and power consumption. link
5 SqueezeNet Read CNNs, CV , Image Architecture, Optimization-No. of params 2016 arXiv Forrest N. Iandola, Song Han Explores model compression by using 1x1 convolutions called fire modules. link
6 Pruning Filters for Efficient ConvNets Pending CNNs, CV , Image Optimization-No. of params 2017 arXiv Asim Kadav, Hao Li link
7 Attention is All you Need Read Attention, Text , Transformers Architecture 2017 NIPS Ashish Vaswani, Illia Polosukhin, Noam Shazeer, Łukasz Kaiser Talks about Transformer architecture which brings SOTA performance for different tasks in NLP link
8 GPT-2 (Language Models are Unsupervised Multitask Learners) Pending Attention, Text , Transformers 2019 Alec Radford, Dario Amodei, Ilya Sutskever, Jeffrey Wu link
9 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Read Attention, Text , Transformers Embeddings 2018 NAACL Jacob Devlin, Kenton Lee, Kristina Toutanova, Ming-Wei Chang BERT is an extension to Transformer based architecture which introduces a masked word pretraining and next sentence prediction task to pretrain the model for a wide variety of tasks. link
10 SAGAN: Self-Attention Generative Adversarial Networks Pending Attention, GANs, Image Architecture 2018 arXiv Augustus Odena, Dimitris Metaxas, Han Zhang, Ian Goodfellow link
11 Single Headed Attention RNN: Stop Thinking With Your Head Pending Attention, LSTMs, Text Optimization-No. of params 2019 arXiv Stephen Merity link
12 Reformer: The Efficient Transformer Read Attention, Text , Transformers Architecture, Optimization-Memory, Optimization-No. of params 2020 arXiv Anselm Levskaya, Lukasz Kaiser, Nikita Kitaev Overcome time and memory complexity of Transformers by bucketing Query, Keys and using Reversible residual connections. link
13 A 2019 guide to Human Pose Estimation with Deep Learning Pending CV , Pose Estimation Comparison 2019 Blog Sudharshan Chandra Babu link
14 A Simple yet Effective Baseline for 3D Human Pose Estimation Pending CV , Pose Estimation 2017 ICCV James J. Little, Javier Romero, Julieta Martinez, Rayat Hossain link
15 Bag of Tricks for Image Classification with Convolutional Neural Networks Read CV , Image Optimizations, Tips & Tricks 2018 arXiv Tong He, Zhi Zhang Shows a dozen tricks (mixup, label smoothing, etc.) to improve CNN accuracy and training time. link
16 Class-Balanced Loss Based on Effective Number of Samples Pending Loss Function Tips & Tricks 2019 CVPR Menglin Jia, Yin Cui link
17 Self-Normalizing Neural Networks Pending Activation Function, Tabular Optimizations, Tips & Tricks 2017 NIPS Andreas Mayr, Günter Klambauer, Thomas Unterthiner link
18 A Comprehensive Guide on Activation Functions This week Activation Function 2020 Blog Ygor Rebouças Serpa link
19 Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet Reading CNNs, CV , Image 2019 arXiv Matthias Bethge, Wieland Brendel link
20 Breaking neural networks with adversarial attacks Pending CNNs, Image Adversarial 2019 Blog Anant Jain link
21 The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Read NN Initialization, NNs Optimization-No. of params, Tips & Tricks 2019 ICLR Jonathan Frankle, Michael Carbin Lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that—when trained in isolation— reach test accuracy comparable to the original network in a similar number of iterations. link
22 All you need is a good init Pending NN Initialization Tips & Tricks 2015 arXiv Dmytro Mishkin, Jiri Matas link
23 Pix2Pix: Image-to-Image Translation with Conditional Adversarial Nets Read GANs, Image 2017 CVPR Alexei A. Efros, Jun-Yan Zhu, Phillip Isola, Tinghui Zhou Image to image translation using Conditional GANs and dataset of image pairs from one domain to another. link
24 CycleGAN: Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks Pending GANs, Image Architecture 2017 ICCV Alexei A. Efros, Jun-Yan Zhu, Phillip Isola, Taesung Park link
25 Language-Agnostic BERT Sentence Embedding Read Attention, Siamese Network, Text , Transformers Embeddings 2020 arXiv Fangxiaoyu Feng, Yinfei Yang A BERT model with multilingual sentence embeddings learned over 112 languages and Zero-shot learning over unseen languages. link
26 Phrase-Based & Neural Unsupervised Machine Translation Pending NMT, Text , Transformers Unsupervised 2018 arXiv Alexis Conneau, Guillaume Lample, Ludovic Denoyer, Marc'Aurelio Ranzato, Myle Ott link
27 Unsupervised Machine Translation Using Monolingual Corpora Only Pending GANs, NMT, Text , Transformers Unsupervised 2017 arXiv Alexis Conneau, Guillaume Lample, Ludovic Denoyer, Marc'Aurelio Ranzato, Myle Ott link
28 Cross-lingual Language Model Pretraining Pending NMT, Text , Transformers Unsupervised 2019 arXiv Alexis Conneau, Guillaume Lample link
29 Word2Vec: Efficient Estimation of Word Representations in Vector Space Pending Text Embeddings, Tips & Tricks 2013 arXiv Greg Corrado, Jeffrey Dean, Kai Chen, Tomas Mikolov link
30 Capsule Networks: Dynamic Routing Between Capsules Pending CV , Image Architecture 2017 arXiv Geoffrey E Hinton, Nicholas Frosst, Sara Sabour link
31 Graph Neural Network: Relational inductive biases, deep learning, and graph networks Pending GraphNN Architecture 2018 arXiv Jessica B. Hamrick, Oriol Vinyals, Peter W. Battaglia link
32 Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs Pending CNNs, Image 2020 arXiv Ari S. Morcos, David J. Schwab, Jonathan Frankle link
33 Arbitrary Style Transfer in Real-Time With Adaptive Instance Normalization Pending CNNs, Image 2017 ICCV Serge Belongie, Xun Huang link
34 How Does Batch Normalization Help Optimization? Pending NNs, Normalization Optimizations 2018 arXiv Aleksander Madry, Andrew Ilyas, Dimitris Tsipras, Shibani Santurkar link
35 WGAN: Wasserstein GAN Pending GANs, Loss Function 2017 arXiv Léon Bottou, Martin Arjovsky, Soumith Chintala link
36 Group Normalization Pending NNs, Normalization Optimizations 2018 arXiv Kaiming He, Yuxin Wu link
37 Spectral Normalization for GANs Pending GANs, Normalization Optimizations 2018 arXiv Masanori Koyama, Takeru Miyato, Toshiki Kataoka, Yuichi Yoshida link
38 One-shot Text Field Labeling using Attention and Belief Propagation for Structure Information Extraction Pending Image , Text 2020 arXiv Jun Huang, Mengli Cheng, Minghui Qiu, Wei Lin, Xing Shi link
39 Perceptual Losses for Real-Time Style Transfer and Super-Resolution Pending Loss Function, NNs 2016 ECCV Alexandre Alahi, Justin Johnson, Li Fei-Fei link
40 Topological Loss: Beyond the Pixel-Wise Loss for Topology-Aware Delineation Pending Image , Loss Function, Segmentation 2018 CVPR Agata Mosinska, Mateusz Koziński, Pablo Márquez-Neila, Pascal Fua link
41 Understanding Loss Functions in Computer Vision Pending CV , GANs, Image , Loss Function Comparison, Tips & Tricks 2020 Blog Sowmya Yellapragada link
42 NADAM: Incorporating Nesterov Momentum into Adam Pending NNs, Optimizers Comparison 2016 Timothy Dozat link
43 Deep Double Descent: Where Bigger Models and More Data Hurt Pending NNs 2019 arXiv Boaz Barak, Gal Kaplun, Ilya Sutskever, Preetum Nakkiran, Tristan Yang, Yamini Bansal link
44 StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks Pending GANs, Image 2019 CVPR Samuli Laine, Tero Karras, Timo Aila link
45 Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? Pending GANs, Image 2019 ICCV Peter Wonka, Rameen Abdal, Yipeng Qin link
46 Improved Techniques for Training GANs Pending GANs, Image Semi-Supervised 2016 NIPS Alec Radford, Ian Goodfellow, Tim Salimans, Vicki Cheung, Wojciech Zaremba, Xi Chen link
47 AnimeGAN: Towards the Automatic Anime Characters Creation with Generative Adversarial Networks Pending GANs, Image 2017 NIPS Jiakai Zhang, Minjun Li, Yanghua Jin link
48 Progressive Growing of GANs for Improved Quality, Stability, and Variation Pending GANs, Image Tips & Tricks 2018 ICLR Jaakko Lehtinen, Samuli Laine, Tero Karras, Timo Aila link
49 BEGAN: Boundary Equilibrium Generative Adversarial Networks Pending GANs, Image 2017 arXiv David Berthelot, Luke Metz, Thomas Schumm link
50 Adam: A Method for Stochastic Optimization Pending NNs, Optimizers 2015 ICLR Diederik P. Kingma, Jimmy Ba link
51 StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation Pending GANs, Image 2018 CVPR Jaegul Choo, Jung-Woo Ha, Minje Choi, Munyoung Kim, Sunghun Kim, Yunjey Choi link
52 IMLE-GAN: Inclusive GAN: Improving Data and Minority Coverage in Generative Models Pending GANs 2020 arXiv Jitendra Malik, Ke Li, Larry Davis, Mario Fritz, Ning Yu, Peng Zhou link
53 Few-Shot Learning with Localization in Realistic Settings Pending CNNs, Image Few-shot-learning 2019 CVPR Bharath Hariharan, Davis Wertheimer link
54 Revisiting Pose-Normalization for Fine-Grained Few-Shot Recognition Pending CNNs, Image Few-shot-learning 2020 CVPR Bharath Hariharan, Davis Wertheimer, Luming Tang link
55 ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning Pending AGI, Dataset, Text 2019 AAAI Maarten Sap, Noah A. Smith, Ronan Le Bras, Yejin Choi link
56 COMET: Commonsense Transformers for Automatic Knowledge Graph Construction Pending AGI, Text , Transformers 2019 ACL Antoine Bosselut, Hannah Rashkin, Yejin Choi link
57 VisualCOMET: Reasoning about the Dynamic Context of a Still Image Pending AGI, Dataset, Image , Text , Transformers 2020 ECCV Ali Farhadi, Chandra Bhagavatula, Jae Sung Park, Yejin Choi link
58 Occupancy Anticipation for Efficient Exploration and Navigation Pending CNNs, Image Reinforcement-Learning 2020 ECCV Kristen Grauman, Santhosh K. Ramakrishnan, Ziad Al-Halah link
59 T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Read Attention, Text , Transformers 2020 JMLR Colin Raffel, Noam Shazeer, Peter J. Liu, Wei Liu, Yanqi Zhou Presents a Text-to-Text transformer model with multi-task learning capabilities, simultaneously solving problems such as machine translation, document summarization, question answering, and classification tasks. link
60 GPT-f: Generative Language Modeling for Automated Theorem Proving Pending Attention, Transformers 2020 arXiv Ilya Sutskever, Stanislas Polu link
61 Vision Transformer: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Pending Attention, Image , Transformers 2021 ICLR Alexey Dosovitskiy, Jakob Uszkoreit, Lucas Beyer, Neil Houlsby link
62 MuZero: Mastering Go, chess, shogi and Atari without rules Pending Reinforcement-Learning 2020 Nature David Silver, Demis Hassabis, Ioannis Antonoglou, Julian Schrittwiese link
63 Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask Read NN Initialization, NNs Comparison, Optimization-No. of params, Tips & Tricks 2019 NeurIPS Hattie Zhou, Janice Lan, Jason Yosinski, Rosanne Liu Follow up on Lottery Ticket Hypothesis exploring the effects of different Masking criteria as well as Mask-1 and Mask-0 actions. link
64 DALL·E: Creating Images from Text Pending Image , Text , Transformers 2021 Blog Aditya Ramesh, Gabriel Goh, Ilya Sutskever, Mikhail Pavlov, Scott Gray link
65 CLIP: Connecting Text and Images Pending Image , Text , Transformers Multimodal, Pre-Training 2021 arXiv Alec Radford, Ilya Sutskever, Jong Wook Kim link
66 Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision This week Image , Text , Transformers Multimodal 2020 EMNLP Hao Tan, Mohit Bansal link
67 SpanBERT: Improving Pre-training by Representing and Predicting Spans Read Question-Answering, Text , Transformers Pre-Training 2020 TACL Danqi Chen, Mandar Joshi A different pre-training strategy for BERT model to improve performance for Question Answering task. link
68 Learning to Extract Attribute Value from Product via Question Answering: A Multi-task Approach Read Question-Answering, Text , Transformers Zero-shot-learning 2020 KDD Li Yang, Qifan Wang Question Answering BERT model used to extract attributes from products. Introduce further No Answer loss and distillation to promote zero shot learning. link
69 TransGAN: Two Transformers Can Make One Strong GAN Pending GANs, Image , Transformers Architecture 2021 arXiv Shiyu Chang, Yifan Jiang, Zhangyang Wang link
70 Interpreting Deep Learning Models in Natural Language Processing: A Review Pending Text Comparison, Visualization 2021 arXiv Diyi Yang, Xiaofei Sun link
71 Symbolic Knowledge Distillation: from General Language Models to Commonsense Models Pending Dataset, Text , Transformers Optimizations, Tips & Tricks 2021 arXiv Chandra Bhagavatula, Jack Hessel, Peter West, Yejin Choi link
72 Chain of Thought Prompting Elicits Reasoning in Large Language Models Pending Question-Answering, Text , Transformers 2022 arXiv Denny Zhou, Jason Wei, Xuezhi Wang link
73 Transforming Sequence Tagging Into A Seq2Seq Task Pending Generative, Text Comparison, Tips & Tricks 2022 arXiv Iftekhar Naim, Karthik Raman, Krishna Srinivasan link
74 Large Language Models are Zero-Shot Reasoners Pending Generative, Question-Answering, Text Tips & Tricks, Zero-shot-learning 2022 arXiv Takeshi Kojima, Yusuke Iwasawa link
75 Flan-T5: Scaling Instruction-Finetuned Language Models Pending Generative, Text , Transformers Architecture, Pre-Training 2022 arXiv Hyung Won Chung, Le Hou link
76 Decoding a Neural Retriever’s Latent Space for Query Suggestion Pending Text Embeddings, Latent space 2022 arXiv Christian Buck, Leonard Adolphs, Michelle Chen Huebscher link
77 Training Compute-Optimal Large Language Models Pending Large-Language-Models, Transformers Architecture, Optimization-No. of params, Pre-Training, Tips & Tricks 2022 arXiv Jordan Hoffmann, Laurent Sifre, Oriol Vinyals, Sebastian Borgeaud link
78 VL-T5: Unifying Vision-and-Language Tasks via Text Generation Read CNNs, CV , Generative, Image , Large-Language-Models, Question-Answering, Text , Transformers Architecture, Embeddings, Multimodal, Pre-Training 2021 arXiv Hao Tan, Jaemin Cho, Jie Le, Mohit Bansal Unifying two modalities (image and text) together in a single transformer model to solve multiple tasks in a single architecture using text prefixes similar to T5. link
79 Scaling Instruction-Finetuned Language Models (FLAN) Pending Generative, Large-Language-Models, Question-Answering, Text , Transformers Instruction-Finetuning 2022 arXiv Hyung Won Chung, Jason Wei, Jeffrey Dean, Le Hou, Quoc V. Le, Shayne Longpre https://arxiv.org/abs/2210.11416 introduces FLAN (Fine-tuned LAnguage Net), an instruction finetuning method, and presents the results of its application. The study demonstrates that by fine-tuning the 540B PaLM model on 1836 tasks while incorporating Chain-of-Thought Reasoning data, FLAN achieves improvements in generalization, human usability, and zero-shot reasoning over the base model. The paper also provides detailed information on how each these aspects was evaluated. link
80 ReAct: Synergizing Reasoning and Acting in Language Models Pending Generative, Large-Language-Models, Text Optimizations, Tips & Tricks 2023 ICLR Dian Yu, Izhak Shafran, Jeffrey Zhao, Karthik Narasimhan, Nan Du, Shunyu Yao, Yuan Cao This paper introduces ReAct, a novel approach that leverages Large Language Models (LLMs) to interleave reasoning traces and task-specific actions. ReAct outperforms existing methods on various language and decision-making tasks, addressing issues like hallucination, error propagation, and improving human interpretability and trustworthiness. link
81 Training language models to follow instructions with human feedback Pending Generative, Large-Language-Models, Training Method Instruction-Finetuning, Reinforcement-Learning, Semi-Supervised 2022 arXiv Carroll L. Wainwright, Diogo Almeida, Jan Leike, Jeff Wu, Long Ouyang, Pamela Mishkin, Paul Christiano, Ryan Lowe, Xu Jiang This paper presents InstructGPT, a model fine-tuned with human feedback to better align with user intent across various tasks. Despite having significantly fewer parameters than larger models, InstructGPT outperforms them in human evaluations, demonstrating improved truthfulness, reduced toxicity, and minimal performance regressions on public NLP datasets, highlighting the potential of fine-tuning with human feedback for enhancing language model alignment with human intent. link
82 Constitutional AI: Harmlessness from AI Feedback Pending Generative, Large-Language-Models, Training Method Instruction-Finetuning, Reinforcement-Learning, Unsupervised 2022 arXiv Jared Kaplan, Yuntao Ba The paper introduces Constitutional AI, a method for training a safe AI assistant without human-labeled data on harmful outputs. It combines supervised learning and reinforcement learning phases, enabling the AI to engage with harmful queries by explaining its objections, thus improving control, transparency, and human-judged performance with minimal human oversight. link
83 Self-Alignment with Instruction Backtranslation Pending Generative, Large-Language-Models, Training Method Instruction-Finetuning 2023 arXiv Jason Weston, Mike Lewis, Ping Yu, Xian Li The paper introduces a scalable method called "instruction backtranslation" to create a high-quality instruction-following language model. This method involves self-augmentation and self-curation of training examples generated from web documents, resulting in a model that outperforms others in its category without relying on distillation data, showcasing its effective self-alignment capability. link
84 Table-GPT: Table-tuned GPT for Diverse Table Tasks Pending Generative, Large-Language-Models, Training Method Instruction-Finetuning 2023 arXiv link
85 Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering Pending Large-Language-Models Prompting, Tips & Tricks 2024 arXiv Dedy Kredo, Itamar Friedman, Tal Ridnik This paper introduces AlphaCodium, a novel test-based, multi-stage, code-oriented iterative approach for improving the performance of Language Model Models (LLMs) on code generation tasks. link
86 Large Language Models for Data Annotation: A Survey This week Dataset, Generative, Large-Language-Models Prompting, Tips & Tricks 2024 arXiv Alimohammad Beigi, Zhen Tan link

About

A list of research papers in the domain of machine learning, deep learning and related fields.

Topics

Resources

Stars

Watchers

Forks