The corpus

Every paper Citeline can answer questions from, and nothing else. Click any title to read the paper itself on arXiv.

131 of 131 papers

Transformers

Attention Is All You NeedVaswani et al. · 2017 BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin et al. · 2018 Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextDai et al. · 2019 FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao et al. · 2022 RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu et al. · 2021 GLU Variants Improve TransformerShazeer · 2020 Root Mean Square Layer NormalizationZhang & Sennrich · 2019 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · 2017 Reformer: The Efficient TransformerKitaev et al. · 2020 Longformer: The Long-Document TransformerBeltagy et al. · 2020 Linformer: Self-Attention with Linear ComplexityWang et al. · 2020

Large Language Models

Language Models are Few-Shot LearnersBrown et al. · 2020 Training language models to follow instructions with human feedbackOuyang et al. · 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsWei et al. · 2022 LoRA: Low-Rank Adaptation of Large Language ModelsHu et al. · 2021 Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerRaffel et al. · 2019 XLNet: Generalized Autoregressive Pretraining for Language UnderstandingYang et al. · 2019 RoBERTa: A Robustly Optimized BERT Pretraining ApproachLiu et al. · 2019 ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsClark et al. · 2020 ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsLan et al. · 2019 GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language UnderstandingWang et al. · 2018 PaLM: Scaling Language Modeling with PathwaysChowdhery et al. · 2022 LLaMA: Open and Efficient Foundation Language ModelsTouvron et al. · 2023 Llama 2: Open Foundation and Fine-Tuned Chat ModelsTouvron et al. · 2023 GPT-4 Technical ReportOpenAI · 2023 Mistral 7BJiang et al. · 2023 Scaling Laws for Neural Language ModelsKaplan et al. · 2020 Training Compute-Optimal Large Language ModelsHoffmann et al. · 2022 Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityFedus et al. · 2021 Constitutional AI: Harmlessness from AI FeedbackBai et al. · 2022 Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov et al. · 2023 Self-Instruct: Aligning Language Models with Self-Generated InstructionsWang et al. · 2022 ReAct: Synergizing Reasoning and Acting in Language ModelsYao et al. · 2022 Toolformer: Language Models Can Teach Themselves to Use ToolsSchick et al. · 2023 Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksReimers & Gurevych · 2019 Generative Agents: Interactive Simulacra of Human BehaviorPark et al. · 2023 Voyager: An Open-Ended Embodied Agent with Large Language ModelsWang et al. · 2023 GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersFrantar et al. · 2022 QLoRA: Efficient Finetuning of Quantized LLMsDettmers et al. · 2023

AI Safety

Deep reinforcement learning from human preferencesChristiano et al. · 2017 Red Teaming Language Models with Language ModelsPerez et al. · 2022 Concrete Problems in AI SafetyAmodei et al. · 2016

Retrieval & Generation

Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al. · 2020 Dense Passage Retrieval for Open-Domain Question AnsweringKarpukhin et al. · 2020 ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTKhattab & Zaharia · 2020

Computer Vision

Deep Residual Learning for Image RecognitionHe et al. · 2015 Very Deep Convolutional Networks for Large-Scale Image RecognitionSimonyan & Zisserman · 2014 Going Deeper with ConvolutionsSzegedy et al. · 2014 U-Net: Convolutional Networks for Biomedical Image SegmentationRonneberger et al. · 2015 Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksRen et al. · 2015 You Only Look Once: Unified, Real-Time Object DetectionRedmon et al. · 2015 Mask R-CNNHe et al. · 2017 EfficientNet: Rethinking Model Scaling for Convolutional Neural NetworksTan & Le · 2019 An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleDosovitskiy et al. · 2020 Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsLiu et al. · 2021 Learning Transferable Visual Models From Natural Language SupervisionRadford et al. · 2021 Segment AnythingKirillov et al. · 2023 NeRF: Representing Scenes as Neural Radiance Fields for View SynthesisMildenhall et al. · 2020

Multimodal

Flamingo: a Visual Language Model for Few-Shot LearningAlayrac et al. · 2022 BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationLi et al. · 2022 Zero-Shot Text-to-Image GenerationRamesh et al. · 2021 Visual Instruction TuningLiu et al. · 2023 Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingSaharia et al. · 2022

Speech

Deep Speech 2: End-to-End Speech Recognition in English and MandarinAmodei et al. · 2015 wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsBaevski et al. · 2020 Robust Speech Recognition via Large-Scale Weak SupervisionRadford et al. · 2022

Generative Models

Denoising Diffusion Probabilistic ModelsHo et al. · 2020 High-Resolution Image Synthesis with Latent Diffusion ModelsRombach et al. · 2021 Generative Adversarial NetworksGoodfellow et al. · 2014 Auto-Encoding Variational BayesKingma & Welling · 2013 Improved Denoising Diffusion Probabilistic ModelsNichol & Dhariwal · 2021 Classifier-Free Diffusion GuidanceHo & Salimans · 2022 Denoising Diffusion Implicit ModelsSong et al. · 2020

Self-Supervised Learning

A Simple Framework for Contrastive Learning of Visual RepresentationsChen et al. · 2020 Momentum Contrast for Unsupervised Visual Representation LearningHe et al. · 2019 Bootstrap your own latent: A new approach to self-supervised LearningGrill et al. · 2020 Masked Autoencoders Are Scalable Vision LearnersHe et al. · 2021 Emerging Properties in Self-Supervised Vision TransformersCaron et al. · 2021

Reinforcement Learning

Playing Atari with Deep Reinforcement LearningMnih et al. · 2013 Trust Region Policy OptimizationSchulman et al. · 2015 Proximal Policy Optimization AlgorithmsSchulman et al. · 2017

Meta-Learning & Few-Shot

Model-Agnostic Meta-Learning for Fast Adaptation of Deep NetworksFinn et al. · 2017 Matching Networks for One Shot LearningVinyals et al. · 2016 Prototypical Networks for Few-shot LearningSnell et al. · 2017

Graph Neural Networks

Semi-Supervised Classification with Graph Convolutional NetworksKipf & Welling · 2016 Inductive Representation Learning on Large GraphsHamilton et al. · 2017 Graph Attention NetworksVeličković et al. · 2017

Interpretability

Grad-CAM: Visual Explanations from Deep Networks via Gradient-based LocalizationSelvaraju et al. · 2016 A Unified Approach to Interpreting Model PredictionsLundberg & Lee · 2017 "Why Should I Trust You?": Explaining the Predictions of Any ClassifierRibeiro et al. · 2016

Efficiency & Compression

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision ApplicationsHoward et al. · 2017 SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizeIandola et al. · 2016 Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only InferenceJacob et al. · 2017

Systems & Scaling

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismShoeybi et al. · 2019 ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsRajbhandari et al. · 2019 GPipe: Efficient Training of Giant Neural Networks using Pipeline ParallelismHuang et al. · 2018

Time Series

N-BEATS: Neural basis expansion analysis for interpretable time series forecastingOreshkin et al. · 2019 Informer: Beyond Efficient Transformer for Long Sequence Time-Series ForecastingZhou et al. · 2020

Recommender Systems

Neural Collaborative FilteringHe et al. · 2017 Wide & Deep Learning for Recommender SystemsCheng et al. · 2016 BPR: Bayesian Personalized Ranking from Implicit FeedbackRendle et al. · 2012

AutoML

Neural Architecture Search with Reinforcement LearningZoph & Le · 2016

Adversarial Robustness

Explaining and Harnessing Adversarial ExamplesGoodfellow et al. · 2014 Towards Deep Learning Models Resistant to Adversarial AttacksMadry et al. · 2017 Intriguing properties of neural networksSzegedy et al. · 2013

Federated Learning

Communication-Efficient Learning of Deep Networks from Decentralized DataMcMahan et al. · 2016

Benchmarks

SQuAD: 100,000+ Questions for Machine Comprehension of TextRajpurkar et al. · 2016 SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding SystemsWang et al. · 2019 HellaSwag: Can a Machine Really Finish Your Sentence?Zellers et al. · 2019 Measuring Massive Multitask Language UnderstandingHendrycks et al. · 2020 Holistic Evaluation of Language ModelsLiang et al. · 2022

Foundations

Efficient Estimation of Word Representations in Vector SpaceMikolov et al. · 2013 Adam: A Method for Stochastic OptimizationKingma & Ba · 2014 Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftIoffe & Szegedy · 2015 Layer NormalizationBa, Kiros & Hinton · 2016 Highway NetworksSrivastava et al. · 2015 Improving neural networks by preventing co-adaptation of feature detectorsHinton et al. · 2012 Distilling the Knowledge in a Neural NetworkHinton et al. · 2015 Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau et al. · 2014 Sequence to Sequence Learning with Neural NetworksSutskever et al. · 2014 Show and Tell: A Neural Image Caption GeneratorVinyals et al. · 2014 SGDR: Stochastic Gradient Descent with Warm RestartsLoshchilov & Hutter · 2016 Decoupled Weight Decay RegularizationLoshchilov & Hutter · 2017 On the difficulty of training Recurrent Neural NetworksPascanu et al. · 2012 mixup: Beyond Empirical Risk MinimizationZhang et al. · 2017 AutoAugment: Learning Augmentation Policies from DataCubuk et al. · 2018 Improved Regularization of Convolutional Neural Networks with CutoutDeVries & Taylor · 2017 Distributed Representations of Words and Phrases and their CompositionalityMikolov et al. · 2013 Enriching Word Vectors with Subword InformationBojanowski et al. · 2016 Universal Sentence EncoderCer et al. · 2018 A Survey on Bias and Fairness in Machine LearningMehrabi et al. · 2019