Awesome Knowledge Distillation

Papers

• Neural Network Ensembles, L.K. Hansen, P. Salamon, 1990
• Neural Network Ensembles, Cross Validation, and Active Learning, Andres Krogh, Jesper Vedelsby, 1995
• Combining labeled and unlabeled data with co-training, A. Blum, T. Mitchell, 1998
• Ensemble Methods in Machine Learning, Thomas G. Dietterich, 2000
• Model Compression, Rich Caruana, 2006
• Learning with Pseudo-Ensembles, Philip Bachman, Ouais Alsharif, Doina Precup, 2014
• Dark knowledge, Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2014
• Distilling the Knowledge in a Neural Network, Geoffrey Hinton, Oriol Vinyals, Jeff Dean, NIPS 2014 Workshop
• Distilling Model Knowledge, George Papamakarios, 2015
• Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization, Baohan Xu, Yanwei Fu, Yu-Gang Jiang, Boyang Li, Leonid Sigal, 2015
• Learning Using Privileged Information: Similarity Control and Knowledge Transfer, Vladimir Vapnik, Rauf Izmailov, 2015
• FitNets: Hints for Thin Deep Nets, Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio, ICLR 2015
• Adapting Models to Signal Degradation using Distillation, Jong-Chyi Su, Subhransu Maji, 2016
• Sequence-Level Knowledge Distillation, deeplearning-papernotes, Yoon Kim, Alexander M. Rush, EMNLP 2016
• Knowledge Distillation for Small-footprint Highway Networks, Liang Lu, Michelle Guo, Steve Renals, 2016
• Deep Model Compression: Distilling Knowledge from Noisy Teachers, Bharat Bhusan Sau, Vineeth N. Balasubramanian, 2016
• Cross Modal Distillation for Supervision Transfer, Saurabh Gupta, Judy Hoffman, Jitendra Malik, CVPR 2016
• Unifying distillation and privileged information, David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik, ICLR 2016
• Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks, Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami, IEEE S&P 2016
• MobileID: Face Model Compression by Distilling Knowledge from Neurons, Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, Xiaoou Tang, AAAI 2016
• Recurrent Neural Network Training with Dark Knowledge Transfer, Zhiyuan Tang, Dong Wang, Zhiyong Zhang, 2016
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, Antti Tarvainen, Harri Valpola, NeurIPS 2017
• Learning from Noisy Labels with Distillation, Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Li, ICCV 2017
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer, Zehao Huang, Naiyan Wang, 2017
• DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer, Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang, 2017
• Revisiting knowledge transfer for training object class detectors, Jasper Uijlings, Stefan Popov, Vittorio Ferrari, 2017
• Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net, Zihao Liu, Qi Liu, Tao Liu, Yanzhi Wang, Wujie Wen, 2017
• Learning Loss for Knowledge Distillation with Conditional Adversarial Networks, Zheng Xu, Yen-Chang Hsu, Jiawei Huang, 2017
• Knowledge Projection for Deep Neural Networks, Zhi Zhang, Guanghan Ning, Zhihai He, 2017
• Moonshine: Distilling with Cheap Convolutions, Elliot J. Crowley, Gavin Gray, Amos Storkey, 2017
• Distilling a Neural Network Into a Soft Decision Tree, Nicholas Frosst, Geoffrey Hinton, 2017
• Do deep convolutional nets really need to be deep and convolutional?, Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed, Matthai Philipose, Matt Richardson, ICLR 2017
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, Sergey Zagoruyko, Nikos Komodakis, ICLR 2017
• Data-Free Knowledge Distillation for Deep Neural Networks, Raphael Gontijo Lopes, Stefano Fenu, Thad Starner, 2017
• Local Affine Approximators for Improving Knowledge Transfer, Suraj Srinivas and Francois Fleuret, 2017
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model, Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, Dhruv Batra, NeurIPS 2017
• Learning Efficient Object Detection Models with Knowledge Distillation, Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, Manmohan Chandraker, NeurIPS 2017
• A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning, Junho Yim, Donggyu Joo, Jihoon Bae, Junmo Kim, CVPR 2017
• Efficient Neural Architecture Search via Parameters Sharing, Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean, ICML 2018
• Interpreting Deep Classifiers by Visual Distillation of Dark Knowledge, Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton, 2018
• Defensive Collaborative Multi-task Training - Defending against Adversarial Attack towards Deep Neural Networks, Derek Wang, Chaoran Li, Sheng Wen, Yang Xiang, Wanlei Zhou, Surya Nepal, 2018
• Deep Co-Training for Semi-Supervised Image Recognition, Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, Alan Yuille, ECCV 2018
• Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples, Zihao Liu, Qi Liu, Tao Liu, Yanzhi Wang, Wujie Wen, 2018
• Multimodal Recurrent Neural Networks with Information Transfer Layers for Indoor Scene Labeling, Abrar H. Abdulnabi, Bing Shuai, Zhen Zuo, Lap-Pui Chau, Gang Wang, 2018
• Large scale distributed neural network training through online distillation, Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, Geoffrey E. Hinton, ICLR 2018
• Born Again Neural Networks, Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar, ICML 2018
• Knowledge Distillation by On-the-Fly Native Ensemble, Xu Lan, Xiatian Zhu, Shaogang Gong, NeurIPS 2018
• Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection, Yongcheng Liu, Lu Sheng, Jing Shao, Junjie Yan, Shiming Xiang, Chunhong Pan, ACM MM 2018
• YASENN: Explaining Neural Networks via Partitioning Activation Sequences, Yaroslav Zharov, Denis Korzhenkov, Pavel Shvechikov, Alexander Tuzhilin, 2018
• Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks, Yuenan Hou, Zheng Ma, Chunxiao Liu, Chen Change Loy, 2018
• A Generalized Meta-loss function for regression and classification using privileged information, Amina Asif, Muhammad Dawood, Fayyaz ul Amir Afsar Minhas, 2018
• Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, CVPR 2018
• Data Distillation: Towards Omni-Supervised Learning, Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, Kaiming He, CVPR 2018
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis, Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, ICML 2018
• Deep Mutual Learning, Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu, CVPR 2018
• Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation, Sarah Tan, Rich Caruana, Giles Hooker, Yin Lou, 2018
• Self-supervised knowledge distillation using singular value decomposition, Seung Hyun Lee, Dae Ha Kim, Byung Cheol Song, ECCV 2018
• KDGAN: Knowledge Distillation with Generative Adversarial Networks, Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi, NeurIPS 2018
• Efficient Video Classification Using Fewer Frames, Shweta Bhardwaj, Mukundhan Srinivasan, Mitesh M. Khapra, 2019
• Knowledge Adaptation for Efficient Semantic Segmentation, Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, Youliang Yan, 2019
• Structured Knowledge Distillation for Semantic Segmentation, Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, Jingdong Wang, CVPR 2019
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks, Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, Jimmy Lin, 2019
• Relational Knowledge Distillation, Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho, CVPR 2019
• A Comprehensive Overhaul of Feature Distillation, Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, Jin Young Choi, ICCV 2019
• Learning Metrics from Teachers: Compact Networks for Image Embedding, Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, Arnau Ramisa, 2019
• Variational Information Distillation for Knowledge Transfer, Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai, CVPR 2019
• Knowledge Distillation via Route Constrained Optimization, Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, Xiaolin Hu, ICCV 2019
• Knowledge Squeezed Adversarial Network Compression, Shu Changyong, Li Peng, Xie Yuan, Qu Yanyun, Dai Longquan, Ma Lizhuang, 2019
• Knowledge Flow: Improve Upon Your Teachers, Iou-Jen Liu, Jian Peng, Alexander G. Schwing, 2019
• Correlation Congruence for Knowledge Distillation, Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, Zhaoning Zhang, ICCV 2019
• Data-Free Learning of Student Networks, Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian, ICCV 2019
• Ensemble Distribution Distillation, Andrey Malinin, Bruno Mlodozeniec, Mark Gales, 2019
• Zero-Shot Knowledge Distillation in Deep Networks, Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, Anirban Chakraborty, ICML 2019
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation, Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, Kaisheng Ma, ICCV 2019
• Deep Face Recognition Model Compression via Knowledge Transfer and Distillation, Jayashree Karlekar, Jiashi Feng, Zi Sian Wong, Sugiri Pranata, 2019
• Distilling Object Detectors with Fine-grained Feature Imitation, Tao Wang, Li Yuan, Xiaopeng Zhang, Jiashi Feng, CVPR 2019
• When Does Label Smoothing Help?, Rafael Müller, Simon Kornblith, Geoffrey Hinton, NeurIPS 2019
• Graph-based Knowledge Distillation by Multi-head Attention Network, Seunghyun Lee, Byung Cheol Song, 2019
• Similarity-Preserving Knowledge Distillation, Frederick Tung, Greg Mori, ICCV 2019
• BAM! Born-Again Multi-Task Networks for Natural Language Understanding, Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, Quoc V. Le, ACL 2019
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation, Yuenan Hou, Zheng Ma, Chunxiao Liu, Chen Change Loy, ICCV 2019
• Self-Knowledge Distillation in Natural Language Processing, Sangchul Hahn, Heeyoul Choi, 2019
• Patient Knowledge Distillation for BERT Model Compression, Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu, EMNLP 2019
• Positive-Unlabeled Compression on the Cloud, Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, Chang Xu, 2019
• On the Efficacy of Knowledge Distillation, Jang Hyun Cho, Bharath Hariharan, 2019
• Improving Generalization and Robustness with Noisy Collaboration in Knowledge Distillation, Elahe Arani, Fahad Sarfraz, Bahram Zonooz, 2019
• Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Framework, Srinidhi Hegde, Ranjitha Prasad, Ramya Hebbalaguppe, Vishwajith Kumar, 2019
• Knowledge Distillation from Internal Representations, Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, Edward Guo, 2019
• Rethinking Data Augmentation: Self-Supervision and Self-Distillation, Hankook Lee, Sung Ju Hwang, Jinwoo Shin, 2019
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf, NeurIPS 2019 EMC^2 Workshop
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision, Tiancheng Wen, Shenqi Lai, Xueming Qian, 2019
• Stagewise Knowledge Distillation, Akshay Kulkarni, Navid Panchi, Shital Chiddarwar, 2019
• Graph Representation Learning via Multi-task Knowledge Distillation, Jiaqi Ma, Qiaozhu Mei, 2019
• Deep geometric knowledge distillation with graphs, Carlos Lassance, Myriam Bontonou, Ghouthi Boukli Hacene, Vincent Gripon, Jian Tang, Antonio Ortega, 2019
• MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks, Yunteng Luan, Hanyu Zhao, Zhi Yang, Yafei Dai, 2019
• The State of Knowledge Distillation for Classification, Fabian Ruffy, Karanbir Chahal, 2019
• Knowledge Distillation with Adversarial Samples Supporting Decision Boundary, Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi, AAAI 2019
• Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons, Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi, AAAI 2019
• Dataset Distillation, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros, ICLR 2019
• Fast Human Pose Estimation, Feng Zhang, Xiatian Zhu, Mao Ye, CVPR 2019
• MEAL: Multi-Model Ensemble via Adversarial Learning, Zhiqiang Shen, Zhankui He, Xiangyang Xue, AAAI 2019
• Distillation-Based Training for Multi-Exit Architectures, Mary Phuong, Christoph H. Lampert, ICCV 2019
• Knowledge Distillation via Instance Relationship Graph, Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, Yunqiang Duan, CVPR 2019
• Retaining Privileged Information for Multi-Task Learning, Fengyi Tang, Cao Xiao, Fei Wang, Jiayu Zhou, Li-Wei Lehman, KDD 2019
• Residual Knowledge Distillation, Mengya Gao, Yujun Shen, Quanquan Li, Chen Change Loy, 2020
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing, Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, EMNLP 2020
• Subclass Distillation, Rafael Müller, Simon Kornblith, Geoffrey Hinton, 2020
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou, NeurIPS 2020
• Regularizing Class-wise Predictions via Self-knowledge Distillation, Sukmin Yun, Jongjin Park, Kimin Lee, Jinwoo Shin, CVPR 2020
• GAN Compression: Efficient Architectures for Interactive Conditional GANs, Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han, CVPR 2020
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks, Lin Wang, Kuk-Jin Yoon, 2020
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou, ACL 2020
• FastBERT: a Self-distilling BERT with Adaptive Inference Time, Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju, ACL 2020
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation, Zaida Zhou, Chaoran Zhuge, Xinwei Guan, Wen Liu, 2020
• ResKD: Residual-Guided Knowledge Distillation, Xuewei Li, Songyuan Li, Bourahla Omar, Fei Wu, Xi Li, 2020
• Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning, Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko, NeurIPS 2020
• Knowledge Distillation Meets Self-Supervision, Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy, ECCV 2020
• MGD: Matching Guided Distillation, Kaiyu Yue, Jiangfan Deng, Feng Zhou, ECCV 2020
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks, Zhiqiang Shen, Marios Savvides, 2020
• Reducing the Teacher-Student Gap via Spherical Knowledge Distillation, Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, Deng Cai, 2020
• Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher, Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Hassan Ghasemzadeh, AAAI 2020
• Contrastive Representation Distillation, Yonglong Tian, Dilip Krishnan, Phillip Isola, ICLR 2020
• Revisit Knowledge Distillation: a Teacher-free Framework, Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang, Jiashi Feng, CVPR 2020
• Self-training with Noisy Student improves ImageNet classification, Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le, CVPR 2020
• TinyBERT: Distilling BERT for Natural Language Understanding, Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu, Findings of EMNLP 2020
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion, Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha, Jan Kautz, CVPR 2020
• General Instance Distillation for Object Detection, Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, Erjin Zhou, CVPR 2021
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation, Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, Il-Chul Moon, CVPR 2021
• Complementary Relation Contrastive Distillation, Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Aijun Yang, Mingzhe Rong, Xiaohua Wang, CVPR 2021
• Emerging Properties in Self-Supervised Vision Transformers (DINO), Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, ICCV 2021
• MobileStyleGAN: A Lightweight Convolutional Neural Network for High-Fidelity Image Synthesis, Sergei Belousov, 2021
• Distilling Knowledge via Knowledge Review, Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia, CVPR 2021
• Hierarchical Self-supervised Augmented Knowledge Distillation, Chuanguang Yang, Zhulin An, Linhang Cai, Yongjun Xu, IJCAI 2021
• Causal Distillation for Language Models, Zhengxuan Wu, Atticus Geiger, Josh Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah D. Goodman, 2021
• Training data-efficient image transformers & distillation through attention (DeiT), Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou, ICML 2021
• Cross-Layer Distillation with Semantic Calibration, Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Yan Feng, Chun Chen, AAAI 2021
• Exploring Simple Siamese Representation Learning, Xinlei Chen, Kaiming He, CVPR 2021
• Channel-wise Knowledge Distillation for Dense Prediction, Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, Chunhua Shen, ICCV 2021
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers, Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei, Findings of ACL-IJCNLP 2021
• Knowledge Distillation: A Survey, Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao, 2021
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language, Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli, ICML 2022
• Progressive Distillation for Fast Sampling of Diffusion Models, Tim Salimans, Jonathan Ho, ICLR 2022
• How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting, Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, Rita Cucchiara, 2022
• Decoupled Knowledge Distillation, Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang, CVPR 2022
• Knowledge Distillation with the Reused Teacher Classifier (SimKD), Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, Chun Chen, CVPR 2022
• Masked Generative Distillation, Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, Chun Yuan, ECCV 2022
• Knowledge Distillation from A Stronger Teacher, Tao Huang, Shan You, Fei Wang, Chen Qian, Chang Xu, NeurIPS 2022
• Localization Distillation for Dense Object Detection, Zhaohui Zheng, Rongguang Ye, Ping Wang, Dongwei Ren, Wangmeng Zuo, Qibin Hou, Ming-Ming Cheng, CVPR 2022
• iBOT: Image BERT Pre-Training with Online Tokenizer, Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, Tao Kong, ICLR 2022
• Focal and Global Knowledge Distillation for Detectors, Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, Chun Yuan, CVPR 2022
• Symbolic Knowledge Distillation: from General Language Models to Commonsense Models, Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi, NAACL 2022
• Information Theoretic Representation Distillation, Roy Miles, Adrian Lopez Rodriguez, Krystian Mikolajczyk, BMVC 2022
• Consistency Models, Yang Song, Prafulla Dhariwal, Mark Chen, Ilya Sutskever, ICML 2023
• TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation, David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, Eric Gu, 2023
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister, Findings of ACL 2023
• UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition, Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, Hoifung Poon, ICLR 2024
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, Hang Zhao, 2023
• Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models, Raviteja Vemulapalli, Hadi Pouransari, Fartash Faghri, Sachin Mehta, Mehrdad Farajtabar, Mohammad Rastegari, Oncel Tuzel, ICML 2024
• MobileSAMv2: Faster Segment Anything to Everything, Chaoning Zhang, Dongshen Han, Sheng Zheng, Jinwoo Choi, Tae-Ho Kim, Choong Seon Hong, 2023
• On Distillation of Guided Diffusion Models, Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, Tim Salimans, CVPR 2023
• Considerations When Learning Additive Explanations for Black-Box Models, Sarah Tan, Giles Hooker, Paul Koch, Albert Gordo, Rich Caruana, 2023
• Curriculum Temperature for Knowledge Distillation, Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, Jian Yang, AAAI 2023
• Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping, Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, Tat-Jen Cham, 2024
• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation, Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, Hai Huang, ICML 2024
• Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation, Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, Ali Thabet, 2024
• Improved Distribution Matching Distillation for Fast Image Synthesis, Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, NeurIPS 2024
• Transferring Knowledge from Large Foundation Models to Small Downstream Models, Shikai Qiu, Boran Han, Danielle C. Maddix, Shuai Zhang, Yuyang Wang, Andrew Gordon Wilson, ICML 2024
• DεpS: Delayed ε-Shrinking for Faster Once-For-All Training, Aditya Annavajjala, Alind Khare, Animesh Agrawal, Igor Fedorov, Hugo Latapie, Myungjin Lee, Alexey Tumanov, ECCV 2024
• Simple Unsupervised Knowledge Distillation With Space Similarity, Aditya Singh, Haohan Wang, 2024
• Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment, Tianyu Peng, Jiajun Zhang, COLING 2025
• Generative Prompt Internalization, Haebin Shin, Lei Ji, Yeyun Gong, Sungdong Kim, Eunbi Choi, Minjoon Seo, NAACL 2025
• ScaleKD: Strong Vision Transformers Could Be Excellent Teachers, Jiawei Fan, Chao Li, Xiaolong Liu, Anbang Yao, NeurIPS 2024
• Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation, Jiaming Lv, Haoyuan Yang, Peihua Li, NeurIPS 2024
• Adversarial Diffusion Distillation, Axel Sauer, Dominik Lorenz, Andreas Blattmann, Robin Rombach, ECCV 2024
• One-step Diffusion with Distribution Matching Distillation, Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, Taesung Park, CVPR 2024
• MiniLLM: On-Policy Distillation of Large Language Models, Yuxian Gu, Li Dong, Furu Wei, Minlie Huang, ICLR 2024
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem, ICLR 2024
• Improved Techniques for Training Consistency Models, Yang Song, Prafulla Dhariwal, ICLR 2024
• Logit Standardization in Knowledge Distillation, Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, Xiaochun Cao, CVPR 2024
• CLIP-KD: An Empirical Study of CLIP Model Distillation, Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, Yongjun Xu, CVPR 2024
• VkD : Improving Knowledge Distillation using Orthogonal Projections, Roy Miles, Ismail Elezi, Jiankang Deng, CVPR 2024
• Understanding the Role of the Projector in Knowledge Distillation, Roy Miles, Krystian Mikolajczyk, AAAI 2024
• Precision Shaking and DORPO: Conceptual Foundations of LLM Knowledge Distillation Methods, Áron Cserveni, 2024
• CrossKD: Cross-Head Knowledge Distillation for Object Detection, Jiabao Wang, Yuming Chen, Zhaohui Zheng, Xiang Li, Ming-Ming Cheng, Qibin Hou, CVPR 2024
• InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, Qiang Liu, ICLR 2024
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion, Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, Stefano Ermon, ICLR 2024
• Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching, Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, Yang You, ICLR 2024
• Online Speculative Decoding, Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhang, ICML 2024
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation, Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, Rishabh Agarwal, ICLR 2024
• Zephyr: Direct Distillation of LM Alignment, Lewis Tunstall, et al., COLM 2024
• MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training, Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel, CVPR 2024
• EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything, Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, Raghuraman Krishnamoorthi, Vikas Chandra, CVPR 2024
• On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm, Peng Sun, Bei Shi, Daiwei Yu, Tao Lin, CVPR 2024
• SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation, Thuan Hoang Nguyen, Anh Tran, CVPR 2024
• AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One, Mike Ranzinger, Greg Heinrich, Jan Kautz, Pavlo Molchanov, CVPR 2024
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, Jeff Wu, ICML 2024
• Knowledge Fusion of Large Language Models, Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi, ICLR 2024
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu, Findings of ACL 2024
• DistiLLM: Towards Streamlined Distillation for Large Language Models, Jongwoo Ko, Sungnyun Kim, Tianyi Chen, Se-Young Yun, ICML 2024
• BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation, Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, Ningyi Xu, ACL 2024
• Knowledge Distillation Based on Transformed Teacher Matching, Kaixiang Zheng, En-Hui Yang, ICLR 2024
• Revisiting Knowledge Distillation for Autoregressive Language Models, Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, Dacheng Tao, ACL 2024
• A Survey on Knowledge Distillation of Large Language Models, Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, Tianyi Zhou, 2024
• SDXL-Lightning: Progressive Adversarial Diffusion Distillation, Shanchuan Lin, Anran Wang, Xiao Yang, 2024
• Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning, Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu, ACL 2024
• PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, Jian Yang, CVPR 2024
• Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation, Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, Robin Rombach, SIGGRAPH Asia 2024
• Scale Decoupled Distillation, Shicai Wei, Chunbo Luo, Yang Luo, CVPR 2024
• Gecko: Versatile Text Embeddings Distilled from Large Language Models, Jinhyuk Lee, et al., 2024
• Elucidating the Design Space of Dataset Condensation, Shitong Shao, Zikai Zhou, Huanran Chen, Zhiqiang Shen, NeurIPS 2024
• Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, Xuefeng Xiao, NeurIPS 2024
• Distilling Diffusion Models into Conditional GANs, Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, Taesung Park, ECCV 2024
• Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation, Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, Jeannette Bohg, RSS 2024
• EM Distillation for One-step Diffusion Models, Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, Ruiqi Gao, NeurIPS 2024
• Phased Consistency Models, Fu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, Xiaogang Wang, Hongsheng Li, NeurIPS 2024
• T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback, Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, William Yang Wang, NeurIPS 2024
• Multistep Distillation of Diffusion Models via Moment Matching, Tim Salimans, Thomas Mensink, Jonathan Heek, Emiel Hoogeboom, NeurIPS 2024
• One-Step Effective Diffusion Network for Real-World Image Super-Resolution, Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, Lei Zhang, NeurIPS 2024
• Depth Anything V2, Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao, NeurIPS 2024
• A Label is Worth a Thousand Images in Dataset Distillation, Tian Qin, Zhiwei Deng, David Alvarez-Melis, NeurIPS 2024
• Dual-Space Knowledge Distillation for Large Language Models, Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, Jinan Xu, EMNLP 2024
• Understanding the Gains from Repeated Self-Distillation, Divyansh Pareek, Simon S. Du, Sewoong Oh, NeurIPS 2024
• Distilling System 2 into System 1, Ping Yu, Jing Xu, Jason Weston, Ilia Kulikov, 2024
• Compact Language Models via Pruning and Knowledge Distillation, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov, NeurIPS 2024
• Theia: Distilling Diverse Vision Foundation Models for Robot Learning, Jinghuan Shang, Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, Laura Herlant, CoRL 2024
• Gemma 2: Improving Open Language Models at a Practical Size, Gemma Team, et al., 2024
• UNIC: Universal Classification Models via Multi-teacher Distillation, Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, Yannis Kalantidis, ECCV 2024
• Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models, Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu, NeurIPS 2024
• LLM Pruning and Distillation in Practice: The Minitron Approach, Sharath Turuvekere Sreenivas, et al., 2024
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models, Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, Tri Dao, NeurIPS 2024
• One-Step Diffusion Distillation through Score Implicit Matching, Weijian Luo, Zemin Huang, Zhengyang Geng, J. Zico Kolter, Guo-jun Qi, NeurIPS 2024
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers, Alexander C. Li, Yuandong Tian, Beidi Chen, Deepak Pathak, Xinlei Chen, NeurIPS 2024
• Jasper and Stella: distillation of SOTA embedding models, Dun Zhang, Jiacheng Li, Ziyang Zeng, Fulong Wang, 2024
• MaKD: Multi-aspect Knowledge Distillation with Large Language Model, Taegyeong Lee, et al., 2025
• A Comprehensive Survey on Knowledge Distillation, Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, Shohreh Kasaei, TMLR 2025
• Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, Benjamin Minixhofer, Ivan Vulić, Edoardo Maria Ponti, NeurIPS 2025
• Autoregressive Distillation of Diffusion Transformers, Yeongmin Kim, Sotiris Anagnostidis, Yuming Du, Edgar Schönfeld, Jonas Kohler, Markos Georgopoulos, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, CVPR 2025
• Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions, Luyang Fang, Xiaowei Yu, Jiazhang Cai, Yongkai Chen, Shushan Wu, Zhengliang Liu, Zhenyuan Yang, Haoran Lu, Xilin Gong, Yufang Liu, Terry Ma, Wei Ruan, Ali Abbasi, Jing Zhang, Tao Wang, Ehsan Latif, Weihang You, Hanqi Jiang, Wei Liu, Wei Zhang, Soheil Kolouri, Xiaoming Zhai, Dajiang Zhu, Wenxuan Zhong, Tianming Liu, Ping Ma, 2025
• CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation, Pardis Taghavi, Tian Liu, Renjie Li, Reza Langari, Zhengzhong Tu, 2025
• Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models, Tiezheng Zhang, Yitong Li, Yu-cheng Chou, Jieneng Chen, Alan Yuille, Chen Wei, Junfei Xiao, 2025
• Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations, Faisal Hamman, Pasan Dissanayake, Yanjun Fu, Sanghamitra Dutta, NeurIPS 2025
• EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data, Grégoire Petit, Nathan Palluau, Axel Bauer, Clemens Dlaska, 2025
• Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs, Nicolas Boizard, Kevin El Haddad, Céline Hudelot, Pierre Colombo, TMLR 2025
• Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models, Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, Ngai Wong, COLING 2025
• Knowledge Distillation with Refined Logits, Wujie Sun, Defang Chen, Siwei Lyu, Genlang Chen, Chun Chen, Can Wang, ICCV 2025
• LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation, Fangxun Shu, et al., ICLR 2025
• Distilling an End-to-End Voice Assistant Without Instruction Training Data, William Held, Ella Li, Michael Ryan, Weiyan Shi, Yanzhe Zhang, Diyi Yang, ACL 2025
• Presto! Distilling Steps and Layers for Accelerating Music Generation, Zachary Novack, Ge Zhu, Jonah Casebeer, Julian McAuley, Taylor Berg-Kirkpatrick, Nicholas J. Bryan, ICLR 2025
• Progressive distillation induces an implicit curriculum, Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel, ICLR 2025
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, Saining Xie, ICLR 2025
• DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis, Yinghao Aaron Li, Rithesh Kumar, Zeyu Jin, ICML 2025
• LoLCATs: On Low-Rank Linearizing of Large Language Models, Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, Christopher Ré, ICLR 2025
• Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, Cheng Lu, Yang Song, ICLR 2025
• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, Tomas Pfister, ICLR 2025
• D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement, Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, Feng Wu, ICLR 2025
• Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step, Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, Hai Huang, ICLR 2025
• LLaVA-KD: A Framework of Distilling Multimodal Large Language Models, Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue, Yong Liu, Xiang Bai, ICCV 2025
• Pre-training Distillation for Large Language Models: A Design Space Exploration, Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, Juanzi Li, ACL 2025
• MiniPLM: Knowledge Distillation for Pre-Training Language Models, Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang, ICLR 2025
• High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws, M. Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, Samet Oymak, ICLR 2025
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time, Justin Deschenaux, Caglar Gulcehre, ICLR 2025
• One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation, Zhendong Wang, Zhaoshuo Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang, Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, Ming-Yu Liu, Yu Zeng, ICML 2025
• Active Data Curation Effectively Distills Large-Scale Multimodal Models, Vishaal Udandarao, Nikhil Parthasarathy, Muhammad Ferjad Naeem, Talfan Evans, Samuel Albanie, Federico Tombari, Yongqin Xian, Alessio Tonioni, Olivier J. Hénaff, CVPR 2025
• Puzzle: Distillation-Based NAS for Inference-Optimized LLMs, Akhiad Bercovich, et al., ICML 2025
• Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement, Qianhan Feng, Wenshuo Li, Tong Lin, Xinghao Chen, CVPR 2025
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, Xun Huang, CVPR 2025
• RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models, Greg Heinrich, Mike Ranzinger, Hongxu Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, Pavlo Molchanov, CVPR 2025
• Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models, Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, Houqiang Li, AAAI 2025
• Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching, Enshu Liu, Xuefei Ning, Yu Wang, Zinan Lin, ICLR 2025
• MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders, Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning Ma, Shanghang Zhang, CVPR 2025
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, DeepSeek-AI, et al., Nature 2025
• Open-Vocabulary Customization from CLIP via Data-Free Knowledge Distillation, Yongxian Wei, Zixuan Hu, Li Shen, Zhenyi Wang, Chun Yuan, Dacheng Tao, ICLR 2025
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models, Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, Takuya Akiba, ICLR 2025
• s1: Simple test-time scaling, Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto, EMNLP 2025
• On Teacher Hacking in Language Model Distillation, Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, Mathieu Blondel, ICML 2025
• LIMO: Less is More for Reasoning, Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu, COLM 2025
• Distillation Scaling Laws, Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, Russ Webb, ICML 2025
• Small Models Struggle to Learn from Strong Reasoners, Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, Radha Poovendran, Findings of ACL 2025
• Advantage-Guided Distillation for Preference Alignment in Small Language Models, Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, Qifan Wang, ICLR 2025
• CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation, Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He, EMNLP 2025
• VRM: Knowledge Distillation via Virtual Relation Matching, Weijia Zhang, Fei Xie, Weidong Cai, Chao Ma, ICCV 2025
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs, Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, Se-Young Yun, ICML 2025
• SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Song Han, Enze Xie, ICCV 2025
• DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers, Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Pau de Jorge, Diane Larlus, Yannis Kalantidis, CVPR 2025
• Di[M]O: Distilling Masked Diffusion Models into One-step Generator, Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton, ICCV 2025
• Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs, Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee, ACL 2025
• Antidistillation Sampling, Yash Savani, Asher Trockman, Zhili Feng, Yixuan Even Xu, Avi Schwarzschild, Alexander Robey, Marc Finzi, J. Zico Kolter, NeurIPS 2025
• ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence, Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, Qingming Huang, ICML 2025
• A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone, Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu, NeurIPS 2025
• ToDi: Token-wise Distillation via Fine-Grained Divergence Control, Seongryong Jung, Suwan Yoon, DongGeon Kim, Hwanhee Lee, EMNLP 2025
• Distilling LLM Agent into Small Models with Retrieval and Code Tools, Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, Sung Ju Hwang, NeurIPS 2025
• MGD³: Mode-Guided Dataset Distillation using Diffusion Models, Jeffrey A. Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah, ICML 2025
• Distillation Robustifies Unlearning, Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, Alexander Matt Turner, NeurIPS 2025
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, Eli Shechtman, NeurIPS 2025
• Reinforcement Learning Teachers of Test Time Scaling, Edoardo Cetin, Tianyu Zhao, Yujin Tang, NeurIPS 2025
• Align Your Flow: Scaling Continuous-Time Flow Map Distillation, Amirmojtaba Sabour, Sanja Fidler, Karsten Kreis, NeurIPS 2025
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis, Yanzuo Lu, Yuxi Ren, Xin Xia, Shanchuan Lin, Xing Wang, Xuefeng Xiao, Andy J. Ma, Xiaohua Xie, Jian-Huang Lai, ICCV 2025
• Cross-Architecture Distillation Made Simple with Redundancy Suppression, Weijia Zhang, Yuehao Liu, Wu Ran, Chao Ma, ICCV 2025
• DINOv3, Oriane Siméoni, et al., 2025
• MobileCLIP2: Improving Multi-Modal Reinforced Training, Fartash Faghri, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, Alexander Toshev, Oncel Tuzel, Hadi Pouransari, TMLR 2025
• EmbeddingGemma: Powerful and Lightweight Text Representations, Henrique Schechter Vera, et al., 2025
• AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders, Yuezhou Hu, Jiaxin Guo, Xinyu Feng, Tuo Zhao, NeurIPS 2025
• Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation, Enshu Liu, Qian Chen, Xuefei Ning, Shengen Yan, Guohao Dai, Zinan Lin, Yu Wang, NeurIPS 2025
• On-Policy Distillation, Kevin Lu, Thinking Machines Lab, 2025
• Black-Box On-Policy Distillation of Large Language Models, Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, Furu Wei, 2025
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation, Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron, 2026
• An Empirical Study of Knowledge Distillation for Code Understanding Tasks, Ruiqi Wang, Zezhou Yang, Cuiyun Gao, Xin Xia, Qing Liao, ICSE 2026
• To Distill or Not to Distill: When Knowledge Transfer Undermines Safety of LLMs, 2026
• Distill Any Depth: Distillation Creates a Stronger Monocular Depth Estimator, Xiankang He, Dongyan Guo, Hongji Li, Ying Cui, Libo Weng, Ruibo Li, Chi Zhang, Findings of CVPR 2026
• OpenThoughts: Data Recipes for Reasoning Models, Etash Guha, et al., ICLR 2026
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data, Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans, Nature 2026
• Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs), Nikita Kornilov, David Li, Tikhon Mavrin, Aleksei Leonov, Nikita Gushchin, Evgeny Burnaev, Iaroslav Koshelev, Alexander Korotin, ICLR 2026
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct, Haoyang Zheng, Xinyang Liu, Cindy Xiangrui Kong, Nan Jiang, Zheyuan Hu, Weijian Luo, Wei Deng, Guang Lin, ICLR 2026
• Distillation of Large Language Models via Concrete Score Matching, Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon, ICLR 2026
• Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency, Kaiwen Zheng, Yuji Wang, Qianli Ma, Huayu Chen, Jintao Zhang, Yogesh Balaji, Jianfei Chen, Ming-Yu Liu, Jun Zhu, Qinsheng Zhang, ICLR 2026
• pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation, Hansheng Chen, Kai Zhang, Hao Tan, Leonidas Guibas, Gordon Wetzstein, Sai Bi, ICLR 2026
• [MotionStream: Real-T