ML Paper Challenge Day 26 — Net2Net: Accelerating Learning via Knowledge Transfer

Papers with Code - Net2Net: Accelerating Learning via Knowledge Transfer

Day 26: 2020.05.07
Paper: Net2Net: Accelerating Learning via Knowledge Transfer
Category: Model/Deep Learning/Technique

A simple but brilliant idea.

Background:

In real life machine learning workflow, we usually have to train many different neural network models to continually find the best model structure for a specific task. What if we can retain the trained information in the net after modifying the structure without retraining from the scratch?

Net2Net:

• rapidly transferring knowledge contained in one neural network to another neural network

Different approaches: