ML Paper Challenge Day 27 — Network Morphism

Papers with Code - Network Morphism

Day 27: 2020.05.08
Paper: Network Morphism
Category: Model/Deep Learning/Technique

This paper handles a similar issue faced by Net2Net, but handles it better.

Network morphism: morph a well-trained neural network to a new one so that its network function can be completely preserved.

Problem of Net2Net (IdMorph):

• Potentially problematic due to the sparsity of the identity layer, and might fail sometimes
• Fails in dealing of the non-linearity in a neural network. It cannot be applied to other commonly used activation functions, such as Sigmoid and TanH, since the idempotent condition is not satisfied.

NetMorph:

• A deconvolution-based algorithm for network depth morphing
• Able to asymptotically fill in all parameters with non-zero elements