Neural Style Transfer on a RESNET-50 architecture

Abstract

This project attempts Neural Style Transfer with the new constraint of sticking to ResNet Architecture.

This work entails a mini research project on aiming to replicate the widely accepted NST performance by commonly used models like VGG or TensorFlow's NST model.

ResNet is not inherently optimal for NST as its capture of feature is not as robust as other architectures.

This can be illustrated be preliminary performance using ResNet50 for NST with standard loss parameters.

I have analyzed different layers of ResNet architecture to find the optimal content and style layers for NST as well as implemented the content and style losses for NST.

Observations in NST trials with RESNET50

Trials by style weight and content weight gave bad results so used the optimal values provided in Kaggles documentation of RESNET50 NST Main trials were in learning rate and layers chosen. Increased learning rate from 0.01 to 20 which made the change in image appear drastically every 100 iterations

Style	Conv(block 2-block 3x2-block 4)	ReLu(blocks 2,3,4,5)

Testing ReLu layers on geometrically defined style images yielded much better results wrt conv layers

Test image	Conv	ReLu

Experiments with weighted ratio of Conv to ReLu layers

Conv	ReLu	Relu+Conv

Some trials with excessive style weight (e8-e12)

Improvements to model robustness:

After analyzing the layerwise feature capture, it was observed that combined conv+relu layers at later stages in the model provide the most robust feature capture.

style_layers = ['conv1_relu','conv2_block1_1_conv',

'conv2_block2_1_relu', 'conv2_block2_1_relu',

'conv2_block3_3_conv','conv2_block2_2_relu',

'conv4_block2_1_conv','conv2_block2_1_relu',

'conv2_block2_1_relu', 'conv2_block3_3_conv',

'conv2_block2_2_relu', 'conv4_block2_1_relu',

'conv4_block2_2_conv', 'conv4_block2_2_relu',

'conv4_block2_3_conv’] 

Trials moving style layers higher to increase robustness.

Before	After