Fast photographic style transfer

This is the torch implementation code for a fast photographic style transfer method based on Fast-Neural-Style (http://cs.stanford.edu/people/jcjohns/eccv16/)

Stylizing an image at a resolution of 852x480 approximately takes 1.40 seconds on a single GTX 1080Ti Card:

In this repo, I also provide the torch implementation of the Domain Transform (Recursive Filter) described in the paper:

Domain Transform for Edge-Aware Image and Video Processing
Eduardo S. L. Gastal and Manuel M. Oliveira
ACM Transactions on Graphics. Volume 30 (2011), Number 4.
Proceedings of SIGGRAPH 2011, Article 69.

Setup

All code is implemented in Ubuntu 16.04 with Torch7 and Lua5.1, thus some necessary packages need to be updated or installed.

First. Install Torch, then update / install the following packages:

luarocks install torch
luarocks install nn
luarocks install image
luarocks install lua-cjson

Second. Install Cuda, if you have an NVIDIA GPU and want to accelerate all operations with CUDA. Then update / install the following packages:

luarocks install cutorch
luarocks install cunn

Third. Install cuDNN to accelerate your convolution operations when using CUDA. You can copy the cuDNN libraries to /usr/local/cuda/lib64/. Then install the Torch bindings for cuDNN:

luarocks install cudnn

Pretrained Models

The pre-trained models are located in ./models/pre-trained/

Running examples

The script fast_photographic_style_transfer_single.sh lets you use a trained model to stylize example images:

bash fast_photographic_style_transfer_single.sh

Running on new images

Change the image paths in fast_photographic_style_transfer_single.sh allow you to use a trained model to stylize new images:

#run fast neural style to stylize an image 
th fast_neural_style.lua \
  -model ./models/pre-trained/newyork_night.t7 \
  -image_size 700 \
  -width 0 \
  -median_filter 3 \
  -timing 1 \
  -input_image [path to your content image] \
  -output_image [path to your temporal output image] \
  -gpu 0
  
#run post-processing step 
th StyleFusion.lua \
  -Type single \
  -content_image [path to your content image] \
  -stylized_image [path to your temporal output image] \
  -output_image [path to your final output image]

Running on videos

Change the video and frame paths in fast_photographic_style_transfer_video.sh allow you to use a trained model to stylize a new video:

#extract frames from a video
ffmpeg -i [path to your video] [path to your frames]/frame_%04d.ppm

#run fast neural style to stylize frames
th fast_neural_style.lua \
  -model ./models/pre-trained/newyork_night.t7 \
  -image_size 700 \
  -width 0 \
  -median_filter 3 \
  -timing 1 \
  -input_dir [path to your frames]/frame_%04d.ppm \
  -output_dir [path to your tmp stylized frames] \
  -gpu 0
  
#run post-processing step 
th StyleFusion.lua \
  -Type video \
  -input_pattern [path to your frames]/frame_%04d.ppm \
  -stylized_pattern [path to your tmp stylized frames]/frame_%04d.ppm \
  -output_pattern [path to your final frames]/frame_%04d.png

#create a stylized video
ffmpeg -i [path to your final frames]/frame_%04d.png [path to your video output]/stylized.mp4

Some example results

Parameters for Fast-Neural-Style

You can control the size of the output images using the -image_size flag.

By default this script runs on CPU; to run on GPU, add the flag -gpu specifying the GPU on which to run.

The full set of options for this script is described here.

Training new models

You can find instructions for training new models here. The training process takes around 2 hours on a single NVIDIA GTX 1080Ti card.

The training example is described in run_train.sh.

The code builds on Perceptual Losses for Real-Time Style Transfer and Super-Resolution by Justin Johnson, Alexandre Alahi, and Li Fei-Fei.

Slow Neural Style for Online Stylization in Arbitrary Styles

If you think train a new model takes a long time, you could use slow_neural_style.lua to stylize images in arbitrary styles and use StyleFusion.lua to play as the post-processing step. The optimization process takes roughly 60-90 seconds for an image at resolution width=512 on a single NVIDIA GTX 1080Ti card.

License

The example images are from Internet, I don't have the copyright. If you are the copyright owners and you think this has copyright conflicts, please contact me to delete the images.

This repo is free for personal or research use; for commercial use please contact me.