Progress

PWC Net

Network architecture

Status: Done.

Link: https://github.com/NeedsMorePie/interpolator/tree/master/pwcnet

TODOs:

• Backward warp.
• Cost volume.
• Feature pyramid.
• Estimator network.
• Context network.
• Loss function.
• Tensorboard summaries.
• Multi GPU.

Dataset

Status: Started and IPR.

Link: https://github.com/NeedsMorePie/interpolator/tree/master/data/flow

TODOs:

• TF record conversion.
• Brightness/hue/saturation/gamma/gain augmentations.
• Stretching augmentations.
• Flipping augmentations.
• Testing on Sintel.
• Testing on Flying Chairs.
• Testing on Flying Things 3D.
• Sintel test data AEPE.

Optimizations

Status: Continually ongoing.

TODOs:

• Cost volume optimizations -- better Tensorflow batching.
• Feature pyramid optimizations -- better Tensorflow batching.
• Custom Tensorflow Op for cost volume.
• Custom Tensorflow Op for backwards warping.

Experiments

Status: Started

TODOs:

• Unsupervised fine tuning.
  • Integrating the UnFlow loss.
  • Training with the UnFlow loss.
• Splined optical flow.
• Context extraction from more layers (maybe deeper ones).

Context aware interpolation

Gridnet Architecture

Status: Done.

Link: https://github.com/NeedsMorePie/interpolator/tree/master/gridnet

TODOs:

• Lateral connection.
• Downsampling connection.
• Upsampling connection.

Dataset

Status: Started and IPR.

Link: https://github.com/NeedsMorePie/interpolator/tree/master/data/interp

TODOs:

• TF record conversion --in frame triplets.
• Test on DAVIS.
• Flipping (x, y, temporal) data augmentations.
• Pipeline to gather and process Youtube videos.

Forward warping

Status: Started and IPR.

Link: https://github.com/NeedsMorePie/interpolator/tree/master/common/forward_warp

TODOs:

• Tensorflow differentiable naive forward warp.
• Non-differentiable C++ warp, hole-filling, and conflict-resolution algorithm wrapped in Python.
• Custom Tensorflow Op for non-differentiable hole-filling, and conflict-resolution forward warp.

Overall architecture

Status: Started.

TODOs:

• Feature extraction from a pre-trained VGG network.
• PWCNet integration.
• Bidirectional warp of image + features.
• Loss function with laplacian pyramid and perceptual loss.
• Tensorboard summaries.
• Instance normalization.

Inference framework

Tensorflow models

Status: Not started.

TODOs:

• Freezing the tensorflow models/graphs.
• C++ server that links the Tensorflow runtime to execute frozen graphs.
• Interop with the front end.

Front end

Status: Started. Not currently IPR.

Link: https://github.com/boxofpasta/interpolator-interface

TODOs:

• Web layout.
• File input and asset browser.
• Inbetweening timeline and bezier curves.
• Inbetweening preview.
• Interop with the inference backend.
• Save/export.