Skip to content

Home

Jump to bottom
Brian Wandell edited this page Jun 22, 2018 · 13 revisions

This project aims to assess the impact optics and sensor parameters on object detection networks. Specifically, we simulate how well machine-learning algorithms (mainly convolutional neural networks) generalize between different camera designs. The camera designs vary lenses, CFAs and other design features. Over time we expect that this repository will host analyses from multiple papers.

Dependencies

This repository depends on methods that produce realistic driving scenes iset3d and methods that implement a variety of camera designs. We use isetcloud to analyze object detection networks from Google's TensorFlow project to understand how optics and sensors choices influence machine-learning behavior.

Creating simulated scenes

  • The assets and scene are produced from CG Trader and various other sources.
  • We convert the scene and assets into a format that can be read by PBRT using RenderToolbox4 (slowish)
  • We use the modified PBRT to produce sensor irradiance image (includes lens description)
  • We use ISET to produce simulated sensor values for training and testing

Current sensor models.

  • fullRGB simulates as if we had three colors at every pixel
  • sRGB is sRGB
  • linearRGB is its name
  • rawSensor is the Bayer pattern at the sensor (linear form)
  • Demosaicking was implemented using bilinear, and then there are several JPEG compression levels.

Demosaicking algorithms that could be compared.

Noise removal algorithms

Optics is on the list; pinhole, lens, light fields

Gamma is on the list

Illumination transforms ... how do we generalize across illuminations

Dynamic range issues

Data management

We store all the data on our server (gray). The main directory with project related data is /share/wandell/data/NN_Camera_Generalization. This directory is meant to be used as 'long-term' storage it is very expensive (i.e.) slow to access.

This directory contains the following sub-directories

  • Assets contains 3D models (meshes, textures, materials), each model is in it's own sub-directory.
  • Renderings contains working folders produces by RenderToolbox4 for different recipes. These primarily contain simulated, multispectral sensor irradiance.
  • Images contains images produced by camera models from the irradiance data stored in Renderings.

Any computation that requires repetitive access to data (such as neural network training) should copy the relevant data to /scratch directory. This is a local, 1T SSD drive which reads and writes data very efficiently.

Neural network approaches

We are going to stick with TensorFlow and figure out the simple ways to script it using Keras???

TensorFlow is installed on gray, so that's good. There is another page describing more about the TensorFlow object_detection methods.

Shun used

Clone this wiki locally