Skip to content

isp.md

Latest commit

 

History

History
254 lines (164 loc) · 5.61 KB

isp.md

File metadata and controls

254 lines (164 loc) · 5.61 KB

Image Signal Processing (ISP)

1. Introduction to ISP

Image Signal Processing (ISP) is the core imaging pipeline that converts raw sensor data into a visually usable image or video stream. In camera systems, the ISP sits between the image sensor (CMOS/CCD) and the encoder/display, performing a sequence of hardware-accelerated and software-controlled processing stages.

Typical ISP responsibilities include:

  • Sensor data normalization
  • Noise suppression
  • Exposure and color correction
  • High Dynamic Range (HDR) reconstruction
  • Image enhancement and sharpening
  • Format conversion for encoding or display

ISP quality directly determines:

  • Image clarity
  • Color accuracy
  • Low-light performance
  • Motion artifacts
  • Overall perceived camera quality

2. ISP Pipeline Overview

Image

A typical ISP pipeline processes frames in a strict, ordered flow. While implementation varies by vendor, the conceptual stages remain consistent.

3. Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF)

AE / AWB / AF form the 3A control loop, continuously adjusting ISP parameters based on scene statistics.

3.1 Auto Exposure (AE)

Goal: Maintain optimal brightness without overexposure or underexposure.

AE Controls

  • Sensor analog gain
  • Sensor digital gain
  • Exposure time (shutter)
  • ISP digital gain

AE Statistics

  • Luma histogram
  • Average luminance
  • Weighted metering zones

AE Control Loop

  1. Collect luminance statistics
  2. Compare against target brightness
  3. Adjust exposure parameters
  4. Apply changes on next frame

Note: Common AE challenges:

  • Flicker under artificial lighting (50/60Hz)
  • Overreaction in high-contrast scenes
  • Exposure pumping during motion

3.2 Auto White Balance (AWB)

Goal: Ensure neutral colors under different lighting conditions.

AWB Techniques

  • Gray-world assumption
  • White-patch detection
  • Color temperature estimation (CCT)

AWB Outputs

  • R/G/B gain ratios
  • Color temperature index
  • Scene classification (indoor / outdoor)

AWB directly affects:

  • Skin tone accuracy
  • Color cast (yellow / blue / green)
  • HDR color stability

3.3 Auto Focus (AF)

Goal: Achieve maximum image sharpness at the focal plane.

AF Methods

  • Contrast-based AF (most common in embedded systems)
  • Phase-detection AF (sensor-dependent)
  • Fixed-focus (no AF loop)

AF Metrics

  • High-frequency edge energy
  • Laplacian variance
  • Gradient magnitude

AF loop is often slower than AE/AWB and operates on fewer frames.

4. Noise Reduction, HDR, and Color Processing

4.1 Noise Reduction (NR)

Noise originates from:

  • Sensor read noise
  • Shot noise (low light)
  • High analog gain

Types of Noise Reduction

Type Domain Description
2D NR Spatial Smooth noise within a single frame
3D NR Temporal Reduce noise across multiple frames
Y NR Luma Preserve edges while reducing grain
C NR Chroma Suppress color speckle

Trade-off:

  • Too much NR → blurry image
  • Too little NR → grainy image

4.2 High Dynamic Range (HDR)

Goal: Preserve detail in both dark and bright regions.

HDR Approaches

  • Multi-exposure HDR (long + short exposure)
  • Line-interleaved HDR
  • Frame-interleaved HDR

HDR Processing Steps

  1. Capture multiple exposures
  2. Align frames (motion compensation)
  3. Merge using tone-mapping curves
  4. Compress dynamic range for display

HDR tuning is highly scene-dependent and computationally expensive.

4.3 Color Processing

Includes:

  • Color Correction Matrix (CCM)
  • Gamma correction
  • Saturation and hue control

Color Correction Matrix (CCM)

A 3×3 matrix mapping sensor RGB to standard color space:

[R']   [a b c] [R]
[G'] = [d e f] [G]
[B']   [g h i] [B]

Proper CCM tuning ensures:

  • Accurate skin tones
  • Correct brand colors
  • Natural-looking images

5. ISP Tuning Concepts

ISP tuning is the process of calibrating ISP parameters to achieve desired image quality under various conditions.

5.1 Tuning Dimensions

Dimension Examples
Lighting Daylight, tungsten, low light
Scene Indoor, outdoor, backlit
Motion Static vs moving objects
Sensor Different CMOS characteristics

5.2 Tuning Artifacts

  • Register dumps
  • XML / JSON tuning tables
  • LUTs (Gamma, Tone Mapping)
  • Per-ISO parameter sets

5.3 Typical Tuning Workflow

  1. Capture RAW frames
  2. Analyze histograms & noise
  3. Adjust ISP parameters
  4. Validate on real scenes
  5. Iterate across lighting conditions

ISP tuning is sensor-specific and lens-dependent. A tuning profile rarely transfers cleanly between hardware variants.

6. Practical Notes for Embedded Camera Systems

  • ISP runs in hard real-time
  • Memory bandwidth is critical
  • Latency impacts AE/AWB stability
  • Poor tuning cannot be fixed in post-processing

In IP cameras, ISP output quality directly impacts:

  • H.264/H.265 compression efficiency
  • AI detection accuracy
  • User-perceived video quality

7. Summary

ISP is not just an image filter chain—it is a closed-loop, real-time imaging system combining:

  • Sensor physics
  • Signal processing
  • Control theory
  • Perceptual tuning

A well-tuned ISP can make a low-cost sensor look premium, while poor tuning can ruin even high-end hardware.