[Discussion] Add best practice check for browser-compatible video codecs in ImageSeries external files

[h-mayorquin]

A core goal of DANDI and the NWB ecosystem is making neuroscience data not just archived but genuinely accessible and reusable. For behavioral video stored as ImageSeries external files, the choice of video codec has a direct impact on storage efficiency and interoperability. Older or camera-native codecs like MJPEG, MPEG-4 Part 2 (mp4v), and DIVX produce significantly larger files than modern alternatives like H.264 for the same visual content, and are poorly supported across platforms and tooling. Modern codecs like H.264, VP9, and AV1 have near-universal support. In practice, most labs don't intentionally choose these older codecs; they inherit them from camera SDK defaults (FLIR SpinView, Basler pylon) or OpenCV scripts. Cameras favor MJPEG because each frame encodes independently, which is computationally cheap enough for onboard hardware and avoids latency or data loss propagation during capture - but these advantages are irrelevant once the data is archived.

The NWB Inspector already checks that ImageSeries external files exist and use relative paths. A natural addition would be a best practice check that probes the codec of each referenced video file and suggests re-encoding to a widely supported modern codec when an inefficient or poorly supported one is detected. Since most of these files are already lossy-compressed, re-encoding to H.264 actually improves compression (smaller files) with negligible additional quality loss at high quality settings (CRF 17-18). The check message could include a ready-to-use ffmpeg command, e.g. ffmpeg -i input.avi -c:v libx264 -crf 18 -pix_fmt yuv420p output.mp4. For the rare case of intentionally lossless video (FFV1, raw), users can ignore the suggestion as with any best practice check.

Detected codec	Suggest	Reason
MJPEG	H.264	No inter-frame compression, very large files, poor tooling support
MPEG-4 Part 2 (mp4v, DIVX, XVID)	H.264	Outdated lossy codec, larger files and worse support than H.264
Raw / uncompressed	H.264	No compression at all, enormous files
FFV1	H.264	Lossless and not widely supported; note that re-encoding is lossy
H.264, VP8, VP9, AV1	No suggestion	Already widely supported and efficient

[CodyCBakerPhD]

Thanks for the detailed writeup

We've seen many of these less optimal codecs used lately and make strong recommendations to the lab for improvements

Sometimes it's a problem of education (just use this flag on ffmpeg that you didn't know existed) sometimes its a limitation of hardware setup

One point of friction you may find is that certain scientists believe raw/lossless videos give better results for their own processing purposes, such as pose estimation - I think that is the state of many such videos currently on DANDI

So what we would need to dissuade them is definitive proof (perhaps similar to the types of figures @alejoe91 did for spike sorting vs. compression for Wavpack), otherwise

I wanted to push on the latter this year, but for now a definitive next step would be to write a best practice recommendation and inspector check RECOMMENDING (perhaps not requiring until we discuss further as a group amongst the archive engineers - a CRITICAL level would invalidate many existing Dandisets I believe)

[CodyCBakerPhD]

I will also note that in discussions with certain groups, they also really don't notice or care about these low-level details and seemed to believe that tools like the converters or DANDI CLI itself should just 'do this' on upload

[pauladkisson]

I think an NWB inspector check would be a great idea! I personally had no clue that a video codec that I can play no problem locally with a tool like VLC would have any issues in the browser. If I were clued into that consideration earlier, I could have recommended appropriate codecs to the labs that I'm working with, and I doubt they would have had any problem with it.

[h-mayorquin]

I have been thinking more about this and this is my current suggestion:

The inspector check for video should follow a simple rule for codecs and a simple rule for containers depending on the lossy vs lossless nature of the codecs:

Lossy codecs:

1. Codec: if the codec is anything other than H.264, VP8, VP9, or AV1, suggest re-encoding to AV1. This covers MJPEG, MPEG-4 Part 2 (mp4v/DIVX/XVID), WMV, DV, and any other legacy codec. If the codec is already H.264, VP8, VP9, or AV1, the codec is fine.

AV1 over H.264 because it is royalty-free (no patent concerns, ever), achieves ~30-50% better compression at the same quality, and has broad browser support (Chrome, Firefox, Edge, Safari 17+). The encoding speed gap is small: SVT-AV1 at preset 8 is ~3x slower than x264 medium while producing ~30% smaller files. DANDI is a write-once, read-many archive, so paying a one-time encoding cost for permanently smaller files is a good tradeoff. That said, I think it is fine to allow H.264 if the data is already provided in that format.

2. Container: if the container is already MP4 or WebM, keep it. Otherwise (AVI, MKV, MOV, WMV, FLV), suggest remuxing or re-encoding into MP4. Remux is sufficient when the codec is already browser-compatible (e.g., H.264 in AVI or MKV); re-encoding is needed when both codec and container need to change.

MP4 and WebM are the only containers browsers can play. Most legacy scientific video is in AVI (camera SDK default) or MKV, neither of which browsers support. When the codec is already H.264, remuxing to MP4 is lossless and takes seconds. When both codec and container need to change, a single re-encoding step produces AV1/MP4.

Lossless / uncompressed codecs:

1. Codec: if the codec is raw/uncompressed, suggest re-encoding to FFV1. If the codec is already FFV1, the codec is fine.

FFV1 provides ~2-3:1 lossless compression with per-frame CRC checksums, strictly better than raw in every dimension (smaller files, same pixel values, added integrity checks). It is the established archival standard for lossless video, adopted by the Library of Congress as a Preferred Format, IASA TC-06 guidelines, the Austrian Mediathek, meemoo, and the Rockefeller Archive Center. It supports high bit depths and scientific pixel formats. Re-encoding FFV1 to a lossy codec destroys the scientific integrity of the original data, so the inspector check should note this clearly.

2. Container: no suggestion needed. The container (AVI vs MKV) makes no practical difference for end users since all tools that read FFV1 (PyAV, OpenCV, VLC) handle both transparently via FFmpeg.

Lossless video will never play in browsers regardless of container (no browser ships an FFV1 decoder, and the bandwidth for lossless streaming is impractical), so there is no container change that improves browser playback.

I'm torn on the lossless recommendation though. FFV1/MKV is the right external format, but for NWB specifically there's an argument that lossless video frames could be stored internally as HDF5/Zarr datasets (with blosc/zstd compression achieving ~1.5-2:1, close to FFV1's ~2-3:1). Internal storage eliminates the broken-reference problem and, with Zarr, enables cloud-native random frame access which is probably the reason why people use lossless formats. The tradeoff is losing video-specific metadata (color space matrices, chroma subsampling, per-slice CRC checksums) and standard video tooling compatibility but that could be improved on the nwb-schema. I plan to open an issue on nwb-schema to discuss this further, as it touches on the broader question of how NWB should handle video data long-term.

[CodyCBakerPhD]

Lossy codecs:

Codec: if the codec is anything other than H.264, VP8, VP9, or AV1, suggest re-encoding to AV1. This covers MJPEG, MPEG-4 Part 2 (mp4v/DIVX/XVID), WMV, DV, and any other legacy codec. If the codec is already H.264, VP8, VP9, or AV1, the codec is fine.

AV1 over H.264 because it is royalty-free (no patent concerns, ever), achieves ~30-50% better compression at the same quality

I do not believe we should make such a strong recommendation for AV1 without more effort (especially in the area of pose quality assessment) to verify understanding of lossy vs. lossless, or ensuring similar effectiveness on neurophysiology-specific videos. Some labs, especially, might be strongly in favor of H.264 so no need to spark division there.

Also, the estimates of 30-50% are taken from what? A published source? AI summary? Has the figure been estimated using systematic re-encodings of DANDI-specific videos (which are quite different than typical videos)?

Note also that the topic is even more nuanced than just straightforward encodings. Sometimes hardware performance (such as how fast a GPU can write the video) is a key concern, would need to test for differences between the suggested codecs. There are practically dozens of extra hyperparameters (rc-lookahead, temporal-aq, surfaces, min/max rates, buffer size, so on) that can effect lossy performance (encoding speed, decoding speed, video quality, compression ratio) depending on codec, these are often chosen and tuned by a lab. So to say 'AV1 beats all' is way too strong and would need a lot more evidence to back it up.

I would simply say instead, "if the codec is anything other than H.264, VP8, VP9, or AV1, then suggest re-encoding to any of those." And add a note about patent concerns for H.264 just to raise awareness.

[CodyCBakerPhD]

@h-mayorquin You will probably get more eyes and opinions if you cross-post to bids-standard/bids-specification#1771

I know some of these concerns are DANDI-specific, but they affect the broader community as well

[yarikoptic]

I would somewhat second @CodyCBakerPhD 's pointer to BIDS that whatever is decided should ideally be consistent between NWB and BIDS, and definitely should not be contradictory (e.g. nwb inspector recommends something BIDS does not support or explicitly advises against )

[yarikoptic]

also re

MJPEG H.264 No inter-frame compression, very large files, poor tooling support

IIRC, this (mjpeg) is exactly what I think we were considering/discussing (with @talmo @satra within ember-tools-sleap slack channel) for using video container to collect training samples of images from the video desired for doing sleap estimation instead of collecting thousands of individual image files. @talmo could you please update us here on what we (you ;) ) ended up deciding on, and whether it was implemented.

[h-mayorquin]

@yarikoptic pointed out that if .mkv contains an open and browser friendly inside (like .webm) it should be fine as we don't want to ask the users to remux.

[yarikoptic]

Example to the above:

• https://datasets.datalad.org/repronim/ReproTube/web/#/channel/ReproNim/video/jzPISfLsaPo?tab=local&wide=1
• which is just a <video>

and here is media info

General
Unique ID                                : 310945486975038975477890082556287732236 (0xE9EDEC2545B43A66F045A1F543014A0C)
Complete name                            : ./ReproNim/videos/2025/12/2025-12-05_littleofsensoring/video.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 116 MiB
Duration                                 : 45 min 18 s
Overall bit rate                         : 357 kb/s
Frame rate                               : 25.000 FPS
Writing application                      : Lavf61.7.100
Writing library                          : Lavf61.7.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 3 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 3 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 45 min 18 s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Title                                    : ISO Media file produced by Google Inc.
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
VENDOR_ID                                : [0][0][0][0]

Audio
ID                                       : 2
Format                                   : Opus
Codec ID                                 : A_OPUS
Duration                                 : 45 min 18 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 32 bits
Compression mode                         : Lossy
Delay relative to video                  : 7 ms
Language                                 : English
Default                                  : Yes
Forced                                   : No

[h-mayorquin]

@CodyCBakerPhD

I would simply say instead, "if the codec is anything other than H.264, VP8, VP9, or AV1, then suggest re-encoding to any of those." And add a note about patent concerns for H.264 just to raise awareness.

I am fine with making the framing weaker for the inspector check.

I think I emphasized performance above, which as you mentioned would need a better defense. I think a better defense of AV1 as the default recommendation can be based on openness instead: AV1 is fully royalty-free (patents cross-licensed by the Alliance for Open Media), its specification is freely available on GitHub under a permissive license, and its reference encoder/decoder (libaom) is BSD-licensed and publicly accessible via git. H.264, on the other hand, has patents managed by MPEG LA's patent pool, its specification is published by ITU-T/ISO (freely downloadable but not redistributable or modifiable), and its open-source coverage through Cisco's OpenH264 depends on a single company's continued willingness to pay the royalties on behalf of its users. I think AV1 is better aligned with open science and DANDI's mission.

So to say 'AV1 beats all' is way too strong and would need a lot more evidence to back it up.

I would like to constrain the goal to move forward with this. I think there are two cases here that we should separate:

• Case 1: People who don't care about codecs and ended up with legacy or uncommon codecs like Indeo, WMV, DV, or MPEG-4 Part 2 because that's what their camera SDK or pipeline outputs by default. Same for containers, where people end up with AVI or MKV simply because that's the default output, not because they chose it deliberately. Another common case is .mov, which is the default on macOS/iOS and many scientific cameras. It is structurally almost identical to MP4, so remuxing is lossless and takes seconds.
• Case 2: People who are opinionated
  about their codec choice, whether that is lossless/uncompressed for scientific reasons (like the MJPEG case @yarikoptic mentioned for pose estimation training, where frame independence matters) or lossy with specific tuning for their application.

I think we should solve these separately. The first step is to add a best practice recommendation with its rationale. Case 2 users who are more technically savvy can ignore it. In the same vein, we can implement an inspector check with something like: "your video uses a codec that is not open or widely supported, consider re-encoding to AV1 in an MP4 container" as a recommendation. The dandi-cli can also prohibit some of the old outdated containers or offer conversion on the spot (the way that we move videos in the dandi-cli).

DANDI might be in a good position to provide guides and recommendations for case 2 in the future, but I think we should start with 1 and slowly build knowledge from there.

Also, the estimates of 30-50% are taken from what? A published source? AI summary?

Yes, I was too loose with that, my mistake. The ~30% figure was my own estimation from re-encoding an H.264 file to AV1 with ffmpeg. The comparisons that I could find are resolution-dependent, ranging from ~10% savings at low resolutions (Netflix production data, 2025) to ~33-40% at 1080p (Uhrina et al. 2024. But I think we should not let this derail the discussion, sorry for this.

[CodyCBakerPhD]

I think I emphasized performance above, which as you mentioned would need a better defense. I think a better defense of AV1 as the default recommendation can be based on openness instead: AV1 is fully royalty-free (patents cross-licensed by the Alliance for Open Media), its specification is freely available on GitHub under a permissive license, and its reference encoder/decoder (libaom) is BSD-licensed and publicly accessible via git. H.264, on the other hand, has patents managed by MPEG LA's patent pool, its specification is published by ITU-T/ISO (freely downloadable but not redistributable or modifiable), and its open-source coverage through Cisco's OpenH264 depends on a single company's continued willingness to pay the royalties on behalf of its users. I think AV1 is better aligned with open science and DANDI's mission.

That's certainly good to educate people on, but still not strong enough on its own for a data archive to make such a single strong recommendation, especially since most labs feel very good about their choice of H.264

It is sufficient to narrow the pool of 'acceptable' codecs to the few you mention, but not enough points altogether to make a strong singular recommendation of that particular one

Now, if you want to gather a bunch of A/V-acquiring people and get a community consensus towards that end, we can certainly revisit. As of right now none of the people discussing this are actual experimentalists, nor even video processing tool developers, nor hardware engineers. A one-sided view of post-hoc video encoding for data archiving is pretty biased perspective.

Such an extended effort should also do a detailed survey of acquisition devices to ensure good performant hardware support across either onboard cameras or microcontroller capture intermediates. When we messed with this kind of thing for the Reprostim setup the tiny affordable capture device had pretty limited support for hardware-accelerated video capture.

The dandi-cli can also prohibit some of the old outdated containers or offer conversion on the spot (the way that we move videos in the dandi-cli).

Re-writing an entire dataset of videos is not a trivial operation; I'd usually offload that kind of thing to the standardization step such as NeuroConv. The closest process in terms of compute load currently in the CLI is the digestion step, which is unavoidable to upload. IDK if we want to weigh it down that much

I am fine with making the framing weaker for the inspector check.

The best practice can also list out your reasoning in a tabular fashion to help encourage people in your direction without explicitly stating one single strong preference

[h-mayorquin]

especially since most labs feel very good about their choice of H.264

Can you share a defense like this from lab? it would be useful for context and for me.