unstable frame rate in x64 release

[fxpena]

The frame rate in the new release is only stable for 30 Hz. When I try to use 20 Hz the actual frame rate oscillates on each recorded frame being either 20 Hz or 100 Hz. Same problem occurs at the 15 Hz and 10 Hz.
This problem doesn't occur when I use the previous release.

[DenisPolygalov]

How do you calculate "the actual frame rate" and it's oscillations? Is your calculation based on time difference between timestamps in the timestamp.dat file or by using some other method?

[fxpena]

My calculation is based on time difference between timestamps in the timestamp.dat file. Also, my calculation matches the value shown in the software GUI. In E for the FPS box.

[fxpena]

Sorry, I closed the issue on accident.

[DenisPolygalov]

@daharoni Daniel, I suggest to be very careful of when you accepting the pull requests. My fork was never intended to be accepted by the upstream. And @phildong wrote on top of that. I suspect there was something in few recent pulls that messes up a lot with the recording code. Also I suspect there are more than a single problem. For example contradicting to information provided by @fxpena - in our case there was always an oscillation of frame rate if I calculated it based on timestamps.dat. Always - mean even in the old release. At the same time there is a bug in GUI - misplaced items in the FPS list box entry. The miniscope camera was programmed to 20 FPS at the moment of launching despite 30 FPS displayed in the GUI, but that not affect actual data. Anyway, mess everywhere.... simpler to re-write from scratch...

[daharoni]

@DenisPolygalov, Thanks for the comment. I have looked through the major changes to the code from @phildong's pull requests and I don't see any changes that would fundamentally change the function of the software negatively. I think the problems you mentioned were present before your modifications. I chat with @phildong quite often and he is in the lab of one of the other original Miniscope developers. While the original Miniscope DAQ code I wrote is a bit of a mess, and still has some bugs, I think @phildong has put his pull request through the necessary tests. I'll keep an eye out to make sure no additional bugs arise from it.

We are now working on some new DAQ software which will replace this package at some point this year which should be more robust and better written.

[phildong]

@DenisPolygalov you are right about the frame rate issue and we observed the same thing! However I believe it's an opencv issue rather than the code themselves, which makes perfect sense why both your version and mine had same issue. See #14 for my two cents.

[ckemere]

Not sure if anyone is still tracking this issue. I have a question about the framerate instability - it seems like the DAQ board should produce a pretty reliable frame rate, but that the timestamps (which are added in software well after the data has reached the computer) might not be. Has anyone validated that the frames really do come too slowly?

(This was super common with PointGrey cameras and libusb or GigE acquisition in Linux.)

Thanks!

[DenisPolygalov]

@ckemere AFAIK the DAQ board is a hardware running an OS-less firmware, therefore it can produce stable frame rate assume the firmware was implemented correctly. On the OS side (host PC) we have a Windows - which is not a hard real time OS (same as Linux) therefore stable frame rate cannot be guarantied in principal. Period. Seems like older version of OpenCV (2.4) is a little bit beter according to comments of other developers, but this is just shamanic dance around Windows 95 PC trying to fix it's freeze. The correct (but still partiall) solution for unstable frame rate is to assign timestamps on the DAQ board's side for example. Noone can validate that "the frames really do come too slowly" or "stable" at any given time. The frame rate can be stable on some particular PC - it depend on what other software is present on that PC and what it is doing. The frame rate can be stable for a an hour, and then you see a whole second without any frames. Or on other PC it can be unstable continiously because it it too slow or because there is a very "smart" antivirus software inside or because Microsoft decided to upgrade Windows in a background.

[DenisPolygalov]

Another partial solution of this problem is used in new Inscopix devices - encapsulate video frame acquisition and preprocessing in one box running Linux (or any other OS with open source code). Doing so will allow developers to do tricks inside of the OS (because it is open source) in order to compensate of non-real time nature of the OS and adjust/interpolate timestamps post hoc. This will remove unstable frame rate problem from end-user's eyes.
Possible real solution is to do not only frame acquistion but also frame processing directly on a hardware by using bare metal non-OS firmaware running on a common CPU or (even better but more expensive) - run frame pre-processing on a FPGA (similar to what Neuralynx doing for it's electrophysiology rigs).

[daharoni]

Hi (copying here what I just emailed to @ckemere),
Yes you are correct that the time stamp is generated right after the openCV grab function. This time stamp is driven by the high resolution CPU timer which is usually accurate to about 1ms. Depending on the openCV version, we can get very very consistent timing of grab() but I also have noticed on the newest versions of openCV the timing of grab() can have pretty huge fluctuations (+30ms) in some cases. USB also has some variable latency but USB3 should keep this under about 5ms at most (usually well under 1ms).

The DAQ box is just jamming data into the computers USB driver without any flow control so if the USB driver cannot keep up with the data rate coming out of the DAQ it will effectively drop a frame and that frame will never trigger a openCV grab(). This means that with running an older openCV version using the Miniscope DAQ software you should see timestamps with interframe intervals all within a couple milliseconds of each other except for when the USB driver completely drops a frame. When this happens you will see a jump in the timestamp by roughly 1 interframe interal. For excample, if you are running at 30FPS, almost all timestamps should be around 33ms but once in a while (on my computer it happens about once every 10,000 frames) a frame is dropped and we see an interframe intervale of ~66ms.

For the behavior camera, the Miniscope software has no control over its frame rate and with many webcams, this framerate can vary pretty wildly during recording. We usually see much larger variation in timestamps for behavior cameras due to this (we are designing out own behavior cameras now for this reason).

You should end up seeing the same number of Miniscope and behavior frames in the .avi recordings and in the timestamp.dat file. Even with the newer openCV version you should still have accuracy of about 1 frame in terms of timing on both Miniscope and behavior frames.

[daharoni]

Oh I should also mention that the DAQ board does have a digital output on one of its SMA connectors which toggles between high and low 3.3Vs on every frame acquired. This happens the moment an "end of frame" arrives to the DAQ from the Miniscope and should have microsecond accuracy. If more accurate timing is needed you can read this signal into other hardware to timestamp it. There is an issue with the occasional dropped frames though. The DAQ will still toggle its output even if the computer's USB ends up dropping a frame. David Tank's cScope paper discusses this and shows a simple solution to accounting for this.

[ckemere]

For behavior, we basically use two webcams - the Logitech C920 and the PS2 Eye. In other contexts, both have produced quite regular frame rates, so I'm definitely curious to dig into the OpenCV code to understand how it is introducing variability. (It seems impossible unless it's actively changing the framerate or operating in a request-one-frame-at-a-time mode, both of which seem unlikely.) I'll do some tests next week with a flashing LED and also look at the Miniscope output pulses. Thanks for the leads!