"Git object file is empty" Error

[vahid-dan]

A Git exception that happens once in a while working with sensor gateways is "Git object file is empty" error. We are not sure about the origin of the error, but apparently it only has happened on battery-powered gateways that are up just for a short time window (about 15 minutes), the second Git pull-commit-push sequence happens just 7 minutes after the first one, and the gateway is forced to shut down several times a day. A plausible cause may be that if the Git process is not finished with the task for some reason, the shutdown command or another Git process sequence may lead to an ungraceful exit and corrupt the Git files.

We can see an actual sample of the problem in the following log.

• On "12/20/21 00:29:31", the Git works fine.
• On "12/20/21 09:29:32", the Git has an unknown error "\00\00...".
• 7 minutes later, on "12/20/21 09:36:43", when the second Git pull-commit-push sequence is scheduled, the Git has "object file is empty" error.

############################ bjorn - 12/20/21 00:29:31 EST ############################

Data:

[bvre-platform-data 06e2936] 12/20/21 00:29:31 EST: Git Backup
 1 file changed, 36 insertions(+)
Already up to date.
To github.com:FLARE-forecast/BVRE-data.git
   55c825d..06e2936  bvre-platform-data -> bvre-platform-data

Logs:

Already up to date.
[bjorn-logs 8c5553cf] 12/20/21 00:29:31 EST: Logs
 5 files changed, 182 insertions(+)
To github.com:FLARE-forecast/BVRE-data.git
   ce960df0..8c5553cf  bjorn-logs -> bjorn-logs
[bjorn-logs 80656b80] 12/20/21 00:29:31 EST: Logs
 2 files changed, 6 insertions(+)
To github.com:FLARE-forecast/BVRE-data.git
   8c5553cf..80656b80  bjorn-logs -> bjorn-logs

############################ bjorn - 12/20/21 09:29:32 EST ############################

Data:

\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
############################ bjorn - 12/20/21 09:36:43 EST ############################

Data:

error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
fatal: loose object 45cf0fcc450fc137e7d30ee9cc097314daaa3fe5 (stored in .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5) is corrupt
error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
fatal: loose object 45cf0fcc450fc137e7d30ee9cc097314daaa3fe5 (stored in .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5) is corrupt
error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
error: object file .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5 is empty
fatal: loose object 45cf0fcc450fc137e7d30ee9cc097314daaa3fe5 (stored in .git/objects/45/cf0fcc450fc137e7d30ee9cc097314daaa3fe5) is corrupt

Logs:

Already up to date.
[bjorn-logs 4c430d90] 12/20/21 09:36:43 EST: Logs
 5 files changed, 215 insertions(+)
To github.com:FLARE-forecast/BVRE-data.git
   80656b80..4c430d90  bjorn-logs -> bjorn-logs
[bjorn-logs 0f1accb9] 12/20/21 09:36:43 EST: Logs
 2 files changed, 48 insertions(+)
To github.com:FLARE-forecast/BVRE-data.git
   4c430d90..0f1accb9  bjorn-logs -> bjorn-logs

A workaround for fixing the issue is to move the local Git repo to a temporary path, clone it again from the remote (GitHub repo), and transferring the observations or logs that may be missing while the Git problem was existing.

Our assumption is that if we have a larger time window for Git tasks and don't start another Git process while a previous one is still running, this problem should not occur in the first place. So, we should increase the time window, if possible, and run the Git pull-commit-push sequence just once in the time window.