Kubernetes: can't create '/var/lib/clamav/bytecode.cvd': Read-only file system

[chasemaier]

Deployed to a kubernetes cluster based on https://github.com/filefrog/clamav/blob/master/deploy/k8s.yml

I'm seeing (the second time the DaemonSet starts), that:

---

The init container seems to work as expected.

$ kubectl logs -f clamd-krlld -c init

seeding main.cvd from image...
'/var/lib/clamav.master/main.cvd' -> '/var/lib/clamav/main.cvd'
seeding daily.cvd from image...
'/var/lib/clamav.master/daily.cvd' -> '/var/lib/clamav/daily.cvd'
seeding bytecode.cvd from image...
'/var/lib/clamav.master/bytecode.cvd' -> '/var/lib/clamav/bytecode.cvd'
done

---

The freshclam container seems to encounter some warnings, but remains in the Running state.

$ kubectl logs -f clamd-krlld -c freshclam

Wed Mar 24 21:02:32 2021 -> --------------------------------------
Wed Mar 24 21:02:32 2021 -> freshclam daemon 0.102.1 (OS: linux-musl, ARCH: x86_64, CPU: x86_64)
Wed Mar 24 21:02:32 2021 -> ClamAV update process started at Wed Mar 24 21:02:32 2021
Wed Mar 24 21:02:32 2021 -> ClamAV update process started at Wed Mar 24 21:02:32 2021
Pruning unwanted or deprecated database file bytecode.cvd.
Wed Mar 24 21:02:32 2021 -> daily database available for update (local version: 25749, remote version: 26119)
Wed Mar 24 21:02:32 2021 -> daily database available for update (local version: 25749, remote version: 26119)
Wed Mar 24 21:02:34 2021 -> WARNING: downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^downloadFile: file not found: https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> ^getpatch: Can't download daily-25750.cdiff from https://database.clamav.net/daily-25750.cdiff
Wed Mar 24 21:02:34 2021 -> WARNING: Incremental update failed, trying to download daily.cvd
Wed Mar 24 21:02:34 2021 -> ^Incremental update failed, trying to download daily.cvd
Wed Mar 24 21:02:37 2021 -> Testing database: '/var/lib/clamav/tmp/clamav-d9833945f9e5f017777108eb8f3654f5.tmp-daily.cvd' ...
Wed Mar 24 21:02:37 2021 -> Testing database: '/var/lib/clamav/tmp/clamav-d9833945f9e5f017777108eb8f3654f5.tmp-daily.cvd' ...
Wed Mar 24 21:02:46 2021 -> Database test passed.
Wed Mar 24 21:02:46 2021 -> Database test passed.
Wed Mar 24 21:02:46 2021 -> daily.cvd updated (version: 26119, sigs: 3965409, f-level: 63, builder: raynman)
Wed Mar 24 21:02:46 2021 -> daily.cvd updated (version: 26119, sigs: 3965409, f-level: 63, builder: raynman)
Wed Mar 24 21:02:46 2021 -> main.cvd database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Wed Mar 24 21:02:46 2021 -> main.cvd database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Wed Mar 24 21:02:46 2021 -> ERROR: NotifyClamd: Can't connect to clamd on 0.0.0.0:3310: Connection refused
Wed Mar 24 21:02:46 2021 -> !NotifyClamd: Can't connect to clamd on 0.0.0.0:3310: Connection refused
Wed Mar 24 21:02:46 2021 -> --------------------------------------

---

But the clamd container runs into an issue -- it appears to want to WRITE to the `` file but the file system is read-only.

$ kubectl logs -f clamd-krlld -c clamd

seeding bytecode.cvd from image...
cp: can't create '/var/lib/clamav/bytecode.cvd': Read-only file system

---

My thought is this line from the freshclam is concerning: Pruning unwanted or deprecated database file bytecode.cvd.

Perhaps the init container creates the file, freshclam deletes it, and clamd attempts to recreate it?

Have you encountered this or is there a clear path forward? I investigated the clamd configuration file options and don't see anything that would disable attempting to write to that location while still maintaining the Bytecode yes default which sounds like the suggested configuration.

I think I roughly understand the purpose of the 3 containers (init gets files ready, freshclam probably keeps them up to date over time, and clamd does the actual scanning) -- so the separation of readOnly file mounts makes sense. Just not sure how to work around this issue -- would it be to remove Bytecode no from the freshclam config?

---

Finally, is ERROR: NotifyClamd: Can't connect to clamd on 0.0.0.0:3310: Connection refused anything to be concerned about?

[chasemaier]

After commenting out Bytecode no from the freshclam config, it does seem to start successfully. Any chance you can speak to why that config option was added as an explicit override on that particular config?

EDIT: Recreation of the pod during changes to the DaemonSet is likely what actually made it restart okay, when clamd container within the new pod OOMs or resarts in place it re-encounters this same error still, even with the modified freshclam config.