rootless podman with runc gives inconsistent error messages when using options that depend on missing cgroup v2 controllers

[ricardobranco777]

Issue Description

Rootless Podman with runc gives inconsistent and misleading error messages when using options that rely on cgroup v2 controllers that are not delegated to the user. Instead of reporting that the controller is unavailable, I get low-level errors like missing files under /sys/fs/cgroup or JSON parse errors.

Steps to reproduce the issue

On a system with memory not being a delegated controller to the user, run repeatedly:

podman --runtime /usr/bin/runc run --rm -it --memory 1G debian

Fedora delegates these controllers by default so it needs a different reproducer: cpu io memory pids.

Describe the results you received

$ podman --runtime /usr/bin/runc run --rm -it --memory 1G debian
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

$ podman --runtime /usr/bin/runc run --rm -it --memory 1G debian
Error: /usr/bin/runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/user.slice/libpod-a1104727d5dc54b23f3bb197babf376fdd431c2dd2f961f604b37f65f418b21d.scope/memory.swap.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected

With crun I get a consistent error message, though a bit misleading:

$ podman --runtime /usr/bin/crun run --rm -it --memory 1G debian
Error: /usr/bin/crun: open `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

podman info output

host:
  arch: amd64
  buildahVersion: 1.41.5
  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: unknown'
  cpuUtilization:
    idlePercent: 99.56
    systemPercent: 0.34
    userPercent: 0.1
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20251020"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2047
  hostname: opensuse
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.17.3-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 54227550208
  memTotal: 62817325056
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.1.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: /usr/bin/runc
    package: runc-1.3.2-2.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.3.2
      commit: v1.3.2-0-gaeabe4e711d9
      spec: 1.2.1
      go: go1.25.3
      libseccomp: 2.6.0
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20250611.0293c6f-3.2.x86_64
    version: |
      pasta 20250611.0293c6f-3.2
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.3-1.1.x86_64
    version: |-
      slirp4netns version 1.3.3
      commit: unknown
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.6.0
  swapFree: 62817320960
  swapTotal: 62817320960
  uptime: 39h 5m 15.00s (Approximately 1.62 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/ricardo/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ricardo/.local/share/containers/storage
  graphRootAllocated: 915833237504
  graphRootUsed: 422961520640
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ricardo/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.2
  Built: 1759469198
  BuiltTime: Fri Oct  3 07:26:38 2025
  GitCommit: ""
  GoVersion: go1.25.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

openSUSE Tumbleweed 20251020

Additional information

No response

[ricardobranco777]

Tracking in openSUSE as https://bugzilla.opensuse.org/show_bug.cgi?id=1252432

[danishprakash]

Having two different error messages appears to be a problem, and the JSON parse error seems to be the actual issue here. Looks like the fallback to read oci-log fails silently when reading from the pipe fails. I'll take a look at this.

[Honny1]

Could the issue be on the conmon side? WDYT? @jnovy

[jnovy]

Yes @Honny1 - there is a race condition in conmon: containers/conmon#617

[Honny1]

@ricardobranco777 Can you please verify that the issue is fixed with the new conmon?

[ricardobranco777]

@ricardobranco777 Can you please verify that the issue is fixed with the new conmon?

I ran make dbuild and copied bin/conmon to /usr/bin and now I consistently get:

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

[jnovy]

@ricardobranco777 Can you please retest with containers/conmon#622 ?

[ricardobranco777]

@ricardobranco777 Can you please retest with containers/conmon#622 ?

Still getting:

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...