Fix DNS resolution in ephemeral guests #167

gursewak1997 · 2025-12-01T23:06:35Z

Configure QEMU user-mode networking to use host DNS servers from /etc/resolv.conf instead of the default 10.0.2.3, which doesn't work when QEMU runs inside containers.

cgwalters

I'm fine with this overall if you are; a few nits.

That said, an integration test (run as part of just test-integration ephemeral) would be both easy and IMO mandatory for changes like this.

crates/kit/src/run_ephemeral.rs

crates/kit/src/qemu.rs

crates/integration-tests/src/tests/run_ephemeral_ssh.rs

gursewak1997 · 2025-12-05T21:03:00Z

Network connectivity test (podman pull) failed: stdout: Error: configure storage: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
Can switch to something simpler which doesn't require podman
Edit: Switching the test to use HTTP

cgwalters · 2025-12-05T21:46:35Z

Error: configure storage: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver

Ah yes that relates to #22 - basically all of /var needs to be equivalent to a VOLUME in docker/podman terms - it needs to be copied up to a tmpfs.

cgwalters

One nit otherwise looks sane (though I'd reiterate we probably really do want a bigger fix of --net=host per the issue, but this helps for now)

crates/integration-tests/src/tests/run_ephemeral_ssh.rs

gursewak1997 · 2025-12-09T00:34:12Z

Reworking this PR as I think it will require a few more changes.

gursewak1997 · 2025-12-12T20:09:50Z

This one's just timing out on test_to_disk_for_image_quay_io_centos_bootc_centos_bootc_stream9 test but should be good to go though

crates/kit/src/run_ephemeral.rs

crates/integration-tests/src/tests/run_ephemeral_ssh.rs

QEMU's slirp reads /etc/resolv.conf from the container namespace, which contains unreachable bridge DNS servers. On systemd-resolved hosts, it only has 127.0.0.53 (stub resolver). Read upstream DNS servers from host's /run/systemd/resolve/resolv.conf, pass them to the container, and write /etc/resolv.conf before starting QEMU. Signed-off-by: gursewak1997 <gursmangat@gmail.com>

cgwalters · 2025-12-18T22:20:45Z

So I get this from every run now:

WARN No usable DNS servers found in system configuration, falling back to public DNS (8.8.8.8, 1.1.1.1). This may not work in air-gapped environments.

Looks like what's happening is when I'm on my work VPN the DNS is switched over to 10.x which matches is_private. It's not totally clear to me why we'd need to filter out private IP addresses.

In general I would say we probably want to get sooner rather than later out of the business of parsing DNS at all and ensuring that libvirt is handling networking consistently.

PR #167 introduced DNS filtering that excluded all private IP addresses (10.x, 172.16-31.x, 192.168.x, fc00::/7) assuming they would be unreachable from QEMU's slirp networking. However, this breaks VPN scenarios where private DNS servers are actually reachable. This change removes the overly aggressive private IP filtering, now only filtering out localhost and link-local addresses. Private network DNS servers are allowed through since they may be reachable (e.g., via VPN or air-gapped networks). If they're actually unreachable, DNS will fail naturally, which is better than prematurely filtering them out. Also downgraded the fallback warning from WARN to debug level since falling back to public DNS is a normal case, not an error condition. Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: gursewak1997 <gursmangat@gmail.com>

gursewak1997 · 2025-12-18T22:32:48Z

So I get this from every run now:

WARN No usable DNS servers found in system configuration, falling back to public DNS (8.8.8.8, 1.1.1.1). This may not work in air-gapped environments.

Looks like what's happening is when I'm on my work VPN the DNS is switched over to 10.x which matches is_private. It's not totally clear to me why we'd need to filter out private IP addresses.

A fix for now: #182

PR #167 introduced DNS filtering that excluded all private IP addresses (10.x, 172.16-31.x, 192.168.x, fc00::/7) assuming they would be unreachable from QEMU's slirp networking. However, this breaks VPN scenarios where private DNS servers are actually reachable. This change removes the overly aggressive private IP filtering, now only filtering out localhost and link-local addresses. Private network DNS servers are allowed through since they may be reachable (e.g., via VPN or air-gapped networks). If they're actually unreachable, DNS will fail naturally, which is better than prematurely filtering them out. Also downgraded the fallback warning from WARN to debug level since falling back to public DNS is a normal case, not an error condition. Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: gursewak1997 <gursmangat@gmail.com>

cgwalters

Assisted-by: OpenCode (Claude Sonnet 4)

AI-generated review based on REVIEW.md guidelines.
Comments prefixed with "AI:" are unedited AI output.

Post-merge cross-check review

The key issue (filtering private network DNS servers) was already identified and fixed in follow-up commit 8f8d301. Overall this is a well-structured fix with good documentation and testing.

cgwalters · 2025-12-19T18:48:15Z

crates/kit/src/run_ephemeral.rs

+
+                // Filter out localhost, link-local, and private network addresses
+                // QEMU runs in user networking mode (slirp) inside a container, which cannot
+                // reach private network addresses (10.x.x.x, 172.16-31.x.x, 192.168.x.x for IPv4,


AI: Important: The filtering of private network addresses (!ip.is_private() for IPv4, ULA check for IPv6) will break DNS for many enterprise/VPN users whose DNS servers are on private networks. Consider only filtering loopback and link-local. (Note: This was fixed in follow-up commit 8f8d301)

cgwalters · 2025-12-19T18:48:15Z

crates/integration-tests/src/tests/run_ephemeral_ssh.rs

+        "-c",
+        r#"
+        if command -v curl >/dev/null 2>&1; then
+            curl -sS --max-time 10 https://quay.io/v2/ >/dev/null


AI: (low) This test requires external network access to quay.io, which may fail in air-gapped CI environments or during quay.io outages. Consider documenting this as an integration test requirement, or adding a skip mechanism for offline testing.

cgwalters · 2025-12-19T18:48:15Z

crates/kit/src/run_ephemeral.rs

+    pub host_dns_servers: Option<Vec<String>>,
+}
+
+/// Parse DNS servers from resolv.conf format content


AI: Good structure - having parse_resolv_conf take &str makes it unit-testable without filesystem dependencies. Consider adding unit tests for edge cases (empty file, malformed lines, mixed IPv4/IPv6).

cgwalters · 2025-12-19T18:56:36Z

⬆️ was a test case for bootc-dev/infra#64

gursewak1997 · 2025-12-19T18:58:25Z

I like how it pointed out an issue and also mentioned (Note: This was fixed in follow-up commit https://github.com/bootc-dev/bcvk/commit/8f8d301a453d5cfeff5e329c8f3da94e18a0cbad)

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch 2 times, most recently from d5e8558 to f693004 Compare December 2, 2025 06:26

cgwalters requested changes Dec 2, 2025

View reviewed changes

crates/kit/src/run_ephemeral.rs Outdated Show resolved Hide resolved

crates/kit/src/qemu.rs Outdated Show resolved Hide resolved

crates/kit/src/qemu.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch 8 times, most recently from 542253f to 8b859ad Compare December 2, 2025 23:18

gursewak1997 marked this pull request as ready for review December 3, 2025 01:25

cgwalters reviewed Dec 3, 2025

View reviewed changes

crates/integration-tests/src/tests/run_ephemeral_ssh.rs Outdated Show resolved Hide resolved

crates/integration-tests/src/tests/run_ephemeral_ssh.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch 3 times, most recently from dcc3ad4 to 9996db4 Compare December 5, 2025 19:52

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 9996db4 to 5e28fd5 Compare December 5, 2025 21:31

cgwalters approved these changes Dec 5, 2025

View reviewed changes

crates/integration-tests/src/tests/run_ephemeral_ssh.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 5e28fd5 to 57af585 Compare December 5, 2025 21:52

gursewak1997 linked an issue Dec 5, 2025 that may be closed by this pull request

DNS resolving not working in ephemeral guests #123

Closed

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 57af585 to eb89500 Compare December 6, 2025 08:00

cgwalters marked this pull request as draft December 10, 2025 17:35

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch 2 times, most recently from 5cf80aa to 3bf0686 Compare December 10, 2025 22:25

gursewak1997 marked this pull request as ready for review December 10, 2025 23:04

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 3bf0686 to 0cd76fc Compare December 11, 2025 22:33

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 0cd76fc to 998adc2 Compare December 13, 2025 00:00

cgwalters approved these changes Dec 15, 2025

View reviewed changes

gursewak1997 force-pushed the fix/dns-resolution-ephemeral-guests branch from 998adc2 to b91195d Compare December 15, 2025 20:39

gursewak1997 requested a review from cgwalters December 15, 2025 21:30

cgwalters approved these changes Dec 15, 2025

View reviewed changes

gursewak1997 merged commit 800e600 into main Dec 15, 2025
7 checks passed

gursewak1997 deleted the fix/dns-resolution-ephemeral-guests branch December 15, 2025 22:03

gursewak1997 mentioned this pull request Dec 18, 2025

Fix DNS resolution for VPN and private network configurations #182

Merged

cgwalters reviewed Dec 19, 2025

View reviewed changes

cgwalters mentioned this pull request Dec 19, 2025

common/tasks: New dir, add AI-driven forge review workflow] bootc-dev/infra#64

Open

Fix DNS resolution in ephemeral guests #167

Fix DNS resolution in ephemeral guests #167

Uh oh!

Conversation

gursewak1997 commented Dec 1, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gursewak1997 commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gursewak1997 commented Dec 9, 2025

Uh oh!

gursewak1997 commented Dec 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Dec 18, 2025

Uh oh!

gursewak1997 commented Dec 18, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Dec 19, 2025

Uh oh!

gursewak1997 commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gursewak1997 commented Dec 5, 2025 •

edited

Loading

cgwalters commented Dec 5, 2025 •

edited

Loading