Skip to content

Conversation

@tthvo
Copy link
Contributor

@tthvo tthvo commented Jul 30, 2025

What type of PR is this?
/kind feature

What this PR does / why we need it:

As of today, CAPA supports IPv6 on EKS, but not self-managed clusters. Thus, these changes bring IPv6 support for self-managed clusters, both single-stack IPv6 and dualstack.

Which issue(s) this PR fixes:

Fixes #2420
Fixes #3381

Special notes for your reviewer:

  • EC2 instance type: Only nitro-based instance type can support IPv6.
  • CNI: The CNI plugin needs to support IPV6. I include sample manifests in test/e2e/data/cni. Calico does not support IPv6 with "IP-in-IP" so we need to use VXLAN.

Checklist:

  • squashed commits
  • includes documentation
  • includes emoji in title
  • adds unit tests
  • adds or updates e2e tests

Release note:

Enable IPv6 support for self-managed kubernetes clusters

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 30, 2025
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jul 30, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot requested review from AndiDog and damdo July 30, 2025 21:03
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added needs-priority cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025
@k8s-ci-robot
Copy link
Contributor

Welcome @tthvo!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 30, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @tthvo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025
@tthvo
Copy link
Contributor Author

tthvo commented Jul 30, 2025

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

@k8s-ci-robot k8s-ci-robot requested a review from nrb July 30, 2025 21:23
@k8s-ci-robot
Copy link
Contributor

@tthvo: GitHub didn't allow me to request PR reviews from the following users: sadasu, patrickdillon.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tthvo
Copy link
Contributor Author

tthvo commented Jul 30, 2025

A quick preview of kubectl get for an IPv6 self-managed cluster can be found here. So far, nodes, pods and services are assigned the expected IPv6 family IPs 😄

@tthvo tthvo force-pushed the singestack-ipv6 branch from ddacd99 to 13e4379 Compare July 31, 2025 00:37
@damdo
Copy link
Member

damdo commented Aug 4, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2025
@damdo
Copy link
Member

damdo commented Aug 4, 2025

/assign @mtulio

Asking you for a review Marco as I know you have been working on this downstream

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2025
@tthvo tthvo force-pushed the singestack-ipv6 branch from 13e4379 to 8816d87 Compare August 5, 2025 18:09
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025
@tthvo tthvo force-pushed the singestack-ipv6 branch from 8816d87 to 5dd2888 Compare August 5, 2025 18:11
tthvo added 26 commits October 30, 2025 20:30
AWS requires that when registering targets by instance ID for an IPv6
target group, the targets must have an assigned primary IPv6 address.

Note: The default subnets managed by CAPA are already set up to assign
IPv6 addresses to newly created ENIs.
The httpProtocolIPv6 field enables or disables the IPv6 endpoint of the
instance metadata service. The SDK only applies this field if
httpEndpoint is enabled.

When running on single-stack IPv6, pods only have IPv6, thus requiring
an IPv6 endpoint to query IMDS as IPv4 network is unreachable.
In the case where egress-only-internet-gateway is deleted, CAPA
reconcilation loop will create a new one. Thus, CAPA needs to modify the
routes to point to the new eigw ID.
This allows IPv6-only workloads to reach IPv4-only services. AWS
supports this via NAT64/DNS64.

More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html
CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number
when creating rules. However, describing a rule from AWS API returns the
protocol name, thus causing CAPA to not recognize it and fail.
…ices

For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the
allowed source IPv4 CIDR for node NodePort services on port 30000-32767.

This extends that field to also accept IPv6 source CIDRs.
We need an option to configure IPv6 source CIDRs for SSH ingress rule of
the bastion host.

This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.
When creating a bastion host for an IPv6 cluster, the instance has both
public IPv4 and IPv6. Thus, we need to report them in the cluster status
if any.

This also adds an additional print column to display that bastion IPv6.
This is a minimal template set to install an IPv6-enabled cluster. Both
the controlplane and worker nodes must use nitro-based instance type
(with IPv6 support).
This combines existing docs for IPv6 EKS clusters with non-EKS ones, and
also properly register the topic page into the documentation TOC.
Validation for specified VPC and subnet CIDRs is added for early
feedback from the webhook.

There are already existing checks for bastion and nodePort CIDRs.
The following is added:
- [BYO VPC] Mention the required route when enabling DNS64.
- [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual
  stack VPC.
There is a brief period where the IPv6 CIDR is not yet associated with
the subnets. Thus, when CAPA creates the default dualstack subnets, it
should wait until the IPv6 CIDR is associated before proceeding.

If not, CAPA will misinterprete the subnet as non-IPv6 and proceed its
reconcilation. The consequence is that CAPA will skip creating a route
to eigw. Route to eigw for destination "::/0" to eigw is required for EC2
instance time sync on start-up.
…ined

When AWSCluster.spec.network.vpc.ipv6 is non-nil, most handlers in CAPA
treats it as "adding" IPv6 capabilities on top of IPv4 infrastructure.
Except security group ingress rules for API LB.

This commit aligns the API LB SG handler with the rest of the code base.

These rules can be overriden in the AWSCluster LB spec to allow only
IPv6 CIDRs if needed.
The field isIpv6 is set to true if and only if the subnet has an
associated IPv6 CIDR. This means the VPC is also associated with an
IPv6 CIDR.
The field targetGroupIPType is added to the loadbalancer spec to allow
configuring ip address type of target group for API load balancers.

This field is not applicable to Classic Load Balancers (CLB).

This commit also defines a new network status field to determine the ip
type of API load balancers.
When creating subnets in a managed VPC with IPv6 enabled, automatically
assign IPv6 CIDR blocks to subnets that have isIPv6=true but no
explicit IPv6CidrBlock specified. This simplifies subnet configuration
by allowing users to enable IPv6 without manually calculating and
assigning individual subnet IPv6 CIDRs, for example, in case where VPC
IPv6 CIDR is unknown pre-provisioning and AWS will assign one during VPC
creation.

Note: This logic only applies when spec.network.vpc.ipv6 is non-nil,
subnets are managed and non-existing.
The field awscluster.spec.network.vpc.ipv6.ipamPool defines the IPAM
pool to allocate an IPv6 CIDR for the VPC. Previously, CAPA only
considers field awscluster.spec.network.vpc.ipamPool, which is used only
for VPC IPv4 CIDR allocation.

Additionally, CAPA should preserve the ipv6 spec fields, provided by the
users, for example, the ipv6 ipamPool. Previously, these spec fields
are lost during vpc reconcilation.
NAT64/DNS64 is meant to be enabled for IPv6-only subnets, in which
instances do not have IPv4 [0].

If we enable DNS64 for dualstack subnets, instances will receive both
A/AAAA records for IPv4-only services. In most cases, OS-level settings
will prefer IPv6, leading to traffic to flow via NAT gateway instead of
using IPv4 directly.

Reference

[0] https://aws.amazon.com/blogs/networking-and-content-delivery/dual-stack-architectures-for-aws-and-hybrid-networks-part-2/
Add new dualstack cluster template and documentation updates
for IPv6 and dualstack cluster configurations. Additionally, docs for
configuring API LB's target group IP type is also added.

New cluster templates and calico manifest are included for creating
dualstack clusters.
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2025
@tthvo
Copy link
Contributor Author

tthvo commented Oct 31, 2025

The latest rebase is to resolve conflicts with upstream/main in conversion webhook 😁 no additional changes...

Copy link
Contributor

@mtulio mtulio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excelent feature, @tthvo ! A few questions related to the matrix of supported scenarios, as well the backward compatibility.

Comment on lines +258 to +259
// Valid values are ipv4 and ipv6. If not specified, defaults to ipv4 unless
// the VPC has IPv6 enabled, in which case it defaults to ipv6.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless the VPC has IPv6 enabled

Is that statement correct even when preferred is IPv4 which has IPv6 enabled in the VPC?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the comment I understood that users would have the path where the VPC supports IPv6 but target would be Ip4, is that correct my understanding?


## Overview

CAPA enables you to create IPv6 and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this following how AWS refers to IPv6 deployment on their docs ("IPv6 only" or "IPv6-only")?

Suggested change
CAPA enables you to create IPv6 and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure.
CAPA enables you to create IPv6 only and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure.

I was also wondering about the "single-stack IPv6" verbiage, but I didn't find many references in AWS docs.


## Enabling IPv6 capabilities

To instruct CAPA to configure IPv6 capabilities for the network infrastructure, you must explicitly define `spec.network.vpc.ipv6` in either `AWSCluster` (for self-managed clusters) or `AWSManagedControlPlane` (for EKS clusters). See [IPv6 CIDR Allocations](#ipv6-cidr-allocations) for different IPv6 CIDR configuration options.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or AWSManagedControlPlane (for EKS clusters)

Is ROSA, managed OCP, also reacts to this field?

kubectl -n kube-system edit cm/coredns
```

**Note**: This CoreDNS workaround is NOT required for dualstack clusters where pods have both IPv4 and IPv6 addresses.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am note sure if I followed completely this statement. I see the CoreOS limitation, but are we going to support IPv6 only self-managed clusters (without requiring NAT64)? I am wondering scenarios where users wants to rid off public IPv4 costs, which includes NAT Gateway (required by NAT64);

<aside class="note warning">
<h1>Warning</h1>

The AWS Cloud Controller Manager (CCM) does **not** currently support dualstack Load Balancers. When creating Services of type LoadBalancer in a dualstack cluster, the Load Balancers will be created with **only** IPv4.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about to mention that CLB does not support IPv6 by AWS too? as well point to the CCM issue that will be supported when resolved: kubernetes/cloud-provider-aws#1219

## Target Group IP Address Type
The secondary load balancer supports the same `targetGroupIPType` configuration as the primary load balancer. By default, the target group IP address type is set based on the VPC configuration:
- If the VPC has IPv6 enabled, the target group uses `ipv6`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the behavior in dualstack preferred ipv4? Is that the default only (discovered ipv6) and user must manually set to ipv4 when wants preferred?

func (s *Service) getNat64PrivateRoute(natGatewayID string) *ec2.CreateRouteInput {
return &ec2.CreateRouteInput{
NatGatewayId: aws.String(natGatewayID),
DestinationIpv6CidrBlock: aws.String(services.NAT64CidrBlock),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is NAT64 required to make an operational cluster on AWS? Can we optionally enable it without impact, specially in preferred ipv6?

One of concerns of users willing to use IPv6 is the IPv4 costs, on AWS we can't fully eliminate it as public LBs still need it, but I wonder if we can reduce the dependency of Nat Gateways with that new proposal. What do you think?

With the preferred IPv6 topology, do we need to allocate one-per-az Nat Gateway (which is a expensive resource)? I wonder if we can create minimum topology (single, or dual, instead per-AZ) of NAT Gateways on IPv6 clusters, helping users to eliminate public IPv4 address from their environment.

topology

Something like: NATGatewayTopologyStrategy: default|single|dual

Where:

  • default: one per az, not changing the default (zonal redundant)
  • single: one NGW for entire cluster (SPOF)
  • dual: dual NGW with HA in two AZs (if cluster is deployed on >1 AZs)

const (
LoadBalancerIPAddressTypeIPv4 = LoadBalancerIPAddressType("ipv4")
LoadBalancerIPAddressTypeDualstack = LoadBalancerIPAddressType("dualstack")
LoadBalancerIPAddressTypeDualstackWithoutPublicIPv4 = LoadBalancerIPAddressType("dualstack-without-public-ipv4")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we using this const / is it a valid value for AWS API?

if subnet.ZoneType != nil && subnet.IsEdge() {
if subnet.ParentZoneName == nil {
allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "ParentZoneName must be set when ZoneType is 'local-zone'."))
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone'."))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I known it would be outside this PR, but let me document it - this comment must be something like this as isEdge() evaluates both Local Zone and Wavelength locations:

Suggested change
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone'."))
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone' or 'wavelength-zone."))

Comment on lines 549 to 553
// Enable DNS64 so that the Route 53 Resolver returns DNS records for IPv4-only services
// containing a synthesized IPv6 address prefixed 64:ff9b::/96.
// This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services.
// We only need to enable on private subnets.
if !sn.IsPublic {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments on this:
1:

This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services... We only need to enable on private subnets.

May this prevent ip6-only workloads running in nodes created in "public subnets" accessing resources in private subnets?

  1. I am not seeing this route in any of private subnets of clusters I've created both with primarily ipv4 and ipv6, I need to investigate closely in the output manifests or logs if I missed some error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-priority ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add dual stack support Add IPv6 support

6 participants