Feature Request: Reduce NSG Destination Prefix Growth When Floating IP Is Disabled

[craigktreasure]

Summary

When service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip: "true" is set on a LoadBalancer Service, the Azure cloud provider uses backend node IPs as NSG rule destinations instead of the load balancer frontend IPs. In larger clusters, this causes NSG destination prefix usage to grow with node count and can exhaust Azure NSG prefix limits.

We would like to request support for a more scalable destination model when floating IP is disabled, such as allowing destination CIDRs/prefixes or allowing operators to opt out of ACP-managed NSG rules for a Service.

Current Behavior

The controller switches destination handling based on azure-disable-load-balancer-floating-ip in pkg/provider/azure_loadbalancer.go.

When floating IP is disabled, reconcileSecurityGroup() uses backend node IPs:

if disableFloatingIP {
    dstIPv4Addresses = append(dstIPv4Addresses, backendIPv4Addresses...)
    dstIPv6Addresses = append(dstIPv6Addresses, backendIPv6Addresses...)
} else {
    dstIPv4Addresses = append(dstIPv4Addresses, lbIPv4Addresses...)
    dstIPv6Addresses = append(dstIPv6Addresses, lbIPv6Addresses...)
}

The destination port also changes from port.Port to port.NodePort in pkg/provider/loadbalancer/accesscontrol.go:

if consts.IsK8sServiceDisableLoadBalancerFloatingIP(svc) {
    p = port.NodePort
} else {
    p = port.Port
}

Those destination addresses are then written into NSG rules via pkg/provider/securitygroup/securitygroup.go and pkg/provider/securitygroup/securityrule.go.

Why This Becomes a Scaling Problem

For Services that need floating IP disabled, the number of destination prefixes in ACP-managed NSG rules grows with the number of backend nodes.

In our deployment shape, a dual-stack Service with:

service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip: "true"
service.beta.kubernetes.io/azure-allowed-service-tags: "Internet"
service.beta.kubernetes.io/azure-allowed-ip-ranges: "fd00::/8"

ends up producing separate allow rules per IP family and source type, while each rule enumerates backend node IPs individually. That makes destination prefix consumption proportional to node count.

This is especially limiting when multiple such Services exist in the same cluster.

Additional Repo Context

• Rule identity is based on protocol, source prefixes, and destination ports in pkg/provider/securitygroup/securityrule.go.
• Because disabled floating IP uses NodePort, Services with different node ports do not naturally converge onto the same allow rule.
• The code already aggregates source prefixes with iputil.AggregatePrefixes() in pkg/provider/loadbalancer/accesscontrol.go, but destination addresses are still handled as individual node IPs.

Requested Capability

Would the project consider one of these approaches for Services with floating IP disabled?

A. Automatically aggregate backend node IPs into destination CIDRs/prefixes before writing NSG rules. The project already aggregates source prefixes with iputil.AggregatePrefixes(), so the analogous idea here would be to convert the backend node IP set into the smallest reasonable set of destination prefixes instead of emitting one destination entry per node IP.

B. Allow a Service annotation that overrides backend node IP enumeration and instead uses operator-supplied destination prefixes, such as worker subnet CIDRs, when generating NSG rules.

C. Allow a Service annotation that skips ACP-managed NSG rule creation so operators can manage NSG rules themselves.

I do not want to be overly prescriptive about the implementation. The main request is to provide a supported way to avoid per-node destination prefix growth in this mode.

From an operator perspective, Option C would be the least preferred of the three because it shifts NSG lifecycle management to the operator and therefore adds ongoing operational overhead. Option A would be the most seamless adaptation because it improves the default behavior without requiring per-Service configuration, while Option B may still be desirable in cases where operators want explicit control over the destination prefixes used.

Deployment-Specific Context

In our environment, some Services require azure-disable-load-balancer-floating-ip: "true" to work correctly with our Azure Service Endpoint usage. That operational requirement is specific to our scenario, but the controller behavior above appears general.

Impact

Today, this behavior can make floating-IP-disabled LoadBalancer Services difficult to operate at larger scale because NSG prefix usage grows with backend node count.