Skip to content

Add Pod Network Rate Limit Chaos ExperimentΒ #760

@neelanjan00

Description

@neelanjan00

Description

Implement a new chaos experiment pod-network-rate-limit that applies network bandwidth throttling to target application pods. This experiment enables testing application resilience under network bandwidth constraints and helps validate behavior during network congestion scenarios.

Requirements

1. Core Functionality

  • Apply network rate limiting using Linux Traffic Control (tc) with Token Bucket Filter (TBF)
  • Target specific pods based on labels or pod names
  • Support configurable bandwidth limits, burst sizes, and rate parameters
  • Integrate with existing network chaos infrastructure

2. Configuration Parameters

  • Network Bandwidth: Maximum allowed bandwidth (e.g., "1mbit", "100kbit")
  • Burst Size: Token bucket burst capacity (e.g., "32kb", "64kb")
  • Limit: Maximum number of bytes that can be queued
  • Peak Rate: Optional peak transmission rate
  • Min Burst: Optional minimum burst size (MTU)

3. Target Selection

  • Support pod targeting via labels: APP_LABEL="app=nginx"
  • Support direct pod targeting: TARGET_PODS="pod1,pod2"
  • Support percentage-based targeting: PODS_AFFECTED_PERC="50"
  • Support container-specific targeting: TARGET_CONTAINER="app"

4. Integration Points

  • Reuse existing network chaos helper infrastructure
  • Leverage common pod selection and filtering logic
  • Support serial and parallel injection modes
  • Integrate with chaos result tracking and probe framework

5. Traffic Control Implementation

  • Use tc tbf (Token Bucket Filter) queueing discipline
  • Apply rate limiting to specified network interface (default: eth0)
  • Support both ingress and egress rate limiting
  • Proper cleanup and reversion after experiment completion

Expected Configuration Examples

Basic Rate Limiting

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-network-rate-limit
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create","delete","get","list","patch","update","deletecollection"]
    env:
    - name: NETWORK_BANDWIDTH
      value: "1mbit"
    - name: BURST  
      value: "32kb"
    - name: LIMIT
      value: "2mb"
    - name: TOTAL_CHAOS_DURATION
      value: "60"
    - name: APP_LABEL
      value: "app=nginx"

Advanced Configuration

env:
- name: NETWORK_BANDWIDTH
  value: "10mbit"
- name: BURST
  value: "64kb" 
- name: LIMIT
  value: "5mb"
- name: PEAK_RATE
  value: "20mbit"
- name: MIN_BURST
  value: "1500"
- name: NETWORK_INTERFACE
  value: "eth0"
- name: TARGET_CONTAINER
  value: "app"
- name: PODS_AFFECTED_PERC
  value: "50"

Traffic Control Command Structure

# Basic rate limiting
tc qdisc replace dev eth0 root tbf rate 1mbit burst 32kb limit 2mb

# Advanced with peak rate
tc qdisc replace dev eth0 root tbf rate 10mbit burst 64kb limit 5mb peakrate 20mbit mtu 1500

Benefits

🌐 Network Resilience Testing

  • Test application behavior under bandwidth constraints
  • Validate graceful degradation during network congestion
  • Simulate real-world network throttling scenarios

⚑ Performance Validation

  • Measure application performance under rate limits
  • Test timeout and retry mechanisms
  • Validate buffering and queuing strategies

πŸ”§ Operational Scenarios

  • Simulate ISP bandwidth limitations
  • Test behavior during network infrastructure maintenance
  • Validate QoS policies and traffic shaping effectiveness

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions