HLD for SmsrtSwitch DPU graceful shutdown #1991

rameshraghupathy · 2025-05-13T01:35:40Z

HLD for SmsrtSwitch DPU graceful shutdown

Related PRs:
sonic-net/sonic-platform-common#567
sonic-net/sonic-host-services#255

mssonicbld · 2025-05-13T01:35:47Z

/azp run

azure-pipelines · 2025-05-13T01:35:53Z

No pipelines are associated with this pull request.

mssonicbld · 2025-05-13T14:19:36Z

/azp run

azure-pipelines · 2025-05-13T14:19:42Z

No pipelines are associated with this pull request.

doc/smart-switch/graceful-shutdown/graceful-shutdown.md

mssonicbld · 2025-05-15T18:15:41Z

/azp run

azure-pipelines · 2025-05-15T18:15:47Z

No pipelines are associated with this pull request.

doc/smart-switch/graceful-shutdown/graceful-shutdown.md

mssonicbld · 2025-05-20T01:40:39Z

/azp run

azure-pipelines · 2025-05-20T01:40:45Z

No pipelines are associated with this pull request.

mssonicbld · 2025-05-20T01:54:32Z

/azp run

azure-pipelines · 2025-05-20T01:54:38Z

No pipelines are associated with this pull request.

mssonicbld · 2025-05-20T03:15:50Z

/azp run

azure-pipelines · 2025-05-20T03:15:56Z

No pipelines are associated with this pull request.

mssonicbld · 2025-05-20T03:19:56Z

/azp run

azure-pipelines · 2025-05-20T03:20:02Z

No pipelines are associated with this pull request.

mssonicbld · 2025-05-20T03:29:58Z

/azp run

azure-pipelines · 2025-05-20T03:30:07Z

No pipelines are associated with this pull request.

gpunathilell · 2025-07-01T18:36:15Z

@rameshraghupathy is the daemon introduced running always? and what is the behaviour on config reload of the switch

doc/smart-switch/graceful-shutdown/graceful-shutdown.md

rameshraghupathy · 2025-07-03T14:37:17Z

@rameshraghupathy is the daemon introduced running always? and what is the behaviour on config reload of the switch

@gpunathilell Fixed

rameshraghupathy · 2025-07-03T14:43:49Z

@rameshraghupathy is the daemon introduced running always? and what is the behaviour on config reload of the switch

@gpunathilell Yes, the daemon is running always. On config reload DB is wiped and repopulated; daemon continues and re-subscribes

mssonicbld · 2025-07-03T14:51:23Z

/azp run

azure-pipelines · 2025-07-03T14:51:30Z

No pipelines are associated with this pull request.

doc/smart-switch/graceful-shutdown/graceful-shutdown.md

mssonicbld · 2025-07-03T22:00:54Z

/azp run

azure-pipelines · 2025-07-03T22:01:00Z

No pipelines are associated with this pull request.

mssonicbld · 2025-07-03T22:45:00Z

/azp run

azure-pipelines · 2025-07-03T22:45:07Z

No pipelines are associated with this pull request.

mssonicbld · 2025-07-03T22:51:46Z

/azp run

azure-pipelines · 2025-07-03T22:51:53Z

No pipelines are associated with this pull request.

Provide support for SmartSwitch DPU module graceful shutdown. Description: Single source of truth for transitions All components now use sonic_platform_base.module_base.ModuleBase helpers: set_module_state_transition(db, name, transition_type) clear_module_state_transition(db, name) get_module_state_transition(db, name) -> dict is_module_state_transition_timed_out(db, name, timeout_secs) -> bool Eliminates duplicated logic and race-prone direct Redis writes. Correct table everywhere Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE). HLD mismatch addressed in code (HLD fix tracked separately). Ownership & lifecycle The initiator of an operation (startup/shutdown/reboot) sets: state_transition_in_progress=True transition_type=<op> transition_start_time=<utc-iso8601> The platform (set_admin_state()) is responsible for clearing: state_transition_in_progress=False optionally transition_end_time=<epoch> (or similar end stamp). CLI pre-clears only when a prior transition is timed out. Timeouts & policy Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants. Typical production values used: startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s. Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase. Boot behavior chassisd on start: Clears stale flags once (centralized sweep). Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state(). Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate. gNOI shutdown daemon Listens on CHASSIS_MODULE_TABLE and triggers only when: state_transition_in_progress=True and transition_type=shutdown. Never clears the flag (ownership stays with the platform). Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon). CLI (config chassis modules …) Uses ModuleBase APIs for all set/get/timeout checks. If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed. Sets transition at the start of startup/shutdown; platform clears on completion. Fabric card flow retained; edits are surgical. Redis robustness Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage. Consistent HGETALL/HSET paths; resilient to connector differences. Race reduction & consistency Centralized writes prevent multi-writer races. All transition writes include transition_start_time; clears may add an end stamp. Existing PCI/file-lock logic left intact; unrelated behavior unchanged. Change scope Minimal, targeted diffs. No background tasks added, no broad refactors beyond transition handling. Behavior changes are limited to making transition semantics correct and uniform across repos. HLD: # 1991 sonic-net/SONiC#1991 sonic-platform-common: #567 sonic-net/sonic-platform-common#567 sonic-utilities: sonic-net/sonic-utilities#4031 sonic-platform-daemons: sonic-net/sonic-platform-daemons#667 How to verify it Issue the "config chassis modules shutdown DPUx" command Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

Provide support for SmartSwitch DPU module graceful shutdown. # Description: * **Single source of truth for transitions** * All components now use `sonic_platform_base.module_base.ModuleBase` helpers: * `set_module_state_transition(db, name, transition_type)` * `clear_module_state_transition(db, name)` * `get_module_state_transition(db, name) -> dict` * `is_module_state_transition_timed_out(db, name, timeout_secs) -> bool` * Eliminates duplicated logic and race-prone direct Redis writes. * **Correct table everywhere** * Standardized on **`CHASSIS_MODULE_TABLE`** (replaces `CHASSIS_MODULE_INFO_TABLE`). * HLD mismatch addressed in code (HLD fix tracked separately). * **Ownership & lifecycle** * The **initiator** of an operation (`startup`/`shutdown`/`reboot`) sets: * `state_transition_in_progress=True` * `transition_type=<op>` * `transition_start_time=<utc-iso8601>` * The **platform** (`set_admin_state()`) is responsible for clearing: * `state_transition_in_progress=False` * optionally `transition_end_time=<epoch>` (or similar end stamp). * CLI pre-clears only when a prior transition is **timed out**. * **Timeouts & policy** * Platform JSON path only: `/usr/share/sonic/device/{plat}/platform.json`; else **constants**. * Typical production values used: * `startup: 180s`, `shutdown: 180s` (≈ `graceful_wait 60s + power 120s`), `reboot: 120s`. * **Graceful wait** (e.g., waiting for “Graceful shutdown complete”) is a **platform policy** and implemented inside platform `set_admin_state()`—not in ModuleBase. * **Boot behavior** * `chassisd` on start: 1. **Clears stale flags once** (centralized sweep). 2. Runs `set_initial_dpu_admin_state()` which **marks transitions** via ModuleBase before calling platform `set_admin_state()`. 3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate. * **gNOI shutdown daemon** * Listens on **`CHASSIS_MODULE_TABLE`** and triggers only when: * `state_transition_in_progress=True` **and** `transition_type=shutdown`. * Never clears the flag (ownership stays with the platform). * Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon). * **CLI (`config chassis modules …`)** * Uses ModuleBase APIs for all set/get/timeout checks. * If a previous transition is stuck, `is_module_state_transition_timed_out()` → auto-clear then proceed. * Sets transition at the start of `startup`/`shutdown`; platform clears on completion. * Fabric card flow retained; edits are surgical. * **Redis robustness** * Helpers handle both stacks (swsssdk/swsscommon); no `hset(mapping=...)` usage. * Consistent HGETALL/HSET paths; resilient to connector differences. * **Race reduction & consistency** * Centralized writes prevent multi-writer races. * All transition writes include `transition_start_time`; clears may add an end stamp. * Existing PCI/file-lock logic left intact; unrelated behavior unchanged. * **Change scope** * Minimal, targeted diffs. * No background tasks added, no broad refactors beyond transition handling. * Behavior changes are limited to making transition semantics correct and uniform across repos. HLD: # 1991 sonic-net/SONiC#1991 sonic-platform-common: #567 sonic-net/sonic-platform-common#567 sonic-utilities: sonic-net/sonic-utilities#4031 sonic-platform-daemons: sonic-net/sonic-platform-daemons#667 How to verify it Issue the "config chassis modules shutdown DPUx" command Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

Provide support for SmartSwitch DPU module graceful shutdown. # Description: * **Single source of truth for transitions** failure_prs.log skip_prs.log All components now use `sonic_platform_base.module_base.ModuleBase` helpers: failure_prs.log skip_prs.log `set_module_state_transition(db, name, transition_type)` failure_prs.log skip_prs.log `clear_module_state_transition(db, name)` failure_prs.log skip_prs.log `get_module_state_transition(db, name) -> dict` failure_prs.log skip_prs.log `is_module_state_transition_timed_out(db, name, timeout_secs) -> bool` failure_prs.log skip_prs.log Eliminates duplicated logic and race-prone direct Redis writes. * **Correct table everywhere** failure_prs.log skip_prs.log Standardized on **`CHASSIS_MODULE_TABLE`** (replaces `CHASSIS_MODULE_INFO_TABLE`). failure_prs.log skip_prs.log HLD mismatch addressed in code (HLD fix tracked separately). * **Ownership & lifecycle** failure_prs.log skip_prs.log The **initiator** of an operation (`startup`/`shutdown`/`reboot`) sets: failure_prs.log skip_prs.log `state_transition_in_progress=True` failure_prs.log skip_prs.log `transition_type=<op>` failure_prs.log skip_prs.log `transition_start_time=<utc-iso8601>` failure_prs.log skip_prs.log The **platform** (`set_admin_state()`) is responsible for clearing: failure_prs.log skip_prs.log `state_transition_in_progress=False` failure_prs.log skip_prs.log optionally `transition_end_time=<epoch>` (or similar end stamp). failure_prs.log skip_prs.log CLI pre-clears only when a prior transition is **timed out**. * **Timeouts & policy** failure_prs.log skip_prs.log Platform JSON path only: `/usr/share/sonic/device/{plat}/platform.json`; else **constants**. failure_prs.log skip_prs.log Typical production values used: failure_prs.log skip_prs.log `startup: 180s`, `shutdown: 180s` (≈ `graceful_wait 60s + power 120s`), `reboot: 120s`. failure_prs.log skip_prs.log **Graceful wait** (e.g., waiting for “Graceful shutdown complete”) is a **platform policy** and implemented inside platform `set_admin_state()`—not in ModuleBase. * **Boot behavior** failure_prs.log skip_prs.log `chassisd` on start: 1. **Clears stale flags once** (centralized sweep). 2. Runs `set_initial_dpu_admin_state()` which **marks transitions** via ModuleBase before calling platform `set_admin_state()`. 3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate. * **gNOI shutdown daemon** failure_prs.log skip_prs.log Listens on **`CHASSIS_MODULE_TABLE`** and triggers only when: failure_prs.log skip_prs.log `state_transition_in_progress=True` **and** `transition_type=shutdown`. failure_prs.log skip_prs.log Never clears the flag (ownership stays with the platform). failure_prs.log skip_prs.log Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon). * **CLI (`config chassis modules …`)** failure_prs.log skip_prs.log Uses ModuleBase APIs for all set/get/timeout checks. failure_prs.log skip_prs.log If a previous transition is stuck, `is_module_state_transition_timed_out()` → auto-clear then proceed. failure_prs.log skip_prs.log Sets transition at the start of `startup`/`shutdown`; platform clears on completion. failure_prs.log skip_prs.log Fabric card flow retained; edits are surgical. * **Redis robustness** failure_prs.log skip_prs.log Helpers handle both stacks (swsssdk/swsscommon); no `hset(mapping=...)` usage. failure_prs.log skip_prs.log Consistent HGETALL/HSET paths; resilient to connector differences. * **Race reduction & consistency** failure_prs.log skip_prs.log Centralized writes prevent multi-writer races. failure_prs.log skip_prs.log All transition writes include `transition_start_time`; clears may add an end stamp. failure_prs.log skip_prs.log Existing PCI/file-lock logic left intact; unrelated behavior unchanged. * **Change scope** failure_prs.log skip_prs.log Minimal, targeted diffs. failure_prs.log skip_prs.log No background tasks added, no broad refactors beyond transition handling. failure_prs.log skip_prs.log Behavior changes are limited to making transition semantics correct and uniform across repos. HLD: # 1991 sonic-net/SONiC#1991 sonic-platform-common: #567 sonic-net/sonic-platform-common#567 sonic-utilities: sonic-net/sonic-utilities#4031 sonic-platform-daemons: sonic-net/sonic-platform-daemons#667 How to verify it Issue the "config chassis modules shutdown DPUx" command Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

Initial version for dpu-graceful-shutdown HLD

01c62c2

Did some minor improvement

2c8b83b

This was referenced May 14, 2025

Module graceful shutdown support sonic-net/sonic-buildimage#22611

Draft

Module graceful shutdown support sonic-net/sonic-platform-common#567

Open

Module graceful shutdown support sonic-net/sonic-host-services#255

Merged

vvolam reviewed May 14, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

vvolam reviewed May 14, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

Addressed review comments

5020c9f

oleksandrivantsiv reviewed May 19, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

Adding two approaches

5fa9881

Did some cleanup

cf02ac4

Did some cleanup

8003e40

Did some cleanup

1d96db3

Did some cleanup

f97db46

gpunathilell reviewed Jul 1, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

Addressed some review comments

4dbf08f

vvolam reviewed Jul 3, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Show resolved Hide resolved

vvolam reviewed Jul 3, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

vvolam reviewed Jul 3, 2025

View reviewed changes

doc/smart-switch/graceful-shutdown/graceful-shutdown.md Outdated Show resolved Hide resolved

Addressed some review comments

490357a

Updated the image

ef08c8e

rameshraghupathy force-pushed the graceful-shutdown branch from 9c66ae9 to ef08c8e Compare July 3, 2025 22:51

vvolam approved these changes Jul 3, 2025

View reviewed changes

vvolam requested a review from gpunathilell July 3, 2025 23:13

gpunathilell approved these changes Jul 10, 2025

View reviewed changes

oleksandrivantsiv approved these changes Jul 15, 2025

View reviewed changes

vvolam merged commit b1ecce9 into sonic-net:master Jul 15, 2025
1 check passed

rameshraghupathy mentioned this pull request Aug 13, 2025

Module graceful shutdown support sonic-net/sonic-utilities#4031

Open

rameshraghupathy mentioned this pull request Aug 25, 2025

Module graceful shutdown support sonic-net/sonic-platform-daemons#667

Open

mssonicbld mentioned this pull request Dec 3, 2025

[action] [PR:255] Module graceful shutdown support sonic-net/sonic-host-services#324

Merged

HLD for SmsrtSwitch DPU graceful shutdown #1991

HLD for SmsrtSwitch DPU graceful shutdown #1991

Uh oh!

Conversation

rameshraghupathy commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented May 13, 2025

Uh oh!

azure-pipelines bot commented May 13, 2025

Uh oh!

mssonicbld commented May 13, 2025

Uh oh!

azure-pipelines bot commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented May 15, 2025

Uh oh!

azure-pipelines bot commented May 15, 2025

Uh oh!

Uh oh!

mssonicbld commented May 20, 2025

Uh oh!

azure-pipelines bot commented May 20, 2025

Uh oh!

mssonicbld commented May 20, 2025

Uh oh!

azure-pipelines bot commented May 20, 2025

Uh oh!

mssonicbld commented May 20, 2025

Uh oh!

azure-pipelines bot commented May 20, 2025

Uh oh!

mssonicbld commented May 20, 2025

Uh oh!

azure-pipelines bot commented May 20, 2025

Uh oh!

mssonicbld commented May 20, 2025

Uh oh!

azure-pipelines bot commented May 20, 2025

Uh oh!

gpunathilell commented Jul 1, 2025

Uh oh!

Uh oh!

rameshraghupathy commented Jul 3, 2025

Uh oh!

rameshraghupathy commented Jul 3, 2025

Uh oh!

mssonicbld commented Jul 3, 2025

Uh oh!

azure-pipelines bot commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 3, 2025

Uh oh!

azure-pipelines bot commented Jul 3, 2025

Uh oh!

mssonicbld commented Jul 3, 2025

Uh oh!

azure-pipelines bot commented Jul 3, 2025

Uh oh!

mssonicbld commented Jul 3, 2025

Uh oh!

azure-pipelines bot commented Jul 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rameshraghupathy commented May 13, 2025 •

edited

Loading