[action] [PR:608] [SmartSwitch] Enhance ModuleBase with graceful shutdown and startup transition handling

[mssonicbld]

Description

HLD: https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/graceful-shutdown/graceful-shutdown.md
These changes build upon enhancements in sonic-platform-common#567

This change introduces enhancements to the ModuleBase class to support graceful shutdown and startup operations for DPU and other module types.

It adds new methods and transition handling logic to ensure platform modules follow an ordered and coordinated shutdown/startup procedure, minimizing hardware inconsistencies and transient errors during reboot or DPU detachment.

Key changes include:
Added transition management APIs:

set_module_state_transition()
get_module_state_transition()
clear_module_state_transition()

Introduced graceful lifecycle handlers:

• _graceful_shutdown_handler() to wait for external transition completion using gnoi_halt_in_progress field with timeout handling
• Implemented database-backed transition tracking (CHASSIS_MODULE_TABLE)

Added helper functions for:

• File-based operation locks to ensure concurrency safety during transitions
• Included caching of transition timeout configuration from platform.json
• Added robust error-handling and logging to prevent partial updates in Redis DB

Motivation and Context

This enhancement is part of the SmartSwitch / DPU graceful shutdown/reboot and state management effort.
Currently, ModuleBase lacks lifecycle orchestration methods for safe shutdown or startup of DPUs and peripheral modules.
By adding transition-aware handling, the system can:

Avoid race conditions between platform daemons during reboot/shutdown
Ensure state transitions are reflected in Redis (CHASSIS_MODULE_TABLE)
Support controlled detach/reattach of PCIe devices and sensor configuration reloads
Enable PMON daemons to coordinate module-level transitions consistently
This work aligns with SONiC’s graceful reboot framework and the upcoming DPU lifecycle enhancements tracked internally.

How Has This Been Tested?

Testing performed on both SmartSwitch (DPU-enabled) and non-DPU platforms:

• ✅ Unit tests added under tests/test_module_base.py covering:
  • Transition management (set/get/clear)
  • Timeout behavior and concurrency lock handling
  • PCIe detach/reattach and sensor config updates
  • Graceful shutdown/startup flows (set_admin_state_gracefully)
• ✅ Verified Redis DB updates for transition keys under CHASSIS_MODULE_TABLE
• ✅ Simulated shutdown and startup sequences:
  • module_pre_shutdown() → safely detaches PCIe and updates state
  • module_post_startup() → rescans PCIe and restores sensor configuration
• ✅ Regression-tested existing platform daemons to ensure backward compatibility

Additional Information (Optional)

[mssonicbld]

Original PR: #608

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).