Prepare settings for ingestion to programmatic settings menu in CLI and GUI

[neubig]

Currently, when new settings are added to the SDK, they need to be manually added to the settings menus in the CLI and GUI. This results in a long time lag from when settings are implemented in the SDK to when they become available to users.

To fix this, we can do the following things:

1. Add extra metadata to the SDK Pydantic data models for settings that specify all of the things that need to be known in order to programmatically generate a settings menu in the CLI and GUI.
2. Implement programmatic settings page generation the CLI (https://github.com/OpenHands/OpenHands-CLI).
   2a. We may also consider whether it is possible to programmatically generate slash commands that call specific API functions within the software agent SDK as well, such as the current one for context compression.
3. Implement programmatic settings page generation for the GUI. (https://github.com/OpenHands/OpenHands/tree/main/frontend)

[malhotra5]

Some examples of SDK settings would be -

1. LLM models
2. Condenser settings
3. Critic settings

The critic is a good example of a new capability introduced in the SDK, and requiring manual configuration in the CLI settings page to enable it: OpenHands/OpenHands-CLI#360

The only concerning part would be where a new setting are programmatically exposed (e.g critic) but the downstream interfaces have not built custom handlings for it (e.g the visualization logic for a critic event, which is controlled by the CLI TUI)

[neubig]

@OpenHands implement this, making sure that all settings are in a single datamodel. The CLI and GUI settings may inherit from the SDK data model if there are any other settings that are needed for only the CLI or GUI. Delete the current settings code in the CLI and GUI and reimplements purely programmatic settings generation (including Slash commands in the CLI) That relies solely on data in the settings datamodel. Create this as 3 prs across the three repos.

[openhands-ai]

I'm on it! neubig can track my progress at all-hands.dev

[openhands-ai]

Final update since the last summary:

What changed

• SDK

  • Fixed the exported settings schema so required is derived from type nullability, not just whether Pydantic provides a default.
  • This prevents generated UIs from treating required settings like llm_model as optional.
  • Added/updated SDK test coverage for the required flag behavior.
  • Recorded this rule in software-agent-sdk/AGENTS.md.
  • Committed and pushed to openhands/issue-2228-sdk-settings-schema (follow-up commit on PR sdk: centralize programmatic settings schema #2361).

• OpenHands GUI/server

  • I started a follow-up refactor toward a more generic schema-driven settings page:
    • generic frontend schema utilities
    • a schema-rendered llm-settings route
    • helper tests
    • i18n additions
    • some backend/model adjustments and API assertions
  • However, after your clarification, I determined this direction was still not fully compliant with the requirement, because it still involved legacy-side mirroring of SDK-managed fields in OpenHands.

Status checklist

• SDK follow-up fixed and pushed
• CLI unchanged since last summary
• OpenHands fully revised to require zero downstream field-specific changes
• All new SDK settings render in GUI with no frontend/server code changes
• OpenHands follow-up committed/pushed

Conciseness check

• SDK changes are concise and relevant.
• OpenHands follow-up is not yet final: it needs another pass to remove the remaining field-specific coupling rather than being pushed as-is.

Bottom line

The new requirement is not fully addressed yet. The main remaining work is in OpenHands: it needs a truly generic schema+values flow so future SDK settings require no frontend or legacy server code changes.

[enyst]

On OpenHands-Web, basically I think this issue is still applicable, although it does need a rewrite to be clearly mapped to today:

• Proposal: OpenHands configuration OpenHands#10947

TL;DR:
OpenHands Web settings system needs a re-architecting, with

• Settings split from user accounts stuff
• Settings split in agent, llm and so on, as they come from the SDK
• offer a little API with, at a minimum, append and filter_out
• which the Saas should use to
  • add its specific stuff (like “solvability”) (append)
  • or display only what it wants to display. (filter_out)

[enyst]

I gave this some thought and I'm not sure about it:

Currently, when new settings are added to the SDK, they need to be manually added to the settings menus in the CLI and GUI. This results in a long time lag from when settings are implemented in the SDK to when they become available to users.

I don't fully understand this reason, TBH. Indeed if the SDK adds a feature that wasn't there before, whether it's a new attribute, a new setting, or a new something else, then downstream applications will either implement with it or they won't. That seems normal to me.

Maybe I misunderstand, but I hear an idea that the SDK should do as much as possible from what client applications need to do, including UI awareness, and I'm not sure at all about that idea.

I think maybe it threatens separation at foundational level:

• the SDK is an SDK, it can run the agent from a script, from python APIs, from REST APIs
• it doesn't know UIs, really
  • the only thing we have is mostly debug-related (a default "visualizer" so we see what we're doing)
• UI is a client application concern.

I think maybe client applications can implement a new menu for a setting if they want to, when upstream adds a new setting, and not implement it if they don't want it?

It makes sense to me that menus are not an SDK functionality. 🤔 Otherwise, I worry that we hard-code assumptions and wishes from a couple of client applications: a tight coupling from the SDK to its downstream (which strikes me as the opposite direction than expected). This may also imply that we will not be as open and friendly as we'd like, to many further third-party applications.

[enyst]

The example you linked, Rohit, indeed has been a lot of work. But I don't know; it's a full new feature that Xingyao added, and it's a large one.

I'm not sure it is unexpected that a large feature to which the CLI wants to work with, add various contents to display, add user interaction, add settings to enable / disable, and a number of other things, means, at its turn, a large implementation to add to the CLI app, though I could be wrong 🤔