Skip to content

Conversation

@julien-nc
Copy link
Member

@julien-nc julien-nc commented Aug 26, 2025

closes #340

Add declarative UI capabilities for clients.
Add one hook for each of our file actions (summarize, stt and tts).

Currently unknown:

  • Which HTTP method is used to reach the hooks
  • How is the file ID passed
  • How to define multiple filters for one hook

Copy link
Member

@marcelklehr marcelklehr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable

@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from a5b2673 to 72e2f65 Compare August 29, 2025 08:20
@julien-nc
Copy link
Member Author

julien-nc commented Aug 29, 2025

When combining the capabilities declared by various apps, the server combines the arrays. For objects, all good. For lists, as the keys are preserved and a Php list is almost an object with integer keys, the combining goes wrong and each list overwrites the other ones.

Here is my suggestion for the declarativeui capability structure:

  • declarativeui is an object indexed by app IDs. So in each item of this object, an app can do whatever it wants, the content will not be combined with others
  • Inside an "appId" object, we can have 2 list of endpoints, one under context-menu and one under create-new
  • An endpoint is an object with those keys:
    • name
    • url
    • method (default: GET)
    • mimetype_filters (comma separated list of mimetype filters, for now these are not regexps, clients will accept files which mime type starts with any of the filters)
    • icon is a URL to the icon
    • bodyParams is an object with all the parameters that should be passed in the request body. The {fileId} placeholder is supported there too. Example: "bodyParams": "{\"fileId\": \"{fileid}\"}",

Object in the declarativeui key of the capabilites:

{
  "assistant": {
    "context-menu": [
      {
        "name": "Summarize",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:text2text:summary",
        "method": "POST",
        "mimetype_filters": "text/, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.oasis.opendocument.text, application/pdf",
        "icon": "/apps-extra/assistant/img/declarativeui/summarize.svg"
      },
      {
        "name": "Transcribe audio",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:audio2text",
        "method": "POST",
        "mimetype_filters": "audio/",
        "icon": "/apps-extra/assistant/img/declarativeui/speech_to_text.svg"
      },
      {
        "name": "Text to speech",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:text2speech",
        "method": "POST",
        "mimetype_filters": "text/, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.oasis.opendocument.text, application/pdf",
        "icon": "/apps-extra/assistant/img/declarativeui/text_to_speech.svg"
      }
    ]
  },
  "cospend": {
    "context-menu": [
      {
        "name": "cospend fake",
        "url": "hello"
      }
    ]
  }
}

@julien-nc
Copy link
Member Author

julien-nc commented Aug 29, 2025

@tobiasKaminsky You can test this branch by checking it out and building it:

npm ci
npm run dev
composer i --no-dev

For the capabilities to be declared by the assistant, you need some AI providers. There are some fake ones in the testing app (summarize and STT).

@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch 4 times, most recently from e4f9dc8 to 7d3d17a Compare September 4, 2025 13:06
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from 3f9001a to 5450fa6 Compare September 23, 2025 14:26
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch 2 times, most recently from 2dd3bc9 to 25154f2 Compare October 6, 2025 14:01
@julien-nc julien-nc marked this pull request as ready for review October 8, 2025 12:14
julien-nc and others added 7 commits October 15, 2025 14:24
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
…add multi filters, add method

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
…n file

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
version 0.1

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from 0e0797c to 211a03d Compare October 15, 2025 12:27
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from 211a03d to c9d4fb8 Compare October 15, 2025 12:36
@julien-nc julien-nc merged commit 8cf4b15 into main Oct 15, 2025
17 checks passed
@julien-nc julien-nc deleted the enh/340/declarative-ui branch October 15, 2025 12:41
@janepie janepie mentioned this pull request Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Declarative UI: audio file -> create a transcript (md)

4 participants