SoundEffect: No (real) prompt adherence

[ScPhilip]

Hey all, I seem to have real problems getting the SoundEffect model to reliably produce audio which follows the prompt - both through ComfyUI (using RS MOSS TSS Loader & RS MOSS TTS Batch Save) and directly via Python. I am on Ubuntu 22.04.3 with rocm (Radeon 9700, 32GB VRAM).

I tried the examples on the web page ("fresh snow crunching under footsteps.") with the parameters listed there, and those seem "okay-ish". However, using anything else (for example: "the ambient sound of gentle ocean waves rolling onto a sandy beach, light wind, distant seagulls") yield audio which does not remotely resemble the prompt.

Am I missing something?

Thank you.

[WangJinbaekhyun]

Hi, thanks for reporting this!

Yes, the current model struggles with complex, multi-event ambient scenes. If you break it down into simpler, single-source
prompts (e.g., "ocean waves" and "seagulls" separately), the results should be much better.

We're actively working on a new version that significantly improves prompt adherence. Stay tuned!

[ScPhilip]

Thank you for answering. I see, so it wasn't my fault, which is reassuring. It would be great if this could get some love, yes - there aren't many SoundEffect targeted models out there ATM.

By the way,

• the installation process for getting the code (or the comfyUI rs nodes) to really generate sound effects was a bit convoluted; the comfy nodes for example insist on input audio although none is required, and the comfyui parameters and the CLI parameters are different, so maybe this could use some work too
• if you can also provide a smaller model which fits into 12GB VRAM, I think you'll have a much larger audience (at least it OOMed on my smaller card, maybe I did something wrong. On 32GB VRAM its fine).

[WangJinbaekhyun]

Thanks for the feedback!

• ComfyUI audio input issue:

  We checked the current ComfyUI nodes, and the MossTTSSoundEffect node itself does not require input audio on our side. It’s possible the TTS node got mixed up with the sound effect node, or there may be something workflow-specific going on. If the issue persists, feel free to share your workflow and we’ll take a closer look.

• Parameter differences:

  The core generation parameters (temperature, top_p, top_k, repetition_penalty, max_new_tokens, etc.) are mostly consistent between the ComfyUI nodes and the reference Gradio/Python implementations. ComfyUI just exposes a few additional controls like seed, head_handle, and tail_handle for finer adjustments, though some naming or defaults may differ slightly.

• 12GB VRAM OOM:

  We’re also working on a lighter model variant, which should help make the model more accessible. Stay tuned!

Thanks again for the suggestions!