[csolution]: fails to handle non-ASCII characters in project names on Windows

[BadreddineBJAST]

Describe the bug
When using Open‑CMSIS "csolution" with a project whose filenames contain non‑ASCII characters, csolution fails to resolve the corresponding .cproject.yml file and reports that the file was not found.

The paths printed by csolution are corrupted , which strongly suggests an encoding / Unicode handling issue for non‑ASCII paths.

This does not affect only simple accented Latin letters; the problem can happen with a wide variety of Unicode characters, including:

Extended Latin letters / accents: é è ê ë á à ä ã å ö ó ò ñ ç š ž ł Ø ß …
Non‑Latin scripts:
Greek: α β γ δ Ω …
Cyrillic: Б Ю Я ж й …
Arabic: ا ب ت ث م …
Hebrew: א ב ג ד ה …
Indic scripts: Devanagari अ आ क ख, Bengali, Tamil, etc.
East Asian: CJK (Chinese), Japanese Kanji, Hiragana あ い う, Katakana ア イ ウ, Korean Hangul 가 나 다 …
Punctuation and typographic symbols:
Curly quotes: “ ” ‘ ’
Dashes: – (en dash), — (em dash)
Ellipsis: …
Non‑breaking space and other Unicode spaces
Currency and specialty symbols:
€, £, ¥, ₹, ₽, ₩, etc.
©, ®, ™
Section/paragraph: §, ¶
Mathematical / technical symbols:
±, ×, ÷, ≤, ≥, ∞, ∑, ∫, √, ≠, ≈, arrows ← → ↑ ↓, etc.

In short, any non‑ASCII character in filenames or paths can trigger the issue.

To Reproduce

Create the following minimal Open‑CMSIS project structure (only filenames are important):

./project
./project/tést_project_françois.cproject.yml
./tést_project_françois.csolution.yml
In tést_project_françois.csolution.yml, reference the .cproject.yml file:

yaml
solution:
  projects:
    - project: project/tést_project_françois.cproject.yml

From the directory containing tést_project_françois.csolution.yml, run:

csolution list contexts -s tést_project_françois.csolution.yml
or:
csolution list configs -s tést_project_françois.csolution.yml

Observe the console output.

Actual behavior

csolution prints a corrupted path and reports that the .cproject.yml file cannot be found, even though it exists:

project/tést_project_françois.cproject.yml - error csolution: cproject file was not found

Notes:

The actual file project/tést_project_françois.cproject.yml exists on disk.
The path printed by csolution (t├⌐st_project_fran├ºois) is the corrupted form of tést_project_françois, indicating that non‑ASCII characters in filenames are being misinterpreted (character encoding / Unicode issue).
If the files are renamed to ASCII‑only names (for example test_project_francois.cproject.yml / test_project_francois.csolution.yml) and the .csolution.yml is updated accordingly, the commands work as expected.
Similar issues occur if the filenames use other non‑ASCII characters from the categories listed above (extended Latin, Greek, Cyrillic, CJK, etc.).

Expected behavior

csolution should correctly handle project and file names that contain Unicode / non‑ASCII characters when:

Reading .csolution.yml and .cproject.yml.
Resolving file paths and opening the corresponding project files.
Printing the paths in diagnostic messages.

The commands:

csolution list contexts -s tést_project_françois.csolution.yml
csolution list configs  -s tést_project_françois.csolution.yml
csolution run -g "GenId" ....
should succeed as long as the referenced files exist, regardless of non‑ASCII characters in their names.

Environment :

Tool / library: Open‑CMSIS csolution (CMSIS‑Toolbox)
Version: 2.11.0
OS: Windows 11 x64
Shell / terminal: Git Bash (MINGW64)

Additional context

The issue is systematic whenever non‑ASCII characters are present in project filenames or paths that are referenced by csolution.
It is not limited to comments or content inside YAML; it specifically affects filenames and paths.
The behavior is consistent with a mismatch between UTF‑8 file encodings / filesystem names and the internal encoding or console output of csolution / its runtime environment on Windows.

Current workaround:
Restrict project names, filenames, and paths referenced by csolution to ASCII characters only.
This may not be practical in localized environments or for users whose default naming uses accents or non‑Latin scripts.

Request

Add robust UTF‑8 / Unicode support in csolution for:
Parsing YAML with Unicode paths.
Handling and passing Unicode file paths on all supported platforms (including Windows 11 with Git Bash / MINGW64).
Printing those paths correctly in logs and error messages.
If full Unicode support is not immediately possible, please:
Detect non‑ASCII characters in project names/paths early, and
Emit a clear, user‑friendly error stating that only ASCII is currently supported, instead of a misleading “file not found” with a corrupted path.

[BadreddineBJAST]

@slhultgren

[brondani]

@BadreddineBJAST Thanks for reporting this issue, we will work on it.
As a generic workaround I can suggest to Enable Unicode UTF-8 for Worldwide Language Support (see tutorial).

However please note CMake is restrictive concerning allowed characters for target names. Project names, target types, build types and execute names are used to form CMake target names, therefore they have the same restrictions. For example this will result in the following CMake error:

CMake Error at .../ExternalProject.cmake:2959 (add_custom_target):
  The target name "tést_project_françois.Release+CS300" is reserved or not
  valid for certain CMake features, such as generator expressions, and may
  result in undefined behavior.

[BadreddineBJAST]

Hi @brondani ,

Thanks a lot for the quick feedback and for the proposed workaround.

I can confirm that enabling “Beta: Use Unicode UTF-8 for worldwide language support” in the Windows region settings solves the issue on my side: after enabling it, csolution correctly handles project and file names containing non‑ASCII characters and the paths are no longer corrupted in the console output.

However, since this is a system-wide setting, we also need to carefully assess the potential impact on Windows users before recommending it broadly. Some legacy tools or scripts may not behave as expected when UTF‑8 is enforced globally, so this change should be applied with awareness of the possible side effects in mixed or older toolchains.

Given this behavior, would it be possible to:

• Document this more explicitly in the CMSIS‑Toolbox documentation, including:
  • The need for UTF‑8 / Unicode support on Windows when working with non‑ASCII project names.
  • A clear explanation that enabling the UTF‑8 beta option can affect other applications, and that users should evaluate this in the context of their environment.
• Clearly state the CMake restrictions on allowed characters in project/target names, with an example of the CMake error that appears when those restrictions are violated.

This would make the current limitations, required settings, and associated trade-offs much clearer for users working in localized environments or with Unicode filenames.

Thanks again for your support and for considering this documentation improvement.

[brondani]

@ReinhardKeil Would you like to take care of this documentation update?