About RELATED_CWEs

[phandat128]

Hi teams, thank you for releasing such an excellent paper and repo, it helped me a lot.
I just have a small question about the RELATED_CWE in your dataset, where did it come from? It seems you are hardcoding it, both in train and test dataset. For example, CWE-20 always comes along with ["CWE-362", "CWE-415", "CWE-269"], this can make model instead of detecting the real vulnerabilities in code, it only focuses on the top 4 CWEs (if not provided, it will hallucinate) and also always return the main CWE in top 4 (because for each right CWE, the related CWEs are not changing, and each CWE in the related CWEs, in other side has another top 3 related ones?). Have you considered this case?

[rucnyz]

Hi @phandat128, great question! Thank you for digging into the details To clarify: in our test setting, we do NOT include related CWEs in the prompt. The related CWEs are only used during part of the training phase, where we found it beneficial to train the model with both: - Arbitrary reasoning (without related CWEs) - helps the model - Focused reasoning (with related CWEs) - helps the model learn structured reasoning within a constrained CWE space This training strategy aims to balance generalization and precision, rather than making the model "memorize" fixed CWE associations. That said, I just checked the released code and I agree it's confusing: use_related_cwe defaults to True in test.py, but the CWE information is only actually included in the prompt when policy is non-empty (via --use_policy), which is not used in our README tutorial. This disconnect can easily mislead users into thinking related CWEs are being used during inference. I'll update the code very soon. Thanks for pointing this out!

…

________________________________ From: Phan Đạt ***@***.***> Sent: Wednesday, January 28, 2026 21:28 To: ucsb-mlsec/VulnLLM-R ***@***.***> Cc: Subscribed ***@***.***> Subject: [ucsb-mlsec/VulnLLM-R] About RELATED_CWEs (Issue #4) [https://avatars.githubusercontent.com/u/106984257?s=20&v=4]phandat128 created an issue (ucsb-mlsec/VulnLLM-R#4)<#4> Hi teams, thank you for releasing such an excellent paper and repo, it helped me a lot. I just have a small question about the RELATED_CWE in your dataset, where did it come from? It seems you are hardcoding it, both in train and test dataset. For example, CWE-20 always comes along with ["CWE-362", "CWE-415", "CWE-269"], this can make model instead of detecting the real vulnerabilities in code, it only focuses on the top 4 CWEs (if not provided, it will hallucinate) and also always return the main CWE in top 4 (because for each right CWE, the related CWEs are not changing, and each CWE in the related CWEs, in other side has another top 3 related ones?). Have you considered this case? — Reply to this email directly, view it on GitHub<#4>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO7WD2IDNIT5UNIW4I4L5RD4JGLAZAVCNFSM6AAAAACTIN5TLCVHI2DSMVQWIX3LMV43ASLTON2WKOZTHA3DQOJRGEYTKMY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[phandat128]

Thank @rucnyz for very quick and informative response.
I still have some wonders: To reproduce test results in paper, you suggest raising flag --use_policy in README, which means hinted the model in test phase. I also test without --use_policy and cannot achieve the results in paper. Did the results from paper use this --use_policy flag too, and what is the performance gap between using it or not?
Thank you!

[rucnyz]

I can reproduce our results with
python -m vulscan.test.test --output_dir results/test_data --dataset_path ./datasets/test/function_level/ ./datasets/test/repo_level/ --language python c java --model UCSB-SURFI/VulnLLM-R-7B --requests_per_minute 1000 --save --use_cot --batch_size 4 --tp 2 --vllm --max_tokens 8192 --random_cwe

Is this the command you are using?

[phandat128]

My command is a bit different because I am using remote vllm server, and my reproduced results if not using --use_policy drop about 10-15% F1 score each benchmark compared to using --use-policy. Anyway, I will try to reproduce again using your original suggested command.
Thank you

[rucnyz]

I see. Let me test with a remote vllm server then. I am not sure if there is any error, it should work fine previously.

…

________________________________ From: Phan Đạt ***@***.***> Sent: Thursday, January 29, 2026 11:21 PM To: ucsb-mlsec/VulnLLM-R ***@***.***> Cc: Yuzhou Nie ***@***.***>; Mention ***@***.***> Subject: Re: [ucsb-mlsec/VulnLLM-R] About RELATED_CWEs (Issue #4) [https://avatars.githubusercontent.com/u/106984257?s=20&v=4]phandat128 left a comment (ucsb-mlsec/VulnLLM-R#4)<#4 (comment)> My command is a bit different because I am using remote vllm server, and my reproduced results if not using --use_policy drop about 10-15% F1 score each benchmark compared to using --use-policy. Anyway, I will try to reproduce again using your original suggested command. Thank you — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO7WD2JKMSQPKGX4NI4UNDL4JMA7RAVCNFSM6AAAAACTIN5TLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMRSGI2TCMRYGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>