Point of clarification on definition of "security reviews"

[jenniferfernick]

I'm wondering what we consider a "security review" for the purposes of this collection:

• A third party security audit of an open source codebase by a security firm? (Assuming yes)
• A technical advisory on a vulnerability in an open source project that has undergone a coordinated disclosure? (Assuming no)
• Some kind of "security review" written by the maintainers of an open source project itself? (Assuming no??)
• Threat models or other documents that fall short of finding specific issues for which proof-of-concept exploits can be demonstrated? (Assuming no)
• A compliance or other non-technical or semi-technical review of an open source project? (Probably not?)
• Results from static analysis tools, fuzzers, etc? (Probably not but is there anywhere that collects these?)

[scovetta]

Some thoughts -- happy to get others' opinions, too!

A third party security audit of an open source codebase by a security firm? (Assuming yes)
Correct, this should definitely be included (though most content as an external reference -- we should decide on whether it's appropriate to upload the artifact (e.g. pdf) to the repository in case links change in the future).

A technical advisory on a vulnerability in an open source project that has undergone a coordinated disclosure? (Assuming no)
I think generally 'no', but there are probably cases where it makes sense to do a writeup.

Some kind of "security review" written by the maintainers of an open source project itself? (Assuming no??)
I don't see why the maintainer couldn't do a security review of their own component, but perhaps we should ask for full disclosure in those cases.

Threat models or other documents that fall short of finding specific issues for which proof-of-concept exploits can be demonstrated? (Assuming no)
Actually, I think threat models would be great in a review.

A compliance or other non-technical or semi-technical review of an open source project? (Probably not?)
Don't see why not... there is a review scope (currently "implementation/full, implementation/partial, or non-implementation) that we can use for things like this.

Results from static analysis tools, fuzzers, etc? (Probably not but is there anywhere that collects these?)
Correct, this shouldn't be a data dump of raw results. Though the researcher(/attacker) community would benefit from such a resource.

[david-a-wheeler]

+1 on @scovetta 's comments.

I think a self-review is okay IF it is clearly identified as such.

A data dump from a tool is not okay, but a review that builds on the data data dump would be fine (e.g.,"I ran tool X using configuration Y on library version V. When I did that, I determined that all but 2 were false positives, and those 2 have since been fixed").

[jenniferfernick]

Okay, this seems good - however, I think setting up a labelling scheme that includes clear labels for specific types of reports would make this valuable.

Maybe something like

ProjectName_ReviewType_YYYY

Where ReviewType is one of:

• ExternalSecurityTest
• SelfSecurityTest
• ExternalThreatModel
• SelfThreatModel

But I'd love to see other options of what to include in such a naming convention and what different review types we could add to the list

[scovetta]

Could we leverage the methodology-summary property, and add a few more options?

methodology-summary: static-analysis;external-security-test;threat-model

We could also add a checkbox to the QuickStart page and text to the template about requiring disclosure within the Methodology section if the author of the review is associated with the project.