Conclusions from Flaw Source Breakdowns

[samm82]

As mentioned in #133, I "need to know exactly what the figure says" and "have clear conclusions that [I] can draw from the data presented in the figure", which obviously also applies to tables. This was feedback directed at my graphs that track which source tiers flaws come from. Inspired by this, I've taken a look at related tables to determine what conclusions I can draw

Flaw Summary Tables

The breakdown of flaws by manifestation and domain; see #155

General

• flaws seem relatively evenly distributed across source tiers, even accounting for differences in length and number of sources comprising them (this may be due to papers that aren't fully investigated)
• the vast majority of flaws are explicit, which helps support our methodology

Manifestations

• contradictions are by FAR the most common way for a flaw to manifest, which makes sense: if two authors do not communicate or work with different resources, there is a higher chance that they will disagree
• contradictions also have the highest ratio of implicit to explicit flaws, which also makes sense: when two authors record the same information, it's possible that only one also records important tacit information, such as prerequisites or caveats
• omissions and redundancies are very rare: omissions are hard to quantity (since proving a negative is tricky) and since these documents are long enough, repetition is undesirable and relatively easy to detect in the review phase

Domains

• traceability is by far the rarest domain since, again, proving a negative is tricky, so we only note traceability information that is incorrect as opposed to that which is missing
• perhaps unsurprisingly, definition flaws are the most common, which is concerning as this is what these documents are mainly used for!
• category flaws are more implicit than explicit, likely due to the lack of standardization of the categories we ourselves use!
• synonym flaws are both the next most common and implicit, again, possible due to uncaptured supplementary information

Flaw Source Table/Pie Charts

These are based on the table form of the original pie charts; I wasn't able to come up with any conclusions specifically from the pie charts, but maybe their more visual nature could be beneficial?

• ironically, standards seem to be the most internally inconsistent, both within documents and between authors, potentially due to their length
• textbooks seem the most correct in isolation, and most consistency flaws are from other documents and/or textbooks
• terminology collections that should (or explicitly claim to) be based on established standards disagree with them the most
  • this extends beyond just collections; standards are the most-disagreed-with source tier
• comparing flaws between sets of authors is slightly biased against standards, since they have the smallest ratio of authors to documents and there are only two non-standard instances of documents with the same set of authors (SWEBOK and Gerrard)

Original Intent

As I mentioned in this comment, I originally included the pie charts to show that

1. "standardized" documents aren't actually standardized
2. even when they are correct, other documents differ from/disagree with these standards
3. documents also disagree with less standardized (but still useful) sources (i.e., terminology collections and textbooks)
4. issues occur even within the same document and between documents with the same author(s)!

[smiths]

This is a good exercise to complete @samm82. Thank you for taking the initiative.

[JacquesCarette]

I agree - what is the current status?

[samm82]

This was essentially just a place for me to collect my thoughts; I'll likely reassess this once the Flaws section has been locked down (so our conclusions are more informed), hence the "low priority" label.