Summary of feedback from ICSE paper submission

[samm82]

Weaknesses of the Paper

(summarized and listed as action items)

• the motivation of the work is based on unsubstantiated claims; are there any cases where problems have arisen due to non-standardized software testing terms?
  • while having crystal-clear definitions for terms would be ideal, in reality, it's challenging because new technologies, tools, and terms are constantly emerging
• it is unconvincing that having these discrepancies has serious consequences and that there exists a practical way to have a standardised and consistent taxonomy
• the research methodology is subjective and not fully explained
• the choice of sources is not clear; what qualifies "trusted"?
• threats to validity are not discussed
• no evidence is provided that the proposed recommendations actually improve anything
• the recommendations are limited to three testing terms (recovery, scalability, and performance); why were these three chosen?
• the paper has various presentation issues
• (because of double-blindness) the supporting tool doesn’t seem to be available, which makes a replication difficult

Questions

(With duplicates combined)

1. How a lack of a standardised and consistent taxonomy can lead to more complicated and expensive testing activities?
2. More in general, what are the main and concrete advantages of having such standardised taxonomy?
3. What are the criteria for selecting sources?
4. When analysing existing terminology, how did you decide what is the "clearest and most concise" version of a definition?
5. Why are the recommendations limited to only three testing terms?
6. How do the recommendations presented in Section VI potentially make testing more productive?
7. How do the authors foresee a strategy to produce a “standardised” taxonomy, meaning that it will become the de facto taxonomy? There is a big gap from producing a taxonomy and having it becoming the standardised one.

[smiths]

Thank you for the summary @samm82. The reviewers were surprisingly harsh. We knew the paper was a longshot, but I'm surprised they didn't see the inherent value in work of this type.

We won't change the reviewer's minds, but it is probably still worth it to write a rebuttal. We don't want the same criticisms of the paper being applied to your thesis. Writing a rebuttal will give us arguments we can be sure to highlight in the thesis.

[JacquesCarette]

I think we're better off writing two rebuttals: one for ourselves and one (very short) for the reviewers. They were quite harsh but nevertheless took their time to properly review the paper.