Sporadic ramblings and interesting information.
The inspiration from this post comes from the TigerHATS parody document entitled "How to Review a Research Article? - A Black Book to Become a Smart Reviewer".
As I have cooperated with several conferences (sometimes as a program chair), I have seen a lot of reviewer comments over the years. Some of them reappeared again, and again, and again (and again, and again, and... again, and...). But those comments were not throwaway comments, they actually were serious advice pointing out problems in the papers.
Therefore, in this post, I present a compilation of useful reviewer comments in the field of computer security (although many of the comments can be applied to other fields, as well), alongside with some personal comments. Also, a disclaimer: The use of these compilation of reviewer comments is entirely at your own risk. I will not be held responsible for any use of the contents of this blog post. :-).
- "Improving the presentation will help to improve the quality of the final paper".
- "Some figures do not show on my computer, or are badly formatted". (Figure quality is a must!).
- "There are many grammar mistakes. English polishing will help to improve the quality of the final paper in the final camera-ready version".
- "The paper does not adhere with the standard formatting style sheet for the proceedings".
- "This conference is specifically oriented to XXXXX, but in your paper you focus on YYYYY - Author(s) should address the relevancy of the work within the XXXX topic". (Link your submission to the theme of the conference).
- "At XX pages long this article ignores the X page limit on submissions". (Always take the page limit into account).
- "The title/abstract/introduction does not capture the content of this paper".
- "The structure of the paper is not clear, and it is quite difficult to follow".
- "The sections do not always follow one another naturally". (It is necessary to maintain a plot thread / storyline in the article).
- "The submission has to be shortened as most of the presented material is quite elementary". (Explain the background of your research according to the expected expertise of the readership).
- "The method can be explained in a clearer way". (Always try to make the paper easy to understand).
- "The timeline and timeframe (at least as presented in the paper) are confusing and unclear". (If we are going to talk about the evolution of something, it must be clearly presented).
- "Lots of the presented content can be expressed in figures and tables to make the paper easier to read, currently its nearly only text". (Use figures and tables to improve the readability of the paper).
- "What is the goal of the work and why is it important to achieve?" (We always need to defend our idea. But we must sell our idea first. Think that you are explaining your idea to someone who does not know anything about the area).
- "No clear context is given". (We have to clearly explain which is our scenario, and where we want to go).
- "It should be made clear which are the additional contributions; compared to XXXX - The main objection I have is the lack of originality". (Always mention explicitly the contributions of our work. In case we improve a previous paper, we must explicitly mention the differences/improvements).
- "The contribution of the paper can be enhanced if it would discuss the advantages of using the proposed method - There is no self-criticism in the proposal". (If we suggest something, we need to explain why it is good... and sometimes discuss why it is bad. In other words, explain where/when your method should be used, and the tradeoffs).
- "Whereas the topic is of importance, the contributions made in this paper are small". (Always try to give weight to the contributions).
- "Consideration of other implications on what is proposed and relevance with other aspects". (Everything we propose in a paper provokes 'a dent in the universe' - we should explain what that 'dent' is).
- "It is not clear how practical the propose scheme is - One detail that was not fully explained in this paper is the deployment model". (Always provide a hint on how the scheme could be applied to the Real WorldTM).
- "For the presentation a clear concentration on a few aspects would be necessary". (Be clear about what you want to explain in the article).
- "The main problem with this paper is that the discussion remains at a too general level and no specific systems or mechanisms are proposed". (It depends on the state of the art and the level of the readership, but it is always good to try to solve a particular problem).
- "On the downside, parts of the paper have quite a narrow focus - While all points raised in the paper are valid, the paper lacks of clear focus". (If we are dealing with a particular problem, we need to take into account all the sides of the dice. However, a paper should focus on a simple set of research questions, in order to avoid being too shallow).
- "Authors cannot pretend to change the specification of the XXX protocol in order to solve the introduced issues". (If we are going to change a standard, our defense of why we want to do that must be 'top-notch').
- "You never evaluate whether your contribution provides such desirable properties". (If you enumerate some properties of your contribution... well, they all must be fulfilled).
- "Comparing the proposed method with existing methods (introduced in the state of the art) could prove the proposed method effectiveness and will improve the quality of the paper".
- "The idea defended by the paper is very trivial and does not add anything to the state of the art - the authors do not even have an originality and novelty claim for their scheme". (We must convince the reviewer that the idea is interesting and promising: sell the idea well!) .
- "The related work section is totally lacking: the authors enumerate too many papers that later are not used in the article" / "the authors only provide a subset of the state of the art" / "there is no state of the art at all". (There is always a reason to have all / few / none SotA in the related work section. Make it explicit!).
- "The complexity of XXXX is a difficult issue and I would suggest addressing this beyond the citation-level". (In some cases [mostly when the context requires it] our assertions need to be backed up with proof, rather than a simple reference).
- "The authors use seven pages to describe [...] and just one page is devoted to the new proposal". (This is a symptom that you do not know how to defend your contribution, or that you need to improve the quality of your contribution [analysis, experiments, comparisons, tradeoffs, ...]).
- "Clearly define what is meant by "XXXX" and what by "YYYY" in the context of this paper - authors do not give a formal definition of the "ZZZZ" property" (Explain any new terms that are introduced in the paper, unless they are well known by the readership).
- "...this term is never used again in the text". (Everything [technical] that is explained in a research paper must be there for a reason).
- "The symbols used in the paper are loosely defined or undefined - No notations table is presented". (If you are going to explain an algorithm, you must explain it clearly, with no ambiguity).
- "The notation does not follow the (de facto / de jure) standards used in the literature". (Always know how things are explained in a certain field).
- "I am concerned that the assumptions are too simplistic". (Assumptions not only need to be mentioned - they also need to be explained and defended!).
- "The model is weak since the assumptions are too strong". (If the assumptions hold, why we need the security mechanism? [e.g. perfect tamper-proof protection for devices located in the Mariana trench]).
- "The proposed scheme assumes some kind of XXXX infrastructure in this context, which does not seem to be really feasible and practical". (We always need to back up our assumptions, explain why we need to include XXXX in this context - and why it can be included).
- "The authors are talking about introducing a "XXXX": it would be good to identify and clearly present the actors in such a XXXX". (If you propose something in an article, its elements / people involved must be explained - or you must explain why you do not give details about them).
- "I am somehow missing the XXXX (human) perspective in the equation". (If we propose something non-technical, we need to provide all the points of view from all actors: citizen / individual / community / governing body / business).
- "Authors focus on protocol XXXX, but it is clear that in the future the protocol YYYY will be used". (If you choose something different than the current trend, explain why).
- "There is no discussion of why these requirements are the right requirements to consider, or whether the list of requirements presented is complete". (If you can provide references or a detailed explanation on this, it would be better).
- "More description of the technical details will help to improve the quality". (This one depends on the readership and its background. Nevertheless, it is good to show the technical details - but not drown the reader in the details).
- "In addition, XXXX is not taken into account for the algorithms". (In some areas [e.g. HW cryptography] there are some factors [e.g. side-channel attacks] that must be considered. We need to discover which are the factors that can affect an algorithm, in order to discuss them).
- "One other point is about the XXXX Algorithm, AAAA et al. managed to break XXXX..." (Always check if the cryptographic algorithms have been broken).
- "Proper analysis can be shown to prove the strength of the approach".
- "No security analysis is given" / "no attacker model is given" / "The selection of attackers seems rather arbitrary". (If we write a security paper, we need to provide the assumptions, the attacker model, and an analysis of why the results are secure for a particular context).
- "I'm also concerned with the performance of the proposed protocol". (Performance is also an important factor that must be analyzed).
- "It has the well-known flaw of (replayability / MiM / DoS / node compromise / etc)". (Check the security of the proposed schemes in the security analysis).
- "There is no self-criticism". (If there is something amiss, do not hide it - fix it, or explain it).
- "The proposed system should be discussed in terms of completeness". (If we are trying to solve a problem with well-known tools, we must make sure that we have considered all the possible ramifications of such tools. Same goes for our proposed solution).
- "More description for experiments should be done" / "Lack of proof-of-concept is another drawback of the paper" / "However the current work remains conceptual". (Experiments are always welcomed - they help to provide an informal proof of the proposed method).
- "The reason of using certain parameters in the experiments should be discussed in more details". (Everything we choose is because of a reason. And the reader must know 'why' in order to be able to duplicate the experiments, or even improve them).
- "The simulation results show that [...] but these results were expected". (If the results of an experiment are as expected [i.e. they are common sense], it usualy means that we need to improve the quality of the experiments).
- "The proposed model overlooks the problems of [...], so the resultant costs are also ignored in the evaluation". (Consider all possible parameters while evaluating your system).
- "The standard deviation in the experiment shows great variance among different users". (If this situation happens, ask yourself why. Do not try to hide it, the reviewer will see it).
- "Did the authors re-implement the algorithms or just take the numbers reported in the other publications?" (When using the results of previous works, we have to be careful on replicating the context of the experiment).
- "The dataset is old". (Always use up-to-date information... if available).
- "A more comprehensive and clearer conclusion is expected".
- "I can't help feeling that (this conclusion) is not properly justified by the preceding analysis". (Every conclusion must come with a justification).
- "Some of the future work described in the paper I would have expected to already be in this one". (Always try to provide a self-contained solution, that can work in a standalone way).
- "The references in this manuscript are out-of-date. Include more recent researches in this field".
- "No/few papers in the references are from recognized *security* conferences" (take this as a warning sign that *security* [or other element] is being downplayed).