Paper Reviews
Overview of Critical Paper Reviews¶
During the course of the class, students will be required to submit two critical paper reviews on any two papers of their choosing (excluding the paper(s) they present) throughout the semester. The instructor will go over the expectations of these paper reviews in class, and students can use the resources below to guide them in writing their critical reviews.
Collaboration Policy¶
You are not allowed to work with other students on your critical reviews. You may discuss high level details of a paper, however, your review must be your own original work.
Critical Review Instructions and Grading¶
The purpose of a critical paper review is to get you to look for, think about, and describe the positive and negative elements of the papers that we will be reading over the course of the semester. No research paper is perfect, but all have positive and negative aspects. Your goal in these reviews is to point out both, similar to the role of a program committee member at a major software engineering or machine learning conference.
Content of the Review¶
- Paper Summary: The paper summary should provide a brief synopsis of the paper including the motivation, key idea of the approach, relevant technical details, and overview of the evaluation methodology, and a brief summary of the results.
- Critical Evaluation Criteria: In this section, you will comment on the good and bad of various aspects of the paper.
- Soundness of Approach: This sub-section should evaluate the technical validity of the paper. More specifically, you should assess the validity of a proposed approach or empirical research methods used. Some questions to ask may be "Does the proposed approach seem to achieve what the authors claim?", "Do the technical components of the approach make sense for the stated problem?", and "Are there any limitations to the approach that are not justified by the authors?".
- Novelty: This sub-section should examine the "newness" of the key ideas presented in the paper. Some questions to ask related to this may be "What has recent past work on this topic accomplished? What is new about this paper?", "What is the key thing that this paper is doing different?".
- Clarity of Relation with Related Work: The extent to which the paper is clearly positioned in the frame of previous research in the field. Questions that may be considered when evaluating this criterion are "To what degree is the related work complete?" "Are important papers missing?" "To what extent does the paper explain how the work in question is motivated by, complements, and/or contradicts the cited literature?"
- Quality of Evaluation & Results: The quality of the performed (empirical) evaluation. Questions that may be considered when evaluating this sub category are "Is the empirical setting designed for the evaluation technically sound?", "Is the evaluation implemented in a rigorous and technically sound manner?", "To what extent is data collected and analyzed in a reliable manner?", "To what extent to the results reflect the authors claims?"
- Ability to Replicate: The extent to which the paper includes information to enable independent replications of the study. Questions that should be considered when evaluating this are "Does the paper provide sufficient detail to foster future replication?", "Are supplemental materials provided? For quantitative work, are data and analysis scripts available to the extent possible? For qualitative work, have survey questionnaires and/or interview guides been shared?", "Are supplemental materials well documented?"
- Quality of Presentation: The extent to which the quality of the paper's writing and content organization meet the standards of the research community. Questions that should be considered when evaluating this are "How well does the clarity of the writing meet expectations?", and "How well does the organization of the paper meet expectations?"
Submission Instructions¶
Critical Paper Reviews should be submitted via Webcourses to the appropriate assignment. Note that a critical review must be turned in before the paper is discussed in class, that is 1:30pm on the day that the paper is being presented in class
Grading Criteria¶
Category |
Professional (100%) |
Adequate (75%) |
Needs Work (50%) |
Serious Problems (25%) |
Grade |
|---|---|---|---|---|---|
| Summary | Full grasp (more than needed) of material in the paper presented in a concise but lucid fashion | Solid description of material in the paper, but some important technical details may not be described with perfect precision | Less than a full grasp of the information in the paper, some important concepts may be missing | No grasp of information, some misinformation, and does not capture enough important parts of the paper | 25% |
| Soundness | Potential issues or positive aspects are described clearly with supporting evidence from the paper. Concepts are used correctly and there is no misinformation | Potential issues or positive aspects are described but one or two important points may be missing. Concepts are used correctly, but there may be some ability when linking the discussion back to the paper. |
Some potential issues or positive aspects are missing or described incorrectly. Come concepts are misunderstood which detracts from the review |
Most potential issues or positive aspects are missing, concepts and links back to the paper content are absent. | 10% |
| Novelty | 10% | ||||
| Related Work | 10% | ||||
| Evaluation & Results | 10% | ||||
| Ability to Replicate | 10% | ||||
| Quality of Presentation | 10% | ||||
| English Usage and Grammar | No misspelled words or grammatical errors | No more than two misspelled words or grammatical errors | Three-five more misspelled words or grammatical errors | More than 5 misspelled words or grammatical errors | 15% |