Skip to content

Research Project Checkpoint 2: Development Sprint Deliverables

Deliverables

Development Sprint – 130 points – due Friday, December 4th, 11:59pm

Process & Implementation Snapshot (60 pts)

By this point every member of your team should have the artifact running locally from Checkpoint 1. Once that is true for everyone, you can start development.

You should follow best practices for collaborative development such as using feature branches, pull requests, and code review. Individual changes should be committed to properly named branches for each core piece of work and make use of pull requests to merge changes together.

Be sure to review each other's code, both from a quality assurance standpoint and so you have a better understanding of the codebase. Although code reviews can be done asynchronously, we encourage the first few to be done synchronously so you can discuss the changes with your teammates and learn more about the artifact.

Every member of your team must contribute to the implementation. One way we will evaluate this is that each team member must have at least one commit as a part of the solution. Failure to do so will result in a significant penalty to your grade.

You must merge all of your changes into the main branch of your team repository before the deadline. All of your work stays inside your team repository: you should not open pull requests or issues against the authors' repository (see the mock pull request below).

Submit a link to your repository and your GitHub project board to Webcourses. For grading, we will take a snapshot of your repository and board right at the deadline and grade based on your progress there.

Reproduction Report

Alongside the code, your repository must contain docs/cen5016/reproduction.md, written from the log you kept during Checkpoint 1 and updated through the sprint. It is the document you wish the authors had shipped. It should cover:

  1. Environment: the operating system, hardware, language and library versions, and any API keys or external services needed, with the cost or quota a full run consumes
  2. How to run it: the exact commands to go from a fresh clone to each result, after your changes
  3. Results: a table with one row per result you attempted, giving the number from the paper, the number you obtained before your changes, and the number you obtain after them, with a short note on any difference
  4. What could not be reproduced, and why, as precisely as you can say
  5. Changes you made that affect how results should be interpreted (for example a newer model, a data subset, a bug fix that changed a number)

AI Agent Requirement

You are required to use an LLM-assisted or agentic software engineering tool to implement at least one of your backlog issues. You are welcome to use it on more.

Keep logging as you did at Checkpoint 1. Commit this sprint's sessions to ai-logs/research-checkpoint-2/<your-github-username>/, and follow whatever your team wrote down in the SDE project's standards.md about flagging AI-assisted work.

Pick one issue to write up in detail: recount, in the issue itself, the positive and negative experiences you had using the agent on it. Label that issue "🤖 AI AGENT ISSUE" so we can find it easily in your repository.

Research code is a good test of an agent's limits. It will happily "fix" a failing experiment by changing what it measures, or declare a result reproduced without running it. Every number in your reproduction report must have been produced by a command a human on your team ran.

Checkpoint Presentation (50 pts)

Your group should prepare a 10-20 minute video presentation describing your work.

Your slides should include the following information (the recommended slide count is in parentheses):

  1. Title Slide (1 slide)
    Include your team name & the names of each team member

  2. The Paper and the Artifact (~1-2 slides)
    What does the paper claim, in one slide? What is the artifact, and what state did your team find it in at Checkpoint 1?

  3. Reproduction Results (~1-2 slides)
    The table from your reproduction report: paper versus you, before and after your changes. What reproduced, what did not, and what you think explains the gaps.

  4. Project Planning (~1 slide)
    Which candidates did your team choose to focus on, and why those?

  5. Development Progress (~1-2 slides per person)
    What have you changed? How did you test it? What have you learned about the codebase? How does this compare with your original milestone goals, and if you did not meet them, why?

    Each team member should describe the contributions they made. In this section, you must include a screen-share demo of the artifact after your changes; if you were not able to get something working, include screenshots that show your current progress and clearly explain what impeded it.

  6. Process Overview (~1-2 slides)
    What process strategies (meeting frequency, code reviews) did your team use? Was it effective? Did you encounter any difficulties or have to make any changes? How did the second project with your team differ from the first?

  7. AI Use Retrospective (~1-2 slides)
    What was your experience using LLMs or AI agents on research code, as opposed to the SDE project's application code? What worked well? What did not? Where did you have to check its work most carefully?

  8. Advice to the Authors (~1-2 slides)
    If the authors asked you what they should do differently next time, what would you tell them? What should the artifact evaluation committee have caught? What is the one change with the biggest payoff that you did not get to?

A Note on Participation

Participation from all team members during the presentation is required.

The presentation slides should be exported into a PDF and a video recording of your presentation should be submitted to Webcourses by the deadline.

Mock Pull Request to the Artifact Repository (20 pts)

We would like you to create a mock pull request in your team repository containing the changes you would offer to the authors. Open it from a branch holding those changes against your imported baseline, so the diff shows exactly what you are proposing. Dr. Moran will review the pull requests and work with teams that want to submit them upstream in a responsible manner. If the authors accept your changes, congratulations: you have made a contribution to open-source research software!

The pull request description should be the kind a maintainer would want to receive:

  • Motivation: what was wrong or missing, with a link to the relevant part of your assessment or reproduction report
  • Changes: what you changed and why, organized by concern, with links to the relevant commits
  • Testing: how you verified it, and whether the artifact's own tests or CI (if any) still pass
  • Reproduction status: what a user of the artifact can now reproduce that they could not before, in numbers

If the artifact exists only as a Zenodo or Figshare archive with no repository, still write the pull request in your own repository; the description doubles as the message Dr. Moran can forward to the authors.

Pull Request Warning

Do NOT submit the pull request to the authors' repository before talking with Dr. Moran. Doing so will result in significant points lost. Simply use a mock pull request in your own team repository. The same applies to opening issues on the authors' repository or emailing the authors: route it through Dr. Moran.

Grading

To receive full credit for the implementation, we expect:

  • Fulfillment of implementation goals as outlined by the planned milestones, or a clearly written justification in the milestone description of why elements fell through
  • docs/cen5016/reproduction.md present and covering every section listed above, with a results table
  • An updated project board for the current progress
  • Reasonable code structure and style, including documentation where appropriate, and a README that lets a stranger run the artifact
  • Coherent commits of reasonable size with meaningful commit messages by all team members
  • High quality usage of Git/GitHub tools, such as issues, milestones, pull requests, and commits
  • AI sessions logged under ai-logs/research-checkpoint-2/<github-username>/ and one issue labelled "🤖 AI AGENT ISSUE"

To receive full credit for the presentation, we expect:

  • All team members to contribute towards presenting their team's progress
  • All of the material described above covered in the presentation, including the reproduction results table and a demo
  • PDF slides uploaded to Webcourses for grading & feedback

To receive full credit for the pull request, we expect:

  • A well-written pull request in your team repository, opened from a branch against your imported baseline, covering all four sections above
  • Links to relevant commits and compliance with any automated tests that the artifact has
  • No pull request, issue, or email to the authors without Dr. Moran's go-ahead

Resources & Documentation

Working in Research Code

Before starting development, take the time to analyze the artifact with your team using the code archaeology techniques from early in the course. A good first exercise is to trace one number in the paper backwards: find the script that prints it, the function that computes it, and the data it reads. Most artifacts have a handful of such paths and everything else is scaffolding.

You will find general development resources at the link below: