Skip to content

Research Reproduction Project

Learning Goals

  • Assess the quality and reproducibility of scientific software against the criteria the research community itself uses (the ACM artifact badges)
  • Reproduce results from a published paper, and document precisely where your results agree with the paper and where they do not
  • Practice contributing to an existing codebase you did not write, under its own conventions
  • Practice version control and development best practices within the context of a group assignment
  • Plan and schedule projects in terms of tasks, milestones, and time estimations, and re-plan as required
  • Improve research code using the software engineering practices covered in this course
  • Use agentic AI tools for environment setup and debugging, and verify what they did

Project Context

Software engineering research has been moving fast. From automated program repair to agentic coding tools, new techniques that build software and study the people who build it are published every month, and the arrival of large language models (LLMs) has accelerated that pace further.

One thing fuelling this is open science: researchers share their code and data so that others can build on it, check it, and study new questions with it. To support this, the major software engineering conferences run artifact evaluation tracks, where the code and data behind a paper are examined by reviewers and awarded badges following ACM's Artifact Review and Badging policy:

Badge What it means
Available The artifact is archived in a permanent, publicly accessible location (for example a Zenodo DOI), not just a GitHub repository that could disappear
Evaluated - Functional Reviewers found it documented, consistent with the paper, complete, and exercisable
Evaluated - Reusable It goes further: it is structured and documented well enough for others to build on it
Results Reproduced / Replicated A different team obtained the paper's main results using the artifact (or independently)

Despite the best efforts of artifact review committees, reviewers have limited time, and artifacts that passed review can still be hard to install, hard to run, and sometimes do not produce the numbers in the paper. The newest generation of artifacts has made this harder still: much current SE research depends on LLMs, so an artifact may need paid API access, a GPU, a model that has since been deprecated, or several days of compute. Working out what actually reproduces, and making an artifact easier for the next person, is a real and valuable contribution.

For this project your team will select the artifact of a paper published this year at ICSE 2026, FSE 2026, or ASE 2026, try to reproduce its results, assess its quality, and then plan and execute a development sprint that improves it. All three conferences have already taken place this year, so these artifacts are new: in some cases you may be the first people outside the authors' group to try them.

Deliverables and Deadlines

This is the second project with your team. There are two (2) deadlines. Each of the core deliverables is described below. This project is worth a total of 200 points.

Detailed information for each of the deadlines has been split into its own subpage on the left.

Tip

This is a large assignment spanning from now until the last Friday of classes. We estimate that it will take each student on the team on average 8 hours/week over the next 5 weeks, for a total of roughly 40 hours. Note that Thanksgiving break falls inside the second sprint; plan for it. We highly recommend reading through the entire assignment before starting so you are aware of our expectations for the later deliverables.

To manage all of the write-ups, we recommend saving the pages as a PDF to print or annotate on as you work through the assignment with your team.

A) Artifact Assessment & Planning – 70 points – due Friday, November 6th, 11:59pm

  • Prerequisite: Artifact Selection & Repository Setup
  • Artifact Quality Assessment (35 pts)
  • Project Planning (35 pts)

B) Development Sprint – 130 points – due Friday, December 4th, 11:59pm

  • Process & Implementation Project Snapshot, including a Reproduction Report (60 pts)
  • Checkpoint Presentation (50 pts) - Short Video
  • Mock Pull Request to the Artifact Repository (20 pts)