Research Note · June 21, 2026 · 7 min read

Open-Source Readiness for GNN Research: A Practical Checklist

Target keyword: open source GNN research reproducible checklist

Open-Source Readiness for GNN Research: A Practical Checklist

You finished the experiments. You wrote the paper. You are about to release the code with the paper. This checklist is what to do before you press publish.

The goal is not perfection. The goal is that a competent third party can clone your repo, follow your README, and reproduce your headline numbers within a small tolerance. Most research code does not meet this bar. The checklist below is designed to get you to it without excessive engineering work.

Repository structure

[ ] README.md at the root. First section: what the project does in two sentences. Second section: how to install. Third section: how to reproduce the headline result.
[ ] LICENSE file. MIT, Apache 2.0, or BSD-3 unless your institution requires otherwise. Avoid "no license" — without a license, others legally cannot use your code.
[ ] CITATION.cff or BibTeX block in the README, so people can cite correctly.
[ ] A small requirements.txt with pinned major versions of dependencies.

Environment

[ ] A lock file. pip freeze > requirements_lock.txt captures the exact versions you ran with. Without this, anyone reproducing later may get a different PyTorch version and get different numbers.
[ ] Python version range. Document the minimum Python version. Test against it.
[ ] CUDA/CPU notes. If you used a specific CUDA version, document it. If your code requires GPU, say so.
[ ] A conda environment file (optional, but very nice for reviewers).

Data

[ ] Data acquisition documented. If you use a public dataset, link to it and document any preprocessing.
[ ] Preprocessing scripts in the repo. Not "I ran some script that's lost now." The exact code that converted raw data into model-ready format.
[ ] A hash of the preprocessed data. So reviewers can verify they have the same preprocessed version you used.
[ ] Sample data for testing. A small subset that lets reviewers verify the pipeline works without downloading the full dataset.

Code

[ ] Working code on the main branch. Not "see the experimental branch." The default branch should run.
[ ] A main.py or equivalent entry point that reproduces the headline result with default arguments.
[ ] Configuration in files, not hardcoded. YAML, JSON, or argparse — anything that lets a reviewer change a hyperparameter without editing source.
[ ] No notebook-only experiments. Notebooks for exploration are fine, but the publication-quality results should be runnable from the command line.

Reproducibility

[ ] All seeds are documented. random.seed(42), numpy.random.seed(42), torch.manual_seed(42), and any framework-specific seeding.
[ ] Multiple seeds reported. Run with at least 3 seeds (5 better), report mean ± std.
[ ] A single command that reproduces the headline number. python main.py --config configs/main.yaml --seed 42 is good. Five sequential commands with manual edits in between is bad.
[ ] Deterministic flag where possible. torch.use_deterministic_algorithms(True) if your operations support it.
[ ] Artifact directories. Each run writes to its own directory with config, metrics, and a system fingerprint.

For TGraphX-based projects:

[ ] tgx.reproducible(seed=42, deterministic=True) wrapping the main experiment.
[ ] tgx.audit_run_dir() shows all expected artifacts present.
[ ] reproducibility_report.json committed for the headline result. This is the system fingerprint that lets reviewers compare environments.

Documentation

[ ] README has a "Reproducing the paper" section. Step by step. Tested by you. Tested by someone else if possible.
[ ] Expected runtime documented. "This should take ~6 hours on an RTX 3090." Saves reviewers time.
[ ] Expected results documented. "Final test accuracy should be around 84% ± 1.5%." Lets reviewers verify they reproduced correctly.
[ ] Known issues documented. If you know that on certain GPUs the results drift by 2 points, say so.

Benchmark reporting

[ ] Mean and std reported. Single-seed numbers are not benchmarkable.
[ ] Hyperparameter search budget disclosed. "We tried 30 combinations" is honest. "We used these hyperparameters" leaves the search budget invisible.
[ ] Split methodology documented. Standard split or custom? If custom, with what seed?
[ ] Compute used disclosed. Hours of GPU time, model size. Helps others scale to their resources.

Testing

[ ] A small test that verifies the code runs. Not testing accuracy — just testing that the pipeline does not crash on a tiny dataset. Run in CI if possible.
[ ] One smoke test that reproduces a single-seed result. Faster than the full benchmark, exercises the same code path.

Sharing

[ ] Public GitHub or GitLab repository. Not a tarball on a lab webpage.
[ ] A tagged release for the paper version. git tag v1.0-paper and push the tag. If you continue developing after publication, the tag preserves the publication state.
[ ] Issues enabled. So reproducers can ask questions.
[ ] A statement about response times. "Best-effort responses to issues" is honest and sets expectations.

Beyond the basics

These are not required for "reproducible" but make the code significantly more useful:

[ ] A Docker image or Containerfile. Eliminates "works on my machine" entirely.
[ ] Continuous integration that runs the smoke test. Catches regressions automatically.
[ ] Pretrained model checkpoints in releases. Lets reviewers verify the headline number without retraining.
[ ] A demo notebook. Runs in 5 minutes, shows the main idea on a tiny dataset.
[ ] A FAQ section in the README. Pre-empts common questions.

What this checklist does NOT solve

It does not guarantee reproducibility across hardware. GPU floating-point varies.
It does not guarantee long-term reproducibility. In 5 years, the pinned PyTorch version may not install on then-current CUDA.
It does not guarantee that anyone will actually use your code. People reproduce papers more often when the code is genuinely accessible.

But it gets you to "a determined person with a similar GPU can reproduce within a small tolerance." That is the realistic bar for GNN research.

Why this is worth the time

The fraction of published GNN papers whose code reproduces their headline numbers is alarmingly low — single-digit percentages in some areas. The reasons are almost always avoidable: missing seed, missing version pin, missing preprocessing script, missing config. Each item on this checklist closes one of those gaps.

The time investment is real — typically 2-5 hours for a polished release. That investment pays off because:

Your future self can reproduce your own work in 12 months when you've forgotten the context.
Reviewers can verify claims without contacting you.
Other researchers can build on your work, increasing citation and impact.
The field's reproducibility culture improves one paper at a time.

The framework's tooling makes much of this easier — the tgx.reproducible() context, the run-directory layout, the audit utilities. If you are using TGraphX, you get the artifacts almost for free. The discipline is still yours.

FAQ

Q: What if I can't release the data?
A: Document the data acquisition clearly and provide synthetic data for testing the pipeline. Document the expected difference between synthetic and real results.

Q: What if my model is too large to share as a checkpoint?
A: Use a model registry (Hugging Face, Zenodo) and link from the README. Or provide a smaller variant for verification.

Q: How much testing is really necessary?
A: One smoke test that runs in under a minute and verifies the pipeline does not crash. More is nice; less is risky.

Q: What about closed-source dependencies?
A: Document them clearly. Note that reproducibility depends on access. If possible, provide a path for reviewers to get access (e.g., free academic license).

Q: How do I handle dependencies that have moved or been deleted?
A: For a polished release, fork the dependency and vendor it. For pragmatic releases, document the original version and trust that reviewers can find archived versions.