TGraphX › Insights › A Practical GNN Reproducibility Checklist with Code
← Back to Insights
Research Note · · 12 min read

A Practical GNN Reproducibility Checklist with Code

Target keyword: GNN reproducibility checklist pytorch code

A Practical GNN Reproducibility Checklist with Code

Reproducibility in GNN research is harder than in standard deep learning. Beyond fixing random seeds, GNN experiments have additional sources of non-determinism: neighborhood sampling order, edge index coalescing, graph construction from raw data, and batch composition. A GNN experiment that is not reproducible is not a reliable result.

This checklist covers every layer of the reproducibility stack for a GNN experiment, with code examples using TGraphX. Each item is marked with its consequence if skipped and how to implement it.

Why GNN Reproducibility Is Harder

In standard image classification, you fix a PyTorch seed and the training is deterministic on the same hardware. In GNNs, additional sources of non-determinism include:

  1. Neighborhood sampling order. NeighborLoader with a different seed samples different neighborhoods, producing different mini-batches and potentially different convergence paths.
  2. Edge coalescing. If your graph construction has duplicate edges and you coalesce them, the order of coalescing can affect which edges survive.
  3. Graph construction from raw data. If you build a kNN graph from embeddings, floating-point distance computation may produce ties that break differently across hardware.
  4. Sparse scatter operations. Some CUDA scatter ops are not deterministic by default (non-atomic vs atomic accumulation order).
  5. DataLoader worker processes. Using num_workers > 0 without per-worker seeding introduces non-determinism.

Checklist Item 1: Seed All RNG Sources

python
from tgraphx.reproducibility import set_seed
        
        # Sets: torch, torch.cuda, numpy (if installed), random, and PYTHONHASHSEED
        set_seed(42)

Or use TGraphX's context manager:

python
import tgraphx as tgx
        
        with tgx.reproducible(seed=42, deterministic=True):
            # All RNG sources fixed for this block
            # deterministic=True also sets torch.backends.cudnn.deterministic=True
            # and torch.use_deterministic_algorithms(True)
            pass

If skipped: Training loss curves are not reproducible. Results cannot be verified by others.

Checklist Item 2: Seed the NeighborLoader

python
from tgraphx import NeighborLoader, Graph
        
        loader = NeighborLoader(
            g,
            fanouts=[15, 10],
            batch_size=64,
            seed=42,          # REQUIRED — seeds the sampling RNG separately
        )

If skipped: Even with set_seed(42), different mini-batches will be sampled across runs. This is because the loader's internal RNG may not be controlled by the global seed.

Checklist Item 3: Fix Train/Val/Test Splits

python
import torch
        from tgraphx.reproducibility import set_seed
        
        set_seed(42)  # seed BEFORE creating masks
        
        N = 1000
        perm = torch.randperm(N)
        train_mask = torch.zeros(N, dtype=torch.bool); train_mask[perm[:600]] = True
        val_mask = torch.zeros(N, dtype=torch.bool); val_mask[perm[600:800]] = True
        test_mask = torch.zeros(N, dtype=torch.bool); test_mask[perm[800:]] = True
        
        # Verify no overlap
        assert not (train_mask & val_mask).any()
        assert not (train_mask & test_mask).any()
        assert not (val_mask & test_mask).any()

If skipped: The same data point may appear in both training and evaluation, leading to data leakage that overstates performance.

Checklist Item 4: Record Exact Versions

python
import tgraphx, torch, platform
        
        print(f"TGraphX: {tgraphx.__version__}")
        print(f"PyTorch: {torch.__version__}")
        print(f"CUDA: {torch.version.cuda}")
        print(f"Python: {platform.python_version()}")
        print(f"OS: {platform.system()} {platform.release()}")

Document this in every experiment. Library updates — even patch releases — can change default initializations, layer behavior, or sampler internals.

If skipped: "It worked with version X" cannot be reproduced if version X is not recorded.

Checklist Item 5: Use Deterministic CUDA Algorithms

python
import torch
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        # For PyTorch >= 1.11:
        torch.use_deterministic_algorithms(True)

Consequence: Some CUDA scatter ops are inherently non-deterministic due to floating-point accumulation order in parallel reductions. Setting deterministic mode enforces atomic accumulation at a throughput cost (~5-20% slower on large graphs).

TGraphX's reproducible(deterministic=True) sets all of these automatically.

Checklist Item 6: Validate Graph Construction

python
from tgraphx import validate_graph
        
        validate_graph(g, strict=True)
        # Checks: edge index range, dtype, feature-label count match,
        #         mask coverage, no duplicate edges (in strict mode)

If skipped: A silent shape mismatch during graph construction can corrupt the entire experiment. See shape-aware validation in TGraphX for what each check covers.

Checklist Item 7: Save the Graph and Model Checkpoint

python
# Save the graph
        g.save("runs/exp01/graph.tgx")
        
        # Save model and optimizer state at best validation epoch
        torch.save({
            "epoch": best_epoch,
            "model_state_dict": model.state_dict(),
            "optimizer_state_dict": optimizer.state_dict(),
            "val_accuracy": best_val_acc,
            "seed": 42,
        }, "runs/exp01/checkpoint.pt")

If skipped: An experiment that cannot be continued or re-evaluated from a checkpoint is not reproducible in the strong sense.

Checklist Item 8: Run Multiple Seeds and Report Variance

A single seed run is not a reliable result. The standard in GNN research is 3-10 seeds:

python
import torch.nn as nn
        import torch.nn.functional as F
        
        results = []
        for seed in [42, 123, 456, 789, 1024]:
            set_seed(seed)
            model = YourGNNModel()
            loader = NeighborLoader(g, fanouts=[15, 10], batch_size=64, seed=seed)
            # ... training loop ...
            results.append(val_accuracy)
        
        import statistics
        print(f"Val accuracy: {statistics.mean(results):.4f} ± {statistics.stdev(results):.4f}")

If skipped: A single result may be a lucky or unlucky seed. Variance across seeds reveals training stability.

Checklist Item 9: Generate a Reproducibility Report

TGraphX can generate a reproducibility report for an experiment:

python
from tgraphx.reproducibility import set_seed
        import tgraphx as tgx
        
        set_seed(42)
        
        with tgx.reproducible(seed=42, deterministic=True):
            result = tgx.easy.train_node_classifier(
                g, model="tensor_gcn", epochs=20, seed=42
            )
        
        result.summary()
        # Includes: seed, library versions, graph stats, metrics per epoch
        # result.to_json("runs/exp01/result.json") saves reproducibility metadata

Checklist Item 10: Separate Code From Results

Store your experimental code, graph data, and results in separate directories:

runs/
          exp01/
            graph.tgx               # saved graph
            config.yaml              # hyperparameters
            checkpoint.pt            # model weights
            reproducibility.json     # versions, seed, metrics
            results.json             # per-seed results
        src/
          model.py
          train.py

This separation ensures that modifying the model code does not overwrite previous results.

Full Minimal Reproducible Experiment Template

python
import torch
        import torch.nn.functional as F
        import tgraphx as tgx
        from tgraphx import Graph, NeighborLoader
        from tgraphx.reproducibility import set_seed
        from tgraphx.layers.vector_gcn import GCNConv
        import torch.nn as nn
        import json, platform
        
        SEED = 42
        set_seed(SEED)
        
        # 1. Build graph
        N = 500
        x = torch.randn(N, 64)
        edge_index = torch.randint(0, N, (2, 2000), dtype=torch.long)
        labels = torch.randint(0, 4, (N,))
        perm = torch.randperm(N, generator=torch.Generator().manual_seed(SEED))
        train_mask = torch.zeros(N, dtype=torch.bool); train_mask[perm[:300]] = True
        val_mask = torch.zeros(N, dtype=torch.bool); val_mask[perm[300:400]] = True
        
        g = Graph(node_features=x, edge_index=edge_index, node_labels=labels,
                  train_mask=train_mask, val_mask=val_mask)
        
        # 2. Define model
        class TwoLayerGCN(nn.Module):
            def __init__(self):
                super().__init__()
                self.c1 = GCNConv(64, 128)
                self.c2 = GCNConv(128, 4)
            def forward(self, x, ei):
                return self.c2(F.relu(self.c1(x, ei)), ei)
        
        model = TwoLayerGCN()
        opt = torch.optim.Adam(model.parameters(), lr=1e-3)
        loader = NeighborLoader(g, fanouts=[15, 10], batch_size=64, seed=SEED)
        
        # 3. Train
        best_val_acc = 0.0
        for epoch in range(20):
            model.train()
            for batch in loader:
                opt.zero_grad()
                logits = model(batch.node_features, batch.edge_index)
                sl = batch.seed_logits(logits); sy = batch.seed_y
                if sl.numel() == 0: continue
                F.cross_entropy(sl, sy).backward()
                opt.step()
        
        # 4. Save metadata
        meta = {
            "seed": SEED,
            "tgraphx": tgx.__version__,
            "torch": torch.__version__,
            "python": platform.python_version(),
        }
        with open("reproducibility.json", "w") as f:
            json.dump(meta, f, indent=2)
        
        print("Done. Reproducibility metadata saved.")

Limitations

Deterministic mode has throughput cost. torch.use_deterministic_algorithms(True) forces atomic scatter, which is 5-20% slower on GPU for large graphs. This is acceptable for reproducibility; for wall-time comparison experiments you must document whether deterministic mode was used.

Different hardware is not fully reproducible. Even with all seeds set, results may differ between CPU and GPU, between GPU models, or between CUDA versions. This is a known limitation of floating-point arithmetic; document hardware in your reports.

This checklist is for standard GNN training. Distributed training, custom CUDA kernels, or multi-process samplers introduce additional non-determinism sources not covered here.

Related Articles