Reproducibility Checklists for Tensor-Valued GNN Experiments

Reproducibility in machine learning has two levels: exact reproduction
(getting the same numbers on the same hardware) and close reproduction
(getting the same conclusions with comparable numbers). Exact reproduction is
often impractical due to hardware variation and non-deterministic operations.
Close reproduction is achievable with disciplined experiment documentation.

This checklist focuses on what to record and how to record it, using
TGraphX's built-in utilities where they apply.

Why Tensor-Valued GNNs Need Extra Attention

Standard GNN reproducibility concerns apply (seed, library version, dataset
split). Tensor-valued GNNs have additional considerations:

• Node feature structure: the shape [N, C, H, W] is part of the
  experiment definition, not just a configuration parameter
• Graph construction method: how edges were built (grid, k-NN, full)
  affects results and must be documented
• Preprocessing applied to tensor features: normalization, augmentation
  that operates on the trailing dimensions

The Reproducibility Checklist

1. Environment

• [ ] Python version (from env_report)
• [ ] PyTorch version (from env_report)
• [ ] TGraphX version (from env_report); pin in requirements.txt
• [ ] CUDA version if using GPU
• [ ] Hardware (GPU model, CPU) from env_report(include_hardware=True)

from tgraphx.performance import env_report
        from tgraphx import write_hardware_report
        
        info = env_report(include_hardware=True)
        # Records: python/torch/tgraphx/cuda versions, GPU name
        write_hardware_report(path="./logs/hardware.json")
        

2. Randomness

• [ ] Random seed set and recorded
• [ ] deterministic=True status recorded (affects reproducibility and speed)
• [ ] Data loading shuffle seed if applicable

from tgraphx.reproducibility import set_seed
        
        SEED = 42
        set_seed(SEED, deterministic=True)
        # Record: seed=42, deterministic=True
        

3. Graph Schema

• [ ] Node feature shape: [N, ...] with all trailing dimensions specified
• [ ] Edge index shape: [2, E]
• [ ] Edge feature shape: [E, ...] if present
• [ ] Node label shape and class distribution
• [ ] Graph construction method (grid, k-NN with k=?, build_fully_connected,
  image_to_patches with patch_size=?)

from tgraphx import write_graph_stats
        
        write_graph_stats(g, path="./logs/graph_stats.json")
        # Writes: node count, edge count, feature shapes, device
        

4. Data Split

• [ ] Train/val/test split method (random, predefined, stratified)
• [ ] Split ratios or sizes
• [ ] Whether split was performed before or after normalization
• [ ] Node-level vs graph-level split for your task

5. Preprocessing

• [ ] Feature normalization applied (mean/std, min-max, none)
• [ ] Parameters of normalization (mean, std values)
• [ ] Any graph augmentation (edge dropout, feature masking)
• [ ] Preprocessing order relative to split

6. Model Configuration

• [ ] Architecture name and type
• [ ] Layer types and dimensions
• [ ] Number of layers
• [ ] Activation functions
• [ ] Regularization (dropout rates, weight decay)

from tgraphx import write_experiment_config
        
        write_experiment_config({
            "architecture": "2-layer TensorGraphSAGELayer",
            "hidden_dim": 128,
            "num_layers": 2,
            "dropout": 0.1,
            "activation": "ReLU",
        }, path="./logs/model_config.json")
        

7. Training Configuration

• [ ] Optimizer type and hyperparameters (lr, weight decay, betas for Adam)
• [ ] Learning rate schedule if used
• [ ] Number of epochs or steps
• [ ] Early stopping criteria and patience (if used)
• [ ] Batch size

8. Evaluation

• [ ] Metric(s) used and their exact definition
• [ ] Whether OGBNodeEvaluator was used (if applicable — Beta v0.5.0+)
• [ ] Evaluation split used (test vs validation for final reporting)
• [ ] Whether multiple runs were performed; if so, seed range

9. Known Limitations

• [ ] Hardware-dependent behaviors noted (e.g., float16 attention upcast
  affects GPU but not CPU)
• [ ] Any non-deterministic operations identified
• [ ] Graph topology choices that might affect generalization

HTML Checklist Widget

For interactive documentation (HTML static sites), use a details/summary
widget for the checklist:

<details open>
        <summary><strong>Experiment Reproducibility Checklist</strong></summary>
        
        <ul>
          <li><input type="checkbox"> Environment: Python, PyTorch, TGraphX versions recorded</li>
          <li><input type="checkbox"> Seed: set_seed(42, deterministic=True) called</li>
          <li><input type="checkbox"> Graph schema: node_features shape documented</li>
          <li><input type="checkbox"> Graph schema: edge construction method documented</li>
          <li><input type="checkbox"> Data split: method and ratios recorded</li>
          <li><input type="checkbox"> Preprocessing: normalization parameters saved</li>
          <li><input type="checkbox"> Model: architecture and hyperparameters in config.json</li>
          <li><input type="checkbox"> Training: optimizer, lr, epochs recorded</li>
          <li><input type="checkbox"> Evaluation: metric definition and split documented</li>
          <li><input type="checkbox"> Limitations: hardware dependencies noted</li>
        </ul>
        
        </details>
        

Note: HTML checkboxes in static sites are presentational only unless backed
by JavaScript state management. They are useful as checklists for human
readers, not for automated tracking.

Putting It Together: Experiment Artifact Structure

A complete reproducibility artifact for a TGraphX experiment:

experiment_v1/
        ├── env/
        │   ├── hardware.json          ← write_hardware_report
        │   └── requirements.txt       ← pinned versions
        ├── data/
        │   ├── graph_stats.json       ← write_graph_stats
        │   └── split_info.json        ← train/val/test sizes
        ├── config/
        │   ├── experiment_config.json ← write_experiment_config
        │   └── model_config.json      ← architecture + hyperparams
        ├── results/
        │   ├── metrics.csv            ← CSVLogger per-epoch output
        │   └── run_metadata.json      ← write_run_metadata
        └── README.txt                 ← human summary of experiment
        

Anyone with TGraphX v1.4.2 and this directory can understand what the
experiment did, verify the shapes and configurations, and attempt reproduction.

What Reproducibility Cannot Guarantee

Even with complete documentation:
- Hardware differences affect floating-point results
- Different CUDA versions have different kernel implementations
- TGraphX's Beta status means future versions may change behavior
- Non-deterministic operations (even with deterministic=True, some
operations may differ across hardware)

The goal is close reproduction — same conclusions, comparable numbers —
not bit-for-bit identical results. Document the limitations alongside the
results, not as a footnote.