Structured Annotation for Graph and Vision Research

The annotation step is often the most labor-intensive part of a research project, and the most overlooked when reporting results. A model evaluated on a poorly annotated dataset is a model with an unknown ceiling. A model evaluated on a well-annotated dataset with documented annotation methodology is one whose results can be trusted to mean what they claim.

This article covers what good annotation infrastructure looks like for graph and vision research, the principles that distinguish it from ad-hoc data labeling, and how it connects to TGraphX workflows. AnnotateX is the working name for a related tool in this space; this overview focuses on the principles, since the project's surface is still evolving.

The annotation problem

A common pattern in graph/vision research:

1. Researcher collects raw data (images, graphs, sequences).
2. Researcher annotates by hand or with a quick script.
3. Annotations are saved as a CSV, a JSON file, or a folder structure.
4. Annotations are loaded with custom code in the training pipeline.
5. Six months later, no one remembers exactly what each label means or how it was assigned.

The result: a dataset that nominally exists but whose annotations cannot be audited, reused, or extended. This is the default in research, and it is a significant source of irreproducibility.

What good annotation infrastructure provides

A defensible annotation pipeline should have:

• Versioning. Each version of the annotations is preserved. Changes are tracked.
• Schema. Labels have explicit types, value ranges, and documentation.
• Provenance. Every annotation records who annotated it, when, with what tool.
• Agreement. Multi-annotator workflows track inter-annotator agreement.
• Round-tripping. Annotations can be exported to and imported from standard formats.
• Validation. A schema-aware validator can catch inconsistencies (missing labels, out-of-range values).

These are not novel ideas. They are standard practice in labeled dataset construction outside ML — they are just rarely applied in ML research because the cost feels high relative to the perceived benefit.

Where graphs add specific requirements

Graph annotation has properties that flat data does not:

• Node-level vs edge-level vs graph-level labels. All three exist and need different storage and validation.
• Subgraph annotations. A motif label applies to a subgraph, not a single element.
• Relational consistency. If node A has label X and node B has label Y, an edge between them may need a label that respects both — these constraints should be schema-encoded.

A general-purpose annotation tool (Label Studio, CVAT) does not handle these graph-specific cases well out of the box. Research projects often end up writing custom annotation tooling, which is fine for one project but does not scale across labs.

What this looks like in a TGraphX workflow

A graph dataset annotated with this kind of infrastructure becomes:

import tgraphx as tgx
        
        # Load the graph
        g = tgx.Graph.load("dataset/cohort_001/graph.tgx")
        
        # Load annotations from a versioned annotation store
        node_labels = load_annotations("dataset/cohort_001/annotations/v2/node_labels.json")
        edge_labels = load_annotations("dataset/cohort_001/annotations/v2/edge_labels.json")
        
        # Use them in training
        result = tgx.classify_nodes(
            x=g.node_features, edge_index=g.edge_index, labels=node_labels,
            model="tensor_gcn", seed=42,
        )
        

The key piece: the annotations are external to the graph file and are versioned independently. Re-labeling produces a new version without invalidating the underlying graph data.

A pragmatic minimum

If you do not adopt a full annotation tool, here is a pragmatic minimum that captures most of the benefit:

1. Version annotations in Git as JSON files alongside the data.
2. Document the schema in a README in the same directory.
3. Validate on load with a small custom validator (a few dozen lines of Python).
4. Record provenance as a provenance.json listing annotator names, dates, and method.
5. Tag releases of the annotation set with semantic version numbers.

This gives you 80% of the benefit without adopting a new tool. It is the discipline more than the tool that matters.

Where dedicated tools help

Dedicated annotation tools become worth the integration cost when:

• Multiple annotators work on the same dataset.
• The dataset is large enough that batch annotation tools save real time.
• Inter-annotator agreement needs to be tracked formally.
• The schema is complex enough that manual validation is error-prone.

For small single-author research datasets, the pragmatic minimum is usually sufficient.

Honest limitations

There is no specific endorsement here of any particular annotation product. The TGraphX framework does not include an annotation tool; it consumes whatever annotations you provide. The principles discussed apply across tools.

The AnnotateX project's specific feature set is not documented inside the TGraphX repository, so this article covers the principles rather than making specific product claims. Check the website's Resources page for the current status of related TGraphX-ecosystem projects.

Where to start

For a new research project, the order is:

1. Define the schema (what labels exist, with what types and ranges).
2. Version the schema in Git alongside the data.
3. Write annotations against the schema using whatever tool fits the data size.
4. Validate annotations programmatically on load.
5. Re-annotate iteratively as the project evolves; new versions, not in-place edits.

This is mundane infrastructure. It is also the kind of infrastructure that distinguishes research that holds up from research that does not.

FAQ

Q: Is AnnotateX available as a package?
A: Check the TGraphX website's Resources page for the latest status. The project's specific availability and feature set may change.

Q: Can I use Label Studio for graph annotation?
A: Label Studio is image- and text-focused. Graph-specific annotation usually requires custom configuration or a dedicated tool.

Q: What about inter-annotator agreement metrics?
A: Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are the standard metrics. They should be computed and reported alongside the dataset.

Q: Should annotations be in the same file as the graph?
A: Generally no. Keep them separate so the graph file can be reused with different annotation versions.

Q: How do I document an annotation schema?
A: A README in the annotation directory with label definitions, value ranges, and examples is the minimum. A formal JSON schema is better.