Multi-Modal Graph Nodes: Mixing Modalities in TGraphX

A multi-modal graph has nodes of different types where each type carries different feature modalities. Consider a healthcare graph: patient nodes have demographic vectors, imaging nodes have [C, H, W] MRI slices, lab-result nodes have time-series vectors, clinical-note nodes have text embeddings. Each modality is naturally different shape and different semantics.

Modeling this in a uniform graph learning framework is hard. The standard [N, D] node feature matrix assumption breaks immediately — different node types have different D (and different rank). This article covers how TGraphX handles the situation and what its limitations are.

The core challenge

In a homogeneous graph, every node has the same feature shape. Message passing aggregates neighbor features into a uniform output. In a multi-modal graph:

• A patient node aggregates from imaging nodes (rank-4) and lab-result nodes (rank-2). What does that aggregation produce?
• A message-passing layer designed for [N, C, H, W] does not know what to do with [N, T, D] input.
• The output shape is also ambiguous — should it be patient-shaped or image-shaped or something else?

The standard solution is to project each modality into a common embedding space at the data-layer boundary. Then message passing operates on the common-space embeddings. This loses some structure but is tractable.

TGraphX's approach

TGraphX has two relevant features:

1. Tensor-valued single-graph nodes. All nodes have the same feature shape, but that shape can be [C, H, W] instead of [D].

2. HeteroGraph (Experimental). Nodes of different types can have different feature shapes. The framework provides projector layers that map each type into a common embedding space.

The first is solid Beta. The second is Experimental — usable but should be evaluated carefully for research before being relied on for production.

Single-modality tensor-valued graphs

If your nodes all share one modality (e.g., all image patches), this is straightforward:

import torch
        import tgraphx as tgx
        
        # 1000 image patches, each [3, 8, 8]
        x = torch.randn(1000, 3, 8, 8)
        edge_index = tgx.knn_graph(x.view(1000, -1), k=8)
        labels = torch.randint(0, 10, (1000,))
        
        g = tgx.Graph(x=x, edge_index=edge_index, labels=labels)
        result = tgx.classify_nodes(
            x=x, edge_index=edge_index, labels=labels,
            model="tensor_gcn", seed=42,
        )
        

This is the supported, stable pattern. Day 1 and Day 17 articles cover it in detail.

Multi-modal heterogeneous graphs (Experimental)

For genuinely multi-modal graphs, the HeteroGraph and related infrastructure provides:

from tgraphx import HeteroGraph
        
        # Different node types with different feature shapes
        patient_features = torch.randn(200, 64)              # rank-2
        imaging_features = torch.randn(150, 3, 32, 32)        # rank-4
        lab_features     = torch.randn(300, 24, 8)            # rank-3 (T=24 timesteps, D=8)
        
        # Edges between types
        patient_to_imaging = torch.tensor([...], dtype=torch.long)  # [2, E1]
        patient_to_lab     = torch.tensor([...], dtype=torch.long)  # [2, E2]
        
        hg = HeteroGraph(
            node_features={
                "patient": patient_features,
                "imaging": imaging_features,
                "lab":     lab_features,
            },
            edge_indices={
                ("patient", "has_imaging", "imaging"): patient_to_imaging,
                ("patient", "has_lab",     "lab"):     patient_to_lab,
            },
        )
        

Each node type has its own feature tensor. Edges are typed by source/relation/target.

For training, the framework provides type-aware GNN layers and a multi-modal projector pattern:

# (Pattern; specific API in tgraphx.kg.multimodal and Experimental hetero subsystem)
        

The implementation is research-grade. For production, evaluate against established multi-modal frameworks.

Honest assessment

What works well:

• Single-modality tensor-valued graphs are stable and well-tested.
• The framework's data layer accommodates the multi-modal pattern cleanly.
• Knowledge graphs with multi-modal entity features are documented in docs/kg_multimodal_tensor_features.md.

What is research-grade:

• HeteroGraph and the type-aware GNN layers are labeled Experimental.
• There are no published benchmarks of TGraphX hetero GNNs against established hetero frameworks (PyG's HeteroData, DGL's heterogeneous graphs).
• Production deployments would benefit from PyG's more mature HeteroData abstraction.

A pragmatic alternative

For multi-modal projects right now, a common pragmatic pattern:

1. Project each modality outside the graph layer. Train per-modality encoders (CNN for images, RNN for sequences, MLP for vectors) that produce same-shape embeddings.
2. Use a homogeneous graph in TGraphX with the projected embeddings. All nodes carry the common-shape embedding.
3. The graph learning layer operates on uniform input.

This decouples the modality-handling problem from the graph-learning problem. It loses some end-to-end joint learning but is simpler and uses only stable framework features.

import torch
        import tgraphx as tgx
        
        # Pre-project each modality to 128-dim embeddings
        patient_emb = patient_encoder(patient_features)   # [200, 128]
        imaging_emb = imaging_encoder(imaging_features)   # [150, 128]
        lab_emb     = lab_encoder(lab_features)           # [300, 128]
        
        # Combine into one homogeneous graph
        all_embeddings = torch.cat([patient_emb, imaging_emb, lab_emb], dim=0)   # [650, 128]
        # (Adjust edge indices to map into the combined index space)
        
        g = tgx.Graph(x=all_embeddings, edge_index=combined_edges, labels=y)
        result = tgx.classify_nodes(x=all_embeddings, edge_index=combined_edges, labels=y, model="gcn")
        

This works today with stable APIs. The downside is that the per-modality encoders are not trained jointly with the GNN. For research where end-to-end learning matters, you need the Experimental hetero infrastructure or a different framework.

When the multi-modal feature pays off

• The modalities carry correlated information about the same underlying entity.
• End-to-end joint training is necessary (gradients flow from the prediction back through the modality encoders).
• The graph structure mixes types in ways a per-modality encoder cannot capture.

For weakly correlated modalities or cases where per-modality features can be precomputed once, the pragmatic separated approach is usually fine.

Where to look next

• docs/kg_multimodal_tensor_features.md — multimodal KG entities, the most mature multi-modal feature in TGraphX.
• docs/hetero_gnns.md — Experimental heterogeneous GNNs.
• PyTorch Geometric's HeteroData documentation — the more mature alternative for heterogeneous graphs.

FAQ

Q: Can I have rank-4 features for some nodes and rank-2 for others in a single tgx.Graph?
A: No. tgx.Graph requires all nodes to share the same feature shape. For mixed-rank features, use HeteroGraph (Experimental) or pre-project to a common shape.

Q: Are message-passing layers type-aware in TGraphX?
A: The standard layers operate on homogeneous graphs. Type-aware layers exist in the Experimental hetero subsystem.

Q: Should I use TGraphX or PyG for multi-modal research?
A: For production-grade multi-modal hetero graphs, PyG's HeteroData is more mature. For research that combines multi-modal hetero with tensor-valued features, TGraphX is differentiated but Experimental.

Q: Can KG entities have multi-modal features?
A: Yes. This is the most stable multi-modal feature in TGraphX. See the documentation in docs/kg_multimodal_tensor_features.md.

Q: How do I project images to a common embedding for the pragmatic approach?
A: Any CNN that produces a fixed-dim output works. A frozen pretrained ResNet's penultimate-layer features are a common choice.