Multi-Modal Nodes: Images, Volumes, and Sequences as Tensor Node States
"Multi-modal" usually conjures elaborate fusion architectures. In tensor-aware graph learning it means something more concrete and more tractable: a node's natural representation is whatever tensor that modality produces — an image patch is [C, H, W], a volumetric block is [C, D, H, W], a sequence is [T, D] — and a graph connects those nodes by whatever relation makes sense. TGraphX is built so that each of these modalities can be a node without being flattened first. This note maps modalities to tensor shapes and shows the builders that turn raw data into graphs.
It deepens the existing Multimodal Graph Nodes article with the construction details.
Modality is just a trailing shape
The unifying idea is that "modality" is encoded entirely in the per-node trailing shape s:
| Modality | Node tensor s |
How a node is obtained | Relation (edges) |
|---|---|---|---|
| Image region | (C, H, W) |
crop / patch of an image | spatial adjacency, visual similarity |
| Volume block | (C, D, H, W) |
sub-volume of a 3-D scan | 3-D adjacency |
| Sequence | (T, D) or (C, L) |
a windowed signal | temporal / relational links |
| Vector | (D,) |
embedding | any graph relation |
Because the framework's layers operate on the trailing shape generically (see Inside Tensor Message Passing), the same message-passing machinery handles all of these — only in_shape/out_shape change. That is the practical payoff of tensor-native design: you do not need a different framework per modality.
Building graphs from images and volumes
TGraphX ships the construction helpers directly. From tgraphx/graph_builders.py:
image_to_patches(...)— split an image into patches, each becoming a[C, H, W]node.patch_grid_shape(...)— compute the patch grid layout so adjacency edges can be built.volume_to_patches(...)— the 3-D analogue, producing[C, D, H, W]nodes.
And a one-call convenience in tgraphx/ux/graph_construction.py:
import tgraphx as tgx
g = tgx.image_to_patch_graph(image, patch_size=8) # patches → tensor-node graph
The result is a graph whose nodes are patch feature maps and whose edges encode patch adjacency — exactly the structure explored in Image Patches as Graph Nodes. For learned per-node embeddings, CNNEncoder (tgraphx/models/cnn_encoder.py) turns image-shaped node tensors into features while keeping the graph structure intact.
Why preserve modality structure
Consider three patches arranged in an image. Flattened to vectors, the model must rediscover that pixels within a patch are spatially correlated; preserved as [C, H, W], a convolutional message can exploit that correlation from the start, as argued in the TGraphX preprint. The same logic extends to volumes (3-D locality) and sequences (temporal locality). The graph layer then adds the relational dimension on top — which patches are adjacent, which regions are similar — giving you both within-node structure and between-node structure in one model. This is the two-geometries idea from the topology-vs-feature-geometry article, applied per modality.
Honest framing
Three honest qualifications. First, "multi-modal" here means structured tensors per node, not arbitrary cross-modal fusion (text + image + audio with learned alignment) — that is a different problem you would build on top, not a built-in feature. Second, you still choose the encoder and the edge-construction rule; the builders give you patches and grids, but the relation that makes the graph meaningful is a modelling decision. Third, whether a tensor-node graph beats a plain CNN or a flat-feature GNN on your task is empirical and should be measured, not assumed — preserving structure is an inductive bias, not a guarantee.
How to describe this accurately
A safe summary: TGraphX represents image, volume, and sequence data as tensor-valued nodes ([C,H,W], [C,D,H,W], [T,D]), with builders (image_to_patches, volume_to_patches, image_to_patch_graph) that construct patch/voxel graphs and a CNNEncoder for learned node embeddings. That is backed by tgraphx/graph_builders.py and tgraphx/models/cnn_encoder.py.
Sequences and the temporal case
Images and volumes get most of the attention, but the sequence case is worth a note because it is where the trailing-shape view pays off again. A node whose feature is a windowed signal has shape (C, L) or (T, D) — a length axis instead of a spatial grid — and the same message-passing machinery applies, with the message function operating over that axis rather than over H, W.
Where this connects to the rest of the framework is the distinction between a sequence as a node feature and a temporal graph whose structure changes over time. The former is just a tensor-valued node, handled by the core layers; the latter is the Experimental temporal subsystem (TemporalGraphSequence, TGN/TGAT) noted in the capability map. Keeping the two apart prevents a common misconception. If your sequences are static per node, treat them as (T, D) tensor nodes and you stay on the Beta core; reach for the temporal subsystem only when the graph itself evolves, and pin your version because that area is still changing.
Related guides
Conclusion
A node's modality is just its trailing tensor shape, and TGraphX's builders turn images, volumes, and sequences into graphs of such nodes without flattening. The same message-passing machinery then handles all of them. Preserving each modality's internal structure is a principled inductive bias — choose your encoder and edges deliberately, measure the benefit, and a single tensor-aware framework covers data that would otherwise need several.