Tensor-Valued Graphs for Multimodal Research Workflows
Multimodal learning — combining images, text, audio, tabular data — is a
growing research area. Most deep learning frameworks handle each modality
separately and combine them at a late fusion step. Graph representations offer
an alternative: different modalities become node features, and edges represent
semantic or structural relationships between data points across modalities.
TGraphX's tensor-valued node feature design makes this natural to represent,
because node features can be any tensor shape, not just vectors.
Important caveat: TGraphX is not a multimodal learning framework. It
does not provide modality-specific encoders, cross-modal attention, or fusion
layers. What it provides is a graph container where nodes can carry tensors
of arbitrary shape — which is useful for multimodal research but requires
substantial additional work from the user.
What Tensor-Valued Nodes Enable
In a standard GNN library, a node has a feature vector. For multimodal data:
- An image must be flattened to a vector before becoming a node feature
- A time series must be averaged or max-pooled to a vector
- A text sequence must be encoded to a vector
Flattening and encoding are valid approaches, but they discard structure:
- Spatial structure in images (CNN conv layers use this structure)
- Temporal structure in sequences (RNNs and transformers use this)
- Hierarchical structure in text
With TGraphX's [N, ...] node feature contract, you can represent:
import torch
# Example: 20 data points, each with an image feature [C, H, W]
image_nodes = torch.randn(20, 3, 224, 224) # [N, C, H, W]
# Example: 20 data points, each with a time-series feature [T, D]
time_nodes = torch.randn(20, 128, 64) # [N, T, D]
# Example: 20 data points, each with a text embedding [seq_len, embed_dim]
text_nodes = torch.randn(20, 512, 768) # [N, seq_len, embed_dim]
All three are valid node_features for a TGraphX Graph. What you do with
them in the GNN layer depends on your architecture.
ASCII Multimodal Node Diagram
Multimodal graph: each node holds a different tensor modality
─────────────────────────────────────────────────────────────────
node_features [N, *] where * varies by modality type
Image nodes (e.g., patient scan): [N, C, H, W]
┌──────────┐ ┌──────────┐ ┌──────────┐
│ [C, H, W]│──│ [C, H, W]│──│ [C, H, W]│
└──────────┘ └──────────┘ └──────────┘
↕ ↕ ↕
Time-series nodes (e.g., vitals): [N, T, D]
┌──────────┐ ┌──────────┐ ┌──────────┐
│ [T, D] │──│ [T, D] │──│ [T, D] │
└──────────┘ └──────────┘ └──────────┘
Note: In a single Graph object, all nodes must share the same feature shape.
For heterogeneous modalities with different shapes, separate Graph objects
or manual padding/projection to a common shape is required.
─────────────────────────────────────────────────────────────────
The Homogeneous Shape Constraint
TGraphX's Graph requires that all nodes share the same node_features
shape. This is because node_features is a single tensor [N, ...] where
the ... is uniform across all N nodes.
For multimodal workflows where different nodes have fundamentally different
shapes (one node is an image [C, H, W], another is a time series [T, D]),
you have several options:
Option 1: Project to a common vector space before construction
from some_image_encoder import encode_image
from some_text_encoder import encode_text
image_embeddings = encode_image(images) # [N_img, d]
text_embeddings = encode_text(texts) # [N_txt, d]
# Both projected to common dim d
all_embeddings = torch.cat([image_embeddings, text_embeddings], dim=0) # [N, d]
Option 2: Separate graphs per modality with cross-graph edges
This requires a heterogeneous graph structure that TGraphX does not natively
support. Users would need to implement this manually.
Option 3: Use the same modality for all nodes
Build a graph where all nodes are images, all nodes are text, etc.
This is the simplest approach and works directly with TGraphX.
A Practical Example: Image Dataset with KNN Graph
A concrete multimodal-adjacent use case: a graph over images where nodes are
images (preserving spatial structure) and edges connect visually similar images:
from tgraphx import Graph
from tgraphx.graph_builders import build_knn_graph
from tgraphx.performance import recommended_device
from tgraphx.reproducibility import set_seed
import torch
set_seed(42)
device = recommended_device()
# 50 images, each [3, 64, 64]
images = torch.randn(50, 3, 64, 64) # [N, C, H, W]
# Build k-NN graph on flattened features for edge construction
feats_flat = images.view(50, -1) # [50, C*H*W]
edge_index = build_knn_graph(feats_flat, k=5) # [2, E]
# Node labels: class labels for 50 images
labels = torch.randint(0, 5, (50,))
g = Graph(
node_features=images, # [50, 3, 64, 64] — full 2D structure preserved
edge_index=edge_index, # [2, E]
edge_features=torch.zeros(edge_index.shape[1], 1),
node_labels=labels,
).to(device)
print(f"node_features: {g.node_features.shape}") # [50, 3, 64, 64]
print(f"edge_index: {g.edge_index.shape}")
Here, edges are built from flattened features (for k-NN similarity) but
node features retain their full [C, H, W] structure for the GNN layers.
Tabular Data as Node Features
Tabular data with multiple columns can be represented as a 1D vector per node:
# 200 samples, each with 30 tabular features
tabular_features = torch.randn(200, 30) # [N, d]
This is the simplest case — vector features. build_knn_graph or
build_fully_connected_graph can construct edges based on feature similarity.
Time-Series Nodes
Time-series data preserves temporal structure:
# 100 sensors, each with 48 timesteps and 4 channels
time_features = torch.randn(100, 48, 4) # [N, T, D]
A GNN layer that aggregates over neighbors can collect time-series from
neighboring sensors and process them as [T, D] tensors. What the layer
does with the temporal structure depends on the layer's architecture.
Honest Scope Boundary
TGraphX provides:
- Graph([N, ...], [2, E], ...) — stores tensor-valued nodes
- TensorMessagePassingLayer — aggregates with scatter (no Python loops)
- Shape validation at construction
TGraphX does not provide:
- Modality-specific encoders (CNN, RNN, transformer)
- Cross-modal attention
- Heterogeneous graph (different node types) support
- Multimodal fusion layers
Using TGraphX for multimodal research means using it as the graph
representation layer and implementing the modality-specific parts yourself.
This is an appropriate split of responsibilities for a Beta research library.
The graph abstraction is useful precisely because it is focused on one thing:
validated, tensor-valued graph representation and message passing.