Multimodal and Vision Use Cases for Tensor-Valued Graph Learning
Computer vision and graph learning have historically developed as separate research tracks. Vision models (CNNs, ViTs) process grid-structured data; GNNs process relationship-structured data. Tensor-valued graph learning is a bridge: it allows graph nodes to carry image-like or volumetric features, passing them through message-passing layers that respect both the spatial structure within each node and the relational structure between nodes.
This article surveys practical use cases where the combination of vision features and graph topology is the natural modeling choice, and shows how TGraphX's architecture supports them.
When Vision and Graphs Naturally Combine
Scene understanding. A scene graph represents a visual scene as a graph where each node is a detected object and each edge represents a spatial or semantic relationship between objects. Node features are object image patches [C, H, W]; message passing propagates context between related objects.
Medical image analysis. In brain MRI analysis, brain regions are nodes and functional connectivity between regions are edges. Each region can carry a spatial activation map as a node feature. In pathology, tissue regions can be nodes connected by spatial proximity, with each node carrying a patch-level feature map.
Point cloud processing. 3D point clouds can be converted to graphs by connecting each point to its k-nearest neighbors. Each node's feature can include a local neighborhood descriptor with spatial extent.
Image segmentation with graph priors. Superpixel graphs represent an image as a graph where each superpixel is a node. Superpixel features can be the raw pixel patch [C, H, W] or aggregated statistics.
Video graphs. Temporal video sequences can be represented as graphs where each frame is a node (with the frame image as a feature) and edges connect temporally or visually similar frames.
Constructing a Scene Graph with TGraphX
import torch
from tgraphx import Graph
from tgraphx.layers.gin import TensorGINLayer
from tgraphx.layers.pooling import GlobalMeanPool
import torch.nn as nn
import torch.nn.functional as F
# 8 objects in a scene, each represented as a [16, 32, 32] patch
N_objects = 8
C, H, W = 16, 32, 32
object_patches = torch.randn(N_objects, C, H, W)
# Spatial relationships: edges between objects
# (in practice, derived from bounding box IoU or spatial proximity)
edge_index = torch.tensor([
[0, 1, 1, 2, 3, 4, 5, 6],
[1, 0, 3, 3, 5, 5, 7, 7],
], dtype=torch.long)
# Edge features: relative position and size [E, 4] (dx, dy, width_ratio, height_ratio)
edge_rel = torch.randn(8, 4)
g = Graph(
node_features=object_patches,
edge_index=edge_index,
edge_features=edge_rel,
)
# GIN layer with vector edge features (edge features encode spatial relation)
gin = TensorGINLayer(
in_channels=C,
out_channels=32,
use_edge_features=True,
edge_dim=4,
edge_features_kind="vector", # [E, 4] edge features projected to channel bias
spatial_rank=2,
train_eps=True,
)
out = gin(g.node_features, g.edge_index, edge_features=edge_rel)
print(out.shape) # [8, 32, 32, 32] — spatial layout preserved
Image-to-Patch Graph Construction
TGraphX provides a utility to convert an image to a graph of non-overlapping patches:
import tgraphx as tgx
import torch
# Single image [C, H, W] = [3, 32, 32]
image = torch.randn(3, 32, 32)
# Convert to patch graph: 16 patches of size 8x8
patches, edge_index = tgx.image_to_patch_graph(image, patch_size=8)
print(patches.shape) # [16, 3, 8, 8] — 16 patches, each [C, ph, pw]
print(edge_index.shape) # [2, E] — edges connecting spatially adjacent patches
g = tgx.Graph(node_features=patches, edge_index=edge_index)
This is the same approach used in the MNIST as graph tutorial and the CIFAR image patch graph tutorial.
kNN Graph from Image Embeddings
For scene graphs or superpixel graphs, kNN construction is common:
import tgraphx as tgx
import torch
# 100 image patches, each embedded as [512]
embeddings = torch.randn(100, 512)
# Build kNN graph (k=5, cosine similarity)
edge_index = tgx.knn_graph(
embeddings, k=5, metric="cosine", make_symmetric=True
)
print(edge_index.shape) # [2, ~1000] — approximately 100*5*2 edges (symmetric)
Multi-Layer Tensor-Valued GNN for Vision Graphs
A typical architecture for processing tensor-valued scene graphs:
from tgraphx.layers.gin import TensorGINLayer
from tgraphx.layers.sage import TensorGraphSAGELayer
import torch.nn as nn
import torch.nn.functional as F
class VisionGNN(nn.Module):
def __init__(self, node_channels, node_h, node_w, num_classes):
super().__init__()
# Conv-based message passing preserves spatial dims
self.gin1 = TensorGINLayer(
node_channels, 32, use_batchnorm=True, spatial_rank=2
)
self.gin2 = TensorGINLayer(32, 64, use_batchnorm=True, spatial_rank=2)
self.sage = TensorGraphSAGELayer(
64, 128, aggr="mean", normalize=True, spatial_rank=2
)
# After pooling: [B, 128, H, W] → flatten → classifier
self.h = node_h
self.w = node_w
self.classifier = nn.Linear(128 * node_h * node_w, num_classes)
def forward(self, x, edge_index):
x = F.relu(self.gin1(x, edge_index))
x = F.relu(self.gin2(x, edge_index))
x = self.sage(x, edge_index)
# Global mean pool over nodes (for graph classification)
x = x.mean(dim=0, keepdim=True) # [1, 128, H, W]
return self.classifier(x.flatten(1))
model = VisionGNN(node_channels=16, node_h=8, node_w=8, num_classes=10)
x = torch.randn(20, 16, 8, 8)
ei = torch.randint(0, 20, (2, 80), dtype=torch.long)
logits = model(x, ei)
print(logits.shape) # [1, 10]
Multimodal Knowledge Graphs with Vision Entities
TGraphX's KG module supports entity features for multimodal knowledge graphs where some entities are represented by images:
import torch
from tgraphx.kg import KnowledgeGraph
from tgraphx.kg.multimodal import MultimodalKGModel
# Entities: 0-49 = image entities, 50-99 = text entities (precomputed embeddings)
num_entities = 100
num_relations = 5
triples = torch.randint(0, num_entities, (200, 3), dtype=torch.long)
# Image entity features: [50, 3, 64, 64] (each image entity has a 64x64 image)
image_entity_features = torch.randn(50, 3, 64, 64)
# Text entity features: [50, 768] (BERT embeddings)
text_entity_features = torch.randn(50, 768)
kg = KnowledgeGraph(triples, num_entities=num_entities, num_relations=num_relations)
For the full multimodal KG tutorial with modality-specific projectors, see knowledge graph embedding with tensor features.
VisionServeX: Serving Vision Models with Graph Context
TGraphX's companion package VisionServeX provides a serving layer for vision models that have been trained on graph-structured data. If your vision GNN produces structured outputs (object detection with relational context), VisionServeX handles the model serving and API layer. See the VisionServeX package page for details.
Limitations
TGraphX's spatial message passing is not a vision foundation model. ConvMessagePassing and TensorGINLayer apply small 1×1 convolutions over node feature maps. They are not equivalent to full convolutional networks applied to the image. For high-quality visual feature extraction, use a pre-trained backbone (ResNet, ViT) and embed node features with it; then apply TGraphX's message passing on the resulting embeddings.
Graph construction from images requires domain knowledge. There is no universally correct way to build a graph from an image. Patch graphs, superpixel graphs, object graphs, and point cloud graphs each make different assumptions about what structure in the image should be captured. TGraphX provides image_to_patch_graph and knn_graph as starting points, not definitive solutions.
No built-in object detector. TGraphX does not include a detection pipeline to extract bounding boxes or segment superpixels. These require external tools (torchvision object detection, SLIC via skimage, etc.).
Spatial dims must be consistent across nodes. TGraphX's message-passing layers require all node feature maps to have the same spatial dimensions [H, W]. If object patches have variable sizes, you must resize them before graph construction.
Related Articles
- MNIST as a graph with TGraphX — digit patches as graph nodes
- Tensor-valued nodes deep tutorial — full training walkthrough
- Multimodal graph nodes in TGraphX — mixed modality graphs
- AnnotateX structured annotation — annotation for vision-graph datasets
- Knowledge graph embedding with tensor features — vision entities in KGs
- Flat vector features are insufficient for GNNs — the spatial structure argument