Graph Learning for Computer Vision: When Image Patches Become Nodes
The dominant paradigm for image processing is convolutional: a kernel slides
over a grid of pixels, building hierarchical features through spatial locality.
But not all image understanding tasks fit the grid well. When the meaningful
structure in an image is relational — objects that relate to each other
regardless of spatial distance, parts that form semantic groups across the
image — a graph representation can better capture that structure.
This article explores the mechanics of turning an image into a graph, using
TGraphX's image_to_patches and build_grid_graph as concrete tools.
When to Consider a Graph for Images
Grid convolutions are the right tool when:
- Spatial locality is the primary signal
- The image has regular structure (photos, medical scans)
- Translation invariance is desirable
Graph representations become interesting when:
- Objects or parts need to be related across the image
- The graph structure itself is part of the representation
- You want to explicitly model pairwise or higher-order relationships
This is not a claim that graph approaches outperform convolutions on standard
benchmarks. They do not, in general. Graph approaches are worth exploring when
the task has a relational structure that convolutions obscure.
image_to_patches: Source-Verified
TGraphX provides image_to_patches in tgraphx.graph_builders. This function
takes an image tensor and a patch size, returning node features where each node
is one patch:
from tgraphx.graph_builders import image_to_patches
import torch
# Image: [C, H, W] — single image, 3 channels, 64x64 pixels
image = torch.randn(3, 64, 64)
# Extract 16x16 patches
patches = image_to_patches(image, patch_size=16)
# Output: [N, C, patch_H, patch_W]
# For 64x64 image with 16x16 patches: N = (64/16)*(64/16) = 16 patches
# patches.shape: [16, 3, 16, 16]
print(patches.shape) # torch.Size([16, 3, 16, 16])
Each of the 16 patches is a [3, 16, 16] tensor. The node count N equals the
number of patches. This is the tensor-valued node feature that TGraphX's
Graph class is designed to hold.
build_grid_graph: Connecting Patches
With patches as nodes, the next step is defining edges. A natural choice for
image patches is a grid graph: each patch connects to its horizontal, vertical,
and optionally diagonal neighbors.
from tgraphx.graph_builders import build_grid_graph, image_to_patches
from tgraphx import Graph
import torch
image = torch.randn(3, 64, 64)
patches = image_to_patches(image, patch_size=16) # [16, 3, 16, 16]
# Build grid edges: 4x4 grid of patches
g_grid = build_grid_graph(height=4, width=4)
# g_grid.edge_index: [2, E] — edges between adjacent patches
# Combine patches with grid edges
N = patches.shape[0] # 16
g = Graph(
node_features=patches, # [16, 3, 16, 16]
edge_index=g_grid.edge_index, # [2, E]
edge_features=torch.zeros(g_grid.edge_index.shape[1], 1),
node_labels=torch.zeros(N, dtype=torch.long), # placeholder
)
print(f"Nodes: {g.node_features.shape}") # [16, 3, 16, 16]
print(f"Edges: {g.edge_index.shape}") # [2, E]
The ASCII Patch Diagram
Image (64x64) → Patches (16x16 each) → Graph Nodes
─────────────────────────────────────────────────────
Original image (4x4 patch grid):
┌───┬───┬───┬───┐
│ 0 │ 1 │ 2 │ 3 │
├───┼───┼───┼───┤
│ 4 │ 5 │ 6 │ 7 │
├───┼───┼───┼───┤
│ 8 │ 9 │10 │11 │
├───┼───┼───┼───┤
│12 │13 │14 │15 │
└───┴───┴───┴───┘
Each cell = one patch node, feature shape [3, 16, 16]
Grid graph edges (4-connectivity):
0─1─2─3
│ │ │ │
4─5─6─7
│ │ │ │
8─9─10─11
│ │ │ │
12─13─14─15
edge_index [2, E]: E = 2 * (3*4 + 4*3) = 24 directed edges
(each undirected edge becomes 2 directed edges)
Using build_knn_graph for Non-Grid Connectivity
Grid connectivity is natural for image patches, but may not capture long-range
relationships. build_knn_graph builds a k-nearest-neighbor graph based on
feature similarity:
from tgraphx.graph_builders import build_knn_graph
# Flatten patches for similarity computation
patch_flat = patches.view(16, -1) # [16, C*H*W]
edge_index_knn = build_knn_graph(patch_flat, k=4) # [2, E]
g_knn = Graph(
node_features=patches, # [16, 3, 16, 16] — preserve 2D structure
edge_index=edge_index_knn, # [2, E] — edges by feature similarity
edge_features=torch.zeros(edge_index_knn.shape[1], 1),
node_labels=torch.zeros(16, dtype=torch.long),
)
A k-NN graph connects each patch to its most feature-similar neighbors,
regardless of spatial position. This can capture texture clusters or
semantic groups that are spatially separated.
What Happens in the Forward Pass
With node features [N, C, H, W], a GNN layer receives patch tensors and
aggregates information from neighbors. The TensorMessagePassingLayer handles
the aggregation via scatter operations. What the layer does with the [C, H, W]
feature is up to its architecture.
For example, a layer might:
1. Flatten [C, H, W] to [C*H*W] and apply a linear projection
2. Apply a CNN to [C, H, W] and produce a vector, then aggregate
3. Aggregate first (via scatter), then apply spatial operations
TGraphX's TensorGraphSAGELayer handles tensor-valued nodes — the exact
behavior with high-dimensional features depends on the layer's documented
forward signature.
Practical Considerations
Patch count as graph size: A 224×224 image with 16×16 patches has 196
nodes. With k=8 edges per node, that is 1568 edges. These are small graphs
by GNN standards, and GraphBatch batching across many images is efficient.
Feature dimensionality: A [3, 16, 16] patch has 768 elements. Scatter
aggregation over 8 edges produces a [3, 16, 16] aggregated feature. The
memory for scatter buffers scales with E * C * H * W — use
estimate_message_memory to check feasibility before scaling up.
This is not a benchmark claim: Converting images to graphs is a valid
research approach for specific tasks (few-shot learning, structured scene
understanding, graph-based attention). It does not generally outperform
convolutional architectures on standard image classification tasks.
Full Pipeline
from tgraphx.graph_builders import image_to_patches, build_grid_graph
from tgraphx import Graph, GraphBatch, GraphDataLoader
from tgraphx.performance import recommended_device
from tgraphx.reproducibility import set_seed
import torch
set_seed(42)
device = recommended_device()
def image_to_graph(image: torch.Tensor, patch_size: int = 16) -> Graph:
C, H, W = image.shape
h_patches = H // patch_size
w_patches = W // patch_size
patches = image_to_patches(image, patch_size=patch_size) # [N, C, ph, pw]
grid_g = build_grid_graph(height=h_patches, width=w_patches)
E = grid_g.edge_index.shape[1]
return Graph(
node_features=patches,
edge_index=grid_g.edge_index,
edge_features=torch.zeros(E, 1),
node_labels=torch.zeros(patches.shape[0], dtype=torch.long),
)
# Convert a batch of images
images = torch.randn(32, 3, 64, 64)
graphs = [image_to_graph(images[i]) for i in range(32)]
batch = GraphBatch.from_graphs(graphs).to(device)
print(f"Batched graph: {batch.node_features.shape}")
# [32*16, 3, 16, 16] = [512, 3, 16, 16]
This pipeline is concrete, runnable, and avoids claims about downstream
accuracy. What the GNN does with the patch graph depends entirely on the
architecture and task — the construction step shown here is the foundation.