Over-Smoothing in Deep GNNs: Causes and Mitigations

Adding more layers to a neural network usually helps — you get more expressive transformations and the ability to capture longer-range dependencies. Graph neural networks are an exception. Stack too many message-passing layers and node representations converge to the same vector regardless of their initial features. This phenomenon, called over-smoothing, is one of the most important practical constraints in GNN design. Understanding its cause, measuring its severity, and applying mitigations is essential knowledge for anyone building GNNs beyond two or three layers.

What This Builds On

Readers should be familiar with the message-passing framework and understand that each GNN layer aggregates information from one-hop neighbors. Concepts from the GIN architecture article (aggregation, neighborhood expansion) are directly relevant. The neighbor sampling article is related but addresses a different problem — scalability rather than depth.

Why Over-Smoothing Happens

At layer 1, each node aggregates features from its immediate neighbors. At layer 2, it aggregates from neighbors of neighbors (2-hop neighborhood). At layer k, it has access to its k-hop neighborhood. For most real graphs (small-world graphs, citation networks, social networks), the k-hop neighborhood of a node covers a large fraction of the entire graph for relatively small k. On the Cora citation network, for example, 3-hop neighborhoods cover most of the graph.

When a GNN's aggregation is a weighted average, repeatedly averaging over larger and larger neighborhoods drives all node representations toward the same stationary distribution — the dominant eigenvector of the propagation matrix. At this limit, every node has the same representation, and the GNN cannot distinguish nodes, regardless of their initial features.

Formally, this is equivalent to repeatedly applying a low-pass filter to the node feature signal. Each message-passing step attenuates high-frequency components (differences between neighboring nodes) while preserving low-frequency components (global trends). After many applications, only the lowest-frequency component — the constant — remains.

Measuring Over-Smoothing: Dirichlet Energy and MAD

Two metrics are commonly used to quantify over-smoothing:

Dirichlet Energy measures how much adjacent nodes differ from each other:

E(H) = Σ_{(u,v) ∈ E} || h_u / ||h_u|| - h_v / ||h_v|| ||^2
        

When node representations are identical (fully smoothed), E = 0. A healthy GNN should maintain nonzero Dirichlet energy throughout all layers.

Mean Average Distance (MAD) measures the average cosine distance between node representations:

MAD = (1 / N^2) Σ_{i,j} (1 - cos_sim(h_i, h_j))
        

A MAD near zero means all node representations are nearly identical (over-smoothed). A high MAD means representations are diverse.

import torch
        import torch.nn.functional as F
        
        def dirichlet_energy(h, edge_index):
            """Compute Dirichlet energy of node representations."""
            h_norm = F.normalize(h, p=2, dim=1)
            row, col = edge_index
            diff = h_norm[row] - h_norm[col]
            return (diff ** 2).sum(dim=1).mean().item()
        
        def mean_average_distance(h):
            """Compute MAD (mean average distance) of node representations."""
            h_norm = F.normalize(h, p=2, dim=1)
            # Pairwise cosine similarities
            sim = h_norm @ h_norm.T
            dist = 1 - sim
            # Exclude diagonal (self-similarity = 1, distance = 0)
            mask = ~torch.eye(h.size(0), dtype=torch.bool)
            return dist[mask].mean().item()
        
        # Demonstrating over-smoothing with repeated averaging
        N, D = 100, 32
        edge_index = torch.randint(0, N, (2, 500), dtype=torch.long)
        h = torch.randn(N, D)
        
        print(f"Layer 0 — Dirichlet energy: {dirichlet_energy(h, edge_index):.4f}, MAD: {mean_average_distance(h):.4f}")
        
        # Simulate k rounds of mean aggregation (no learned weights)
        for layer in [1, 2, 4, 8, 16]:
            h_current = h.clone()
            for _ in range(layer):
                agg = torch.zeros_like(h_current)
                agg.scatter_add_(0, edge_index[1].unsqueeze(1).expand(-1, D), h_current[edge_index[0]])
                # Normalize by degree
                deg = torch.zeros(N).scatter_add_(0, edge_index[1], torch.ones(edge_index.size(1)))
                h_current = agg / deg.clamp(min=1).unsqueeze(1)
            energy = dirichlet_energy(h_current, edge_index)
            mad = mean_average_distance(h_current)
            print(f"Layer {layer:2d} — Dirichlet energy: {energy:.6f}, MAD: {mad:.6f}")
        

Running this code will show the energy and MAD collapsing toward zero as the number of aggregation steps increases.

Mitigation 1: Residual Connections

The simplest mitigation is to add a skip connection that bypasses aggregation:

import torch.nn as nn
        from tgraphx.layers.sage import TensorGraphSAGELayer
        
        class ResidualGNNBlock(nn.Module):
            def __init__(self, dim):
                super().__init__()
                self.conv = TensorGraphSAGELayer(dim, dim)
                self.norm = nn.LayerNorm(dim)
        
            def forward(self, x, edge_index):
                return self.norm(x + self.conv(x, edge_index))
        

The residual connection ensures that the original node features are preserved alongside aggregated neighborhood information. The normalization layer prevents the residual signal from exploding in magnitude. In practice, residual connections allow GNNs to be trained at 4–8 layers without severe over-smoothing.

Mitigation 2: Jumping Knowledge Networks (JK-Net)

JK-Net (Xu et al., 2018) addresses over-smoothing differently: instead of connecting the input directly to the output, it connects all intermediate layer representations to the final prediction layer. This way, a node can use representations from any depth:

class JKNet(nn.Module):
            def __init__(self, in_dim, hidden_dim, out_dim, num_layers):
                super().__init__()
                self.layers = nn.ModuleList()
                self.layers.append(TensorGraphSAGELayer(in_dim, hidden_dim))
                for _ in range(num_layers - 1):
                    self.layers.append(TensorGraphSAGELayer(hidden_dim, hidden_dim))
                # JK aggregation: concatenate all layer outputs
                self.out_proj = nn.Linear(hidden_dim * num_layers, out_dim)
        
            def forward(self, x, edge_index):
                import torch.nn.functional as F
                representations = []
                for layer in self.layers:
                    x = F.relu(layer(x, edge_index))
                    representations.append(x)
                # Concatenate representations from all layers
                h_all = torch.cat(representations, dim=1)
                return self.out_proj(h_all)
        

JK-Net effectively allows each node to choose the effective reception field that works best for it. Shallow aggregation is used for nodes in high-homophily local neighborhoods; deeper aggregation for nodes that benefit from global context.

Mitigation 3: DropEdge

DropEdge (Rong et al., 2020) randomly removes edges during each training step, reducing the propagation rate and slowing the convergence to the smooth limit:

def drop_edge(edge_index, drop_rate=0.3, training=True):
            """Randomly drop edges during training."""
            if not training or drop_rate == 0.0:
                return edge_index
            num_edges = edge_index.size(1)
            keep_mask = torch.rand(num_edges) > drop_rate
            return edge_index[:, keep_mask]
        
        # In a training loop:
        # edge_index_train = drop_edge(g.edge_index, drop_rate=0.3, training=model.training)
        # out = model(x, edge_index_train)
        

DropEdge acts as a form of data augmentation and implicitly regularizes the model against over-reliance on any specific edge. In the paper, it was shown to improve performance on Cora and Citeseer when stacking 8 or more GNN layers.

Mitigation 4: Normalization

Pair-norm (Zhao and Akoglu, 2020) and other normalization schemes explicitly maintain the spread of node representations:

class PairNorm(nn.Module):
            """Normalize node representations to maintain diversity."""
            def __init__(self, scale=1.0):
                super().__init__()
                self.scale = scale
        
            def forward(self, x):
                x = x - x.mean(dim=0, keepdim=True)   # center
                rms = (x.pow(2).sum(dim=1).mean()).sqrt()
                return self.scale * x / (rms + 1e-8)
        

Inserting a PairNorm layer between GNN layers prevents the Dirichlet energy from collapsing, though it does not address the root cause. It is most effective combined with residual connections.

Mitigation 5: Initial Residual and Identity Mapping (GCNII)

GCNII (Chen et al., 2020) combines two ideas into a deep GNN architecture that reliably trains to 64 layers without over-smoothing:

1. Initial residual: Each layer adds a fraction of the layer-0 (initial) features: H^(l+1) = σ( ((1-α) A_norm H^(l) + α H^(0)) W^(l) ). The α H^(0) term ensures that every layer has direct access to the original features, preventing convergence to the constant.

2. Identity mapping: The weight matrix is initialized close to the identity: W^(l) = (1 - β) I + β Θ^(l). This initialization, combined with the initial residual, allows gradient signals to flow through many layers without vanishing.

The resulting architecture can use many more layers than standard GCN for tasks that require long-range dependencies, at the cost of the additional hyperparameters α and β.

TGraphX does not currently implement GCNII as a dedicated layer, but the pattern can be constructed manually:

class GCNIILayer(nn.Module):
            """GCNII-inspired layer with initial residual and identity mapping."""
            def __init__(self, dim, alpha=0.1, beta=0.5):
                super().__init__()
                self.alpha = alpha
                self.beta = beta
                self.conv = VectorGCNLayer(dim, dim)
                self.linear = nn.Linear(dim, dim, bias=False)
        
            def forward(self, x, edge_index, x0):
                # Aggregation term
                h_agg = self.conv(x, edge_index)
                # Combine: initial residual
                h = (1 - self.alpha) * h_agg + self.alpha * x0
                # Identity mapping: W = (1 - β) I + β Θ
                h = (1 - self.beta) * h + self.beta * self.linear(h)
                return torch.relu(h)
        

Practical Depth Recommendations

Based on empirical results across many published papers, the following rough guidelines apply:

• 2 layers: Safe for most node classification tasks. Use this as your starting depth unless you have strong reasons to go deeper.
• 3–4 layers: Usually feasible with residual connections or batch normalization. Monitor MAD during training.
• 5–8 layers: Requires deliberate anti-smoothing measures (JK-Net, DropEdge, PairNorm). Test carefully.
• > 8 layers: Rarely helpful in practice. Very few published results show consistent gains beyond 8 layers. If you need to capture long-range dependencies, consider attention mechanisms or positional encodings rather than depth alone.

Limitations and Honest Notes

Over-smoothing is one of several reasons that deeper GNNs often underperform shallower ones, but it is not the only reason. Gradient vanishing, over-fitting on small graphs, and the mismatch between graph diameter and effective reception field also contribute. Mitigating over-smoothing (e.g., with residual connections) does not automatically fix these other problems.

The metrics — Dirichlet energy and MAD — measure representation diversity, but high diversity does not guarantee useful representations. A model can maintain high MAD while learning useless features. These metrics are diagnostic tools, not optimization targets.

DropEdge and PairNorm introduce their own hyperparameters (drop rate, scale) that require tuning. For small datasets, this tuning can eat into the labeled data budget. The benefit of these techniques is most clearly demonstrated on deeper networks; for 2-layer GNNs, they rarely make a meaningful difference.

The over-smoothing literature is dominated by homophilic benchmarks (Cora, Citeseer, Pubmed). On heterophilic graphs, the dynamics are different — deeper GNNs may actually benefit from longer range, and the appropriate mitigation strategies differ. Be cautious about applying homophily-era intuitions to heterophilic settings.

Frequently Asked Questions

Is over-smoothing the same as vanishing gradients?
No. Over-smoothing is a property of the forward pass — node representations converge to the same value. Vanishing gradients are a property of the backward pass — gradient signals become too small to train early layers. Both can affect deep GNNs, but they have different causes and different fixes.

Does batch normalization help with over-smoothing?
Batch normalization standardizes the feature distribution per dimension and helps with gradient flow, but it does not directly prevent over-smoothing. PairNorm is a normalization scheme specifically designed to maintain inter-node representation diversity.

What depth do state-of-the-art GNNs use?
Most competitive models on standard benchmarks use 2–4 layers. Some architectures like GCNII (Chen et al., 2020) achieve 64 layers with a specialized residual formulation, but this is an outlier rather than the norm.