Classes and methods for Geometric Deep Learning and related topics to support π¬ Substack - Newsletter and π¬ LinkedIn Articles
.
.
.
Facing challenges with high-dimensional, densely packed but limited data, and complex distributions?
Geometric Deep Learning (GDL) offers a solution by enabling data scientists to grasp the true shape and distribution of data. This newsletter explores the diverse techniques and frameworks shaping the field of Geometric Deep Learning.
Data Scientists or machine learning engineers involved in fields of research or applications such as Computer Vision, Robotics, Physics-informed Modeling, Molecular Science, Telecommunication Network Optimization, Transport Systems, Cosmology, Fluid Dynamics or Design Optimization.
Learn about Geometric Learning, Graph Neural Networks, Manifolds, Equivariance and Lie Groups or Topological data analysis and unlocks the ability to model complex, structured, and relational data.
Geometric Deep Learning (GDL) is a field of machine learning that generalizes deep learning to non-Euclidean domainsβsuch as graphs, manifolds, groups, and other structured geometric spacesβby leveraging their symmetries and topologies.
.
The key concepts are:
β
Invariance: Output stays the same when input is transformed.
β
Equivariance: Preserves transformations. The output transforms predictably when input is transformed.
β
Manifolds: Operates directly on Non-euclidean or curved spaces or tangent spaces.
β
Locality: Uses local node structures and infinitesimal neighborhoods.
Here are some references to get you feet wet!
- Geometric deep learning: going beyond Euclidean data M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, P. Vadergheynst - 2017
- Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges M. Bronstein, J. Bruna, T Cohen, P. VeliΔkoviΔ - 2021
- Theory of Graph Neural Networks: Representation and Learning S Jegelka, Dept. of EECS and CSAIL, MIT, 2022
- AMMI 2022 Course "Geometric Deep Learning" Lectures 1 to 12 M. Bronstein, J. Bruna, T Cohen, P. VeliΔkoviΔ - YouTube - 2022
- Topological Deep Learning: Going Beyond Graph Data M Hajij et All - 2023
- A Brief Introduction to Geometric Deep Learning - J. McEwen, Medium - 2023
- A Hands-on Introduction to Geometric Deep Learning, with Examples in PyTorch Geometric - G. Santin, A minitutorial at the SIAM Conference on Computational Science and Engineering, 2023
- Michael Bronstein - Geometric Deep Learning EEML Community - YouTube - 2024
- Introduction To Geometric Deep Learning Patrick Nicolas Substack - 2025
Differential geometry offers a solution by enabling data scientists to grasp the true shape and distribution of data.
Differential geometry is a branch of mathematics that uses techniques from calculus, algebra and topology to study the properties of curves, surfaces, and higher-dimensional objects in space. It focuses on concepts such as curvature, angles, and distances, examining how these properties vary as one moves along different paths on a geometric object.
Differential geometry is crucial in understanding the shapes and structures of objects that can be continuously altered, and it has applications in many fields including physics (I.e., general relativity and quantum mechanics), engineering, computer science, and data exploration and analysis.
Euclidean space, created from a collection of maps (or charts) called an atlas, which belongs to Euclidean space. Differential manifolds have a tangent space at each point, consisting of vectors. Riemannian manifolds are a type of differential manifold equipped with a metric to measure curvature, gradient, and divergence.
The directory geometry contains the definition and implementation the various elements of smooth manifolds using Geomstats module.
- Covariant and contravariant vectors and tensors
- Intrinsic and Extrinsic coordinates
- Riemannian metric tensor
- Tangent space, Exponential and Logarithmic maps
- Levi-Civita connection and parallel transport
- Curvature tensor
- Euclidean, hypersphere and Kendal spaces
- Logistic regression and K-Means on hypersphere
- Frechet mean
- Push-forward and pull-back
Lie groups play a crucial role in Geometric Deep Learning by modeling symmetries such as rotation, translation, and scaling. This enables non-linear models to generalize effectively for tasks like object detection and transformations in generative models.
Lie groups have numerous practical applications in various fields:
β
Robotics: Lie groups model the motion of robots, particularly in the context of rotation and translation (using groups like SO(3) and SE(3)).
β
Control Theory: Lie groups are used in the analysis and design of control systems, especially in systems with rotational or symmetrical behavior.
β
Computer Vision: They help in image processing and 3D vision, especially in tasks involving rotations and transformations.
β
Differential Equations: Lie groups are instrumental in solving differential equations by leveraging symmetry properties.
The directory Lie illustrates the various element of Special Orthogonal Group of 3 dimension (SO3) and Special Euclidean Group in 3 dimension (SE3) using Geomstats library.
Note: Visualizing the SO(3) manifold in three dimensions is inherently challenging. In this example, we represent the space of rotations as a solid ball, where the center corresponds to the identity rotation and each point within the ball encodes a rotation using the axis-angle representation. Reference: SO3 Visualization
π Lie-equivariant networks are neural networks that are equivariant to transformations from a Lie group, such as rotations, translations, scaling, or Lorentz transformations. These networks are designed to respect the continuous symmetries inherent in many scientific and geometric problems.
π Group-equivariant Convolutional Neural Networks (G-CNN) are a generalization of standard Convolutional Neural Networks designed to respect symmetries in data, especially group symmetries (translational, Rotational, Reflectional and Permutational).
Graph Neural Networks (GNNs) are a class of deep learning models designed to operate directly on graph-structured data. Unlike standard neural networks that work on fixed-size vectors or grids (like images or sequences), GNNs handle data where relationships are represented as nodes and edgesβlike social networks, molecules, recommendation systems, and more.
There are 3 levels or categories of tasks:
- Graph-level task: Predict the property of the entire graph such as classification problems with MNIST or CIFAR images or sentiment analysis for a document or paragraph.
- Node-level task: Predict if a node belongs to a specific class (i.e. Karate club) or image segmentation (identify the role of a pixel in an image) or part of speech a word belongs to.
- Edge-level task: Predict the relationship between node (i.e. Interaction between users) that can be classified (discovery of connections between entities or nodes). The task also consists in pruning a fully connected graph into a sparse graph.
The key Attributes that define the complexity of a Graph Neural Network are
- Number of graph convolutional and attention layers
- Graph pooling layer
- Node or edge neighbor sampling method
- Residual connections
- Heteromophilic nodes
- Inclusion of isolated nodes
- Aggregation method
- Data transformation prior aggregation
There are many types of Graph Neural Networks as new ones are regularly introduced in research papers. Here is the list of the 5 common types of Graph Neural Networks:
β
Graph Convolutional Networks
β
Graph Attention Networks
β
GraphSAGE
β
Spectral Graph Neural Networks
β
Graph Transformers
The homophily ratio in a Graph Neural Network (GNN) quantifies how often nodes in a graph are connected to other nodes with the same label or class. Itβs a key measure for understanding how well GNNs are likely to perform, especially those that rely on message passingβwhich typically assumes that connected nodes share similar features or labels.
There are 3 homophily ratios:
β
Node homophily: average fraction of same-label neighbors per node.
β
__Edge homophily__: fraction of edges connecting nodes with the same label
β
__Class-insensitive edge homophily__: adjusts for class imbalance by computing same-label edge fractions per class, then averaging. across the C classes.
Since its introduction in early 2019, PyTorch Geometric has expanded significantly, incorporating new models, graph samplers, and transformations, continuously evolving to align with the latest research publications.
The key Features of PyTorch Geometric are:
- Efficient Graph Processing: Optimizes memory and computation using sparse graph representations.
- Flexible GNN Layers: Covers GCN, GAT, GraphSAGE, GIN, and other advanced architectures.
- Batching for Large Graphs: Supports for mini-batching for handling graphs with millions of edges.
- Seamless PyTorch Integration: Provides full compatibility with PyTorch tensors, autograd, and neural network modules.
- Diverse Graph Support: PyTorch Geometric handles directed, undirected, weighted, and heterogeneous graphs.
The most important PyG Modules are:
β
torch_geometric.data to manages graph structures, including nodes, edges, and features.
β
torch_geometric.nn to provide data scientists prebuilt GNN layers like convolutional and gated layers.
β
torch_geometric.transforms to pre-process input data (e.g., feature normalization, graph sampling).
β
torch_geometric.loader to handle large-scale graph datasets with specialized loaders.
Information geometry applies the principles and methods of differential geometry to problems in probability theory and statistics. It studies the manifold of probability distributions and provides a natural framework for understanding and analyzing statistical models.
The directory informationgeometry focuses on the Fisher Information Metric (FIM).
Statistical manifolds generalize the concept of Riemannian manifolds to the space of probability distributions. While their geometric propertiesβsuch as exponential and logarithm maps or distanceβare often intractable, univariate distributions typically allow for closed-form or analytical expressions of these properties.
Most of Statistical manifolds (with 2 or more parameters) have intractable exponential, logarithm and inner production formulation. However single parameters distribution have closed-form formulation:
- Exponential Distribution
- Poisson Distribution
- Geometric Distribution
- Binomial Distribution with fixed draw
Example: Geometric Distribution
Reference [Geometry of Closed-form Statistical Manifolds]
The Fisher Information Matrix plays a crucial role in various aspects of machine learning and statistics. Its primary significance lies in providing a measure of the amount of information that an observable random variable carries about an unknown parameter upon which the probability depends on.
The Fisher information matrix is a type of Riemannian metric that can be applied to a smooth statistical manifold. It serves to quantify the informational difference between measurements. The points on this manifold represent probability measures defined within a Euclidean probability space, such as the Normal distribution. Mathematically, it is represented by the Hessian of the Kullback-Leibler divergence.
The directory dl implements a framework of reusable neural blocks as key components of any deep learning models such as:
β
Feed forward neural network/Multilayer perceptron
β
Convolutional network
β
Variational auto-encoder
β
Generative adversarial network
β
Automatic generation (mirror) of encoder/de-convolutional blocks.
Configuring the parameters of a 2D convolutional neural network, such as kernel size and padding, can be challenging because it largely depends on the complexity of an image or its specific sections. Fractals help quantify the complexity of important features and boundaries within an image and ultimately guide the data scientist in optimizing his/her model.
A fractal dimension is a measure used to describe the complexity of fractal patterns or sets by quantifying the ratio of change in detail relative to the change in scale.
For ordinary geometric shapes, the fractal dimension theoretically matches the familiar Euclidean or topological dimension.
There are many approaches to compute the fractal dimension of an image, among them:
- Variation method
- Structure function method
- Root-mean-square method
- R/S analysis method
- Box counting method
The directory 'fractal' contains the implementation of the box counting method for images and 3D objects.
Sampling sits at the core of data science and analysis. The directory mcmc explores a category of numerical sampling techniques, known as Markov Chain Monte Carlo, and how they can be implemented via reusable design patterns, using the Metropolis-Hastings model as an example.
A Kalman filter serves as an ideal estimator, determining parameters from imprecise and indirect measurements. Its goal is to reduce the mean square error associated with the model's parameters. Being recursive in nature, this algorithm is suited for real-time signal processing.
The directory control contains the implementation of Kalman filters.
The source tree is organized as follows:
- Features in python/
- Unit tests in tests/
- Newsletter specific evaluation code in play/
- Animation in tests/anim
Files and classes in this repository follow a strict naming convention as illustrated in the following example:
π Features file: python/topology/simplicial/abstract_simplicial_complex.py
π Features Class: AbstractSimplicialComplex.py
π Unit test file: python/topology/simplicial/abstract_simplicial_complex_test.py
π Unit test class: python/topology/simplicial/AbstractSimplicialComplexTest.py
π Substack newsletter evaluation code: play/AbstractSimplicialComplexPlay.py
Here are the list of exceptions used in this library.
| Exception | Description |
|---|---|
| Standard Library | |
| TypeError | Incorrect type of argument |
| ValueError | Incorrect value of argument |
| KeyError | Incorrect key lookup in dictionary |
| IndexError | Index in sequence out of range |
| AssertError | Internal validation failed |
| NotImplementedError | Not Implemented |
| Geometric Learning | |
| DatasetException | Failure of execution for datasets |
| GraphException | Failure of execution for Graph Neural Networks |
| MetricException | Failure of execution for Metrics |
| TrainingException | Failure of execution during model training |
| MLPException | Failure of execution for Multi-layer perceptron |
| ConvException | Failure of execution for Convolutional neural networks |
| MCMCException | Failure of execution for Markov Chain Monte Carlo |
| TopologyException | Failure of execution for Topological data analysis or deep learning |
| InformationGeometryException | Failure of execution for statistical manifold and information geometry |
| ManifoldException | Failure of execution for smooth, Riemannian manifold |
| LieException | Failure of execution for Lie groups and algebras |
| GenerativeException | Failure of execution for Generative and Auto-encoders models |
| Library | Version |
|---|---|
| Python | 3.12.9 |
| SymPy | 1.12 |
| Numpy | 2.1.3 |
| Pydantic | 2.4.1 |
| Shap | 0.43.0 |
| torch | 2.6.0 |
| torchVision | 0.20.1 |
| torch-geometric | 2.6.1 |
| torch_sparse | 0.6.18 |
| torch_scatter | 2.12 |
| torch_cluster | 1.6.3 |
| Scikit-learn | 1.5.2 |
| Geomstats | 2.8.0 |
| Jax | 0.4.34 |
| PyTest | 8.3.3 |
| matplotlib | 3.10.0 |
| ffmpeg | 1.14 |
| latex | 0.7.0 |
| manim | 0.19.0 |
| memory-profiler | 0.61.0 |
| networkx | 3.4.2 |
| optuna | 4.4.0 |
| pillow | 11.1.0 |
| pytorch-lightning | 2.5.1 |
A block is defined as a logical grouping of neural components, implemented as Pytorch Module. All these components are assembled into a sequential set of torch modules.
class NeuralBlock(nn.Module):
def __init__(self, block_id: Optional[AnyStr], modules: Tuple[nn.Module]):
super(NeuralBlock, self).__init__()
self.modules = modules
self.block_id = block_id
A MLP block a fully-connected layer, an activation function, and possibly a drop-out component.
.
class MLPBlock(NeuralBlock):
def __init__(self,
block_id: AnyStr,
layer_module: nn.Linear,
activation_module: Optional[nn.Module] = None,
dropout_module: Optional[nn.Dropout] = None) -> None:
# A MLP block should contain at least a fully connected layer
modules = [layer_module]
# Add activation module if defined
if activation_module is not None:
modules.append(activation_module)
# Add drop out module if specified
if dropout_module is not None:
modules.append(dropout_module)
super(MLPBlock, self).__init__(block_id, modules)
self.activation_module = activation_module
A Convolutional block may include a convolutional layer, kernel, batch normalization and possibly a drop-out components of type Module.
class Conv2dBlock(ConvBlock):
valid_modules = ('Conv2d', 'MaxPool2d', 'BatchNorm2d', 'Dropout2d')
def __init__(self,
block_id: AnyStr,
conv_layer_module: nn.Conv2d,
batch_norm_module: Optional[nn.BatchNorm2d] = None,
activation_module: Optional[nn.Module] = None,
max_pooling_module: Optional[nn.MaxPool2d] = None,
drop_out_module: Optional[nn.Dropout2d] = None) -> None:
# The 2-dimensional convolutional layer has to be defined
modules = [conv_layer_module]
# Add a batch normalization is provided
if batch_norm_module is not None:
modules.append(batch_norm_module)
# Add an activation function is required
if activation_module is not None:
modules.append(activation_module)
# Add a mandatory max pooling module
if max_pooling_module is not None:
modules.append(max_pooling_module)
# Add a Drop out regularization for training if provided
if drop_out_module is not None:
modules.append(drop_out_module)
super(Conv2dBlock, self).__init__(block_id, modules)
class GConvBlock(nn.Module):
def __init__(self,
block_id: AnyStr,
gconv_layer: GraphConv,
batch_norm_module: Optional[BatchNorm] = None,
activation_module: Optional[nn.Module] = None,
pooling_module: Optional[SAGPooling | TopKPooling] = None,
dropout_module: Optional[nn.Dropout] = None) -> None:
super(GConvBlock, self).__init__()
self.block_id = block_id
# Iteratively build the sequence of Torch Module according
# to the order of the arguments of the constructor
modules: List[nn.Module] = [gconv_layer]
if batch_norm_module is not None:
modules.append(batch_norm_module)
if activation_module is not None:
modules.append(activation_module)
if pooling_module is not None:
modules.append(pooling_module)
if dropout_module is not None:
modules.append(dropout_module)
self.modules = modules
The current hierarchy of neural blocks is defined as:
Neural models are dynamic sequence of neural blocks that are assembled and converted into a sequence of torch Module instances.
The Base class for Neural model is defined as
class NeuralModel(torch.nn.Module, ABC):
def __init__(self, model_id: AnyStr, modules_seq: nn.Module) -> None:
super(NeuralModel, self).__init__()
self.model_id = model_id
self.modules_seq = modules_seq
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.model(x)
Each model inherits from NeuralModel (i.e. Convolutional neural network type : ConvModel)
A Multi-layer Perceptron generated from reusable neural blocks
class MLPModel(NeuralModel):
def __init__(self, model_id: AnyStr, neural_blocks: List[MLPBlock]) -> None:
self.neural_blocks = neural_blocks
# Define the sequence of modules from the layout of neural blocks
modules = [module for block in neural_blocks
for module in block.modules]
super(MLPModel, self).__init__(model_id, nn.Sequential(*modules))
and it associated builder pattern.
with implementation....
class MLPBuilder(NeuralBuilder):
keys = ['in_features_list', 'activation', 'drop_out', 'output_activation']
def __init__(self, model_id: AnyStr) -> None:
super(MLPBuilder, self).__init__(model_id, MLPBuilder.keys)
# Default configuration parameters that can be overwritten
self._attributes['activation'] = nn.ReLU()
self._attributes['drop_out'] = 0.0
def build(self) -> MLPModel:
# Instantiate the model from the dictionary of Configuration parameters
mlp_blocks = self.__create_blocks()
# Validation
MLPBuilder.validate(mlp_blocks)
return MLPModel(self._attributes['model_id'], mlp_blocks)
A convolutional neural network is generated from reusable neural blocks using the Builder recursive pattern.
class Conv2dModel(ConvModel):
def __init__(self,
model_id: AnyStr,
input_size: Conv2DataType,
conv_blocks: List[Conv2dBlock],
mlp_blocks: Optional[List[MLPBlock]] = None) -> None:
super(Conv2dModel, self).__init__(model_id, input_size, conv_blocks, mlp_blocks)
with its builder:
class Conv2dBuilder(NeuralBuilder):
keys = ['input_size', 'in_channels_list', 'kernel_size', 'stride',
'padding', 'is_batch_norm', 'max_pool_kernel', 'activation',
'in_features_list', 'output_activation', 'bias', 'drop_out']
def __init__(self, model_id: AnyStr) -> None:
super(Conv2dBuilder, self).__init__(model_id, Conv2dBuilder.keys)
# Provide default values that may be overwritten.
self._attributes['stride'] = (1, 1)
self._attributes['padding'] = (1, 1)
self._attributes['is_batch_norm'] = True
self._attributes['activation'] = nn.ReLU()
self._attributes['bias'] = False
self._attributes['drop_out'] = 0.0
def build(self) -> Conv2dModel:
# Instantiate the model from the dictionary of Configuration parameters
model_id = self._attributes['model_id']
# Generate the convolutional neural blocks from the configuration attributes dictionary
conv_blocks = self.__create_conv_blocks()
# Generate the fully connected blocks from the configuration attributes dictionary
mlp_blocks = self.__create_mlp_blocks()
# Validation
Conv2dBuilder.__validate(conv_blocks, mlp_blocks, self._attributes['input_size'])
return Conv2dModel(model_id, self._attributes['input_size'], conv_blocks, mlp_blocks)
class GConvModel(nn.Module):
def __init__(self,
model_id: AnyStr,
gconv_blocks: List[GConvBlock],
mlp_blocks: Optional[List[MLPBlock]] = None) -> None:
super(GConvModel, self).__init__()
self.model_id = model_id
self.gconv_blocks = gconv_blocks
# Extract the torch modules for the convolutional blocks
# in the appropriate order
modules: List[nn.Module] = [module for block in gconv_blocks
for module in block.modules]
# If fully connected are provided as CNN
if mlp_blocks is not None:
self.mlp_blocks = mlp_blocks
# Flatten the output from the last convolutional layer
modules.append(nn.Flatten())
# Extract the relevant modules from the fully connected blocks
[modules.append(module) for block in mlp_blocks for module in block.modules]
self.modules = modules
The current class hierarchy for Neural models is defined as:
-Introduction to Geometric Deep Learning
-Riemannian Manifolds: Foundational Concepts
-Riemannian Manifolds: Hands-on with Hypersphere
-Reusable Neural Blocks in PyTorch & PyG
-Modular Deep Learning Models with Neural Blocks
-Taming PyTorch Geometric for Graph Neural Networks
-Practical Introduction to Lie Groups in Python
-Demystifying Graph Sampling & Walk Methods
-Modeling Graph Neural Networks with PyTorch
-FrΓ©chet Centroid on Manifolds in Python
-Visualization of Graph Neural Networks
-Limitations of the Linear Kalman Filter
-Introduction to SO3 Lie Group in Python
-Introduction to SE3 Lie Groups in Python