Bayesian deep learning, at DTU PhD school
We can characterize these manifolds
through the full-data GGN
(aka the Fisher-Rao metric)
In practice
A toy-example
Consider the (absurdly) simple neural network
A toy-example
Consider the (absurdly) simple neural network
Note that we can write this as
This implies that the weights
and
correspond to the same function.
Let's look at this in weight space:
A toy-example
Consider the (absurdly) simple neural network
Note that we can write this as
This implies that the weights
and
correspond to the same function.
Let's look at this in weight space:
Let's look at this in weight space:
Now, apply your favorite Bayesian approximation...
Let's look at this in weight space:
Now, apply your favorite Bayesian approximation...
Unlikely under posterior
Likely under posterior
Unlikely under posterior
Likely under posterior
Let's look at this in weight space:
Let's look at this in weight space:
Now, apply your favorite Bayesian approximation...
At this point, our approximate posterior
violates the basic rules of probability:
it assigns multiple densitiy values to a
single function, such that the posterior
probability density
function
(pdf) is not
a function...
WTF ?!
At this point, our approximate posterior
violates the basic rules of probability:
it assigns multiple densitiy values to a
single function, such that the posterior
probability density
function
(pdf) is not
a function...
You can show that... (but I won't have time)
Consider a (smooth) overparametrized
model. At parameter , the space
of parameters locally decompose into
two Riemannian manifolds:
You can show that... (but I won't have time)
Consider a (smooth) overparametrized
model. At parameter , the space
of parameters locally decompose into
two Riemannian manifolds:
the kernel manifold:
a smoothly
connected set of parameters that
yield identical functions.
You can show that... (but I won't have time)
Consider a (smooth) overparametrized
model. At parameter , the space
of parameters locally decompose into
two Riemannian manifolds:
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical functions.
connected set of parameters that
yield strictly different functions.
You can show that... (but I won't have time)
Consider a (smooth) overparametrized
model. At parameter , the space
of parameters locally decompose into
two Riemannian manifolds:
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical functions.
connected set of parameters that
yield strictly different functions.
If you change these manifolds
change as well.
You can show that... (but I won't have time)
You can show that... (but I won't have time)
An eigen-decomposition of this yields
You can show that... (but I won't have time)
An eigen-decomposition of this yields
where
is tangential to the kernel
maniflold
You can show that... (but I won't have time)
An eigen-decomposition of this yields
where
is tangential to the kernel
maniflold
is tangential to the image
maniflold
You can show that... (but I won't have time)
An eigen-decomposition of this yields
where
is tangential to the kernel
maniflold
is tangential to the image
maniflold
In essence:
we can get a coordinate system
in which some dimensions never
change the function, while the
others always change the function.
In practice, things are slightly different
If we replace the full-data GGN
with an average over the training data
this changes things.
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical functions.
connected set of parameters that
yield strictly different functions.
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical predictions on the
training data.
connected set of parameters that
yield strictly different predictions
on the training data.
(points in the kernel manifold retain predictions
on training data)
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical predictions on the
training data.
connected set of parameters that
yield strictly different predictions
on the training data.
(points in the kernel manifold retain predictions
on training data)
Our eigen-decomposition
now gives a coordinate system
where
changes the function every-
where except at the training
data.
changes the function every-
where.
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical predictions on the
training data.
connected set of parameters that
yield strictly different predictions
on the training data.
(points in the kernel manifold retain predictions
on training data)
Our eigen-decomposition
now gives a coordinate system
where
changes the function every-
where except at the training
data.
changes the function every-
where.
can be estimated from data
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical predictions on the
training data.
connected set of parameters that
yield strictly different predictions
on the training data.
(points in the kernel manifold retain predictions
on training data)
Our eigen-decomposition
now gives a coordinate system
where
changes the function every-
where except at the training
data.
changes the function every-
where.
these cannot
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
Define an approximate posterior with covariance
3
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
Define an approximate posterior with covariance
COMPUTATIONAL
Gigantic!
Cubic complexity
von Neumann to the rescue!
Fortunately, we only care about the kernel (null space)
of the generalized Gauss Newton (GGN) matrix
John von Neumann
from a Danish perspective
von Neumann to the rescue!
Fortunately, we only care about the kernel (null space)
of the generalized Gauss Newton (GGN) matrix
John von Neumann
from a Danish perspective
This kernel is the intersection of the kernels
of mini-batched GGN estimates
von Neumann to the rescue!
Fortunately, we only care about the kernel (null space)
of the generalized Gauss Newton (GGN) matrix
John von Neumann
from a Danish perspective
This kernel is the intersection of the kernels
of mini-batched GGN estimates
von Neumann's alternating projections algorithm
can then project to the GGN kernel. This can be
implemented in (with some Jax tricks)
space complexity, and
time complexity.
model
parameters
number of
data
This scales to
large models!
iteration
counter
It actually works (even if it's clearly dumb)
Standard tests on in-distribution performance.
In-distribution performance is unchanged.
It actually works (even if it's clearly dumb)
Standard tests on out-of-distribution performance (AUROC, bigger is better).
Seems to be best in class. In all experiments, in-distribution performance
is unchanged.
Boldly goes where Bayes barely goes
Scales to contemporary models and dataset.
Improves out-of-distribution performance.
SWIN transformer (
28 million params
) trained on ImageNet-1K;
tested on Places365.
Boldly goes where Bayes barely goes
Scales to contemporary models and datasets (AUROC).
Improves out-of-distribution performance.
Visual Attention Network (VAN,
4 million params
) trained on CelebA.
Boldly goes where Bayes barely goes
Scales to image generative models
Can flag OoD image content
Original
Reconstruct
MSE
Uncertainty
Perceptual VAE (
50 million params
) trained on Fitzpatrick17k.
MSE
A: Light
B: Mixed
C: Dark
A: Light
B: Mixed
C: Dark
Uncertainty
Strong correlation between uncertainty and reconstruction error
(you should be paying attention if you use generative models to create training data)
Correlation: 0.92
MSE
Uncertainty
Similar trends when testing on out-of-distribution data (PASSION)
Summary: uncertainty is an excellent proxy for accuracy; here potentially
useful for revealing bias in model/data.
Define an approximate posterior with covariance
In theory
The space of model parameters
always decompose into
the image manifold:
a smoothly
the kernel manifold:
a smoothly
connected set of parameters that
yield identical predictions on the
training data.
connected set of parameters that
yield strictly different predictions
on the training data.
Disregarding this structure breaks
the usual Bayesian approximations.
Summary (and stuff I didn't tell you)
Scalable algorithms allows
you to project onto the tangent
spaces of these manifolds.
This is enough to build (locally)
reparametrization invariant
approximate posteriors.
And it even works pretty well.
Summary (and stuff I didn't tell you)
I didn't have time to tell you that:
You can say a lot of fun things
about the geometry of the
image and kernel manifolds.
Using Riemannian diffusion you
can build better models (but it's
too expensive to be worth it).
You can do full variational training
of our approximate posterior
(and it works).
You can use techniques from
matrix sketching to get even
faster algorithms (logarithmic
in number of parameters for
some tasks).
Selected references
VIKING: Deep variational inference with stochastic
projections.
Fadel, Roy, Krämer, Zainchkovskyy,
Syrota, Valverde Mahou, Ek, and Hauberg
.
NeurIPS 2025.
Bayes without Underfitting: Fully Correlated Deep
Learning Posteriors via Alternating Projections.
Marco Miani, Hrittik Roy and Søren Hauberg.
AISTATS 2025.
Gradients of Functions of Large Matrices.
Nicholas Krämer, Pablo Moreno-Muñoz,
Hrittik Roy and Søren Hauberg.
NeurIPS 2024.
Reparameterization invariance in approximate
Bayesian inference.
Hrittik Roy, Marco Miani,
Carl Henrik Ek, Philipp Hennig, Marvin Pförtner,
Lukas Tatzel and Søren Hauberg.
NeurIPS 2024.
Sketched Lanczos uncertainty score: a low-
memory summary of the Fisher information.
Marco Miani, Lorenzo Beretta and Søren Hauberg.
NeurIPS 2024.
Summary (and stuff I didn't tell you)
I didn't have time to tell you that:
You can say a lot of fun things
about the geometry of the
image and kernel manifolds.
Using Riemannian diffusion you
can build better models (but it's
too expensive to be worth it).
You can do full variational training
of our approximate posterior
(and it works).
You can use techniques from
matrix sketching to get even
faster algorithms (logarithmic
in number of parameters for
some tasks).
Selected references
VIKING: Deep variational inference with stochastic
projections.
Fadel, Roy, Krämer, Zainchkovskyy,
Syrota, Valverde Mahou, Ek, and Hauberg.
NeurIPS 2025.
Bayes without Underfitting: Fully Correlated Deep
Learning Posteriors via Alternating Projections.
Marco Miani, Hrittik Roy and Søren Hauberg.
AISTATS 2025.
Gradients of Functions of Large Matrices.
Nicholas Krämer, Pablo Moreno-Muñoz,
Hrittik Roy and Søren Hauberg.
NeurIPS 2024.
Reparameterization invariance in approximate
Bayesian inference.
Hrittik Roy, Marco Miani,
Carl Henrik Ek, Philipp Hennig, Marvin Pförtner,
Lukas Tatzel and Søren Hauberg.
NeurIPS 2024.
Sketched Lanczos uncertainty score: a low-
memory summary of the Fisher information.
Marco Miani, Lorenzo Beretta and Søren Hauberg.
NeurIPS 2024.
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
Define an approximate posterior with covariance
Just requires us to draw
a Gaussian sample
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
Define an approximate posterior with covariance
Just requires us to draw
a Gaussian sample and
projecting this to the
kernel dimensions
Bayesian training works as well
We can also estimate by directly maximizing the usual ELBO lower
of the marginal likelihood
This can be evaluated stochastically with the projection algorithm.
Empirically, this stabilize training, allows you to automatically infer
prior precision (regularization strength) and posterior variances.
Bayesian training works as well
We can also estimate by directly maximizing the usual ELBO lower
of the marginal likelihood
This can be evaluated stochastically with the projection algorithm.
Empirically, this stabilize training, allows you to automatically infer
prior precision (regularization strength) and posterior variances.
Bayesian training works as well
It even applies to not-too-tiny models...
(to those of you who know my work on latent space geometries, you will
know that I link uncertainty to topology, which is kinda why I'm doing this stuff...)
Part three
In which we see the underlying problem...
Here's a funny observation...
Recall the generalized Gauss-Newton Hessian approximation
Here's a funny observation...
Recall the generalized Gauss-Newton Hessian approximation
Jacobian of neural network at
Here's a funny observation...
Recall the generalized Gauss-Newton Hessian approximation
Jacobian of neural network at
Another intepretation of the associated Laplace approximation appears
by linearizing the neural network with respect to the weights (but not
x
)
Under this approximation, the neural network Hessian is the GGN,
i.e., if we allow ourselves to linearize the neural network (wrt. weights)
then we don't have to approximate the Hessian.
Here's a funny observation...
Recall the generalized Gauss-Newton Hessian approximation
Jacobian of neural network at
Another intepretation of the associated Laplace approximation appears
by linearizing the neural network with respect to the weights (but not
x
)
Under this approximation, the neural network Hessian is the GGN,
i.e., if we allow ourselves to linearize the neural network (wrt. weights)
then we don't have to approximate the Hessian.
But really...
who cares???
A subtle implication is that now we should also linearize the network when making
posterior predictions, i.e.,
Here's a funny observation...
This should blow your mind!
Replacing the neural network with a linearized version
(without having a mathematical need!)
makes predictions better!
This is also seems to work with large networks.
This is freakin' weird! I demand an explanation!
Starting with linear functions
Consider the linear (in parameters) function
and the 'reparametrization function'
Starting with linear functions
Consider the linear (in parameters) function
and the 'reparametrization function'
We then say that
and
are equivalent if
We write this
Starting with linear functions
Consider the linear (in parameters) function
and the 'reparametrization function'
We then say that
and
are equivalent if
We write this
Starting with linear functions
Consider the linear (in parameters) function
and the 'reparametrization function'
We then say that
and
are equivalent if
We write this
nullspace
Starting with linear functions
Consider the linear (in parameters) function
and the 'reparametrization function'
We then say that
and
are equivalent if
We write this
nullspace
Linearized neural networks
Consider a linearized neural network
where we evaluate at the entire training set, i.e.
(i.e.
x
is a stacking of all observations)
Linearized neural networks
Consider a linearized neural network
We then say that
and
are equivalent if
We write this
nullspace
where we evaluate at the entire training set, i.e.
(i.e.
x
is a stacking of all observations)
Linearized neural networks
Consider a linearized neural network
We then say that
and
are equivalent if
We write this
nullspace
where we evaluate at the entire training set, i.e.
(i.e.
x
is a stacking of all observations)
If we now reintroduce
we see that
Why Linearized Laplace works
With our basic understanding of linear reparametrizations, we can understand
why Linearized Laplace works better than 'classic' Laplace.
Why Linearized Laplace works
With our basic understanding of linear reparametrizations, we can understand
why Linearized Laplace works better than 'classic' Laplace.
Laplace
Linearized
Laplace
Recall
Why Linearized Laplace works
With our basic understanding of linear reparametrizations, we can understand
why Linearized Laplace works better than 'classic' Laplace.
Laplace
Linearized
Laplace
Recall
If we make an eigendecomposition
Any sample from
q
can then be decomposed
image contribution
kernel contribution
In Linearized Laplace we see that the kernel part of the sample disappears
(it is absorbed into the Jacobian nullspace)
such that reparametrizations disappear under the linear model.
Linearized Laplace is, thus, invariant to reparametrizations (of the linear function).
FMINST with 10,000 observation
Number of parameters
Effective number of parameters (%)
(
(
Laplace only fails when reparametrizations dominate;
less so for linearized Laplace
Recall that the GGN rank
gives the dimension of the
effective space of functions,
while the remaining dimensions
are reparametrizations.
As we add more parameters
to our neural nets, we
increasingly emphasize
reparametrizations.
Linearized Laplace is, thus, invariant to reparametrizations (of the linear function).
Who cares?
Summary: Laplace works fine when reparametrizations are not a big deal
(e.g. in non-neural models), but as we increase neural model capacity,
we increase the reparametrization issue. Linearization gets rid of
reparametrizations infinitesimally, which helps quite a bit.
We can characterize these manifolds
through the full-data GGN
(aka the Fisher-Rao metric)
We can characterize these manifolds
through the full-data GGN
(aka the Fisher-Rao metric)
We can characterize these manifolds
through the full-data GGN
(aka the Fisher-Rao metric)
We can characterize these manifolds
through the full-data GGN
(aka the Fisher-Rao metric)
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
Define an approximate posterior with covariance
Just requires us to draw
a Gaussian sample and
projecting this to the
kernel dimensions
The dumbest Bayesian approximation
Compute the empirical Gauss Newton matrix
1
Compute its eigenvectors
2
3
(if low-noise data)
Define an approximate posterior with covariance
Just requires us to draw
a Gaussian sample and
projecting this to the
kernel dimensions
Summary (and stuff I didn't tell you)
Summary (and stuff I didn't tell you)
I'm not trying to sell you a method (we currently do not
recommend our own stuff). I'm telling you that the
problem is hard. Be skeptical of sales people in this field!
Summary (and stuff I didn't tell you)
Standard point estimates (max. likelihood and MAP) allow you to reason about
data uncertainty, but not model uncertainty (which is what you usually want).
Summary (and stuff I didn't tell you)
Standard point estimates (max. likelihood and MAP) allow you to reason about
data uncertainty, but not model uncertainty (which is what you usually want).
Bayesian approaches gives you model uncertainty, but they require (sometimes pretty crazy)
approximations (which fail most of the time).
Summary (and stuff I didn't tell you)
Standard point estimates (max. likelihood and MAP) allow you to reason about
data uncertainty, but not model uncertainty (which is what you usually want).
Bayesian approaches gives you model uncertainty, but they require (sometimes pretty crazy)
approximations (which fail most of the time).
Point estimates work, but doesn't give you what you want!
Bayesian methods give you what you want, but they don't work.
Upcoming book
Primer on the underlying geometry;
to appear with Cambridge Uni Press.
www2.compute.dtu.dk/~sohau/weekendwithbernie/
Summary (and stuff I didn't tell you)
I didn't have time to tell you that:
You can say a lot of fun things
about the geometry of the
image and kernel manifolds.
Using Riemannian diffusion you
can build better models (but it's
too expensive to be worth it).
You can do full variational training
of our approximate posterior
(and it works).
You can use techniques from
matrix sketching to get even
faster algorithms (logarithmic
in number of parameters for
some tasks).
Selected references
VIKING: Deep variational inference with stochastic
projections.
Fadel, Roy, Krämer, Zainchkovskyy,
Syrota, Valverde Mahou, Ek, and Hauberg.
NeurIPS 2025.
Bayes without Underfitting: Fully Correlated Deep
Learning Posteriors via Alternating Projections.
Marco Miani, Hrittik Roy and Søren Hauberg.
AISTATS 2025.
Gradients of Functions of Large Matrices.
Nicholas Krämer, Pablo Moreno-Muñoz,
Hrittik Roy and Søren Hauberg.
NeurIPS 2024.
Reparameterization invariance in approximate
Bayesian inference.
Hrittik Roy, Marco Miani,
Carl Henrik Ek, Philipp Hennig, Marvin Pförtner,
Lukas Tatzel and Søren Hauberg.
NeurIPS 2024.
Sketched Lanczos uncertainty score: a low-
memory summary of the Fisher information.
Marco Miani, Lorenzo Beretta and Søren Hauberg.
NeurIPS 2024.
Upcoming book
Primer on the underlying geometry;
to appear with Cambridge Uni Press.
www2.compute.dtu.dk/~sohau/weekendwithbernie/
Thanks for the attention!
Bayesian deep learning, at DTU PhD school
August 2026
Søren Hauberg
1