Bayesian deep learning, at DTU PhD school
The beginning will be basic, but hang in there if you already know it -- there's a point!
My ambition is not to sell you on a solution, but to communicate some of the
difficulties of the problem. Hopefully, making it easier for you to be skeptical...
My usual story
advertise book
Søren Hauberg
Technical University of Denmark
sohau@dtu.dk
from basics, through why it fails, towards something sensible
Bayesian deep learning
The big question: why are we here?
Uncertainty is an inherent part of real-world data and decision-making,
yet traditional machine learning models often overlook it. In recent years,
the field of uncertainty quantification has emerged as a vital area of research,
aiming to equip models with the ability not only to make predictions, but
also to accurately express how confident they are in those predictions.
This is especially important in domains where the cost of errors is high,
such as healthcare, autonomous systems, and scientific discovery.
https://uncertainty.compute.dtu.dk/
The big question: why are we here?
Uncertainty is an inherent part of real-world data and decision-making,
yet traditional machine learning models often overlook it. In recent years,
the field of uncertainty quantification has emerged as a vital area of research,
aiming to equip models with the ability not only to make predictions, but
also to accurately express how confident they are in those predictions.
This is especially important in domains where the cost of errors is high,
such as healthcare, autonomous systems, and scientific discovery.
https://uncertainty.compute.dtu.dk/
That's not quite true...
The big question: why are we here?
Uncertainty is an inherent part of real-world data and decision-making,
yet traditional machine learning models often overlook it. In recent years,
the field of uncertainty quantification has emerged as a vital area of research,
aiming to equip models with the ability not only to make predictions, but
also to accurately express how confident they are in those predictions.
This is especially important in domains where the cost of errors is high,
such as healthcare, autonomous systems, and scientific discovery.
https://uncertainty.compute.dtu.dk/
That's not quite true...
Up until the rediscovery of neural networks in ~2012, the leading
machine learning methods were probabilistic and naturally had a
notion of uncertainty...
The big question: why are we here?
Uncertainty is an inherent part of real-world data and decision-making,
yet traditional machine learning models often overlook it. In recent years,
the field of uncertainty quantification has emerged as a vital area of research,
aiming to equip models with the ability not only to make predictions, but
also to accurately express how confident they are in those predictions.
This is especially important in domains where the cost of errors is high,
such as healthcare, autonomous systems, and scientific discovery.
https://uncertainty.compute.dtu.dk/
That's not quite true...
Up until the rediscovery of neural networks in ~2012, the leading
machine learning methods were probabilistic and naturally had a
notion of uncertainty...
but when we moved to neural networks, these approaches stopped
working and people forgot about them...
My plan for the day
1) We'll review the basics of probability and Bayesian inference.
2) We'll apply this stuff the neural networks (spoiler: this will fail!)
3) We'll talk about why this failed and what current efforts are to fix stuff.
My plan for the day
1) We'll review the basics of probability and Bayesian inference.
2) We'll apply this stuff the neural networks (spoiler: this will fail!)
3) We'll talk about why this failed and what current efforts are to fix stuff.
Interrupt me at any time!
That makes life more fun for everyone
in the room... :-)
What's all this uncertainty fuzz about?
Part zero
In which we figure out what we want...
Part one
In which we do point estimates...
Part two
In which we become Bayesian...
Part three
In which we see the underlying problem...
0
1
Totally unsure about
the answer being correct
Totally sure the
answer is correct
somewhere in between
this line contains all
possible answers
(equivalently)
(equivalently)
Photo by Konrad Jacobs - http://owpdb.mfo.de/detail?photoID=7493, CC BY-SA 2.0 de, $3
How's the weather tomorrow?
No accurate physics model;
Incomplete data;
Noise data...
What's all this uncertainty fuzz about?
How's the weather tomorrow?
No accurate physics model;
Incomplete data;
Noise data...
We will have to settle for an
imprecise answer to our query!
Quantified imprecision
Since we target an imprecise answer, we should quantify this imprecision.
Let's do this with a number between 0 and 1 (totally arbitrary choice!):
Speaking about the weather...
One possible (totally lame!) answer is:
Since we are totally sure about that answer, we have
Another answer could be "it's not freezing"
or perhaps "it is freezing"
Let's agree on a rule
(assuming the answers are mutually exclusive)
Kolmogorov's Probability Axioms
We made some simple requirements on how to quantify the plausibility
of our answers, and we arrived a probability theory (
good news!
).
Photo by Konrad Jacobs - http://owpdb.mfo.de/detail?photoID=7493, CC BY-SA 2.0 de, $3
Kolmogorov's Probability Axioms
We made some simple requirements on how to quantify the plausibility
of our answers, and we arrived a probability theory (
good news!
).
But did we pick the right rules?
If we want rules that:
agree with traditional logic when we are sure of everything;
are consistent (i.e. if you can combine your knowledge
in different ways, then you arrive at the same conclusion),
then probability theory is the answer.
In other words: if you are working with non-probabilistic
uncertainty, you need a damn good reason!
Okay, we'll do probability stuff... What else?
We need something that works! Sounds mundane, but...
Okay, we'll do probability stuff... What else?
We need something that works! Sounds mundane, but...
Conceptually, we are considering systems that produce predictions and uncertainties:
AI thingy
Input (data)
prediction
uncertainty
Okay, we'll do probability stuff... What else?
We need something that works! Sounds mundane, but...
Conceptually, we are considering systems that produce predictions and uncertainties:
AI thingy
Input (data)
prediction
uncertainty
This must be reliable even
when the prediction fails.
Otherwise, the uncertainty
will be misleading and directly
harmful!
How to build a probabilistic model
Let's first establish basic notation:
Input data (e.g. an image)
Output data (e.g. a class label)
Parameters (weights) of a neural network
The function (neural network) induced by the weights
The neural network applied to an input
Informally, we want
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
For example, regression tasks are often modeled with Gaussian likelihoods
shorthand
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
For example, regression tasks are often modeled with Gaussian likelihoods
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
For example, regression tasks are often modeled with Gaussian likelihoods
shorthand
Usual squared
error
loss
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
For example, regression tasks are often modeled with Gaussian likelihoods
shorthand
Usual squared
error
loss
This starts to matter
when talking about
uncertainties
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
For example, regression tasks are often modeled with Gaussian likelihoods
shorthand
Usual squared
error
loss
This starts to matter
when talking about
uncertainties
How to build a probabilistic model
The probabilistic ingredients are
The data
likelihood
measures how well a parameter fits the data
The
prior
explicates our preference among parameters
Likewise, we often rely on Gaussian priors (we'll see that this is basically weight decay)
Point estimates: maximum likelihood
The most basic learning scheme is to find that parameters that maximize
the likelihood of your data, e.g., with a Gaussian likelihood
How is that for UQ?
How is that for UQ?
Neat that we can do this, but
just captures the observation noise
of the data. It doesn't reflect the
accuracy of our prediction...
How is that for UQ?
Neat that we can do this, but
just captures the observation noise
of the data. It doesn't reflect the
accuracy of our prediction...
Perhaps just replace with a
neural network to give it more power?
How is that for UQ?
Neat that we can do this, but
just captures the observation noise
of the data. It doesn't reflect the
accuracy of our prediction...
Perhaps just replace with a
neural network to give it more power?
unchanged, except now we have
an extrapolation problem...
Point estimates: maximum a posteriori
So far, we've disregarded the idea of having a prior...
Often, we prefer having small weights as the resulting networks ttends
to "wiggle" less and therefore do less crazy extrapolation:
This may be tempting,
but don't fall for it...
Point estimates: maximum a posteriori
So far, we've disregarded the idea of having a prior...
Often, we prefer having small weights as the resulting networks ttends
to "wiggle" less and therefore do less crazy extrapolation:
We can leverage this prior through Bayes' rule
Point estimates: maximum a posteriori
So far, we've disregarded the idea of having a prior...
Often, we prefer having small weights as the resulting networks ttends
to "wiggle" less and therefore do less crazy extrapolation:
We can leverage this prior through Bayes' rule
Point estimates: maximum a posteriori
So far, we've disregarded the idea of having a prior...
Often, we prefer having small weights as the resulting networks ttends
to "wiggle" less and therefore do less crazy extrapolation:
We can leverage this prior through Bayes' rule
Maximizing this posterior gives
Point estimates: maximum a posteriori
So far, we've disregarded the idea of having a prior...
Often, we prefer having small weights as the resulting networks ttends
to "wiggle" less and therefore do less crazy extrapolation:
We can leverage this prior through Bayes' rule
Maximizing this posterior gives
We can leverage this prior through Bayes' rule
Maximizing this posterior gives
So: the Gaussian prior just gave us a standard weight decay-style regularizer.
This is often how people think of MAP estimation: we have regularized our loss.
This also implies that nothing changes in terms of uncertainty.
The beginning will be basic, but hang in there if you already know it -- there's a point!
My ambition is not to sell you on a solution, but to communicate some of the
difficulties of the problem. Hopefully, making it easier for you to be skeptical...
What have we seen so far?
Point estimates in probabilistic models allow us to arrive at standard
loss functions, but with the extra benefit that we can easily reason about
observational noise.
What have we seen so far?
Point estimates in probabilistic models allow us to arrive at standard
loss functions, but with the extra benefit that we can easily reason about
observational noise.
Valuable, but doesn't tell us when
we shouldn't trust predictions...
What have we seen so far?
Point estimates in probabilistic models allow us to arrive at standard
loss functions, but with the extra benefit that we can easily reason about
observational noise.
Valuable, but doesn't tell us when
we shouldn't trust predictions...
Fundamentally, we care about the
uncertainty of the
model
, instead of the
data
What have we seen so far?
Point estimates in probabilistic models allow us to arrive at standard
loss functions, but with the extra benefit that we can easily reason about
observational noise.
Valuable, but doesn't tell us when
we shouldn't trust predictions...
Fundamentally, we care about the
uncertainty of the
model
, instead of the
data
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
What if we just drop the idea of having a "best set of parameters"?
That's what being Bayesian is all about...
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
What if we just drop the idea of having a "best set of parameters"?
That's what being Bayesian is all about...
Assume we have training data
Then the prediction associated with a new input, given the training data, is
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
What if we just drop the idea of having a "best set of parameters"?
That's what being Bayesian is all about...
Assume we have training data
Then the prediction associated with a new input, given the training data, is
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
What if we just drop the idea of having a "best set of parameters"?
That's what being Bayesian is all about...
Assume we have training data
Then the prediction associated with a new input, given the training data, is
The Bayesian prediction
With point estimates we
1. phrased a probabilistic model;
2. maximized to find the "best" parameters;
3. made predictions using the "best" parameters.
What if we just drop the idea of having a "best set of parameters"?
That's what being Bayesian is all about...
Assume we have training data
Then the prediction associated with a new input, given the training data, is
In other words: instead of making predictions with the "best" model,
we sample possible models from the posterior and consider the average prediction.
We "just" need the posterior...
This approach requires us to compute the posterior distribution
That's really hard, and that's why most people don't bother with Bayesian stuff!
The rules of the game
Just as a reminder, the basic rules working with probability
(what Neil Lawrence calls the
calculus of uncertainty
) are
Sum rule:
Product rule:
Bayes' rule:
Independence:
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
Recall our previous MAP estimator
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
Recall our previous MAP estimator
Let's assume that we have access to this and that
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
Recall our previous MAP estimator
Let's assume that we have access to this and that
If we do a second-order Taylor expansion of the loss at the mode, we get
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
Recall our previous MAP estimator
Let's assume that we have access to this and that
If we do a second-order Taylor expansion of the loss at the mode, we get
The simplest approximation
Because the posterior is generally intractable, Bayesian stuff is all about approximations.
Laplace's method
is, arguably, one of the simplest posterior approximations.
It's really rather crude, but it is enough for us to proceed.
Recall our previous MAP estimator
Let's assume that we have access to this and that
If we do a second-order Taylor expansion of the loss at the mode, we get
Our loss is the log-posterior (disregarding constants), so let's take the exponential
This is what we call
the Laplace approximation
:
Figures from MacKay's
book, which is still the
best source!
The recipe is simple:
1. optimize your usual loss;
2. the Hessian of loss is your approximate inverse covariance.
The recipe is simple:
1. optimize your usual loss;
2. the Hessian of loss is your approximate inverse covariance.
But if I use a bigger network...
Historically, Laplace approximations worked okayish, ...
...but when applied to non-tiny neural nets, it truly sucks...
Large models
Small models
Historically, Laplace approximations worked okayish, ...
...but when applied to non-tiny neural nets, it truly sucks...
Large models
Small models
Unfortunately, this pattern repeats itself with
other Bayesian approximations, so it isn't
just Laplace being simplistic...
What could be the source of the error?
Theory 1:
We assume that
That's probably not true... Does that mess up things?
What could be the source of the error?
Theory 1:
We assume that
That's probably not true... Does that mess up things?
You can still perform the Taylor expansion when the gradient is not zero,
which just leads to a slightly different Laplace approximation.
Empirically, there is no useful difference to assuming a zero-gradient
(for reasonably well-optimized models).
What could be the source of the error?
Theory 2:
We use the Hessian as an inverse covariance. Covariance are supposed
to be positive definite, but the Hessian doesn't need to be.
Does that mess up things?
What could be the source of the error?
Theory 2:
We use the Hessian as an inverse covariance. Covariance are supposed
to be positive definite, but the Hessian doesn't need to be.
Does that mess up things?
Here, the answer is Yes. This does mess up things.
What could be the source of the error?
Theory 2:
We use the Hessian as an inverse covariance. Covariance are supposed
to be positive definite, but the Hessian doesn't need to be.
Does that mess up things?
Here, the answer is Yes. This does mess up things.
What could be the source of the error?
Theory 2:
We use the Hessian as an inverse covariance. Covariance are supposed
to be positive definite, but the Hessian doesn't need to be.
Does that mess up things?
Here, the answer is Yes. This does mess up things.
positive semi-definite
often small;
e.g. for a Gaussian likelihood:
Here, the answer is Yes. This does mess up things.
positive semi-definite
often small;
e.g. for a Gaussian likelihood:
This gives the
generalized Gauss-Newton (GGN)
Hessian approximation:
This approximation is important step, but the resulting approximate posterior still gives
us the same old crappy samples:
So, here's what people do (brace yourself...)
Since stuff seems to work with smaller models, ...
So, here's what people do (brace yourself...)
Since stuff seems to work with smaller models, common tricks involve
only looking at subnetworks (i.e. a subset of the weights).
So, here's what people do (brace yourself...)
Since stuff seems to work with smaller models, common tricks involve
only looking at subnetworks (i.e. a subset of the weights). Most commonly,
just the last layer.
So, here's what people do (brace yourself...)
Since stuff seems to work with smaller models, common tricks involve
only looking at subnetworks (i.e. a subset of the weights). Most commonly,
just the last layer.
This sometimes works.
Results are hit-or-miss.
Part one
In which we do point estimates...
Part two
In which we become Bayesian...
Part three
In which we see the underlying problem...
Point estimates: maximum likelihood
The most basic learning scheme is to find that parameters that maximize
the likelihood of your data, e.g., with a Gaussian likelihood
Søren Hauberg
August 2026
Bayesian deep learning, at DTU PhD school
1