Pre-requisites

Pre-requisites#

Reinforcement learning and control theory are interdisciplinary fields which build on other subjects from mathematics and computer science.

The formal pre-requisites are listed in the course base. However, I personally feel that a list of courses as pre-requisites can be difficult to put to practical use.

For instance, DTUs statistics course emphasize that you can apply appropriate statistical tests to a dataset – however, this course will apply exactly 0 statistical tests to 0 datasets, but instead we will use more basic concepts such as probabilities (which was part of the statistics course).

For that reason, I think it is helpful to list some examples of concrete skills from the courses that I hope that most who take this course will feel are familiar. This table provide an overview of how I would evaluate different levels of knowledge:

I know what what it means and I can do it

I hope many (most) of the concrete tasks will go here, especially those that seem highschool-adjacent such as \(\frac{d}{dx} x\sin(x)\)

I am supposed to know and do that but..

I think many will have something in this category and that’s fine – it means that you know where to find the information.

I can only do that with a computer

This is a more dangerous category. If for instance you can only solve a linear inequality such as \(3x - 5 \geq 8x + 2\) with Maple (but not by hand!) it is likely you are missing background knowledge. In this course we often don’t care about concrete answers, but instead that you know what the symbols or concept mean and can be applied – so therefore I would recommend that you use your previous courses, Khan Academy or AI to move skills in this category into one of the categories above.

I have not seen that before

Don’t panic. Try to gauge what you don’t know, for instance by talking with me or a TA, and find appropriate resources to fill the gab. The course is about control theory and reinforcement learning and not about math, so a few hours will get you a long way!.

Programming#

Many of the course exercises and about 1/3rd of the exam involves programming in python. To implement environments, and many of the algorithms, you will also need to work with concepts from object oriented programming 02002. You can look at the first three chapters in [Her25] to get an overview of the python functionalities we use.

Linear algebra#

You should be familiar with vectors, matrices and other foundational concepts from linear algebra. for instance:

  • Solve a linear system: \(A \mathbf{x} = \mathbf{b}\),

  • Compute determinants \(\textrm{det}(\begin{bmatrix} 2 & 1 \\ -1 2 \end{bmatrix} )\)

  • Calculate with inverses and products of matrices: \(A^{-1}\) or \(ABA^{-1}\)

  • Compute Eigenvalues and Eigenvectors \(\lambda \mathbf{x} = A \mathbf{x}\).

Analysis#

We will use various tools from analysis during the control section. For instance:

  • Function: For instance, if \(f(x) = 3x \cos(x)\) then compute \(f(0)\)

  • Equations and inequalities: Solve these for \(x\) (i) \(3x - 5 \geq 8x + 2\) or (ii) \(2x^2 - 3 = 5x + 1\).

  • Compute derivatives \(\frac{d}{dx} x\sin(x)\)

  • Compute simple integrals such as \(\int_0^5 x^3 dx\) or \(\int \frac{1}{x}dx\).

Differential equations will play an important role in control theory. However, unlike most math courses, our focus will not be to solve the differential equations (which is the hard part!) but to use them as a tool to formulate the control problems – however, it is still a really good think if you feel you understand what an equation such as:

  • What is an ODE, for instance \(y'' - \cos(x) y = 3\)

  • Does \(y = f(x) = \cos(x)\) solve \(y'' + y' = -y\)?

  • What a system of linear differential equations is (see mat1)

Probability theory#

Note

Although [SB18] formulate many/most results using stochastic variables, I am aware this is a difficult subject for many and we will not rely on a particular in-depth understanding of this concept beyond what you know from e.g. 02450 or statistics.

You should have a good operational grasp of basic concepts from probability theory such as probabilities, densities, and mean values. For instance:

  • What is the normal distribution \(p(x) = \frac{1}{ \sqrt{2\pi\sigma^2} }e^{ -\frac{(x-\mu)^2}{2\sigma^2} }\)

  • The rules of probability. For instance \(p(x, y) = p(x) p(y | x)\)

  • Expectations over continuous densities: \(\mathbb{E}[f(x)] = \int f(x) p(x) dx\) if \(x\) is a continuous variable with density \(p(x)\)

  • Expectations over discrete densities: \(\mathbb{E}[f(x)] = \sum_{i=1}^n f(x_i) p(x_i) dx\) if \(x_i\) is discrete

  • Rules involving expectations such as \(\mathbb{E}[ax + b] = a \mathbb{E}[x] + b\)

If this is causing you problems you can find a statistics-free brush-up in Chapter 5 and 6 of [HMS24].

Contents