I've tried descending down the RL branch, always seem way out of my depth with those formulas and star-this, star-that.

Yeah, the formalisations can be hard to crunch through (especially because of [1]). But this book in particular is quite well laid out. I'd suggest getting a math background on the (very) basics of "contraction mappings", as this is something the book kind of assumes you have the knowledge of.

[1] There's a lot of confusing naming. For example, due to its historic ties with behavioural psychology, there are a bunch of things called "eligibility traces" and so on. Also, even more than the usual "obscurity through notation" seen in all of math and AI, early RL literature in particular has particularly bad notation. You'd see the same letter mean completely different things (sometimes even opposite!) in two different papers.

What is your background? Unfortunately I did not find it very accessible.

That book is a joy. Strong recommend.

You mean "Reinforcement Learning: An Introduction"? Or did they write another one?

Yeah that one. Messed up the name.