That infinite things can be stored in finite space isn't weird at all. Programmers do it all the time (e.g. circular lists). The trick is that you cannot store all infinite things in finite space (circular lists can only store repeating sequences). There are only a finite number of infinite things you can think of.

For example, the number 0.33333 is infinite if you write it this way. If you use another representation (1/3) then it isn't. Similarly, you can use a computer program that generates the digits of pi as a representation of pi.