Surely the point is not to "know the numbers", but to understand the mechanics and the relative positions of each peice and their relative efficiencies in system architecture.
when I'm doing optimisation, my goal is not to explicit say "oh, L1 cache is X nanoseconds", it's to say "the profile pattern of the performance we see indicates that X is the current performance bottleneck, and if we can change programming/design, we'll gain Y". Especially since, by the time we're optimising heavily, it generally assumes something was amiss in your original coding or expectations of the system anyway, so your memorised numbers of what performance should be are often shown to be wrong in the first place, and your real world measurements will tell you the numbers that are relevant to your particular use case.
The point is not to memorize the numbers, but to have used them enough that they are at your fingertips at all times. Awareness of costs is essential to engineering economics.
And it is always better not to need to optimize, because the code is already fast enough. You may leap up to proclaim against premature optimization, but only if you have forgotten what Knuth actually wrote.
