I guess you can abstract away a lot, but there will always be questions that require you to go back to the math. For example: a use would be to set up a transformation that projects points onto a plane and a library could cover that. But consider the question of having a bunch of points and (paired with them) projected points and given the task of computing the original projection transform. If the library doesn't contain that operation you'd have to do it yourself.