That's really not how it works at all.

Frameworks are layered, and the acceleratored tensor libraries are useful independently of the neural network architectures built on top of them.

It's basically a numpy/scikit like relationship, except both come from the same project.