But the TeX math is interesting to discuss: it turns out that you can skip the usual multi-second download/parse/render/reflow workflow of MathJax JS libraries on a static website by preprocessing the final HTML pages using https://github.com/mathjax/MathJax-node . This gets you pretty much the best of all worlds: it renders instantly without JS, looks good, works cross-browser, and is dead-simple to set up as you just pipe into a tool. Definitely the best way I've found for static sites to render math.
Exactly. This lets you test it out for you and actually can pick up on some highly nonlinear behavior. Actually applied this myself to housing price prediction with better results than single functions on a layer.
Makes sense. Out of all of the permutations of functions, I see no reason why using the same function at every node would lead to an optimized network.
I really commend the ML people out there that have developed an intuition for network architecture and which functions to use... If there's a Feynman for machine-learning, I wanna listen to some of his lectures.