This makes sense in a way given the API of typical ML libraries. But there is no fundamental reason this needs to be the case.
Or, more correctly stated: model == code for sure, but said code need not have any rights to perform side effects. For some reason e.g. TensorFlow has stuff like tf.io.write_file [1] (is that actually an operation you can put in a model???), but one could easily imagine a more appropriate domain-specific model language that your code is compiled to, that can by design not perform any IO. Imagine that a model you distribute is not random Python code that may or may not run a model, but instead the model itself, i.e. the graph encoded in that domain-specific language.
Then downloading a random model from some random untrusted place is no different from downloading some random data from some untrusted place: you're going to execute the model, which may DOS you, but nothing much else will happen.
Unfortunately the ML world is too stuck in the imperative mindset for this (IMO more sensible) way of doing things. :)
At that point you'd need a machine learning DSL and runtime. Currently, it's all python libraries, so you can do everything python can... Which is everything, essentially.
It's highly unlikely that the market for running these models like an appliance securely in an untrusted context will ever manifest. It's just too much of a niche, as it would also reduce their extensibility/usability significantly
Something like this may grow out of the GGML project, which is gaining traction. They already have a weights format which can be loaded with mmap, though AFAIK the model architecture still needs to be defined in C++.
This makes sense in a way given the API of typical ML libraries. But there is no fundamental reason this needs to be the case.
Or, more correctly stated: model == code for sure, but said code need not have any rights to perform side effects. For some reason e.g. TensorFlow has stuff like tf.io.write_file [1] (is that actually an operation you can put in a model???), but one could easily imagine a more appropriate domain-specific model language that your code is compiled to, that can by design not perform any IO. Imagine that a model you distribute is not random Python code that may or may not run a model, but instead the model itself, i.e. the graph encoded in that domain-specific language.
Then downloading a random model from some random untrusted place is no different from downloading some random data from some untrusted place: you're going to execute the model, which may DOS you, but nothing much else will happen.
Unfortunately the ML world is too stuck in the imperative mindset for this (IMO more sensible) way of doing things. :)
[1]: https://www.tensorflow.org/api_docs/python/tf/io/write_file