A lot of work at larger orgs is put into reproducible training runs. It's about the only way to debug 'did the small parameter tweak tank performance because of a hardware failure or is there something special about that parameter?'