MoE inference wouldn't be terrible. That being said, there's not a good MoE mode...

		bick_nyers 8 months ago \| parent \| context \| favorite \| on: Nvidia's latest AI PC boxes sound great – for data... MoE inference wouldn't be terrible. That being said, there's not a good MoE model in the 70-160B range as far as I'm aware.