The point you are missing is that with one diet you would out of control crave far more calories. While on a different diet you would have to force yourself to eat beyond feeling satisfied to reach the same number of calories.
I'm not missing that, it was just not something that was discussed. I was talking about weight gain and weight loss, not how easy or difficult it is in practice.
What you eat will of course matter when it comes to nutrition, muscle gain, satiety and many other factors but that in itself does not generally remove that a calorie is a calorie (unless extreme diets).
You asked why it makes a significant difference. And it looks like you answered your own question. A calorie being a calorie is one of the less interesting factors when it comes to actual practice. And yet it gets repeated ad nauseum every time the discussion comes up.