It turns out a good fitting mask is essential for comfort. I had a problem with masks being hard to breathe in and fogging up my glasses. Then I found a good set of N95 masks that have a real solid fit. It's loose and flexible enough to be breathable, and has a solid seal across the nose so moisture doesn't get near my glasses. You can tell it's a good fit because it deflates/inflates likes a balloon when you breathe.

I did manage to find sports masks I wear when I got the gym that do make it much easier to breath, although they fog up my glasses still so I take those off when I'm there.