For comments, yes, for stories, only flag but not downvote.

But both flag and downvote are karma-gated (I think the threshold is 500 for both, but I could be wrong.)