It depends on the type of anxiety and how you're using the medication. If you it routinely to handle everyday stress, you're heading down a very bad road. If you have events a few times a year that are extremely stressful, medication can really make a difference.
I had the same fears and I'm not gonna say you should try medication, but I will share it my psychiatrist did. She wouldn't let me have more fifteen low-dose pills at any one time.
Consider your relationship with alcohol. If you already have a problem with it or completely abstain from it, I would strongly recommend avoiding medication.
I'm the same, and similarly, the idea of taking a pill to make it go away is terrifying.
I have to keep myself very disciplined on the "helpful" medications I do allow into my life because I know how easy it'd be for me to get hooked and start depending on the meds to avoid my problems. Instead I've been working on training myself to see through my nerves.
Alcohol at least has the drawback that the few times I've had it, I've had such an awful experience that I don't associate it with avoiding my problems.
I mean, that's how a lot of prescriptions work anyway. I take daily medication to prevent anxiety and depression. It's not addictive, but I try pretty hard to never miss a day, because anxiety and depression will get in the way of me living my life. Assuming Xanax is safe to take daily (I have no idea), the only difference is that a supply disruption would have side-effects. But if I run out of my medication, I'll get depressed within weeks, almost guaranteed, withdrawals or no.
I've been an anxiety basket case my whole life and the idea that an addictive pill could take it all away terrifies me - I'd be hooked for life.