I can relate to what you are going through. I'm female; 55; had to have a stent placed in my LAD (Left Anterior Descending) artery almost 2 years ago. Had angina pains for about 10 days before seeing the doctor...failed my stress test and had an angiogram (cardiac cath) the next day. They found a 99% blockage and put in a stent. I was very, very lucky not to have had a heart attack; it would probably have killed me. I'm doing well now, at least physically, though dealing with some minor liver function test elevations, probably due to the meds I have to take to keep my cholesterol at the numbers they need to be for people with CAD (coronary artery disease).
If I were in your shoes, I would talk at length with the cardiologist to weigh your risks of a heart attack versus the risks of the cardiac cath. If you can't take statin drugs, it will be tough to get your numbers at the level they should be, and even if you do, that won't do anything much for the blockage you already have. The cardiac cath is not only a diagnostic tool; it is the way they fix the problem with a stent (if you are a candidate for a stent; you might need a bypass, but they can't tell that without the cath either). Or, at least this is my understanding. You really need to talk at length with your doctor about your individual risks and options. Don't worry about asking questions again of him/her soon, as you are likely in so much shock you didn't hear much they said the first time (at least that was my experience).
Hope you are at least taking an aspirin every day, as it sure sounds like you have CAD. If the blockage is in your LAD (they can't tell for sure from the stress test you had, but they can get a very good idea) you may be at very high risk of a heart attack and death. Some people refer to a blockage in the LAD as the widowmaker: as I understand it, that's about the worst artery to have a blockage in should you have a heart attack.
The fear you are experiencing in normal...hang in there; it will help you make the right decision.
I would add that the risks of the cardiac cath are real; they accidentally stopped my heart the first time I had one done, and had to revive me with the defrillator, BUT it was also what saved my life. I had a second a year later with no problems. They've been doing them at least since the early 70s...my Dad had several before he died of a 3rd heart attack at age 64 (before stents existed...he could have been a guinea pig for the early bypass experiments but decided again it). If you are in a good hospital with a good interventional cardiologist, you should have few problems.
Again, this is my own personal experience, and I can't give you advice: only you can decide what is best for you in consultation with a good cardiologist. You WILL have to drastically change your lifestyle whatever you decide to do re: treatment.