While I don't know all of your story I do know a thing or two about
PCa. Your PSA is small. And your doc told you that "some people" will die in three years with your numbers...
I disagree very much with your doc trying to "scare" you into treatment that way. It's inappropriate and its more likely false. It is true that radiation can be beneficial for you, but I would get a second opinion as well.
I had 38 treatments of IMRT radiation and did very well with some slight fatigue during treatment. You should do well with radiation, even able to keep your exercise regimen.
Please note that I do not mean to down play the use of radiation. But I don't like your radiation oncologists approach.
Advanced Prostate Cancer at age 44 (I am 49 now)
pT3b,N0,Mx (original PSA was 19.8) EPE, PM, SVI. Gleason 4+3=7
Da Vinci Surgery ~ 2/16/2007
Adjuvant Radiation Therapy ~ IMRT Completed 8/07
Adjuvant Hormone Therapy ~ 28 months on Casodex and Lupron.