Your PSA is still quite low, and as your doctor has said, it hasn't changed in a year. Therefore if you really do have PCa it will be very slow growing.
If your PSA went from one to two in a year, then that would be cause for a biopsy. There is an obvious rise that would need to be investigated.
If your uro says you're OK, then chances are very good that you are OK (at least in the prostate department). You can't do much to prevent prostate cancer. However if you want to worry about
something, worry about
heart disease -- at least worrying about
that might lead you to some effective preventative action.
In your situation, I would get out and enjoy life while you still have it, and especially do all those things that those of us without prostates can't do