I had my catheter removed this morning. It's definitely no big deal. As spinbsicuit said earlier, the worst part of the whole thing is the plumbing cleaning you do the day before surgery. To be perfectly blunt, it's only bad cause your get so sore due to the number of wipes you must do. Advise using as much lotion as possible early and often.
PS - I had no prior hospital time either and hated the idea. As many have stated here, the anticipation is worse than the event. That includes my biopsy, surgery and catheter removal.
Edit - You asked for details. They pour in clear liquids through the catheter tube prior to xrays - no pain. Barely feel the addition to the bladder. Take a few more xrays. Nurse seats you on side of table, removes liquid from bubble in cath in your bladder with syringe - no pain. Tells you to take a deep breath, then as you breathe out she pulls catheter out - no pain - slight tugging sensation - it's over quickly and is more of a relief than and discomfort. You have a small container to empty your bladder a few seconds after the catheter is removed. You gladly begin filling the container, then sit it on the counter because you were done, then begin dribbling down your leg and grab the sheet she covered you up with during the photo session. Laugh at the situation, dribble some more then begin the next phase of your recovery!