You might try taking a multi-vitamin, like Centrum. Certain medicines for IBS, altered diets, and even IBS itself can cause you to not get enough vitamins. I am on a medicine that interferes with my absorption of fat-soluable vitamins, which includes Vitamin A, which is what boosts your immune system. Before I started taking a multi-vitamin pretty regularly, I felt run down and without energy all the time; I also caught every little cold and bug that went around. Within a couple of days of starting a multi-vitamin, I felt normal again and don't get sick nearly as often.
IBS can also run in cycles (a flare up and then remission or at least lessened symptoms), so it may be that your symptoms will ease in the future and you will be able to resume your exercise routine. At any rate, I wouldn't knock walking or cycling at an easy pace. Some exercise is better than none at all, and it doesn't do you any good if you try too hard and then make yourself sick.
Personally, I subscribe to "do myself no harm." Now, it may be great for most people's health to drink a glass of red wine everyday, but it gives me diarrhea. That is worse on my health than whatever heart or cholesterol benefits I can get from the wine. So no wine. It may be that vegetables are great for the average person, but for you they are harmful becaues they upset your guts. So do yourself no harm and skip the veggies. You need to realize that you are no longer normal, and that normal rules of diet and exercise, what is good and what is bad, no longer applies to you. Or, at least, doesn't necessarily apply to you. Apples are good, right? They're terrible for me and for pretty much everyone else that has IBS. Processed foods like white bread and white rice are evil, right? They're something that's almost universally tolerated by people with IBS. You just have to figure out what's good and bad for YOU and ignore what's good and bad for the rest of mankind.