Small amounts shouldn't be a problem however, you can see if eliminating them makes a difference. If your symptoms remain the same after leaving the sugar out of your tea and cereal for a few days, then you can assume that small amounts are okay for you.
The Mayo Clinic has some IBS Diet tips here: http://www.mayoclinic.com/health/irritable-bowel-syndrome/DS00106/DSECTION=lifestyle-and-home-remedies
Moderator Crohn's disease, Osteoarthritis and Irritable Bowel Forums
CD, Ankylosing Spondylitis, small fiber peripheral neuropathy, avascular necrosis, peripheral artery disease, pulmonary hypertension, degenerative disc disease, asthma, severe allergy and a host of other medical problems.