Lactose is highest in milk and ice cream; processing of milk such as cooking (such as in pancakes) or culturing (such as for yogurt/cheese) reduces lactose. Or you can try Lactaid brand milk - it is lactose-free. Many people with IBD are lactose intolerant, especially when flaring. Also, you may not be aware of what other symptoms of yours are from dairy, because they may be delayed onset (i.e., occur hours to days after eating/drinking dairy products) or because you are only sensitive to whey proteins (which are destroyed with cooking). If I were you, I'd avoid all dairy for 2-3 weeks, then try various dairy products one at a time to see which one(s) cause a change in symptoms for you. Some with IBD and/or IBS actually find their disease becomes more manageable once they eliminate foods they are sensitive to.
As for which comes first, the dairy intolerance or the bowel trouble, it depends on the individual... Usually, though, the bowel trouble starts out asymptomatically (i.e., without obvious symptoms), followed by development of the sensitivity, which then can make the bowel trouble worse, which can lead to a worstening of the dairy sensitivity and/or development of other food intolerances/sensitivities, etc. If your 4+ to dairy was IgG antibodies, avoiding all dairy for 6-18 months may enable you to eat dairy again. However, some find they must limit their consumption once they re-introduce the food to once every 3-5 days or they tend to re-develop the sensitivity. Rice, soy, nut and/or oat milk may be adequate for milk substitution, and are usually widely available in most grocery stores today (and the taste has improved over the years too). These milk alternatives are also suitable for cooking, and some companies also make other dairy products based on these dairy alternatives....but be careful, most "dairy-free" cheese still contains a milk protein called Caseine (ingredient list on the label says caseine or caseinate). Some with dairy sensitivity can still tolerate butter, but you may need to avoid that too - Nucoa and some other brands of margarine are truely dairy-free, while most other margarines do still contain some dairy ingredients (watch those labels!).
For more info on dairy allergy, check out http://www.nomilk.com/
Good luck & take care,
Chronic Lyme Disease, Gluten & Sulfite Sensitivity, Many Food/Inhalant/Medication Allergies & Intolerances, Asthma, Gut issues (dysmotility, non-specific inflammation), UCTD ("Secondary Lupus-Like Syndrome"), Osteoporosis, Anemia, Lymphopenia, intermittant Pancytopenia, chronic malabsorption/malnutrition, etc.; G Tube; Currently TPN-dependent.
Meds: Zofran, Pulmicort, Heparin (to flush PICC line), IV Cipro (for G-tube site infection).