Yep, sounds like you're having the exact same problems I had. Aren't those attacks in the middle of the night fun?
No, sludge is not innoccuous. My stepmother had her GB taken out because of sludge and her doctor said he thought it was worse on the body than gall stones, because you can sometimes have gall stones for a long time without them ever really getting in the way (but boy, you know it when they finally do!), but he said that the semi-solid sludge (pre-stones) dripped out of the gall bladder and was bad on the intestines.
The word as I know it is sludgy gall bladders have got to go.