We do have one person on the Forum who said that he was afraid to have a colonoscopy because his father had something wrong with him... lesions or something... and that caused them to perforate his colon during the scope and he died from it afterwards and the poster had this same problem and was afraid that the same thing might happen to him.
So yes, you can have a colon that's so messed up that it's too difficult and dangerous to perform a colonoscopy. The colonoscopy is really only necessary for finding colon cancer. If it shows that you have an IBD, great... now what? Not much more than can do about an IBD than IBS, so seeing which you have is almost pointless, except that knowing you have an IBD makes severe problems, like blood in the stools, not a cause for concern. Detecting colon cancer early gives you a high chance of survival (detecting it late gives you a very low chance), but if having a colonoscopy is very risky for you and especially if you don't have a family history of colon cancer, then better to not have it and to just guess at IBD versus IBS.