Your GI is right. Crohn's does tend to follow a pattern and it does tend to re-appear at the same spots. So if you've had Crohn's colitis for years without any small bowel involvement, it is probably unlikely you will develop small bowel disease in the future. However, that said, there aren't any guarantees. Some people are just unlucky. Hopefully you won't be one of them but there is always that possibility you are and the Crohn's returns.
It's a good idea to get your rectum removed eventually, if you have no intention of being reconnected. There's no rush, but a defunct rectum does more than just sit there and gather dust - it can develop diversion colitis or Crohn's and, in the long term, is a cancer risk. If you start developing urgency and passing mucus or blood, your rectum is almost certainly inflamed. You can treat it the same way you would normal proctitis: with mesalazine or steroid suppositories.
I had a colectomy five years ago, but like you my rectum was left. It was fine for a few months but eventually I started developing symptoms of diversion colitis, i.e. urgency and daily passing of mucus and occasional blood. I had a flex sig where biopsies were taken and I was told it wasn't Crohn's. Partly on that information I had a reversal, but it was bollocks: the Crohn's returned almost straight away. I would probably have elected to have stayed with the ileo if I knew what I knew now, but I'm not in dire straits so I'm carrying on as I am for the time being. If I do go back to having an ileostomy again, though, my rectum is coming out as well. I've had enough of problems in that area to last me a lifetime.
As for small bowel disease, I've always had that so the fact it's come back in my small bowel as well as my rectum isn't a surprise really. At least it's in a relatively small area :-/
Dx Crohn's in June 2000. (Yay )
Tried: 5-ASAs, azathioprine, 6MP, Remicade, methotrexate, Humira, diets.
1st surgery 20/2/13 - subtotal colectomy with end ileostomy.
2nd surgery 10/7/15 - ileorectal anastomosis. Stoma reversed and ileum connected to the rectum.
Current status: Chronic flare. Do I have any other kind?
Current meds: 50mg 6MP; Entyvio (started 3/11/16)