Stop the spinning thoughts hun and just tell your neighbor that you like that you are not comfortable with her friend and explain that you would be glad to have her come to visit but to please not let the obnxious friend know where you have gone to.
If the person you don't care for shows up you don't invite them in. Just be assertive and no guilt as that is a wasted emotion.
I hope this helps you feel better. We must stop worrying about what people that are not nice think about us. It really does not matter to us. They will go their way and we will follow the road we choose.
Blessings to you