Does your husband take anything for bipolar? I think that you should talk to him about the situation. He really should be on medication. This would take a load off of your mind. Also maybe you need to see a therapist to help you deal with your situation. Especially if you don't want to take medications. His situation could dedfinately affect you especially if he has extreme mood swings. Bipolar can be really hard to deal with.
You need to keep your relationship open with him. Maybe he doesn't realize how he is affecting you. But yes I would recommend therapy to help you to understand what he is going through and he needs some kind of help.
I wish you luck with this and in the meantime, keep posting here, we willl help you in any way that we can.