That is the core question isn't it? Is it worth it? I've been dealing with the same thing. Being jittery, anxious, and very forgetful are the worst for me, and they are causing a lot more anxiety and stress in my life which in turn makes my depressed or manic symptoms worse. I have been doubting if the side effects are worth the change, but I keep thinking about the big picture.
I am young and the disorder has already effected my life in so many ways. I only realize this looking back with an open mind. I have lost friends, wrecked cars (once almost killing myself and two good friends), dropped out of school, and learned to accept a life that I feel isn't worth living. And now I'm told that it only gets worse with age. Looking back on the last year, I mostly remember the good times (I've also learned to forget about bad times), and I think "man, I'm crazier now than I was then", but I know that I am a risk to myself and others, and that if I can smooth out some of these kinks I can do great things and live a truly happy life.
So for me, it's worth it. For you, I can't say. You might want to try different meds, or just stick with it. If you wouldn't mind, let us know what you are on and how much of it, and how long you've been on it. Have you tried other drugs? There are many out there! For now just keep following the docs advice, but remember it is your body and you need to make the final decision!