I personally do endorse staying on meds during pregnancy - the risk to the fetus is extremely small, but relapses of depression/mania are guaranteed if you stop, and are worse for the fetus than the meds. It has been studied that depression can actually kill a fetus.
Your doc is right; the postpartum depression is very severe and happens most every time for BPs. This is a very serious state for even non BPs, and prohibits bonding with the baby, and that effects the bonding for life.
Ask your doc about trying Lamictal. It is also a mood stabilizer although it's not effective at controlling mania by itself; it is used as an antidepressant, and only has the benefit of AD for bipolars, not clinical depression. It keeps us from going to low, just like lithium keeps us from going too high. There have been no deliberate studies on the effect of Lamictal on pregnancies, other than a small one on 100 women, all of whom delivered perfectly healthy babies. SSRI medications are thought to be safe, but are very dangerous for BPs, because they tend to bring mania and even psychosis to us. Mood stabilizers with it don't seem to stop it. Lithium has been thought to have a little influence on controlling depression, too, so it might be a combo worth asking your doc about.
Bottom line, it's your choice in the end.
Variety is the spice of life, BP is the key