I can relate hazelnut. I just started seeing a psychiatrist for my panic attacks/ anxiety. I've been working with a homeopath for several months already though and the psychiatrist was so quick to dismiss the value of any of the holistic approaches, remedies, etc. that I was using.
I see a homeopath (a type of holistic doctor). He has a traditional MD license as well. When I was in the hospital following a two-hour attack a few weeks ago I was given Xanax. This is the first time I've ever taken any traditional, western-based psychotropic med. While the psychiatrist couldn't answer questions about possible interactions between the homeopathic stuff I was taking and the Xanax, the homeopathic doctor could.
So my suggestion is that if you'd like to consider herbs, flowers, etc. that you look for a holistic doctor that you trust. They'll be more understanding of your interest and probably more familiar with more "traditional" meds as well.
For the record, I haven't heard about passion flower but I am taking another flower that was prescribed to help elicit emotions more freely so that the anxiety doesn't build up inside of me until I implode in the form of a panic attack. Part of my issue was holding things in and trying to handle everything on my own.