Many doctors blow off having a low grade fever of unknown origin. However, if you have this on a chronic basis it means something is going on somewhere in your body. I have heard that infectious disease doctors can be somewhat arrogant, so be prepared. I hope you get a good one. Please forgive me if I sound like a broken record - but it would be a good idea to have a careful dental evaluation if you don't find another cause. My dentist let me walk around with an absess for YEARS. The University of Maryland School of Dentistry discovered the abcess, and then my dentist ADMITTED that he didn't tell me. After I had the wisdom tooth pulled this past December, the fever went away, and I feel much better all around.
If you don't get satisfaction with this doctor let me know. There is a woman in the Silver Spring area who has a reputation for being formidable in helping patients track down and deal with tough problems. I think she could also take over as your primary. She will send you to Johns Hopkins if she can't find the problem. I can send you her contact information. You know where to find me.
Hang tough. Be strong. Don't let anyone intimidate you. Sometimes it takes a while to get to the bottom of "hidden" health issues.