It's not uncommon to have seizures and then go several years without one.
The seizure activity could be burried deep in the brain and can't be detected on most tests so in many cases it's hard to diagnose.
You could go several years with no seizure, it sort of goes into "remission" and then at a later date, sometimes years, something triggers it again.
Your Neurologist may choose to keep you on meds the whole time as a bit of an insurance policy but if you insist on going off all meds, you should go through your Doctor and you will have to give up your drivers licence for about two years and go seizure free during that time.
I speak from 28 years experience with epilepsy. My case was almost the same as yours. I had my first seizure in 1979 and then nothing 'til 2002, 2004, 2006.
I was able to keep my drivers licence the whole time cuz I was on medication and my seizures were controlled ~'til 2002 cuz I had a seizure in a parking lot and hit another car.
Diagnosed with Epilepsy & Ulcerative Colitis.
Current Medications: Diantin 275mg, Pheonobarb 150mg, Asocol 3,000 mg.