Unfortunately, love does not always accompany sex. It sounds like there was a very real attraction from his end, but on a physical level and not emotional. It is human nature to want what we can't have. You need to start looking at what you do have - a loving husband, for example - and appreciating it and trying to make the most of it.
You need to cut ties with this other man. Do not see him, do not speak to him. Do not let him be the one to use you and then leave - leave him now, even if you were never really "together" to begin with. Do not write to him, and try to keep yourself busy so you do not think of him. If you choose not to share this secret with your husband, then at least try to work on your marriage. Rekindle some romance and passion. You believe your husband loves you - that is more than what many others have.
We all feel terrible when we look at these things in hindsight. We wonder why we didn't see it sooner, but emotions have a way of blinding us from the truth. What you experienced with this man was real, but he does not reciprocate. Do not give him more power to make you feel terrible about all of this. Just walk away from it. If you still feel unlovable, then make yourself feel lovable by working on your healthy relationships with your husband and other family and friends. I think a reason why you feel unlovable is because of this secret that perhaps makes you feel unworthy. Make things right by ending it. He has had many chances to be with you if he really wanted to be, so you are not missing out on anything if you end your relationship - both romantic and friendship - with him.
"Life's journey is like driving at night in the fog.
You can only see as far as your headlights,
but you can make the whole trip that way."
Chronic Lyme, Depression, Anxiety, 12+ Years