i used to get like that when i was really depressed. The reason you think things are worse when you are feeling better is because you tend to go out more when you're feeling better and experience more interactions with people and situations, than when you are depressed and hiding away, or trying not to be noticed.
I used to think poeple were staring at me, or ignoring me, but half the time they hadn't even noticed me, or they had noticed me but were so screwed up with something in their own lives they didn't acknowledge me. It's not necessarily because people don't care, believe it or not the majority of them have their own anxieties, worries, and fears in their lives, and they don't differ all that much from our own. They are working hard to fit in too, and not to embarrass themselves too.
I find these days because although i cope a lot better now, i still say or do the wrong thing, or get nervous and end up babbling about something and getting wierd stares! I usually just say to someone, 'sorry i'm feeling a bit nervous', or 'i don't know what to say', or 'i'm feeling a bit down today' etc.... Most people usually accept honesty and usually respond well to it, as long as i don't end up going off in great graffic detail about my problems!
The way i did it was to just go out a bit at a time, a little every day. Just a walk around the shops even if i'm not buying anything, a walk in the park, a sandwich in a cafe. Anything to get me out, eventually it lessons the fear of going out and being around people.
Eventually when i was comfotable going out i'd slowly start talking to people, small steps again. At first just saying 'hello, hi, or good morning etc....' until i was comfortable doing that much, and then building up to having small conversations with people; 'this queue is long' etc.... It's all about small managable steps and building them up gradually. That old expression, not learning to run before you can walk etc.... Decide what you can manage and do a little bit every day.
Apologies for above, i'm training to be an occupational therapist, my whole life has become about breaking big tasks down into smaller managable ones!
I used to distance myself from people too, because at least there's no risk of getting hurt by anyone, but i like taking a bit of a risk now and getting to know people, because those relationships and friendships are essential in life. I used to put myself down a lot too, because that way if i was already at the bottom of the pile, no one could push me down further. I still hunch over when i'm walking around to make myself smaller and less noticeable, and eye contact is still a huge problem, but again small steps. I make an effort to hold my head up and look at people when they are talking to me. You just have to go by what kind of day you're having, some days feel better than others, and you can do more in than others. The key is to listen to yourself and see what you can manage, and look after yourself, so you're not doing too little and hiding away day after day, and you're not doing too much and not coping with it all.
I have full confidence in you that you can overcome your social anxiety because you go to work every day, which is a lot more than i could manage back then, so you're doing really well already. I think you need to treat yourself to whatever makes you happy once in a while; a video, a takeaway etc...., and learn to praise yourself as well as berate yourself; 'this i did well today, that was ok, that was not so good but i'll try again another day'. It's ok to not get things right, that's part of being human, we make mistakes. And it's ok to make slow progress as long as you are always moving forwards, it doesn't matter at what rate, whatever is managable to you,
Best Wishes, and take care,