It feels to me like you are having anticipatory anxiety. You start worrying how anxious and upset your will feel when he is out.
Why the mind tends towards the negative rather than the positive is a mystery. Yet, there is no doubt that anticipatory anxiety is purely a mind game. You can win the game if you are aware of your thinking and able to challenge the irrational, unrealistic thinking and replace it with more realistic thinking. Realistic thinking is not necessarily positive thinking, it is more objective thinking sometimes called scientific thinking because it is based on evidence, not conjecture. So, next time you start to feel anxious, become a scientific thinker and examine the evidence. You may find yourself pleasantly surprised to find the source of your anxiety vanish like clouds dispersing after a storm.
Many years ago when I was home with 4 small kids and my hubby would be out on his motorcycle, I would anticipate all kinds of things which included him talking to young slim gals.
I have learned to trust him and not get into that mode anymore but I did some really dumb thing on some of those occasions so you are not alone......
You have us now to help you recognize the behavior for what it is and help you move forward.