Your body is trying to tell you something. I wouldn't just brush it off. I would listen to it and try and determine what it is trying to tell you. When that happens to me, it is usually for a reason. It doesn't necessarily mean anything bad about your relationship with this guy, but worth listening to.