That would be your Ulnar nerve. (It runs along your Ulna, the bone on the outside of your forearm.) It's also the nerve that is called the "funny bone."
If it is just on one side, you may have a bone spur, or scar tissue, or some other issue that is affecting that nerve. If it is on both sides, you may have a larger problem going on in the body. It couldn't hurt to get it checked out. If you are diabetic, you could get numbness in your feet as well. If you are anorexic, or there is an imbalance in your electrolyte level, you will get peripheral nerve numbness too.
Your doctor should be able to give you a good answer. Better safe than sorry.