I am so sorry you are going through this anxiety but I know this story well as most families have relatives like this in their lives.
It is very hard to respond to you without sounding judgemental as I know I have some relatives that are complete idiots and care less about anyone other then themselves to the point of being not only irresponsible but narcissistic.
Here comes the tough part, this child is not yours nor is it going to help for you to start to tell your brother how to behave. If the child were home and you felt the child's welfare were in danger then there are reporting procedures to go through to seek help for the baby.
I know your Mother is hurt, I have been there and very recently but you must remember again that this is your Mom's feelings and you cannot make her feel better by starting a family feud or insisting others behave the way you feel they should.
For your own sake I would tread lightly here as the conquesquences may be that your brother tells you to get lost and you will not get to spend any time with this wonderful new niece.
Just FYIs from my own experience and they have been vast.
You have a right to feel anxious and upset over this situation but again remember you do not have the power to make the world alright.
Special hugs to you and congratulations on your new role Auntie.