Why is it selfish to not want children?

I am married, but my husband is in a relationship with another woman. He recently decided that he wants more children (he already has kids by other women). He also knows that I have serious health issues that can be fatal if I try to carry another child. He doesn't work or try to help me with the bills or anything else for that matter. Even if I risked my life and successfully had a child, I would not be able to afford to take care of him. Everyone is calling me selfish for not giving him children (mostly women). What gives?