In the scarlet letter by nathaniel hawthorn...?

Hester is an adulteress, and is forced to live in practical exile with a giant scarlet "A" on her chest as a reminder. The father of her child, the town pastor, is revered and loved. What's the difference between the two? Hester is a woman. When her husband, whom she presumed was dead overseas, comes back (Chillingsworth) he even discriminates against her and attempts to make her already terrible life even worse because of her "sin"...but never questions who the other party involved was.