Hmmm... that's generally the sign of a man who is weak, or sees himself as weak (which are pretty much the same thing, as far as one's actions.) How well a person holds up when things aren't so rosy is the true measure of their character. I'm not sure if there is a way to deal with someone like that long-term. Since he is no longer in his formative years, the odds that he will grow a backbone are pretty much between slim and none. If I were you, I'd be looking to protect myself and my interests. If I'm sounding a bit harsh, it's because I was once in a similar situation.

If he is violent and hurts you, you really do need to seriously consider leaving him. This sort of behavior only ever escalates.

If he hurts you, he doesn't love you enough for you to stay.