I believe his major motive behind this was he did not want to endanger anybody.
He was very wary when
Natasha tries to get intimate with him.
He isn't very comfortable with this, & is trying to protect her from the "other guy". He also says to the effect that he trusts her, it's the "other guy" who he doesn't.
We have seen this in the past as well.
- In Hulk (2003 film) he leaves civilized settlements & goes to live in a forest alone.
- In Incredible hulk (2008 film) yet again we see him living in a remote place.
- In Avengers we see him living in a remote village in Calcutta.
Please note that in both cases he leaves everything so as to put his love away from the "other guy"s path.
I believe that is something he does at a personal level. He doesn't want to endanger people around him. We hear this from time to time.
Remember, in Avengers he says
"The last time I was in New York I kinda broke Harlem"...
This clearly shows some level of remorse.
Natasha Romanoff: You want to think about removing yourself from this environment, Doctor?
Bruce Banner: [chuckles] I was in Calcutta, I was pretty well removed.