I think it was definitely because he let the Wildlings through.
There appears to be a few times when they could have done it:
These are not facts just my speculation.
In between announcing that we was going to save the Wildlings and actually leaving.
When he got back with the Wildlings.
When they actually killed him.
Why? What are the chances that the Wildlings are going to agree to come with the leader of their oldest enemy? I would think he would never make it back alive and he almost didn't. Remember the Nights Watch think that the Wildlings are exactly as their name suggests. Wildlings vs Free Folk. As others have said there are also witnesses aplenty.
If they try to kill him here and succeed you would think that however many Wildlings were there would do something about it. i.e. attack the wall, yeah they had women and children but I doubt that the Nights Watch would have been able to fend off that many. Also witnesses.
Since he managed to return with the Wildlings, that 'plan' was now ruined. Middle of the night the only witnesses were the ones committing the act. No backup in the form of Wildling friends.