This does happen, albeit in the books more than the TV series.
In Season 6, Episode 7 it is stated that the visitors to the camp are looking for Money, Food or Steel as payment for protection.
Also remember that (most) of the characters that we follow are:
a) rich and
b) honorable
