In the second book, when Harry tells the Weasleys about Dobby, Fred says that "whoever owns him will be an old wizarding family, and they'll be rich", and George adds that house-elves "come with big old manors and castles and places like that".
Other mentions of owners (or masters) of house-elves in the books include Barty Crouch who is presumably well-off, Hepzibah Smith and the Black family, who are known to be very rich, and the house-elves of Hogwarts (which surely has some important numbers in their account at Gringotts).
Also, no mention of house-elves is done in the context of poor families, or even what could be middle-class families.
So, how come house-elves always seem to be associated with riches, when their work is not paid? Were they paid servants in some distant past?
To partially answer or offer perspective, I don't really follow the theory that the elves actually "come with the castle/house/manor", because it is explained more than once that the elves are bound to serve a family, and I take that to mean they keep serving them even if the family moves to a different house.