Poverty doesn't make for good fighters.
The people in the outlying districts live on the brink of starvation. They're deliberately oppressed, kept poor and ignorant and most importantly weak. Poverty implies malnutrition, and malnourished people don't tend to fight well. For the same reason, armies (especially in poor countries) tend to be more well-fed and in better health than the majority of the population. But what army is there in Panem? The Peacekeepers come from the Capitol or - like Career Tributes - from District 2. From the authorities' point of view, there's no need to ensure any of the people in the outlying districts are in good health or properly fed, especially as children when they're not so useful as labour power.
Sure, some of the tributes from District 7 might be able to use axes, or those from District 11 might be good at living rough, but that usually doesn't make up for the years of combat training that the battle-hardened Career Tributes from Districts 1, 2, and 4 have been through. Career Tributes, brought up in relative luxury, have been able to afford the time to be trained, while those from poorer districts haven't.
Add to this the fact that the tributes are only children. Sure, District 7 has big strong men who can handle axes, but a boy or girl of fifteen won't be as strong and certainly not as efficient in personal combat. This, though, is something that puts District 12 at more of a disadvantage than the others: as Katniss notes in Catching Fire, kids in District 7 are put to work on the trees at an earlier age than those in District 12 are allowed to work in the mines. Which brings us to ...
Maybe there are more volunteers in other districts.
Canonically we know that volunteering is almost unknown in District 12:
hasn't had a volunteer in decades and the protocol
has become rusty. The rule is that once a tribute's
name has been pulled from the ball, another eligible
boy, if a boy's name has been read, or girl, if a girl's
name has been read, can step forward to take his or
her place. In some districts, in which winning the
reaping is such a great honor, people are eager to risk
their lives, the volunteering is complicated. But in
District 12, where the wordtribute is pretty much
synonymous with the word corpse, volunteers are all
-- The Hunger Games, Chapter 2
But it sounds as though there could well be occasional volunteers from other districts, not just the Careers:
One by one, we see the other reapings, the names
called, the volunteers stepping forward or, more
-- The Hunger Games, Chapter 3
You're assuming that volunteering is almost unheard of in all districts outside of 1, 2, and 4, but I don't think the text of the book actually says that. It could be that it's most common in the districts of the Career Tributes, less common but still occasional in other districts, and rarest of all in District 12.