The short answer is that there's more to a large scale construction project, especially in a wintry clime, then you are giving credit for, and when you take them into account a more northerly location may be extremely foolish.
Although magic was built into the wall, it was not built exclusively (or easily) with just magic. Actual manual labor was involved.
For such a large construction project one must take logistics into account. While there may be a "narrower" span of land further North, the further north you go the more difficult it becomes to get labor and supplies (for both the construction and the labor) to the site. This would also complicate keeping the wall garrisoned after its construction. Furthermore, attempting to do construction during (permanent) winter conditions makes things radically more difficult than building in an area where there is a more hospitable, warmer period during which to build. Modern northern (US) cities do essentially zero construction during the winter months, waiting instead for warmer periods; I know Michigan natives like to quip "In Michigan we have two seasons: Winter and Construction".
It also pushes you further into contested territories, making encounters with wildlings or Others more likely, which makes the construction all the more difficult.
You also haven't accounted for the terrain. The "narrower" bits further north may be very unsuitable for the construction. It may be very far from flat, and the ground underneath may be of a material that is bad at supporting such a large structure (you wouldn't want it to collapse because of a sinking foundation, sinkhole, etc). The scenes we have seen from the TV show depicting the regions north of the wall all seem to be hilly, rocky, undeveloped terrain that would likely be problematic for construction. Trying to clear and level an otherwise smaller stretch of land and reinforcing and enlarging the foundation would take more time and resources than where it was ultimately built. We have no records of what sort of surveying Bran may have done, but it's plausible that he considered a more northerly location and ultimately decided against it because there was a far more suitable construction site further south.
Adding to the weather problems, the further north you go the more it becomes "permanent winter". Winds will naturally deposit snows at the foot of the wall, creating a snow bank far deeper than otherwise would exist; the foot on the side where you don't want enemies to be able to simply scale up a huge mound of snow to surmount the wall, at that. Such a build-up is what allows Theon and Sansa to escape Winterfell: they jump off a wall into a large build-up of powdery snow. Eventually, at such a massive height scale, the weight of the built-up snow may begin to compromise the wall itself. It may no longer be able to bear the weight of all the snow pressing against it; a large build-up would also eventually start compressing the ground beneath it, which may compromise the structural foundation of the wall. This compression also results in heating, which can cause a temperature differential between parts of the wall, which leads to stress. If the heat of the compression was eventually sufficient to melt the snow at the base, you then get additional problems: the water will start eroding the ground, seeping into any cracks in the structure or accumulating beneath the base and compromising the foundation, and if it refreezes later it will expand and put additional pressures on the wall and its foundation. All of these work against the wall being a (semi-)permanent barrier, as it may permit nature herself to destroy the wall in a matter of years.
We don't have a lot of details about the exact functions of the wall, especially the magic that was intertwined with it. Perhaps the magic not only acts as a barrier to the Others, but also was meant to be functional enough to counter some of these problems. After all, we see indications of it snowing several times at both the top and base of the wall, but there doesn't seem to be any significant build-up at the base. We see in this shot from the TV show that there is a few meters of snow built up at the gate, which has been cut through to allow transit through the gate:
It is mentioned that part of the duties of the Night's Watch is maintenance and repair of the wall, and to keep the forest from getting too close (this may simply be to prevent giving enemies cover, but also may be to prevent root systems from compromising the foundation). So a certain amount of damage from time and nature was already presupposed, meaning they must have tried to control how much damage could possibly accumulate so that it stayed manageable. But the forces of men and magic may not have been strong enough to counter what the wall would have to endure further north. One way or another, when we account for the complications we are likely to encounter by moving the wall further north, it becomes reasonable that at some point it becomes untenable to build there. And, seeing as the wall is where it is and lasted thousands of years, it seems it was determined that the best location for the wall is exactly where it is.
