As explained in a few earlier questions, the Wall does not extend all the way to the sea on the East, and the castle there is essentially undefended anyway. Thus, wildlings -- and Stannis' army -- were able to go around it undetected.
We know that the Others and the wights they raise can't cross through the Wall, and I think we can assume that it's built to stand up to any attacks they make trying to bring it down. But there does not appear to be any reason they can't just go around like anyone else.
Possibly related: in the TV Show, we actually see the attempted evacuation of Hardhome, and the Night King allows Jon Snow and his human army and civilians to flee back to their boats to safety, even after raising a massive army of wights. You could interpret this as merely a psychological ploy, but it may be a hint that the dead can't cross the water either.
Has there been any indication (books or show) that the Others/White Walkers and their army of wights are blocked by open water, or are somehow tied to the land of the Frozen North, or that there's any other obstacle keeping them from just sailing/swimming around the Wall?