Out of phase, from a physics perspective, means you have two waves that are not in sync:
In audio this causes a weird effect, because the sound coming out of the right speaker reaches you at a slightly different time than the sound coming out of the left speaker. Then when you consider the fact that sound is a wave, the pressure can interfere with each other causing all kinds of other phenomenon.
In Donald.McLean's answer, he quoted this:
Geordi: Well, whatever or whoever is there, we're out of phase with it. But we're only talking by a fraction of a second.
Warf: A fraction of a second would make them invisible?
Geordi: A millisecond, a year -- it wouldn't make any difference. If what we're reading is true, then we're occupying the same space but in a different time.
Applying this idea of being out of phase, it appears then that in Star Trek time is a waveform. And, in this particular case, the Enterprise and her crew were out of phase with these other folks. If you treat the Y-axis of the above graph as a coordinate in 3-d space, and the X-axis as time, if the blue line is the Enterprise and the red line is the other party you can see that aside from the two intersection points, they would always be occupying a different time when they are at the same space.
This actually reminds me of an interesting episode where the Enterprise and her crew were being sucked into this wormhole thing, and Deanna Troy was having problems.
As it turned out
They were actually trapped by 2-dimensional beings that were trying to get "home". Turned out that all they had to do was either dive or raise and they were able to escape. They just happened to be on the same 2-d plane as the beings.