There is, apparently, conflicting evidence in the books
In Deathly Hallows Chapter 1 Voldemort's borrowed wand gets
destroyed. He immediately requests a replacement, and by the time he
gets one Harry is safe:
Face level with the handlebars, Harry could see nothing but distant
lights growing nearer and nearer: He was going to crash and there was
nothing he could do about it. Behind him came another scream, “Your
wand, Selwyn, give me your wand.”
This seems to indicate that Voldemort was unable to attack Harry wandlessly.
Similarly, later in Deathly Hallows Harry, Ron, and Hermione
discuss whether Moody actually died:
“Even if the Killing Curse missed, Mad-Eye still fell about a
thousand feet,” said Hermione, now weighing Quidditch Teams of
Britain and Ireland in her hand.
“He could have used a Shield Charm — ”
“Fleur said his wand was blasted out of his hand,” said Harry.
Harry's response seems to indicate that Moody could not have cast a Shield Charm without a wand.
If wandless spellcasting was difficult but possible, who would be able to do it if not the most powerful wizards like Voldemort and Moody? Thus, it seems that wandless spellcasting is either impossible, or too difficult to have been used in those situations.
Additionally, throughout the series, losing one's wand seems to be
equated with powerlessness – even for powerful wizards. (I don't
think there is a single example in the series of a wizard using magic
to defend himself or attack an opponent after losing his wand.) The most prominent example of this is Dumbledore when he gets killed in Half-Blood Prince:
“Dumbledore cornered!” he said, and he turned to a stocky little
woman who looked as though she could be his sister and who was
grinning eagerly. “Dumbledore wandless, Dumbledore alone! Well done,
Draco, well done!”
Indeed, Dumbledore fails to perform any magic once he had lost his wand. If wandless spellcasting were at all possible, you would expect Dumbledore to do some in this grave situation. Also, the mere fact that the Death Eaters taunt him about this indicates that they did not think that there was a possibility that Dumbledore could still defend himself.
In fact, Dumbledore explicitly tells Malfoy that he cannot defend himself as he does not have his wand:
And after all, you don’t really need help. ... I have no wand at the
moment. ... I cannot defend myself.”
(Even if Dumbledore was lying to goad Malfoy, we still see that Malfoy did not object and say that Dumbledore could defend himself without a wand.)
If even Dumbledore was unable to do wandless spellcasting, who else would be able to do it?
It is further implied that wandless spellcasting is impossible from
McGonagall's comment in Deathly Hallows on how Snape could survive
jumping out a window:
“No, he’s not dead,” said McGonagall bitterly. “Unlike Dumbledore, he
was still carrying a wand ... and he seems to have learned a few
tricks from his master.”
Another implication of wizards being unable to perform wandless spellcasting
comes from a comment of Ron's in Deathly Hallows:
“The right to carry a wand,” said the goblin quietly, “has long been
contested between wizards and goblins.”
“Well, goblins can do magic without wands,” said Ron.
The goblins apparently complain that they cannot carry wands, and Ron's response is that goblins can do magic without wands. The implication, then, is that wizards cannot do magic without wands. (Of course, as this is Ron talking,
it's probably the exact opposite or something, unless we take into account that Harry and Hermione didn't correct him.)
An additional possible indication against wandless spellcasting comes from
Voldemort in Deathly Hallows:
“No volunteers?” said Voldemort. “Let’s see ... Lucius, I see no
reason for you to have a wand anymore.”
Lucius Malfoy looked up. His skin appeared yellowish and waxy in the
firelight, and his eyes were sunken and shadowed. When he spoke, his
voice was hoarse.
“My Lord?”
“Your wand, Lucius. I require your wand.”
In context it seems as though the taking away of the wand is akin to taking away the magic.
Furthermore, the existence of wandless spellcasting would create major complications, such as keeping prisoners in Azkaban. In fact Sirius specifically mentions in Prisoner of Azkaban that he would not have been able to get past the Dementors without a wand:
But I was weak, very weak, and I had no hope of driving them away
from me without a wand. ...
Similarly, Bart Crouch Jr. tells us that he was not allowed to have a wand while imprisoned in his father's house, seemingly as a measure to prevent him from doing magic:
I had not been allowed a wand since before Azkaban.
Moreover, when Voldemort relives his attack on the Potters in Deathly Hallows we see him completely dismiss the possibility that James would be able to defend himself without a wand:
Hold him off, without a wand in his hand! ... He laughed before
casting the curse. ...
Elsewhere (at his rebirthing party in Goblet of Fire) Voldemort tells us explicitly that spells require the use of a wand:
Nevertheless, I was as powerless as the weakest creature alive, and
without the means to help myself . . . for I had no body, and every
spell that might have helped me required the use of a wand. ...
Another possible piece of evidence is the description of Ron trying to escape Malfoy Manor in *Deathly Hallows:
Ron was now trying to Disapparate without a wand.
The possible implication here is that in general spells cannot be cast without a wand.
When Bellatrix is about to perform the Unbreakable Vow in the beginning of Half-Blood Prince, Snape reminds her that she needs her wand, indicating that it could not have been done wandlessly:
"You will need your wand, Bellatrix," said Snape coldly.
When the Dark Mark is conjured in Goblet of Fire, both Mr. Weasley and Mr. Bagman state that conjuring it requires a wand:
“Come off it, Amos,” said Mr. Weasley quietly, “you don’t seriously
think it was the elf? The Dark Mark’s a wizard’s sign. It requires a
wand.”
“No!” he said. “Winky? Conjure the Dark Mark? She wouldn’t know
how! She’d need a wand, for a start!”
While in context they say this to prove that it couldn't have been a house-elf, the fact that they specifically say that it requires a wand rather than saying that only a wizard can conjure it seems to indicate that a wizard would actually require a wand to conjure it.
In addition, in Order of the Phoenix there is actually an explicit case of a wizard saying incantations without a wand and nothing happens because he did not have a wand:
Snape let out a stream of mixed swearwords and hexes, but his wand
being ten feet away nothing happened.
However, when Harry was attacked by Dementors in the beginning of
Order of the Phoenix he was able to cast a spell even though he was not holding his wand:
Wand!” Harry muttered frantically, his hands flying over the ground
like spiders. “Where’s — wand — come on — Lumos”
He said the spell automatically, desperate for light to help him in
his search — and to his disbelieving relief, light flared inches from
his right hand — the wand tip had ignited. Harry snatched it up,
scrambled to his feet, and turned around.
This was not wandless spellcasting per se, because the light still came out of the wand. It seems that Harry was simply able to channel his magic through the nearby wand. Thus, this would only show that it is not necessary to be holding a wand in order to channel magic through it (and the wand presumably has to be very close by because in the above example Snape's wand was only 10 feet away from him and his hexes still didn't work.)
In a somewhat similar vein, Harry tried to summon his nearby wand when Malfoy used the Full Body Bind on him in the beginning of Half-Blood Prince:
He tried to make a sound, even a grunt, but it was impossible. Then he
remembered that some wizards, like Dumbledore, could perform spells
without speaking, so he tried to summon his wand, which had fallen out
of his hand, by saying the words “Accio Wand!” over and over again in
his head, but nothing happened.
While this didn't actually work, it might have been because Harry was unable to perform non-verbal spells. Indeed he seems to have thought that it would be possible to perform the spell even though he was not holding his wand.
On the other hand, in Deathly Hallows there is an almost identical situation where once again the spell does not work:
"Yes," whispered the voice. "Yesss... hold you... hold you..."
"Accio... Accio Wand..."
But nothing happened
We do find several instances of apparently pure wandless
magic. For example, when learning how to fly in Philosopher's
Stone:
“Stick out your right hand over your broom,” called Madam Hooch at
the front, “and say ‘Up!’ ”
“UP!” everyone shouted.
Harry’s broom jumped into his hand at once, but it was one of the few
that did. Hermione Granger’s had simply rolled over on the ground,
and Neville’s hadn’t moved at all.
Quirrel and Snape both cast spells on Harry's broom at the
quidditch match in Philosopher's Stone:
Ron grabbed the binoculars. Snape was in the middle of the stands
opposite them. He had his eyes fixed on Harry and was muttering
nonstop under his breath.
Your friend Miss Granger accidentally knocked me over as she rushed
to set fire to Snape at that Quidditch match. She broke my eye
contact with you. Another few seconds and I’d have got you off that
broom. I’d have managed it before then if Snape hadn’t been muttering
a countercurse, trying to save you.”
There is no mention of either of them using their wands, and indeed it is unlikely that they used their wands because then it would have been obvious what they were doing.
Quirrel again uses wandless magic against Harry when they fight for
the Philosopher's Stone:
Quirrell snapped his fingers. Ropes sprang out of thin air and
wrapped themselves tightly around Harry.
He later undoes this wandlessly as well:
He clapped his hands once, and the ropes binding Harry fell off.
Dumbledore also does magic by clapping his hands in Philosopher's
Stone:
He clapped his hands. In an instant, the green hangings became
scarlet and the silver became gold; the huge Slytherin serpent
vanished and a towering Gryffindor lion took its place.
It is perhaps telling that all of the above examples of wandless magic are from the first book in the series. The explanation might simply be that the author had not yet fully crystallized all the rules for magic at this point.
It is also perhaps possible to distinguish between doing general magic and casting specific spells. Apparition, animaging, and legilimency all seem to be done fairly often without wands, but they also don't seem to require an actual spell. (Snape does use a spell for legilimency, so perhaps it is some hybrid of magic that can be a spell or not a spell.) Similarly, one could argue that the "up" that brings the broom to the wizard's hand is not a spell per se but some kind of magical activation. Even so, it is harder to argue this for some of the other examples of wandless magic, so it still seems to be somewhat of a contradiction..
While it is perhaps possible to create a very nuanced theory that would allow for very specific instances of wandless magic, it seems more likely that this is just one of those discrepancies that result from the extremely difficult task of having seven books' worth of information (written over the course of a decade) all present in the author's mind at all times.
The above is all in reference to wizards who have already begun magical training with wands. It is well established that young children can do magic without wands, often unintentionally. (Then again, this might also fall under the category of "magic-but-not-a-spell" if there is any truth to the aforementioned theory.)