I understand that he had four major projects come out in 2005 - all of which got nominated for major awards, so I understand why he did not do Goblet of Fire. However, was there any particular reason ever given by him or the directors/producers/etc involved, why he did not return for 5-8?
John Williams was asked back to score Harry Potter and the Deathly Hallows - Part 2 by director David Yates; however, Williams's schedule and Yates's schedule "did not align" -- Yates would have had to have provided Williams with a raw cut of Deathly Hallows - Part 2 much sooner than was feasible if he were to compose the music for the film. The John Williams Fan Network (I'm unsure of the quality of this source) cites either a lack of time on Williams's part, or the director wanting someone local to the production, as possible reasons behind why Williams did not go on to score Goblet of Fire after the first three films. Fans at the Film Score Monthly forums seem to lend credence to these possibilities.
Clearly, The Powers That Be secured the rights to, for example, Lumos! Hedwig's Theme YouTube 1:42 (SFW), which is the main Harry Potter theme, which would allow them permission for any subsequent composer to incorporate Hedwig's Theme into future Potter scores as needed. Indeed, Hedwig's Theme is heard in all eight Potter films in one incarnation or another.
In Harry Potter and the Goblet of Fire, Hedwig's Theme can be heard in the opening credits. In Order of the Phoenix, it can also be heard in the opening credits. In Deathly Hallows - Part 2, the second part of Hedwig's Theme, which corresponds with the opening credits of Chamber of Secrets, is heard at the 19 Years Later scene, which is the last scene of all the Harry Potter movies. Offhand I can't pinpoint off the top of my head where exactly Williams's work is used in Half-Blood Prince or Deathly Hallows - Part 1, but according to John Williams's Wikipedia page, it was indeed used.
As an aside, here YouTube 4:56 (SFW) is a clip of the London Symphony Orchestra performing Hedwig's Theme for the Proms for the BBC, which is pretty cool because it shows which instrument plays each part of the theme. I hadn't known before, and GorchestopherH pointed out to me, that the instrument that plays the opening measures of Hedwig's Theme -- the bell-like instrument -- is something called a celeste or celesta, which is grouped in the keyboard family although it's technically a percussion instrument (TangoOversway pointed this out to me; originally I had said it was not a percussion instrument. It's an idiophone.). "Celesta" or "celeste" means "heavenly" in French.
He had already passed the torch.
Many creative people are like that - once they have to let go of something or move on, they would rather not go back to revisit old works. The other side to it is something a bit like courtesy. Once you've turned it over to someone else, let them have it - don't grab it back.
In terms of consistency, once a producer of a film series has moved on from using one person in a work (whether they're a composer, director, or actor), it's a step down from what might be called "total unity" and it makes sense, once you change direction, to not go back. (Unless the new direction is a disaster - such as with On Her Majesty's Secret Service in the 007 series. It did lousy at the box office, and they went back to Sean Connery as Bond, and ened up with what's generally considered one of the worst Bond films ever made (Diamonds are Forever) and knew they'd have to move on anyway.)
One of the main reasons why John Williams never came back to Compose for Goblet of Fire was due to his involvement of 'Revenge of the Sith - Episode III' that same year. Thus, his scheduling made it nearly impossible to compose both films at the same time, so Williams generously acknowledged Mike Newell (the director) to use one of his longtime friends and brilliant composer Patrick Doyle, whom went on to compose the film in his honor, using samples of motifs from John throughout the score. In terms of his being asked back or to leave, Warner Brothers and the producers themselves were upset to Williams leaving the project due to scheduling conflicts, yet never insisted Williams to be let go due to his "style". The stylistic choice for his music made the tone absolutely set for the films. Had John stayed on for all 8 (presumably 7, if they had at the time), I think the musical score would still have worked brilliantly, since Williams is known for keeping things fresh and unique, with a hint of familiarity to counter-balance it all out, like what he recently did for Star Wars: The Force Awakens. The fact that the series garnished new composers for almost each movie made the scores distinct from one another, complementing the scope, subtly, and tone of the individual films with said scores.