Plausibly most aspects of Gandalf's look might derive from the depictions of Odin in the answer from @Cugel but it seems that Odin's hat was shown as having a short rounded top in those pictures, so I'll focus on where the idea of wizards wearing tall pointy hats came from, since Tolkien does specify in The Hobbit that Gandalf "wore a tall pointed blue hat".
This page has a depiction of a wizard from a late 19th century lantern slide:
(some other magic lantern slides with wizards/sorcerers can be found here and here)
It sort of seems like this guy is somewhere between the modern Merlin/Gandalf vision of a wizard and older depictions of the "renaissance magus" like the one shown here of Dr. Faustus from a 1620 edition of Faust:
Another similar Dr. Faustus, found on this blog which features a lot of old witch & sorcerer imagery:
And here is an illustration of Merlin by Gustave Doré from 1868, you can see the image of the wizard already included the robes and the long white beard but perhaps the conical hat wasn't yet part of standard wizard regalia in the public imagination:
And this illustration of Merlin from the 1889 edition of Mark Twain's A Connecticut Yankee in King Arthur's Court has a short conical hat, but again not the really tall kind we now associate with wizards:
Some of the films of silent film pioneer Georges Méliès feature characters with robes and conical hats decorated with stars and crescent moons, as is common in depiction of wizards...earliest example on my Méliès DVD seems to be "The Magician" from 1898, below:
He also sometimes depicted astronomers in a similar costume, as in "The Astronomer's Dream" from 1898, and his most famous film, "A Trip to the Moon" from 1902:
He had a background in pantomime theater, so it's possible this type of costume was used in pantomimes to represent people who studied the heavens, whether for astrological magic or astronomy. The conical hat may be based on a type of hat used by the medieval philosopher Duns Scotus, which later became an object of mockery (and the basis for the 'dunce cap') as explained in this straight dope article:
What does this all have to do with those silly pointy hats?
Well, one of the more mystical things Duns accepted was the wearing of
conical hats to increase learning. He noted that wizards supposedly
wore such things; an apex was considered a symbol of knowledge and the
hats were thought to "funnel" knowledge to the wearer. Once humanism
gained the upper hand, Duns Scotus's teachings were despised and the
"dunce cap" became identified with ignorance rather than learning.
Humanists believed learning came from internal motivation rather than
special hats, and used the public shame of having to wear a dunce cap
to motivate slow learners to try harder.
Here is a cover to Twain's The Mysterious Stranger from 1916 which again features the pointed hat and stars-and-crescent-moons decorations (this page mentions that the image is of a character, "the astrologer", who was added to the first heavily-rewritten edition of the book and was not present in Twain's original text):
Perhaps the "The Sorceror's Apprentice" from Disney's Fantasia helped cement the modern iconic image of a wizard's hat in public consciousness (in the same way Wizard of Oz cemented the image of a witch, inventing the notion that they had green skin for example)--in this cartoon, Mickey gained temporary magic powers from "borrowing" the hat of his wizard master, which was pointed, blue, and had the star-and-crescent moon pattern often seen in later images of wizards. Fantasia came out in 1940, only 3 years after The Hobbit, so I don't know how likely it is that the Disney animators were influenced by Gandalf (especially since according to the "Design and Animation" section of the Fantasia wiki article, "Animation on The Sorcerer's Apprentice began on January 21, 1938", and this article says "Houghton Mifflin Co. of Boston and New York published the first American edition of The Hobbit in spring of 1938 following its September, 1937 debut in the United Kingdom from George Allen & Unwin LTD").
This page has some of the story of how Disney artists came up with that particular design for the hat:
For the actual filming of the animated classic, a well-known silent
film star, Nigel de Brulier, was brought in to provide live-action
reference footage for the character of the sorcerer. Disney writer
Carl Fallberg went to the famous Hollywood costume rental house,
Western Costume, to find the appropriate robe and pointed hat for the
performer to wear during the shooting. Fallberg had to paste white
stars and crescent moons onto the hat to get the look he wanted.