You are remembering correctly
The textual history of this passage can be seen in The Annotated Hobbit and The History of the Hobbit.
1937 first edition and 1951 second edition
The mother of our particular hobbit — what is a hobbit? I suppose hobbits need some description nowadays, since they have become rare and shy of the Big People, as they call us. They are (or were) small people, smaller than dwarves (and they have no beards) but very much larger than lilliputians. There is little or no magic about them, except the ordinary everyday sort which helps them to disappear quietly and quickly when large stupid folk like you and me come blundering along, making a noise like elephants which they can hear a mile off.
1966 third edition
The mother of our particular hobbit — what is a hobbit? I suppose hobbits need some description nowadays, since they have become rare and shy of the Big People, as they call us. They are (or were) a little people, about half our height, and smaller than the bearded Dwarves. Hobbits have no beards. There is little or no magic about them, except the ordinary everyday sort which helps them to disappear quietly and quickly when large stupid folk like you and me come blundering along, making a noise like elephants which they can hear a mile off.
c.1930 First Draft
The mother of this hobbit – what is a hobbit? I meant you to find out, but if
you must have everything explained at the beginning, I can only say that hobbits are small people, smaller than dwarves (and they have no beards), and on the whole larger than lilliputians. There is little or no magic about them, except the ordinary everyday sort which helps them to disappear quietly and quickly when ordinary big people like you or me come blundering along, making a noise like elephants which they can hear a mile off;
This quote was also used in marketing material, such as this 1938 advertisement (reproduced in The Annotated Hobbit)
Why lilliputians was removed
It can be helpful to look at some other writings, where Tolkien rejects comparisons between his Hobbits and Swift's lilliputians, and where he rejects the idea that Swift should even be classified as Fairy.
Letter to Milton Waldman (1951)
The Hobbits are, of course, really meant to be a branch of the specifically human race (not Elves or Dwarves) – hence the two kinds can dwell together (as at Bree), and are called just the Big Folk and Little Folk. They are entirely without non-human powers, but are represented as being more in touch with 'nature' (the soil and other living things, plants and animals), and abnormally, for humans, free from ambition or greed of wealth. They are made small (little more than half human stature, but dwindling as the years pass) partly to exhibit the pettiness of man, plain unimaginative parochial man – though not with either the smallness or the savageness of Swift, and mostly to show up, in creatures of very small physical power, the amazing and unexpected heroism of ordinary men 'at a pinch'.
1951 Letter to Milton Waldman (Letters of J.R.R. Tolkien #131)
On Fairy-Stories (delivered 1939, published 1947)
But what is to be said of the appearance in the Blue Fairy Book of A Voyage to Lilliput? I will say this: it is not a fairy-story, neither as its author made it, nor as it here appears “condensed” by Miss May Kendall. It has no business in this place. I fear that it was included merely because Lilliputians are small, even diminutive—the only way in which they are at all remarkable. But smallness is in Faerie, as in our world, only an accident.
"On Fairy-Stories" - "Fairy-story" §16
Some Tolkien scholars have commented on Tolkien's inclusion and later removal of the term.
John Rateliff says that this was a term appropriate in children's literature at the time, pointing to Mistress Masham’s Repose as another example. He suggests that the later removal was because it no longer matched the more somber tone used in The Lord of the Rings.
The lighthearted comparison to lilliputians, surprising as it is to readers approaching The Hobbit from the more somber perspective of The Lord of the Rings, survived into the published text and was only removed almost three decades later, in the third edition of 1966. [Footnote:] That Swift’s bitingly satiric invention had come to be considered appropriate children’s fare (albeit usually in carefully bowdlerized versions), a shift that seems to have occurred during the Victorian era, is also attested to by T. H. White’s modern-day sequel, Mistress Masham’s Repose [1946].
The History of the Hobbit - I(b) i. "Baggins of Bag-End"
Douglas Anderson notes that Andrew Lang had classified that Lilliputs as fairy. He suggests that Tolkien removed the term to avoid such a close tie to another work of literature.
Tolkien probably removed the reference to lilliputians because of the unsuitability of directly referring to elements from another work of literature. In Gulliver’s Travels (1726) by Jonathan Swift, the people of Lilliput are about six inches tall. The association of lilliputians with fairy tales is seen in the children’s version of the story (rewritten and abridged by May Kendall) that appears in Andrew Lang’s Blue Fairy Book (1891) under the title “A Voyage to Lilliput.” In his essay “On Fairy-Stories” Tolkien objected to its classification as a fairy-story, both in its original and condensed forms.
The Annotated Hobbit - Chapter 1 "An Unexpected Party"
Anderson and Flieger contrast Tolkien's usage of the word with his objection to Lang's classification, perhaps suggesting this as the eventual cause of its removal.
included merely because Lilliputians are small - At variance with this judgment is Tolkien's inclusion of lilliputians [sic] with hobbits and dwarves as "small people" in chapter one of the first edition of The Hobbit. The reference to Lilliputians was removed in the 1966 revision to the text.
Tolkien On Fairy-Stories - Part One - Editor's Commentary §16