I think it might be a throwaway.
Grindelwald also seriously considered raising an army of Inferi, and while it's not explicitly stated how many people Grindelwald killed (the only known victim is possibly Arianna Dumbledore), there's nothing in canon to imply that Grindelwald was a darker wizard than Voldemort. As well, their agendas were different -- immortality for Voldemort, taking over of the Muggle world by the Wizarding world for Grindelwald.
Tales of Beedle the Bard says [We] remain incapable of raising the dead, and there is every reason to suppose this will never happen. Vile substitutions have, of course, been attempted by Dark Wizards, who have created Inferi, but these are ghastly puppets, not truly reawoken humans. - TOBTB - Page 164. It indicates that Necromancy is the Dark Art used to conjure the dead, but it is a branch of magic that has never worked. (Page 160)
I can't recall or locate anything in canon that indicates Inferi can only be conjured by the Dark Wizard who initially killed them. And if Necromancy doesn't work, then is there a spell? I suppose that's a separate question.
ETA: I forgot I was going to mention this bit of text:
‘The Lestranges should stand here,’ said Voldemort quietly. ‘But they are entombed in Azkaban. They were faithful. They went to Azkaban rather than renounce me ... when Azkaban is broken open, the Lestranges will be honoured beyond their dreams. The Dementors will join us ... they are our natural allies ... we will recall the banished giants ... I shall have all my devoted servants returned to me, and an army of creatures whom all fear ...’
Goblet of Fire - Page 564 - British Hardcover
The last sentence might imply the Inferi, but I take it neutrally as to your question. It neither confirms nor denies whether or not the wizard must use his/her own murder victims for Inferi. I still err on the side of it being a toss-away comment, given the lack of canon indicating otherwise. :)