Most likely not, they were incompatible.
In a section on her old website, JKR addressed the possibility of Luna and Neville “hooking up” in a later book. Considering that Harry Potter is a relatively clean series, “hook up” in this case most likely refers to the possibility of them dating rather than a physical encounter. JKR considered herself to be quashing the rumor with her response, stating Luna and Neville were too different to ever be able to feel much more for each other than friendship.
Luna and Neville will hook up in HP&THBP
The Luna/Neville shippers are much less vehement and scary than the Harry/Hermione, Ron/Hermione tribes, so I hope I won’t receive too much hate mail for quashing this rumour. I see Neville and Luna as very different kinds of people and while they share a certain isolation within Hogwarts, I don’t think that’s enough to foster true love - friendship, perhaps, although I think that Neville would always find Luna’s wilder flights of fancy alarming.
- Rumors (JKR’s old website)
This statement was from before all the books were published, and certainly before the Deathly Hallows movie in which Neville wished to confess a crush, so it is possible that JKR could have changed her mind on their relationship. However, in the books it is never even hinted at that either Neville or Luna had any sort of crush on the other. Therefore from the information given in the books, there is no reason to believe they would have dated. Additionally, JKR never mentions Luna and Neville dating in any later interviews. Considering it is never mentioned anywhere but the movies, which often had some inconsistencies with the books, there is no real evidence to imply that Luna and Neville were meant to have dated in book canon.