Trelane originally had no species given, and we had no indication that any more than three of his kind existed; he was originally just a Sufficiently Advanced Alien.
However, he was later retconned into being one of the Q, in the TNG novel Q-Squared. From the book jacket, which can be found in its entirety at that link:
This renegade Q is named Trelane - also known as the Squire of Gothos, who Captain Kirk and his crew first encountered over one hundred years ago.
EDIT: I should learn to read the question before I answer; I just saw that you don't consider Q-Squared to be canon. The canonicty of Star Trek novels is, of course, a matter of some debate, but in fairness Q is the (identified) species that best fits what we know of Trelane.
Your reasoning for why he can't be a Q is based on two points, one of which I believe to be flawed and one which I believe can be explained within the context of the show.
Your first point, that Trelane can't be a Q because he has respect for humans in general and the Enterprise crew in particular, is factually incorrect. We have no real basis for saying that this attitude is atypical of the Q: with few exceptions, they don't seem to really care about us. That one exception (John deLancie's Q), showed through his ongoing character development that he had a great deal of regard for the human race (And all but said this explicitly in the TNG series finale).
Your second point, that Trelane appears to rely on technology, is to me a better point and one that I want to expand on.
At the end of Squire of Gothos, we're introduced (in voice) to beings that act like Trelane's parents; they chastise him, clean up after him, and "send him to his room." Interestingly, this is pretty much exactly what John deLancie's Q does in the Voyager episode Q2: he appears on Voyager, yells at his son ("DON'T ANTAGONIZE THE BORG!"), cleans up after him (restoring Neelix's vocal cords, to the dismay of a large number of fans), and then turns the boy into an amoeba. The major difference is that Trelane's parents actually seem like, well, parents. This in contrast to John deLancie's character, who acts consistently like a man-child - however, we know that he's an atypically childish Q, so this doesn't bother me too much.
The reason I bring up Q2 is because the existence of Trelane's parents introduces some complexity with him being a Q: namely that the Q don't reproduce, and haven't since they ascended to the Continuum. John deLancie's son is the first Q to be born in the recorded history of the Q. So, if Trelane is a Q, how does this work?
One possibility is that Trelane is Q's son, much later in his personal timeline. But I don't like this explanation because of his reliance on that machine, As you say, it's possible the machine was a ploy, but that doesn't seem like a Q's style.
What I consider to be a more compelling explanation (And I fully admit that this is total speculation on my part) is that Trelane is a Q from before the formation of the Continuum. It's hinted many times over the course of the franchise that the Q used to be mortal, and then ascended. My theory is that Trelane is a Q from before the species became what we would consider "Q", but after they had developed technology such as matter-energy conversion, teleportation, time travel, etc. It's not that far-fetched - we see many of these technologies used in later shows (Including no less than 4 time-travel devices)