The Harry Potter Wiki page on Time explains what once happened with time travel:
In 1899, the Department of Mysteries conducted its last experience concerning time-travelling back in time for more than a few hours. Eloise Mintumble was sent to the year 1402, wherein she became stuck for a period of five days. When she was finally retrieved to the present, her body had aged five centuries, and, irreparably damaged, she died in St Mungo's Hospital for Magical Maladies and Injuries.
Her excursion to the past provoked a great disturbance to the life
paths of all those she met, changing the present so dramatically that
no fewer than twenty-five of their descendants vanished in the
present, having been "un-born". Moreover, there were a few more
alarming signs that time itself had been disturbed: Tuesday following
her reappearance lasted two and a half full days, whereas Thursday
shot by in the space of four hours.
The Ministry of Magic had a great deal of trouble covering up the
mishap and, since then, the most stringent laws and penalties have
been placed around those who study time travel.
Now, this particular incident was what happened from travelling back about 450 years, albeit substantially longer than the period that occurred in Prisoner of Azkaban. Nevertheless, consider the following regulation, one of hundreds:
The longest period that can be travelled back in time without serious
chance of harm to the traveller or time itself is around five hours.
Now, to address this specific question, if you saw yourself from the past, even if your past-self did know you had a time turner, surely the temptation to know what the future entails would be too much ("Hey - what's in the exam in a couple of hours?") Consider the effects that this could have in terms of creating paradoxes.
If you didn't know that you had a time turner, you would probably turn mad at seeing yourself! You would at least be quite disturbed by it and this would probably cause a paradox, as if you were disturbed prior to getting the time turner and didn't end up getting the time turner, how could you travel back in time and see yourself? This is more of a general time-travelling paradox i.e not specific to the Potterverse, but it's probably the reason behind Hermione saying what she did.