I don't know if it is ever established exactly when the first Fables arrived, but it was definitely before 1955.
In 1001 Nights of Snowfall, at the end of the story "Fair Division" (p. 134 in my copy), King Cole arrives in our world and meets Snow White. Here is the dialog:
KING COLE: Thank you for meeting our ship, Miss White. I have your letter. So this is Fabletown?
SNOW WHITE: No, this is the colony of New Amsterdam. We're secretly building Fabletown further up the island where we can be safely isolated.
The following pages go on to mention the renaming of the city to New York while it talks about the election of King Cole as the first mayor of Fabletown. New Amsterdam was renamed on September 8, 1664. From this we can assume that Fabletown was established sometime around then.
Thus, we know for certain the first Fable refugees arrived in our world no later than 1664. Given the Fables came to our world over time and it probably would've been years before they organized and formed Fabletown, I'd say 17th century is a fair estimate. Maybe someone will find a reference to earlier arrivals though.
This is assuming you're referring specifically to the residents of Fabletown. I don't know if it's ever established whether the first Fables to arrive in our world were definitely fleeing the Adversary. After all, the gates may have existed long beforehand. Perhaps it is possible that one or more Fables arrived long before the wave of refugees that formed Fabletown.