In the leaked script, which seems pretty accurate from a skim and after watching the film, he's noted as being in his 30's. As this comes from the leaked script though there may have been changes since so it might not be totally accurate.
CLOSE ON JOKER (30's), tears in his eyes from laughing so hard. He's trying to get it under control. His greasy, black hair is matted down. He's wearing an old, faded red hooded zip-up sweatshirt, a threadbare gray scarf, thin from years of use, hangs loosely around his neck.
Joker - Leaked script
As Eric's answer notes, Arthur also states Penny worked for the Waynes "over 30 years ago".
Arthur Fleck: You worked for him over 30 years ago. What makes you think he would help you?
Joker - Leaked script
The clerk at Arkham State Hospital also notes that Penny's file is from "over 30 years ago".
Clerk: Sorry for the wait. All our records that are 10 years or older are
stored in the basement. You're talking over 30 years ago,-- I had to do some serious digging.
Joker - Leaked script
The script then notes that Arthur was 3 years old in the photographs from the case file. As he was 3 then and this is over 30 years ago and he's in his 30's that would mean he's got to be late 30's.
ANGLE ON JOKER, catching his breath. He opens the file, flipping through the records, finds the page the clerk was reading. As he reads it over for himself, he HEARS his mother being interviewed for her psychiatric assessment, over 30 years ago.
[...]
Dr. Stoner keeps going through the file, pulls out black & white forensic photographs of three-year-old Joker's body--
Joker - Leaked script
Lastly, Thomas Wayne's accounts of what happened also seems consistent with everything else. So, whether or not he was adopted or really is their son doesn't seem to matter too much for how old he is.
Thomas Wayne: She never told you? Your mother adopted you before she even started working for us. She was arrested when you were four years old and committed to Arkham State Hospital. She's batshit crazy.
Joker - Leaked script