The in universe explanation is that the rebels got lucky/were just that good I suppose
Star destroyers are formidable things as per their specifications here
Imperial II-class Star Destroyer
TECHNICAL SPECIFICATIONS
LENGTH
MGLT
MAXIMUM ATMOSPHERIC SPEED
ENGINE UNIT(S)
- KDY Destroyer-I ion engines (3)
- Cygnus Spaceworks Gemon-4 ion engines (4) (emergency use)
HYPERDRIVE RATING
- Class 2 (primary)
- Class 8 (backup)
HYPERDRIVE SYSTEM
POWER PLANT
SHIELDING
- Deflector shield generator domes (2)
SENSOR SYSTEMS
ARMAMENT
- Turbolaser batteries (60)
- Heavy ion cannon emplacements (60)
- Tractor beam projectors (10)
COMPLEMENT
- TIE line starfighters (72), usually:
- TIE/ln space superiority starfighters (48)
- TIE/sa bombers (12)
- TIE/IN interceptors (12)
- All Terrain Armored Transports (20)
- All Terrain Scout Transports (30)
- At least 2 Gozanti-class cruisers
- Lambda-class T-4a shuttles (8)
DOCKING BAYS
CREW
- Officers (9,200)
- Enlisted crew (27,850)
PASSENGERS
CONSUMABLES
COMMUNICATION SYSTEMS
OTHER SYSTEMS
https://starwars.fandom.com/wiki/Imperial_II-class_Star_Destroyer
The OOU is simply that we were only shown a tiny cross section of battles that likely occured across the empire. You can't build an empire at all without winning fights after all
So I suspect they won a lot more than they lost.
I'll add to my answer part of Chronocidal's comment to your question, Fighters ARE anti fighter weaponry and as you can see the Star Destroyer has a decent complement.