You've got a misunderstanding of how the Sacred Timeline works and it wasn't really until the last episode that I realised this myself. The Sacred Timeline actually consists of all the universes in the multiverse squashed down together into the same timeline, so the overall flow of time is the same. However, this still allows for differences in appearance because they don't necessarily affect the flow of time. It can also allow for slight differences that don't affect the flow of time.
When Miss Minutes gives her introductory speech to Loki we're told the TVA are just protecting the proper flow of time. They're not making sure everything and ever detail is 100% correct but rather that the outcome of everything is always correct.
Miss Minutes: Long ago, there was a vast multiversal war. Countless unique timelines battled each other for supremacy, nearly resulting in the total destruction of... well, everything. But then, the all-knowing Time-Keepers emerged, bringing peace by reorganizing the multiverse into a single timeline, the Sacred Timeline. Now, the Time-Keepers protect and preserve the proper flow of time for everyone and everything.
Miss Minutes: But sometimes, people like you veer off the path the Time-Keepers created. We call those Variants. Maybe you started an uprising, or were just late for work. Whatever it was, stepping off your path created a nexus event, which, left unchecked, could branch off into madness, leading to another multiversal war.
Loki, Season 1 Episode 1, "Glorious Purpose"
Also when Loki and Mobius are talking about apocalyptic events they briefly touch on what causes a nexus event and it is only mentioned about doing something they're not supposed to do. I.e. once again only about changing the flow of time.
Loki: Nexus events happen when someone does something they're not supposed to do, right?
mobius: Well, it's a little more complicated, but, yeah.
Loki, Season 1 Episode 2, "The Variant"
Again when He Who Remains is talking to Loki and Sylvie all he mentions is protecting the timeline and flow of time.
He Who Remains: Once I isolated our timeline, all I had to do was manage the flow of time and prevent any further branches. Hence, the TVA. Hence, the Time-Keepers and a highly efficient bureaucracy. Hence, ages... and ages of cosmic harmony.
Loki, Season 1 Episode 6, "For All Time, Always"
This means that some physical differences and even minor differences are okay as long as they don't lead to a nexus event and so the timeline branching. Then the timeline timeline proceeds as expected/written and the TVA do not need to get involved.
It's also worth noting that the TVA aren't fool proof, Sylvie can only hide out because she's already found a hole in their system: they're not really monitoring large scale apocalyptic events because any changes there won't cause nexus events. In a similar vein they might not even be monitoring for physical differences as in the grand scheme of things they probably don't cause nexus events.
Loki: If everything and everyone around you is destined for imminent destruction, then nothing that I say or do will matter, because the timeline's not gonna branch. 'Cause it gets destroyed. Hence, the Variant could be hiding in the apocalypse and do whatever he wants, and we wouldn't know!
Loki, Season 1 Episode 2, "The Variant"
I just want to address a misconception you have as well. It's not that Sylvie didn't do anything or didn't have a Nexus Event, we just don't know what it is and Ravonna refuses to say.
Ravonna: What do you wanna say to me, Variant?
Sylvie: What was my nexus event? Why did you bring me in?
Ravonna: What does it matter?
Sylvie: It was enough to take my life from me, lead to all of this. Must have been important. So, what was it?
Ravonna: I don't remember.
Loki, Season 1 Episode 4, "The Nexus Event"
Note that with a season 2 on the cards it's possible we may find out there. Kate Herron, the director, has spoken about her nexus event but doesn't divulge what it is just yet:
My interpretation of it is that it's definitely open for discussion, because I wouldn't want to shut that down. I think it's kind of fun seeing everyone and their read on it. But I think there's something tragic, in the sense that we see her as a little girl and she's playing with her toys, and it doesn't seem like it's a big thing that she's done. So I think that's the really key thing with her is that we don't necessarily know what the exact event was, but it wasn't villainous, and it didn't seem [that way]. I think that's the clear thing, particularly with Episode 4, is that obviously our perspective on the TVA started to shift. For example, it was really important for me showing her going through the same process as Loki but it's like, 'Okay, how do we show that through the eyes of a child that's also innocent?' And we should feel, as an audience, wrongfully there. So I think that was really key for us.
[...]
But in terms of like, what exactly it was, I would kind of just leave that open to the fans to discuss because yeah, I think it's fun. I have my own idea of what it is, but I think in my head, it's definitely something innocent and something out of her control. Which kind of plays into the fact that not everyone arrested by the TVA is necessarily like Loki and has stolen a tesseract and created this branch. Sometimes you accidentally do just step onto the wrong leaf and you create this branch. Do those people, where it's accidental, do they deserve to go through this process where ultimately they're deleted by the TVA? Probably not. Or maybe they do, for the better of servicing and protecting the timeline. So yeah, that's kind of all part of the discussion.
Cinema Blend, Loki Director On Sylvie's Nexus Event And Why We Didn't See More Of Her Timeline Reset Attack