Why in each season and film of Star Trek does the effect of the transporter change? I can understand them changing over centuries and different ships (upgraded system) but the effects even change on the same ships about 5 years apart.
Just like all the different warp effects throughout the series, the transporter effect changes no due to any in universe technological change, but out of universe improvement in filmography, cinematic, and special effects techniques. And budget. The films get better effects because they had much much more money to spend instead of the shoe-string budget of the tv series.
As for differences in species technology, there is more than one way to transport people. Every species have their own technological advancements and discoveries, and while they might be interoperable, the specifics behind their tech make it look different. See Federation Phasers vs Romulan Disruptors vs Borg beams. Out of universe, it's done so the audience can immediately know what is happening, by giving each species a unique look.
Changes in the visible transporter effect seem to be due to different underlying technology. With the wide variation of effects we have seen, it seems that the effects vary easily, possibly even with only small changes to the underlying systems - at the same time, it is reasonable that little attention would be paid to achieving a particular visual effect; technical considerations would be much more important.
That explains why transporters of different civilizations have considerably different effects (based on totally divergent technologies), and Starfleet transporters are mostly somewhat similar, but still exhibit considerable changes over time. This is best visible in a scene from TNG: Relics, which first has the TNG away team materialize with the Enterprise-D's transporter, featuring the TNG effect, and then uses the Jenolan's (older) transporter systems to materialize Captain Scott, which happens with something similar to the TOS effect.