I haven't heard of any canon-source (other than the one mentioning already listed in the comments) for this. Also there are only about 215 hits on Google, so I assume there's indeed no other source other than the TNG Technical Manual:
[...] simultaneously transferring energy and separating from the previous layer at velocities between 0.5c and 0.9c. This is well within the bounds of traditional physics, effectively circumventing the limits of General, Special, and Transformational Relativity. During force coupling the radiated energy makes the necessary transition into subspace, applying an apparent mass reduction effect to the spacecraft.
Due to this, the following is pure speculation, but I'd assume this is as far as this can get, unless there's some other, pretty unknown information not readily available on the internet (or there's some more common, alternate name for this):
The general as well as the special relativity are terms established in real life by scientists such as Einstein. I don't think there's any real term called transformational relativity.
However, the very next sentence (the last one quoted above), tries to explain some concept, that might be what in-universe is defined by the transitional relativity: "transition into subspace".
If you'd like to accelerate some mass to the speed of light, you'll need the special relativity. If you'd like to fiddle around with subspace, you'll need the transitional relativity.