In Rogue One: A Star Wars Story we see troopers wearing all black: they are called Death Troopers. The only time we see them is in Rogue One. They seem strong for all the explosives and weapons they carry, but in A New Hope we see mostly the normal white Stormtroopers.
Why did the Empire have such good technology and troopers in Rogue One and suddenly abandon the idea of them, leaving no trace in A New Hope?
The Empire does. Although as of January 30, 2017, we don't know why, they do, as per the Star Wars Rebels Season 3 Mid-Season trailer.
Anyone is free to update this if new information comes about.
As far as rogue one is concerned the death troopers shown there are the body guards of the director himself. As he is pretty high ranking it can be assumed that he would have elite soldiers as bodyguards. Thus even though normal stormtroopers are the best troops the empire has these bodyguards are probably the best of these best.
Additionally as Krennic is only one of the top officers of the empire and not THE top officer it can be assumed that others also get this service.
So all in all the reason we don't see others in the films.....we don't really see any high ranking imperial officer get attacked in the films. Vader is an exception there as he has no need for bodyguards (I even assume they wouldn't live long around him as he could easily feel offended by someone assigning him bodyguards which drastically reduces the lifespan of the guards and the one who assigned them).
Their status as bodyguards also explains why they have even better equipment than regular stormtroopers (in the film the firerate of their blasters honestly surprised me though!). As bodyguards to high ranking officers they need to be able to take down even stormtroopers. Thus they not only need better training, but also better equipment to do that with any reliable chance (and their charge surviving this).
According to the director of Rogue One, all of the Death Troopers were on Scarif when the Death Star annihilated the planet. This also explains the absence of U-wings.
Out-of-universe, the idea was to tie up any discontinuities so that the film could be watched before Star Wars: A New Hope without breaking the existing canon.
That is such a great story. Was there any discussion about why we don’t see U-Wings or Death Troopers after Rogue One? Can you sort of talk about putting characters or types of characters in that might not be seen again, and was there any discussion about that?
EDWARDS: Yeah. I mean, you don’t see Snow Trooper or Speeder Bikes in A New Hope, so I think it’s fair to say that the Empire and Star Wars in general has different vehicles and different types of soldiers for particular areas or warfare. Being such a fan of Star Wars, the dream is to be able to add something to the equation and bring something to the table like Death Troopers and ships.
But then you have to justify them not existing, you don’t want to be contradicting the canon of the movies, so you make sure, whether it comes across or not, I’m not sure, but if you watch the movie, all the U-Wings are kind of like the helicopters of Star Wars and they all go in through the shield gate to Scarif, so they all get destroyed in the Battle of Scarif; and the Death Troopers, same thing happens to them. So we had to make sure anything significant like that which was new was gonna get eradicated through the events of that film.
Gareth Edwards on ‘Rogue One’, Darth Vader, and What Happened to the Death Troopers
The out-of-universe answer is obviously they weren't invented yet. I think we will probably see them more in new materials going forward.
In-universe, there just wasn't a lot of them compared to plain Stormtroopers. They are highly specialized troops that operate almost exclusively as guards for officers within the Tarkin Initiative. That means after Tarkin dies and the Tarkin Initiative is disbanded they were probably reassigned to other units.
Other limits to their numbers is the fact that the physical requirements are much higher than standard troopers. A small fraction of recruits are qualified to even go through Death Trooper training. Presumably not everyone that starts the training graduates.
They also have a great degree of cross-training, which is something the Empire doesn't like. That's one reason the TIE Striker wasn't seen in widespread use, the navy wanted ships that were specialized for one job.
Finally is cost. Death Troopers go through extensive training in exotic locations. Their equipment is also much more expensive than standard trooper equipment. Their helmets have more sophisticated sensors and their armor is coated with materials that make them harder to detect by enemy sensors. Each Death Trooper also goes through secret medical procedures to make them faster, stronger and probably smarter than a standard human. All this extra expense means the Empire would limit their number, the same way Vader's TIE was never mass produced because the hyperdrive and shields made it too expensive.
All this information is taken from The Rogue One Ultimate Visual Guide.
I think there's two main possibilities here:
Personally I'd tend towards theory 1.
I’m going to argue that the Empire could have. I mean if they have the money, time and resources to make a Death Star “go unnoticed for a while”, get it blown up and make a bigger one and on top of that make thousands of ships, other vehicles, and Star Destroyers then really I think they would be able to employ better stormtroopers.
