I believe this to be a version of SGI Dogfight (usually just known as dog or flight to SGI/Silicon Graphics enthusiasts.)
More precisely, I believe this to be an unauthorised mod of the game that was created, and spread, in violation of the NDA Silicon Graphics placed on the game's source code. This mod added X-Wings, TIE Fighters and Klingon battlecruisers from TOS-era Star Trek to the set of available planes, as well as two WW1-era craft and the Grumman X-29. Since the original set of planes included four different types of fighter jet, this means that a Klingon battlecruiser and a fighter plane could have fought each other in-game.
More information on this mod is available from these links:
(The reference to "Mountain View" above is because that's where SGI's headquarters were at the time. In fact, Google's Mountain View campus is exactly the same one Silicon Graphics used to have! There are some cool photos here and here.)
EDIT/MAJOR UPDATE: I recently found a scan of two pages from SGI's old in-house magazine, IRIS Universe, containing more information on this mod:
The above scan reveals that it was created at the University of Toronto, that the author was one Shannon McKenzie, and that at one time (presumably before lawyers objected) SGI didn't mind about the NDA violation and were willing to distribute the modded version to anyone who wrote in asking for it! It also contains a black-and-white screenshot of the game's X-Wing model.
END OF EDIT.
I'll be relying heavily on two Youtube videos as sources. One of them is from 1990. The other has no date on it, but must have been filmed between 1988 and mid-September 1991. I'll explain why later.
From the first video, starting at 01:54:
Here we see a fighter plane flying through a canyon, as described. Maybe even low-altitude enough to count as "nap of the earth".
In retrospect, it seems that a fighter jet would be travelling too fast for it to make narrative sense for a stationary camouflaged ship and the jet to be in the same frame for more than a second
If you watch the video, you'll notice that it's flying very slowly - in fact, unusually slowly given that it's meant to be a fighter jet! But this wasn't slowed down footage. The jet really was flying that slowly in the software. In messages at the end of the video, the following text was displayed:
All of the computer imagery in this presentation, shown at actual speed, was produced in real-time on Silicon Graphics workstations.
The game offered four fighter jets as options (as well as other planes like a Cessna C150 and Boeing 747.) The twin-engine, two-tailed plane you see in the screenshots could have been any one of the F-14, F-15 or F-18. Having no way of playing the game myself, I can't tell you which of them it would have been. But I suspect the F-14, since this Usenet thread mentions "the skeleton mark on the tail of F-14", and since the black tails with yellow lines on top match a picture from this site:
(NB: I have tried to run that Windows port under Wine on Linux, but it fails every time.)
Now, let's look at screenshots from the second video, starting at 1:27 in:
Once again, we have a strangely slow-moving fighter jet, and this time there is some other craft in the distance. Whatever this craft is, it's too small and blurry to say for sure, but it appears to consist of two segments with the front one smaller than the rear. Of all the planes in the original set, plus the craft added by the mod, a Klingon battlecruiser is the closest match for this shape - in fact, it's the only one I would say has a good match. Though I still would have expected the wings to stick out a bit more - maybe it's the angle it's viewed from?
NOTE: This is not hard evidence that you're seeing the Klingon craft in this particular video. It could be, for instance, that what we're seeing there is the twin engines of another fighter jet seen from behind. The rest of the jet, being light grey, would blend with the sky and be less visible.
And once again, the fighter jet was indeed moving that slowly in the actual software on the SGI - because the second video showed the same message as the first:
All of the computer imagery in this presentation, shown at actual speed, was produced in real-time on Silicon Graphics workstations.
From the interview starting at 03:35 in this video, we can tell that it was filmed when Tom Jermoluk was still vice president of the Advanced Systems Division (i.e. between 1988 and 1991, according to Crunchbase). He was promoted to a different role at some point prior to September 17 1991, according to this source.
So between this and the first video, I'd say the most likely year for your brother to have seen this is 1990, with 1991 being the second most likely. And one of those years would also fit in with his being impressed by what he saw. Because after that...
... and those games massively raised the bar for 3D graphics in the early 90s!
came back describing an animation he'd seen demonstrated.
You mention your brother seeing an animation, not someone playing the game, so it might be that the video you saw was the software using its -i option to read files containing the flightpaths of various aeroplanes from earlier playthroughs, and then replay them. But I don't know if planes being "replayed" in this fashion could interact with other planes, so there may still have been human players operating the two craft you saw, since one was able to shoot the other down.
Note also that dog was a very early example of a networked game, so one of the players might have been sitting at another workstation somewhere your brother didn't see.
Or maybe someone filmed a video of the game, and your brother saw a promotional VHS playing. Or an SGI machine playing a video file.
So... would SGI actually have shown this illicit version of their tech demo game to impress people? Hard to say. But they certainly knew about it. I'll repeat this quote from earlier:
There is a version of the old flight from a university with a bunch of
unusual planes(X-wing, Tie-fighter, Klingon). This can not be shipped by
SGI as these are copyrighted things. One of the first demos was snoopy
on a doghouse flying around. We were soon contacted and told we could not use it.
(source)
That quote was from Rob Mace, who took over as lead developer for flight when Gary Tarolli moved on to other things.
Finally, here's another picture - just to show that the quality of the graphics was a bit inconsistent. Note the texture mapping on the green hills versus the flat colour on the purple ones and the ground:
The game was officially forked to allow for more sophisticated graphics on the GT, GTX and VGX machines, but even then I suspect there was internal inconsistency as new terrain and features were added many years later than the original assets.
