According to Mike Hopkins, Sound Designer for the film (in the 'Production Commentary' on the Extended Edition Bluray), the horn sound was not from a single instrument, but rather a digital composition made from a combination of various sounds including a ship's horn.
Mike Hopkins: The horn was, we knew it was going to be something that has to really shake the theatre, and has to go on. So it was about coming up with a sound that has that kind of power and we ended up going with a combination of, well, it's a sound that's mostly based on a ship horn.
Under the circumstances, it seems unlikely that there was an actual horn, just a set of stock audio clips being manipulated on a mixing desk.