In Captain America: The First Avenger, the US military was shown to have normal weapons of the WW2 era. Even Stark's aeroplane (which dropped Captain America in enemy territory) was of that time.
Hydra was initially working for Hitler, whom they later planned to overthrow. If they initially agreed to work under Hitler, it means they weren’t that powerful initially (probably there was a money issue too).
Later, the only special thing they had was the Tesseract. This explains the advanced weapons able to vaporize people (or destroy New York). But what explains the Valkyrie, advanced submarines and bombs having rotors which were clearly ahead of time? How did Hydra achieve that much in such a short time? Enhancing aerodynamic design etc requires decades of research.