In Legends, you had Maw Installation where Tarkin had the Death Star developed. They constructed a prototype (basically just the frame and super-laser) and it was powerful enough to destroy the moon of Kessel.
Canon no longer indicates that there was any such prototype. There's also indications that the Death Star itself had been in planning stages for a very long time before the Clone Wars. It's first mentioned in Episode II, where the plans were in Count Dooku's possession and were a closely guarded secret. There's no indications that the plans were still being worked on at the time. This seems to be a final design.
And then, at the end of Episode III, we see the framework is underway
The canon novel Tarkin mentions nothing about a prototype or testing either. There's just the main construction project, which suffers from setbacks and delays (plus the whispers of a super-weapon beginning to stir). That he was able to apparently hide such a massive project from public view for 20 years (and even from the rest of the Imperial forces) was an accomplishment itself.
Per Rogue One:
The Death Star was first tested on Jedah City in "single reactor mode". There was some uncertainty expressed by Tarkin before the test. The destruction was blamed on a mining accident.
We know pretty much nothing about StarKiller Base's construction. Testing of SKB seems even less likely than testing the Death Star. As mentioned in another answer
Also, how do you fire it (in any direction) without giving yourself away?
So let's talk practicalities here and speculate a bit. Both of these are epic projects in their own right. We're talking major logistics just to design something like this, let alone build it. So if you're going to even entertain the idea of building it (where you're committing the resources necessary to do this), you're going to have to have a fairly high degree of certainty that the underlying theories it's built on are well founded. Probably with some sort of small scale mockup. In other words, they probably did some sort of testing in the early design phases. Blow up an asteroid or two. The only thing that would change in the final project would be the size and scope (scalability). Still, I wouldn't want to be an engineer on that thing, hoping that the button push works the first time. Vader wasn't happy with failure.
TL;DR - They had to have a high degree of certainty that it would work before building it.