Because the holodeck is in a confined amount of space, you can determine you are in one by testing the boundaries with real physical objects.
[In] "Ship in a Bottle" Data needs to try to test and figure out if, [he], Picard, and Barclay are indeed still in the holodeck. Instead of grabbing any other object in the holodeck, Data takes off his combadge to throw towards the holodeck wall.
So if you knew the items on your person were real, this would be a good test. Using the phaser in Insurrection is the same kind of test. Use a real object to interact with the boundaries of the holodeck (such as shooting the holographic emitters).
On the other hand, if you woke up in a holodeck, and your clothing and everything was provided by the holodeck, it might be very difficult to determine if you were in the real world.
Here is a list of possible ideas:
- Attempt to interact with the holodeck computer.
- Are there safeties enabled? Attempt to get slightly injured.
- Try to go some place where there wouldn't ordinarily be atmosphere, like outer space. I've never heard of the holodeck removing atmosphere.
- See how far you can go, a lot of holodeck programs are limited in scope. They may be for a town, or continent, or planet. But if you can get off the planet and go to the farthest reaches of the galaxy, you might exceed the bounds of the holodeck program.
All of this is based on the idea that, for some reason, you suspect you are not in the real world.