Having now seen Iron Man 3, I can add my opinion. The Avengers adds significant character development for Tony Stark, and world-building to the Marvel Cinematic Universe. Skipping it means your knowledge of the characters and world is a step behind. You'll be able to follow along, but there will be a decent amount of wondering what you missed. In addition, the fragments of backstory they reference in Iron Man 3 not only spoil The Avengers, it manages to spoil crucial parts of that storyline. Consider:
- At the end of Iron Man 2:
- Tony and Pepper got together.
- He cured his palladium poisoning.
- He built the Mark VI armor (the one with the triangular chest hole).
- He defeated Vanko, the first person to duplicate the miniaturized Arc Reactor tech that is a foundation of the Iron Man technology.
- The events in Thor and The Incredible Hulk had begun.
- At the end of Avengers: (I'm avoiding spoiling anything that isn't covered by the trailers)
- Iron Man, Hulk, Thor, Captain America, Black Widow, and Hawkeye teamed up and fought Loki.
- Iron Man, Hulk, Thor, Captain America, Black Widow, and Hawkeye all interacted, worked together, and these characters developed as a result.
- They battled Loki's attack on Earth.
The big ones are the events of Thor, and a good chunk of Avengers. These are things that changed this world, and those changes impact Iron Man 3 in a major way. In addition to the occasional name-drop-style references, these events impacted Tony and Pepper personally. They also affect the in-universe society at large by taking what is essentially our world, and having supernatural elements (i.e. Thor & Loki) impact and change that status quo. Iron Man 3 takes this status quo and fleshes it out, but if you don't know what happened in Thor and Avengers, it is confusing.
I'd say that if you're a fan of the Iron Man films, you owe it to yourself to sit down and watch The Avengers prior to watching Iron Man 3. It is a great film in its own right, and it advances Tony Stark and the Marvel film world in an interesting way that's developed further in Iron Man 3.
And yes, I'll admit that this opens up another can of worms, as the question mentions, since full appreciation of The Avengers is helped by watching Thor. So I guess I am saying you should spend a half-dozen hours watching Marvel movies before going to see Iron Man 3.