I can't readily find an on-line reference to it; however, I remember from the mid-90's or so, not too long after the start of the Vertigo line, a decision to avoid having DC characters appear in Vertigo books, or vice-versa, based on the simple fact that the Vertigo books couldn't be sold to younger fans, due to the content. Having John Constantine guest star in an issue of JLA could lead to someone wanting to buy HELLBLAZER; all to the good, unless that person was 10 years old, and selling them an issue of HELLBLAZER including nudity could get people at comics shops arrested.
This was around the same time period when a number of comics shops and employees got in trouble for selling inappropriate material to youngsters/minors. It's one of the reasons the Comic Book Legal Defense Fund was created.
It was not a wholly unreasonable action for DC/Vertigo to take. One could easily imagine someone claiming that an issue of HELLBLAZER with, say, Superman on the cover would logically be marketed to the same people as SUPERMAN and ACTION COMICS; thus, leaving DC Comics itself open to charges of some sort. Oh, and, of course, looking out for the folks selling the comics by minimizing the likelihood of DC/Vertigo books being involved in this sort of thing.
Occasionally, where story reasons dictated it, we'd see DC characters in Vertigo books, or vice-versa - but it was very rare. And, one character (Zatanna) seemed to straddle the divide, being used in DC books sometimes, and in Vertigo books at other times. However, for the most part, characters did not cohabitate both universes. When DOOM PATROL was brought back a few years after its Vertigo run ended, the characters were no longer seen in Vertigo books (not that they had been, with any regularity).
Perhaps the most significant violation of this rule was the "Stars and Sand" storyline in STARMAN (a DCU title), which guest-starred Wesley Dodds, DC's original Sandman, then appearing in SANDMAN MYSTERY THEATRE (A Vertigo book). Sandman was even on the covers of STARMAN - but was not named.
A couple of things should perhaps be noted: First, this was not a direct crossover with SANDMAN MYSTERY THEATRE, as that book was set in the late 1930's, not the 1990's. And, STARMAN was perhaps more like the DC books that became the core of Vertigo when it launched than it was like the rest of the DCU books published at the time. Finally, STARMAN had an on-going theme about the legacies of older heroes, and a strong, well respected creative team (James Robinson and Tony Harris) to pull for the story. All that said, Sandman was hardly such a recognizable figure as to be likely to cause problems, and STARMAN had an older-skewing readership to begin with.
So, I'd argue that the appearance of Death in ACTION COMICS shortly before the start of "the New 52" was the first really significant appearance of a Vertigo character cover-featured in a DC book since that split began. And, shortly thereafter, Swamp Thing and John Constantine were brought back to the main DCU, with others (mostly short-lived - Shade, the Changing Man appeared in early issues of JUSTICE LEAGUE DARK, following his role in the Flashpoint series SECRET SEVEN) following.
All that said - that's my memories. I have no links to point to that explicitly back me up, unfortunately. For that matter, the decision to keep DC and Vertigo characters firmly separated wasn't something that was announced via a press release or anything; it was intimated by various creators and comics insiders, not an official public policy, as best I recall.
(Ok, now you can all come, point out all the stuff I've forgotten that disproves everything I've said, and provide a real answer with links I didn't find :-)
