Obviously the in-universe analysis is going to be pure supposition.
To answer your ultimate question we have to first clarify what the suits were for,
I would suggest the answer, in part, is: Radiation.
The suits were primarily for hazardous material protection rather than extravehicular excursions. In fact, I'd argue that they would work very poorly in the exposed vacuum of space.
There are troubles with this hypothesis but first to the supporting evidence. Radiation was clearly a contributing factor in the death of Scotty's nephew, Peter Preston, when the USS Reliant attacked the Enterprise during it's training mission (Star Trek Wrath of Kahn) even though he was wearing said suit.
Next, I believe the suits first appeared in Star Trek:TMP and it looks like they might be in this still:
Engineers in the background are wearing the suits and helmets while in close proximity to conduits. This also appears to imply they are more likely to encounter a hazard due to close proximity (in the foreground our heroes are unsuited).
Finally, this website selling props refers to the suit specifically as an "Engineering Radiation Suit."
Now, such a suit would be expected to be even better than hazmat suits of today and specifically tailored to the Engineering hazards that could be encountered. This would mean that some form of life-support would be included: cooling systems, atmospheric protection and thermal protection.
We might even go so far as to suppose that the suits were designed to operate in a vacuum, such as a compromised internal structure, but not full extravehicular excursions, rather situations where engineers would be inside the ship structure (protected from the harshest effects of everything from extreme temperature swings to solar radiation) effecting damage repairs or maintenance.
Now that we've properly established the purpose of the suit. I think it's more easy to address your big question: Why wasn't such a similar suit used before or after that time period.
Future without Suits
The "After" seems easier to address: Forcefields and shielding. Even in Star Trek Generations we see the Engineering compartment has very effective forcefields, we can safely assume that advances in shielding helped protect Engineers from exposure to hazards that were more common in a previous era. (If we were to use modern day nuclear power as an example, initially the equipment was bulkier and the risk greater. As technology has advanced the equipment needed has become smaller, simpler and safer; relative to the past systems)
Before the Suits
The "Before" is more difficult as I don't recall similar suits in TOS or ENT. That means the only logical suppositions are (a) they didn't care or (b) the risk was less severe.
If we toss aside option (a) then what would explain the difference is that the risk was less severe risk. Again, technological advancement is a viable theory. Some change in the Warp system made the warp drive more efficient or faster but increased the hazards associated with the system. Perhaps experimentation with transwarp, perhaps dilithium crystals that were more effective but more toxic, perhaps some of the suppression and control systems associated with high-speed travel were simply more dangerous and generated more radiation in the Engineering compartment. For example, some rocket fuel in old Earth eras (aka, today) is much more toxic than other fuel available but is still used because it is more effective.
Why would Starfleet switch from a safer warp technology to a more dangerous one? Well, the 20th century US Navy switched from oil and coal to nuclear because it seriously extended the range of their vessels despite the serious radiation risk. I'm not sure I can guess what technological advancement was developed between the USS Enterprise NCC-1701 and the refit.
It does, though, seem clear the warp drive was substantially redesigned in Star Trek:TMP because of the references to it being a new ship, "Admiral, this is an almost totally new Enterprise. You don't know her a tenth as well as I do," (Will Decker) and concern about the warp drive not working and causing the wormhole in, "Captain, it was our anti-matter imbalance that created the wormhole in the first place. It will happen again if we don't correct it." (Scotty)
It is a bit of a stretch to say that these are problems associated with a new technology (Scotty: "Admiral, the Enterprise has just finished eighteen months redesigning and refitting. She needs testing, a shakedown...") instead of a plain lack of time but it is possible that it was a combination of new technology and a need to test it.
Radiation being a struggle with the Enterprise (refit) warp drive was reaffirmed in Star Trek II it was also radiation that took one of Enterprise's most revered officer's lives whereas toxic radiation protection appears less present in the future or past warp drive incarnations.
There are other possibilities of why the suits were necessary but given what we know about the suit and the technology at the time, it seems likely that the simplest answer is that between 2270-2350 warp technology in Starfleet vessels was more dangerous to organic life.