Lost novel? A group exploring a space where reality is distorted or malleable

Question

This story is unusual in that it involves a group of humans exploring a space/location where the laws of Physics break down. They were mainly scientists, possibly with protection from an armed security team.

As the story progresses and they explore they are killed off singly and in groups. They gain access to a shelter and even there they have to be wary of the dangerous local conditions.

One of the concepts involved non-linear distances between points. Another may have involved a shifting of wavelenths of light and sound causing strong visual and auditory confusion.

I have read two stories separated by many years both of which explored this theme, so I may be conflating the two. It would be great if both were found by this request.

The reality may have been malleable, being altered by the observer.

There may have been another species, or human team, in the distance altering their portion of reality. This may have lead to conflict between the groups.

I apologise for the lack of well defined detail in this request, I read this/these sometime between the mid 1970's and the early 1990's. I got the impression that it/they were writen in the 1960's to 1970's.

I was reminded of this by a conversation about avant-guarde SF, mainly recalling the existance of stories from the really-alien point of view, where the author takes on the full alien persona and describes events mapped against the truely alien motivations and expectations of the protagonist.

Update: The Jack Vance book is similar in style and I do recall the books cover. However the story I am looking for is if anything even weirder with the protagonists struggling against a different reality where it was extremely dangerous to make a mistake. This next is probably true but I am not certain: I recall that the area being investigated consisted of mathematical curves and planes. I have the impression of a lack of colour. The 'Shelter' they found was approximately a rectangular structure but they could not work out at first how to enter it.

Answer 1

It's hard to tell from your description, but here are some novels I have run across that involve the types of "laws of physics breaking down" and dangerous exploration you describe. Perhaps one of these may be relevant (or perhaps they may help someone else who stumbles upon this question and is looking for similar novels):

"Evolution's Shore" or "Chaga" by Ian McDonald: An alien biosphere takes root on Earth and starts spreading. Various attempts are made to explore it. It seems to alter the laws of physics.

"Roadside Picnic" by Arkady and Boris Strugatsky: A strange zone with altered physics is entered by various exploration teams known as "Stalkers". There were movies and games also based on this story.

As others have mentioned, Annihilation by Jeff Vandermeer also has similar themes although in that case most of the odd effects of the zone (the "shimmer") have biological, rather than physical, effects.

House of Leaves by Mark Z. Danielewski: A house starts expanding internally, getting bigger on the inside, into a space where the rules of physics no longer seem to apply. Attempts are made to explore this space, some of which are fatal.

As a bonus, Son of Man by Robert Silverberg also contains exploration of zones with strange physics, but I don't think it involves exploration by teams as your description suggests. The exploration is mostly done by the protagonist.