This story is unusual in that it involves a group of humans exploring a space/location where the laws of Physics break down. They were mainly scientists, possibly with protection from an armed security team.
As the story progresses and they explore they are killed off singly and in groups. They gain access to a shelter and even there they have to be wary of the dangerous local conditions.
One of the concepts involved non-linear distances between points. Another may have involved a shifting of wavelenths of light and sound causing strong visual and auditory confusion.
I have read two stories separated by many years both of which explored this theme, so I may be conflating the two. It would be great if both were found by this request.
The reality may have been malleable, being altered by the observer.
There may have been another species, or human team, in the distance altering their portion of reality. This may have lead to conflict between the groups.
I apologise for the lack of well defined detail in this request, I read this/these sometime between the mid 1970's and the early 1990's. I got the impression that it/they were writen in the 1960's to 1970's.
I was reminded of this by a conversation about avant-guarde SF, mainly recalling the existance of stories from the really-alien point of view, where the author takes on the full alien persona and describes events mapped against the truely alien motivations and expectations of the protagonist.
Update: The Jack Vance book is similar in style and I do recall the books cover. However the story I am looking for is if anything even weirder with the protagonists struggling against a different reality where it was extremely dangerous to make a mistake. This next is probably true but I am not certain: I recall that the area being investigated consisted of mathematical curves and planes. I have the impression of a lack of colour. The 'Shelter' they found was approximately a rectangular structure but they could not work out at first how to enter it.