What is this place that the Bene Gesserit are not able to look and why? Also why is it so terrifying?
The Bene Gesserit can only bear to look at the feminine but cannot bear to look at the masculine. The Kwisatz Haderach will be able to see both feminine and masculine.
Paul Atreides and Reverend Mother Gaius Helen Mohiam, on the subject of men undergoing the spice trance:
"The drug's dangerous," she said, "but it gives insight. When a Truthsayer's gifted by the drug, she can look many places in her memory — in her body's memory. We look down so many avenues of the past . . . but only feminine avenues." Her voice took on a note of sadness. "Yet, there's a place where no Truthsayer can see. We are repelled by it, terrorized. It is said a man will come one day and find in the gift of the drug his inward eye. He will look where we cannot — into both feminine and masculine pasts."
The spice agony is an ordeal in which an acolyte of the Bene Gesserit takes a poisonous "awareness spectrum" narcotic and, by internally changing the substance and neutralizing its toxicity, awakens her inner self and the selves of all her female ancestors (called Other Memory).
Only women have ever survived the agony, but through their breeding program the Bene Gesserit sought the male Kwisatz Haderach who would be able to change an illuminating poison. A Kwisatz Haderach is given abilities different from those of a Reverend Mother. During the Spice agony, there are two areas of the soul that the acolyte may visit — the part that gives, and the part that takes; a Reverend Mother cannot access the memories of her male ancestors, and is terrified by the psychic space within her that the masculine memories inhabit. http://en.wikipedia.org/wiki/Bene_Gesserit
The place a Reverend Mother dare not look is the ancient force that "takes"
Masculine and Feminine are broken down into a more fundamental concept, Creation and Destruction. Looking inward to the force opposite your gender will be "overwhelming."
Paul said, "There is in each of us an ancient force that takes and an ancient force that gives. A man finds little difficulty facing that place within himself, where the taking force dwells, but it's almost impossible for him to see into the giving force without changing into something other than man. For a woman the situation is reversed."... "These things are so ancient within us that they are ground into each separate cell in our bodies, we are shaped by such force. You can say to yourself, yes I see how such a thing may be, but when you look inward, and confront the raw force of your life unshielded, you see your peril. You see that this could overwhelm you. The greatest peril to the giver is the force that takes. The greatest peril to the taker is the force that gives. It's as easy to be overwhelmed by giving as it is by taking.
what is the force that takes? (relating to physics)
The B.G. ability to see into maternal ancestry is a product of awakening to the "giving" force. A Reverend Mother can see the succession of bio-reproductive creation which culminated in her birth, giving them an unparalleled understanding of the past. Prescience, a common thread in dune, is the cognition of the future. Quite the opposite of other memory which deals only in the past. In physics, Entropy is the destructive force which moves a system from ordered to disordered. To glimpse the future is to see death and destruction of the present (reason why it's frightening). A few times in Dune, it is said that "time becomes space." Observing possible futures would take an understanding of the possible paths entropy could take on it's journey to the entropic heat death of the universe. To understand the future you must understand destruction. It's a reasonable postulate that the "force that takes" is the root of prescience, which Paul (nearly) masters after drinking the water of life.
I had always understood this to refer, not to the lives of male ancestors, but rather to the future. Paul's power to see possible futures appears to come from his unique status as Kwisatz Haderach, and the idea that the multitude of futures would be terrifying to the female mind but manageable by the male one fits with the in-universe mechanics of sexual dimorphism. Women cannot become mentats because they are incapable of the intensely focused complex analysis required. It seems likely that sifting through all the possible futures would require a similar set of skills.