When Elves awoke, the Valar wanted to protect the Elves and therefore summoned them to Aman:
Then again the Valar were gathered in council, and they were divided in debate. For some, and of those Ulmo was the chief, held that the Quendi should be left free to walk as they would in Middle-earth, and with their gifts of skill to order all the lands and heal their hurts. But the most part feared for the Quendi in the dangerous world amid the deceits of the starlit dusk; and they were filled moreover with the love of the beauty of the Elves and desired their fellowship. At the last, therefore, the Valar summoned the Quendi to Valinor, there to be gathered at the knees of the Powers in the light of the Trees for ever; and Mandos broke his silence, saying: 'So it is doomed.' From this summons came many woes that afterwards befell.
Later, after the departure of Noldor and the dying of the Two Tress, we know that the Valar hid Valinor to protect it from Morgoth, and of course Men had no part in this:
And in that time also, which songs call Nurtalë Valinóreva, the Hiding of Valinor, the Enchanted Isles were set, and ail the seas about them were filled with shadows and bewilderment. And these isles were strung as a net in the Shadowy Seas from the north to the south, before Tol Eressëa, the Lonely Isle, is reached by one sailing west. Hardly might any vessel pass between them, for in the dangerous sounds the waves sighed for ever upon dark rocks shrouded in mist. And in the twilight a great weariness came upon mariners and a loathing of the sea; but all that ever set foot upon the islands were there entrapped, and slept until the Change of the World. Thus it was that as Mandos foretold to them in Araman the Blessed Realm was shut against the Noldor; and of the many messengers that in after days sailed into the West none came ever to Valinor - save one only: the mightiest mariner of song.
Then Men awoke:
To Hildórien there came no Vala to guide Men, or to summon them to dwell in Valinor; and Men have feared the Valar, rather than loved them, and have not understood the purposes of the Powers, being at variance with them, and at strife with the world.
It seems to me that the Valar held different attitudes towards Elves and Men. Is this true? And if so, why?
I think it was a plot device Tolkien used so that Elves could live in Aman and Men could live in Middle Earth separately. And as Aman and Elves were hidden from the world, only Middle-earth remained and there were only Men in it. Tolkien tried to explain how this world was shaped.