This is Robert Silverberg's Nightwings.
Roum is a city built on seven hills. They say it was a capital of man in one of the earlier cycles. I knew nothing of that, for my guild was Watching, not Remembering; but yet as I had my first glimpse of Roum, coming upon it from the south at twilight, I could see that in former days it must have been if great significance. Even now it was a mighty city of many thousands of souls.
Nightwings is set in the far future, where Earth is prophesied to be taken by invaders from another star. The people are grouped in various "guilds", each with a job they do not step outside of.
The narrator is a Watcher, so it's his job to scan the heavens for a sign of the invasion, and give the signal. He is traveling with the Flier Avluela and Gormon, a changeling outside of any guild.
They enter the city of Roum. He is refused lodging at his own guild's inn (too full), but become guests of Roum's Prince, who is taken by Avlulela's beauty.
One night, as the narrator performs his Watch, he sees troubling signs that the invasion might be happening.
I evaluated my position and decided that I did not have valid grounds for alarm. I reflected that Gormon had placed suggestive ideas in my mind that evening. I might possibly be reacting only to his jeering talk of imminent invasion. I could not act. I dared not jeopardize my standing by hasty outcry. I mistrusted my own emotional state.
I gave no alarm.
As it turns out
Gormon was an advance agent of the invasion force, and brags about this to the Watcher. He hastily Watches again, sees the invasion fleet, and gives the alarm. It's too late.
You had one job, man!