In order to hover (approximately) in place over the surface of the rotating habitat, the helicopter does need to exert a force "upwards", i.e. towards the axis of rotation.
This is because the habitat is rotating, which means that its surface is constantly moving around the axis. But the surface is also constantly being pulled towards the axis by the habitat structure supporting it — if it weren't, the habitat would simply fly apart. This centripetal force exerted by the habitat structure keeps the habitat together and the surface moving in a circle.
In order to stay above a particular spot on the surface, the helicopter must also move in the direction of the rotation at the same speed as the surface (or, actually, slightly more slowly, since it will be closer to the axis). But since the helicopter is not supported by the habitat structure, it must provide its own centripetal force (e.g. by thrusting against the surrounding air). If it did not, the helicopter would keep moving straight on a free-fall trajectory, which would quickly lead to a collision with the habitat wall:

OK, you ask, but what if the helicopter did not try to stay above a particular spot on the surface, but simply floated in space at a constant distance and direction from the axis, letting the surface spin beneath it?
Well, if the interior of the habitat was in vacuum, this would actually work. (Of course, helicopters are kind of useless in vacuum, so in that case you'd better take a spacecraft instead.) However, if the habitat cylinder is filled with air (which it needs to be, to allow helicopter flight — not to mention letting the inhabitants breathe), there's a problem: wind.
You see, the air in a rotating habitat will also be rotating at (approximately) the same speed as the surface. How fast is that?
Well, according to Wikipedia, Rama has an interior radius of about 8 km (making its circumference a little over 50 km) and a rotation period of 4 minutes. Thus, the interior surface is moving around the axis at about 50 km / 4 minutes, or 750 km/h (= 466 mph). Near the surface, the air will also be moving at about this speed, so a low-flying helicopter trying not to follow the surface, but to remain stationary with respect to the central axis, would have to fight a 750 km/h headwind(!).
The current world record for fastest helicopter flight is a little over 400 km/h (249 mph), so it doesn't seem feasible for a helicopter built using current or near-future technology to pull this off, at least not at "ground level". Of course, going higher up towards the axis would reduce the required speed; for example, halfway up towards the axis, i.e. 4 km above the interior surface, the average speed of the rotating air inside cylinder would be only 375 km/h, which should be achievable in principle. Yet, even so, the power needed to achieve such extreme airspeeds would certainly be much greater than that needed for simply hovering over the surface.
(That said, if you did have a big rotating space station and a fast helicopter, and managed to pull off this trick, you'd observe something interesting: when flying anti-spinward at exactly the right speed to cancel out the rotation of the surrounding air, the helicopter would be able to remain aloft even with the propeller tilted sideways, at a 90° angle to the surface. Also, anyone inside the helicopter would effectively be in free-fall.)