To answer this question, it's important to note that Star Trek physics often aren't very consistent or accurate or anything. But I guess most fans consider this as a fact and ignore it, so let's ignore that for now as well.
The short answer already mentioned by you: action-reaction force (or more specific: actio = reactio); so... yes, there's some impact.
First (or more specific second part) of the question:
Would there be any difference with the deflector field being off?
Sure there would! But in a bit different way. The intention of the deflector is right in its name: It's there to deflect anything that's in the way, no matter whether we're talking about particles, dust, scrap, or anything else. Without that, there'd definitely be some difference, even if we're just talking about space particles crashing into the ship's hull. Given the Star Trek typical vulnerability of the engines, this would most likely kill the ship instantly anyway (unless the deflector fails for some added dramatic effect).
So would there be some difference? Definitely. You have to consider that the deflector field essentially changes the overall "shape" of the ship (bare ship exteriors vs. "bubble"). Starting from there it should be easy to assume that there will be differences in friction/drag/etc. (while ignoring catastrophic impacts etc.).
But is there an in-universe explanation or example?
Of course, there are, lots of them actually.
Excuse me for not having all episodes (and especially their names) in my head, but there are lots of different examples where external forces are applied to a ship despite the deflector shields fully working:
More than once a ship is shown being steered into some kind of shockwave to "ride" on it and/or to prevent damages. This can be seen several times in TNG as well as VOY and ENT (and possibly others as well). If the deflector field would just neutralize all drag/push/whatever, this wouldn't be necessary nor possible.
There are at least two episodes (TNG and VOY), where the ship is stuck "swimming" with a swarm of aliens that will simply drag the ship with them. Once again, if the deflector would neutralize external influences, this wouldn't be possible.
So back to the initial question:
Would the deflector/particle drag impact the movement of the ship?
Yes it would (even if just very, very tiny and possibly neglectable). Just imagine a small ship in a bath tub. Whenever you apply external force (waves), it will react to it. No matter whether the shape of the "ship" is round, a classic ship form, a rectangle, etc. The shape doesn't matter unless it's magically able to negate all outer influences.