According to an interview with James B. Sikking (Captain Styles) on Startrek.com, we learn that it was a swagger stick, which was a military accessory common in the 19th and early 20th centuries.
Q. Whose idea was the swagger stick that Styles wielded?
SIKKING: I think… I think… I don’t know. I’m not sure if it was mine, but it was certainly a good idea. When I was studying acting
years and years and years and years ago, we’d have a class and they’d
say, “Go to the bus depot or to the train station or the airport. Take
a little bag with you. Wear dark glasses. And go sit somewhere and
observe people.” It’s true; you sit there and watch people, and how
they move their bodies is far more communicative than dialogue. So any
kind of a prop that reveals a character is well worth having in your
hand or next to you.
Startrek.com - "Hill Street Blues' Lt. Hunter Was... Star Trek III's Captain Styles"
The goal was to visually convey the script direction about Styles' personality
The elevator door opens, and as Scott is preparing to step in, CAPTAIN
STYLES steps out. Styles is an officer about Kirk's age. If he's a
little stuffy, it's pardonable; he does, after all, have the plum
assignment in all of Starfleet.