After having done some more research I can confirm this book is The Night They Stole the Alphabet by Sesyle Joslin (1968). To quote information I found on it:
Victoria would never have rushed off in the middle of the night after three ordinary robbers, but these had stolen the letters of the alphabet from her bedroom wallpaper, and the pages of all her books were empty of printing as well, so she did not hesitate. Her search for the 26 letters that were scattered far and wide led her first to a baby with a B in its bonnet, and then to the terrifying Madame Muss, a turnabout mermaid who told her to mind her P'S and Q's; a gentle and hospitable owl who invited her to a refreshing cup of T, and a gypsy who predicted that she would soon receive an important letter. And all the while Victoria was pursuing the shadowy robbers...or were they pursuing her?
https://www.amazon.com/gp/customer-reviews/R2VORKYE8FBCO0/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=0152574409
Like Alice down the rabbit hole. Victoria on the trail of the letters lifted from her wallpaper enters a realm where everything is askew but somehow familiar -- either a reminder of home or a satire of mores and manners. The reader knows that she'll eventually retrieve each of the twenty-six letters and suspects that then she'll wake up; and there's a constant threat in the persons of master witch Madame Muzz and her cockney minions the three Robbers, some help from Alexandra the Russian reverse mermaid (fish head, female feet) and tiny hockey goalie Gibson who scorns being called a fairy...
https://www.kirkusreviews.com/book-reviews/sesyle-joslin-4/the-night-they-stole-the-alphabet/
This has the episodic structure, the alphabet wallpaper, the young girl hero and the light tone I mentioned in the question, and matches my sense that it was a older book when I read it in the 80's. The name "Victoria," the shadowy robbers, and the detail of the fish-headed mermaid all sound right.
EDIT: After requesting this on interlibrary loan, I can now confirm this as the correct answer. The illustrations are very familiar to me, including one with a letter inside a snowglobe.