Minsc and Boo were inventions of James Ohlen, lead designer for the original Baldur's Gate duology, created during their personal tabletop adventures.
In an interview with Rock, Paper, Shotgun, he goes into some elucidating details:
Ohlen picked his side early. He was telling BioWare stories even before he joined the company. The meeting of Minsc and Boo, one of the most enduring partnerships in PC gaming, came about in a tabletop Dungeons & Dragons game he ran as a teenager.
BioWare programmer Cam Tofer played Minsc in one of those campaigns, “as a guy who’s basically been knocked on the head too many times in fights”. A merchant NPC of Ohlen’s invention sold him Boo, the miniature giant space hamster, in an apparent scam. Tofer ran with it, declaring that Boo would be Minsc’s animal companion, and holding one-sided conversations with the confidant that lived in his pocket. “In my campaign he was just a hamster,” Ohlen says. “I always thought of him as just a hamster.”
In another interview on the Beamdog blog, Cameron Tofer (formerly BioWare, founder of Beamdog), who apparently played that first incarnation of the bald good-hearted ranger, tells a little about these sessions:
“[Minsc] was based off some horrible rolls - his Intelligence, I just fumbled that. And because I joined late I was a couple of levels behind everyone.”
It was this level disparity that inspired one of the most adored animal companions of all time. “[Minsc] was knocked out in the first round of pretty much every fight,” Tofer says. “When I finally levelled up enough to get a companion, that's how I got Boo.”
“Eventually when we started leveling up, Minsc was pretty badass,” Tofer recalls. [..] “It was just Minsc lying unconscious and Boo running around doing everything.”
Minsc and Boo had “some kind of mental link” says Tofer, which made it easier to play the hamster. Perhaps Boo is the one influencing Minsc in Baldur's Gate.
But being a ranger, Minsc has a magical bond with Boo due to his class's Animal Companionship, so this doesn't really say anything about the nature or origins of Boo.
And, finally, on the original character sheet for Minsc, we can see how Boo is a "hamster":
For completeness' sake, I will also note that miniature giant space hamsters do exist in the Forgotten Realms, but were introduced, it seems, only in 2013, in the 5th Edition Encounters adventure Murder in Baldur's Gate.
I'm not sure this is considered canon in the first place.