Has any canon or (more likely) Expanded Universe source suggested a familial link between them?
There's no indication in any of the films, TV shows or other media that she's related to the Montgomery Scott.
Kirk and Picard briefly meet up in the EU Novel Star Trek: Preserver. Tryla Scott is mentioned. Kirk doesn't know who she is, nor is any mention made of her relative being someone that Kirk knows intimately.
“Focusing only on Starfleet command appointments made in the past twenty years,” Savrin answered, “we have identified thirty seven orders generated with a lack of traceable authorization codes.”
“Are you familiar with Captain Tryla Scott?” Picard asked Kirk.
“She broke your record for youngest starship captain. Her orders can’t be traced.”