During the confrontation between Malfoy and Dumbledore right before the latter's death at the end of Half-Blood Prince, they discussed how Malfoy had been communicating with Madam Rosmerta:
Tell me, how have you been communicating with Rosmerta? I thought we had all methods of communication in and out of the school monitored.”
“Enchanted coins,” said Malfoy, as though he was compelled to keep talking, though his wand hand was shaking badly. “I had one and she had the other and I could send her messages —”
“Isn’t that the secret method of communication the group that called themselves Dumbledore’s Army used last year?” asked Dumbledore. His voice was light and conversational, but Harry saw him slip an inch lower down the wall as he said it.
“Yeah, I got the idea from them,” said Malfoy, with a twisted smile.
Apparently, both Dumbledore and Malfoy were aware of the DA's secret method of communication. How did they find out about this?
While it is possible that Dumbledore knew of the existence of the DA because the original meeting in the Hog's Head was overheard (as argued in the answers to this question), the coin method of communication wasn't introduced until the fourth meeting, which was not overheard, as documented in Chapter 19 of Order of the Phoenix (my emphasis):
Hermione soon devised a very clever method of communicating the time and date of the next meeting to all the members in case they needed to change it at short notice, because it would look so suspicious if people from different Houses were seen crossing the Great Hall to talk to each other too often. She gave each of the members of the D.A. a fake Galleon (Ron became very excited when he saw the basket at first, convinced that she was actually giving out gold).
“You see the numerals around the edge of the coins?” Hermione said, holding one up for examination at the end of their fourth meeting. The coin gleamed fat and yellow in the light from the torches. “On real Galleons that’s just a serial number referring to the goblin who cast the coin. On these fake coins, though, the numbers will change to reflect the time and date of the next meeting. The coins will grow hot when the date changes, so if you’re carrying them in a pocket you’ll be able to feel them. We take one each, and when Harry sets the date of the next meeting he’ll change the numbers on his coin, and because I’ve put a Protean Charm on them, they’ll all change to mimic his.”
It is unlikely that they found out through Marietta's betrayal, because Umbridge said that Marietta stopped talking right after mentioning that there would be a meeting that night:
“Oh, very well, you silly girl, I’ll tell him,” snapped Umbridge. She hitched her sickly smile back onto her face and said, “Well, Minister, Miss Edgecombe here came to my office shortly after dinner this evening and told me she had something she wanted to tell me. She said that if I proceeded to a secret room on the seventh floor, sometimes known as the Room of Requirement, I would find out something to my advantage. I questioned her a little further and she admitted that there was to be some kind of meeting there. Unfortunately at that point this hex,” she waved impatiently at Marietta’s concealed face, “came into operation and upon catching sight of her face in my mirror the girl became too distressed to tell me any more.”
(It's possible that Marietta could have mentioned it earlier in the conversation, but that doesn't seem to be the type of information that would be the first thing to tell about.)
Additionally, during the entire confrontation in Dumbledore's office, Umbridge made no mention of this communication method.
So, to reiterate, how did Dumbledore and Malfoy come to know about the coin method of communication?