It definitely appears to have originated in the late 19th century as more of a medical term, at least according to the OED listings. Since it seems to align with the earlier, slangier "molly" of the 18th/early 19th centuries, though, in terms of the idea of gay men/sodomites as feminine, it seems vaguely plausible that it was an earlier term/idea. Though if it was a medico-legal term earlier on it seems that it would've turned up in the OED as such, and if it wasn't it seems unlikely the medical establishment would've co-opted a slang term.

(Really, I think the novel was just being anachronistic in having its early 19th-century characters use it, but I didn't want to just write it off reflexively, I suppose.)