I'm just jumping in to mention that Reddit is officially beta-testing a modmail abuse filter, from what I know it detects keywords much like a bot would. Could wait for that to come out or apply to beta test it, which would be the same as what you want. Only issue is that it isn't as customizable but it's worked fairly well for modmails on the subs I mod.

If you're curious: https://www.reddit.com/r/modnews/comments/r61d8f/join_the_modmail_harassment_filter_beta/