Six steps to (mostly) perfect email sorting

  1. Make sure you’re moving Inquiries out of Undecided, whether they’re spam or not. The spam filter’s decision process is trying to figure out not just whether something is spam or not, but where to put it other than Undecided. If you are closing cases in Undecided, or changing them to a different category, then you’re not training the filter with those cases. Autosort will only leave training mode and start sorting once you have moved 15 Inquiries out of Undecided and into an Area other than Spam.
  2. Close Autosorted Inquiries in the project and area they’re sorted into. If you move the case out of the project, FogBugz Autosorter loses track of it. This is not ideal operation, but is due to the need to support multiple Autosort projects.
  3. Limit the number of Autosort areas to five or six per project. This kind of sorting takes a lot of computational resources. Deciding between seven areas is about three times as intense as deciding between four areas. Fewer choices also give you better sorting.
  4. Have an “Undeliverable” area. Bounces from email servers and out-of-office auto-replies will create new cases. Because they’re so predictably structured, though, it’s easy to get the sorter trained to sort these out into their own area. We use “Undeliverable” because it sorts nicely to the bottom of the Inbox filter.
  5. Turning on Autosort will create “Spam”, “Not Spam”, and “Undecided” areas. If there are already areas with these names in the project, then Autosort may encounter errors. Be sure to rename any such areas before turning on Autosort for a project.
  6. Add areas slowly and organically. (This is taken from Spooky’s answer.) Your first order of business should be to train spam vs. nonspam. Once you have mail reliably being sorted into Not Spam, you can add areas one by one to segment out certain types of mail. The more areas you have, the less accurate it will be. Also, it almost never works to send emails that you think will look like the emails you intend to receive. The signals that the sorter listens to go far beyond the actual text content of the email, so it’s really not worth your while to send 100 messages with your product name into the Inbox and sort them to “pre-train” the filter. Rather, just open it up and train it naturally. It will respond very quickly to things like web form submissions and auto-generated emails.

Details on how the auto-sorter is trained

Nitty-gritties: how training works