My Almost-Perfect Spam Filter

Try Blocking Spam With Gmail (it works for any inbox)

According to Statista, a global research firm that provides market and consumer data, the ratio of legit emails to spam is improving. In the beginning of 2011, spam accounted for 80 percent of all emails. Today, that number has decreased by almost half, but that's still not low enough. One factor in that decline is machine learning, an application of artificial intelligence, which has been part of Gmail’s spam filter program since its inception.

Every time a user clicks the “Report Spam” or “Not Spam” button on a message, Gmail learns something that helps it filter spam better. But Google is also using AI technology to help Gmail learn on its own, without the user’s clicks.

The Gmail spam filter uses an artificial neural network (ANN) that can detect and block the “especially sneaky” spam that sometimes slips past users’ radar. What’s an “artificial neural network,” you ask? The oversimplified answer is, a lot of computers connected to each other in an attempt to simulate the interconnections of human brain cells. It’s expensive to build an artificial neural network; most are tiny, with 1 to 10 million connections.

Google has built an ANN with over one billion connections using the processors in its vast and far-flung empire of data centers. That sounds awesome, until you learn that the human brain contains several TRILLION connections! Nonetheless, Google’s ANN is capable of rudimentary human-like self-learning. That means you don’t have to teach it what spam is, because it's trained to understand the context and nuances of language, making it highly effective at distinguishing spam from legitimate messages.

In one experiment, Google’s ANN was fed millions of still images of cats culled from YouTube videos. The images were not labeled as “cats” and programmers did not tell the ANN what a “cat” is. The ANN figured it out for itself, learning to recognize cats in virtually any image. If it can do that, it can recognize spam with greater accuracy than most humans can. In 2019, Google announced that thanks to their machine learning framework, they were able to block an additional 100 million spam messages daily. With recent advances in AI, one might assume that number is higher (especially if the messages have anything to do cats).

Unidentified Frying Objects?

But the spam-eating technology is still not perfect. Just as humans see UFOs in pictures of streetlights, Gmail’s filter sometimes sees spam in legitimate messages and mistakenly consigns them to the spam folder. Google claims that only about 0.01 percent (1 in 10,000) of legitimate emails are falsely labeled as spam these days. As the system learns more about email, that figure should fall even further.

Take a look in your Gmail spam folder, and see how it's working. If you do find certain messages are being incorrectly flagged as spam, you can create a Gmail filter for them, with a "Never send to Spam" action, so they'll be delivered to your inbox.

On the flip side, occasionally, I do get a batch of spammy messages that recur over a number of days. Recently I was getting several emails daily emails about investing in bitcoin. It was clearly spam, but perhaps because of other tech-related stuff that I do read, Gmail thought otherwise. I created a filter to send them to the Trash, problem solved.

How does Gmail’s spam filter compare to other free Webmail services, such as Microsoft’s Outlook.com and Yahoo? I haven’t been able to find any independent, rigorous studies of this question. I’ll just note that Yahoo seems rather overzealous about spam filtering, and quite a few of my outgoing messages to Yahoo accounts end up wrongly labelled as spam, or just don't get delivered at all. If you have any relevant data on this, please do share.

In addition to flagging spam based on the content of the message, Gmail, Yahoo, and other popular webmail services assess the reputation of the sender's IP address and domain name. Emails from known spam sources and messages lacking proper authentication are flagged as potential spam. [Tech note: Techniques such as SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), and DMARC (Domain-based Message Authentication) are used to verify the authenticity of the sender. Messages that fail these authentication checks are more likely to be considered spam or phishing attempts.]

Another spam-fighting technique that Gmail employs doesn’t really tackle spam, in my opinion. Gmail attempts to learn individual users’ reading preferences and filter out even legitimate mail that goes unread. The company gives the example of email newsletters. One user reads them assiduously while another just lets them pile up unread, or deletes them without reading. Why people don’t just unsubscribe from newsletters is one of life’s mysteries. Hopefully, Gmail can distinguish between legit opt-in email newsletters and the ones that just arrive without your consent.

Finally, Gmail is providing help for legitimate, large-volume email publishers whose communications are sometimes wrongly condemned as spam. The Gmail Postmaster Tools will help airlines, banks, credit card companies, and other well-known firms analyze the fate of their emails and improve their delivery and reading rates. That’s a good thing, because more and more large companies are encouraging customers to receive their monthly statements by email, rather than paper copies in the mail.

Even if you don't use or like Gmail, you can still use it to filter spam. Here's one technique that some people use to "pre-filter" their incoming emails. Instead of providing your actual email address when asked, give out a Gmail address that you've created. Configure that Gmail account to simply forward everything to your actual address. Gmail does spam-filtering BEFORE forwarding, so the messages that do get forwarded are virtually spam-free.

Gmail’s spam filter is so reliable and accurate that I hardly ever check my spam filter for false positives anymore. I get hundreds of emails daily, and I rarely find myself clicking on the “Report Spam” button. The spam filter just works.

Do you use Gmail? How is the spam filtering working for you? Your thoughts on this topic are welcome. Post your comment or question below.