Why the email you're urgently waiting for doesn't arrive

There are emails that can arrive whenever they please.

The newsletter from the shop where we bought a screw in 2019, for instance, can quite happily turn up next Tuesday. The unmissable offer to renew an antivirus we have not owned since 2014 may also take its time.

Then there are the other emails.

The code you need to log in. The booking confirmation. The client's message that has to arrive now. The password-reset email while you are sitting in front of the screen, browser open, one increasingly nervous finger already resting on F5.

And that is precisely when email decides not to arrive.

Five seconds.

Refresh.

Ten seconds.

Refresh.

One minute.

Check spam.

Nothing.

At this point, modern man — who until three minutes ago regarded email as one of the fundamental infrastructures of Western civilisation — begins to wonder whether it might have been wiser to entrust the message to a pigeon.

Quite often, behind all this, there is a charming little invention called greylisting.

First came the good and the bad

In the beginning, things were relatively simple.

There were white lists: addresses or servers we trusted.

And there were black lists — pardon, these days we prefer block lists — containing those it would be prudent not to trust.

The principle was the classic one used at private parties.

Mr Rossi is on the list: do come in.

Mr Spammer is on the wrong list: terribly sorry, sir, but I am afraid it will not be possible this evening.

Simple.

Until someone noticed that spammers had the tiresome habit of changing IP addresses, servers and identities with the same casual ease with which a politician changes parties.

Something subtler was required.

Enter Evan Harris

In 2003, Evan Harris proposed a new anti-spam technique and called it greylisting.

The idea was almost childishly simple.

When an unknown server tries to deliver an email to us for the first time, our server does not say:

“No.”

It says:

“Not now.”

Technically, it returns a temporary SMTP error from the 4xx family.

And here we come to one of those small rules of good manners on which the Internet rests.

A proper mail server, when it receives a temporary error, must try again later.

So the sending server goes away, waits a while and comes back.

“Good evening. It's me again. I have this email for Mr So-and-so.”

“Ah, certainly. Do come in.”

And the message gets through.

Greylisting normally identifies the attempt using a combination of information such as the sending server's IP address, the sender and the recipient. The first time that combination appears, it is unknown and is temporarily rejected; when it reappears after the expected interval, it is treated as more trustworthy.

Why should this odd little ritual stop spam?

Because when the technique was conceived, many systems used to send spam operated according to a rather different philosophy.

They had millions of addresses to write to.

If a server replied “try again later”, they often did not try again at all.

They simply moved on to the next one.

A legitimate mail server, on the other hand, disciplined and respectful of SMTP etiquette, came back.

It was a sort of test of perseverance.

If you come back, you are probably a respectable person.

Or at least the server of a respectable person.

And one has to admit that, from a system administrator's point of view, there is a certain elegance to it.

There is no need to analyse the contents of the email.

No need to decide whether “EARN €50,000 WORKING FROM HOME!!!” is, on balance, a suspicious commercial communication.

You do not even have to accept the message.

You simply shut the door politely in its face and see whether it comes back.

The problem is that your email comes back too

And now we reach the interesting part.

Suppose you have just requested a password reset from a website.

The site generates the email immediately.

Its SMTP server immediately tries to deliver it.

And your server replies:

451 – Temporary failure. Try again later.

From a technical point of view, absolutely nothing abnormal has happened.

The system is working exactly as designed.

From your point of view, however, the Internet is broken.

Because the sending server is under no obligation to try again ten seconds later.

It may do so after a few minutes.

Or later still, depending on the configuration of its SMTP queue.

And there is the absurdity.

We have built an extremely sophisticated system capable of moving information from one side of the planet to the other at practically the speed of light.

Then, to make sure that the communication is legitimate, one computer says to the other:

“Come back later.”

When five minutes are five minutes too many

For an ordinary business email, the problem is almost invisible.

If someone writes to me at 10:03 and the message arrives at 10:08, in all likelihood nobody will notice.

But over the last twenty years we have begun using email for operations with entirely different characteristics.

Password resets.

Magic links.

Registration confirmations.

Verification codes.

Bookings.

Automated alerts.

Messages we are not vaguely expecting “today”.

We are expecting them now.

And this is where a delay that is perfectly acceptable for traditional email becomes a dreadful user experience.

The user does not know that somewhere, two SMTP servers are performing an elaborate ceremony of mutual recognition.

All the user knows is that the screen says:

We have sent you an email.

And the email is not there.

Things, naturally, become more complicated

Like every ingenious system invented to tell the good from the bad, greylisting eventually had to deal with the fact that the bad learn.

Spammers can retry deliveries.

Large providers, meanwhile, may send later attempts through different servers and therefore from different IP addresses.

That can complicate recognition of the sender if the greylisting system uses the IP address too rigidly as part of its identity.

In other words, the butler we stationed at the door to recognise undesirable guests has become a little too zealous.

“Do you know the master of the house?”

“Yes.”

“Have you been here before?”

“Many times.”

“Do you have identification?”

“Certainly.”

“Splendid. Come back in five minutes.”

So is greylisting a bad idea?

Not necessarily.

It is a clever, inexpensive and historically effective technique.

But it highlights something about email that we tend to forget:

SMTP is designed to be resilient, not instantaneous.

Email was built around the idea of store and forward.

If delivery does not work now, try again.

If a server does not answer, wait.

If there is a temporary problem, the message stays in the queue.

It is precisely this stubbornness that makes email extraordinarily robust.

And, rather ironically, it is the same stubbornness that allows greylisting to work.

The reliable server is rejected and thinks:

“Never mind. I'll come back.”

The problem is that we, sitting on the other side of the screen, do not possess the same patience.

We have just clicked:

SEND ACCESS CODE

and eleven seconds later we have already concluded that the entire global email infrastructure has been administered by incompetents.

So the question remains: is that little delay worth putting up with?

Historically, the answer is rather embarrassing.

In some measurements on real mail servers, greylisting managed to stop between 85 and 95 percent of spam before it ever had to be examined by more sophisticated filters. And in 2022, almost twenty years after its invention, with spammers by then perfectly well aware of the trick, a USENIX study still measured a reduction in spam of around 50 percent.

Not bad for a system that does not read the message, does not analyse the words, does not consult ChatGPT, does not assign scores and does not attempt to determine whether the Nigerian prince genuinely has urgent need of our bank details.

It simply says:

“Come back later.”

Proper mail servers come back.

Many spammers do not.

Naturally, we are the ones who pay the price for this brilliant exercise in natural selection, precisely when we have just clicked SEND ACCESS CODE and, seventeen seconds later, are hammering F5 with all the composure of a British gentleman who has just been informed that the bar has run out of gin.

But one must give greylisting its due: for an anti-spam system that, in essence, does nothing but waste time, it has wasted a remarkable amount of the spammers' time.