Email Security Gateways and Phishing Defense: Four Layers from Authentication to Process

Email Security Gateways and Phishing Defense: Four Layers from Authentication to Process

For most teams the first move in email security is to configure SPF, DKIM, and DMARC. Done correctly, that step blocks someone from sending under your brand name to your customers. Yet almost every risk an employee faces sits on the other side: someone impersonating a colleague, a supplier, or a bank and writing to people inside the company. The first problem is solved by outbound configuration. The second exists only because of inbound protection, and the two never use the same technology.

What makes inbound defense hard is the shape of its payoff. An attacker needs to succeed once; the defender needs to succeed every time. A gateway should therefore not be positioned as something that blocks all threats, but as something that removes cheap attacks entirely, downgrades the suspicious into quarantine, and hands the rest to process. Understanding that division of labor matters more than comparing feature lists.

The order below follows the layers from inside out: classify the attack surface first, then what each of the four layers solves, then authentication, reputation and intelligence, content and links, and identity and process in turn, and finally selection criteria and an incident response checklist. For the record level detail of email authentication, see SPF, DKIM, and DMARC configuration.

1. More Than One Kind of Attack Surface

1.1 Credential Phishing, Malicious Attachments, and Malicious Links

These three carry the most volume and the most mature tooling, and they share one property: there is a detectable carrier. A link pointing at a fake login page, a document carrying a macro, an executable inside an archive. They can be detected because the carrier itself has content or reputation features.

Credential phishing aims to walk a person onto a page that looks like a login screen and collect the password. Malicious attachments aim to run code on the local machine. Malicious links sit between the two, sometimes serving the phishing page directly, sometimes redirecting to a download. All three share one weakness: the victim has to take an action, and that action usually happens after the click. That is precisely why time-of-click detection earns its place.

1.2 Internal Sending After an Account Takeover

Harder than external phishing is a message that genuinely comes from an internal account. The attacker signs in with stolen credentials and then writes to colleagues, customers, and the finance team from that account. The message passes every authentication check: right domain, right IP, valid DKIM signature, aligned DMARC, because the sending is legitimate in every mechanical sense.

At that point any judgment based on whether the sender domain is trustworthy stops working. The only remaining signals are behavioral: who this account usually writes to, when it writes, how often, whether it suddenly sends to a large external list, whether it signed in from an unusual location or device. That is work for the identity layer, not the authentication layer.

1.3 Why BEC Is the Hardest to Stop

Business email compromise has three absences: no malicious link, no attachment, and nothing unusual about the language. The content is usually along the lines of a request to arrange a payment today before paperwork catches up, or a notice that the bank account has changed and here are the new details. There is no technical feature anywhere in the message to detect, because its attack surface is human judgment rather than software.

Technology is close to useless here. What works is process: any change of payment details must be confirmed by calling back a number already known from the contract, large payments require dual approval, and there needs to be a hard rule that a change of payee account is never accepted by email. These are not product features. They are part of the finance control environment, and they do more against BEC than any filter.

One detail is routinely missed. BEC usually opens with a payload-free reconnaissance message, such as a bare check on whether someone is at their desk or a question about who owns a project, used to confirm the mailbox is live and to sample the signature and tone. Those messages look clean in any gateway, but they are valuable evidence: the same sender probing first and making a demand days later is a pattern a behavioral rule can catch.

2. What Each of the Four Layers Solves

The four layers are not substitute products. They form a depth in which each layer removes something different and whatever slips through is caught by the next one.

2.1 What the Layers Contribute

The authentication layer answers whether this message truly came from the domain it claims. The reputation and intelligence layer answers whether that sending domain and IP have a record and how new they are. The content and link layer answers what happens if a link is clicked or an attachment is opened. The identity and process layer answers whether the account currently behaves like its owner.

Mixing the outputs of all four into a single block-or-allow switch is the usual reason a gateway underperforms. The better design keeps each verdict explainable and graded: pass, tag, quarantine, reject. A user in the quarantine queue who can read why a message was held learns more from that reason than from an aggregate score.

3. Authentication Judges Forgery, but Only of Your Domain

3.1 What DMARC Covers and What It Does Not

DMARC judges alignment: is the domain of whichever check passed, SPF or DKIM, the same as the domain in the From header. When an attacker writes your domain into From and fails your authentication, DMARC reports failure and the receiver applies the p= policy. That is anti-spoofing.

Three things sit outside it. First, lookalike domains: examp1e.com, example.co, and exampIe.com are not your domain, and DMARC has no jurisdiction over them. Second, sending after a legitimate account is taken over, which passes everything. Third, display name deception, where the From display name reads like the CEO while the actual address is an external free mailbox, mechanically legitimate at the authentication level. The third is the cheapest and most common, and receivers can only respond with heuristics such as flagging an external address carrying an internal display name. For alignment modes and spec evolution, see the DMARC specification update.

3.2 Lookalike Domains and Homoglyphs

Homoglyph attacks exploit characters that look alike: the Latin l and the digit 1, the Cyrillic а and the Latin a, and the display differences that internationalized domain names introduce after encoding. The defensible move is to enumerate and monitor in advance rather than judge after the fact.

import itertools
import os

brand = os.environ["BRAND_DOMAIN"].split(".")[0]
tld = os.environ.get("BRAND_TLD", "com")
homoglyphs = {"a": "а", "e": "е", "o": "о", "p": "р", "c": "с", "x": "х"}

variants = set()
for i, ch in enumerate(brand):
    if ch in homoglyphs:
        variants.add(brand[:i] + homoglyphs[ch] + brand[i + 1:])
    if ch in "lo":
        sub = "1" if ch == "l" else "0"
        variants.add(brand[:i] + sub + brand[i + 1:])
for a, b in itertools.combinations(brand, 2):
    variants.add(brand.replace(a, "", 1))
print(sorted(f"{v}.{tld}" for v in variants))

The script is a minimal demonstration of the enumeration idea. A real program also inserts hyphens, concatenates words, and substitutes top level domains, and it feeds the results into registration and resolution monitoring rather than running a one-off check. For domain asset ownership as a whole, see the sections on unified domain management in business email service selection.

4. The Reputation and Intelligence Layer

4.1 Domain Age and First-Seen Senders

A newly registered domain has no history, so it cannot be judged by whether it sent spam in the past. Inverted, the absence of history is itself suspicious in a business context, because your supplier did not register its domain yesterday. Domain registration age is therefore a cheap and effective negative signal: the newer the domain, the lower the score, compounded by whether that sending domain has ever appeared in the organization history.

First appearance is a signal too, with two sides. A genuine new partnership, a newly connected SaaS notification stream, or a customer who changed domains all trigger first appearance. The reasonable response is not silent rejection but a warning banner, a quarantine entry, or a request for confirmation.

4.2 IP Reputation and Threat Intelligence Matching

IP reputation reflects the past behavior of the sending IP: presence on blocklists, complaint history, and whether it belongs to a hosting range that is heavily abused. Its limitation is that it is shared. One large mail provider egress IP carries thousands of senders, and one bad customer damages the reputation of the whole range, which produces false positives and false negatives at the same time.

Threat intelligence matching compares against known malicious infrastructure: known phishing domains, known malicious file hashes, known command and control addresses. Its value is speed and its hit rate on attacks that are already circulating. Its weakness is blindness to new infrastructure. This layer removes the known at low cost; it cannot be the primary judge.

5. The Content and Link Layer

5.1 Link Rewriting and Time-of-Click Detection

The older approach fetches links at delivery time and compares them. The problem is that an attacker can time the payload: the page is clean when the message arrives and turns into a phishing page hours later, or it returns different content based on user agent or geolocation so that a sandbox sees a clean page while a real recipient sees the fake.

Link rewriting moves the decision to the moment of the click. Links in the message are rewritten to a gateway domain, and the verdict and redirect happen in real time. The cost is privacy and usability: the gateway observes every click, and the link no longer looks like the original domain in the message body. The availability of the rewriting path also has to be guaranteed, because a gateway hiccup becomes company-wide inability to open mail links. Both trade-offs need assessment, not just detection rates.

5.2 Sandbox Detonation and Macro Blocking

Attachment sandboxing opens the file in an isolated environment and observes behavior: outbound connection attempts, writes to autostart locations, spawned child processes. That catches samples whose static features look harmless, at the cost of latency ranging from tens of seconds to minutes, so time-sensitive mail needs a separate policy.

Macros and scripts are simpler and more effective to handle: block Office macros by default, block executable attachment types, and block password-protected archives, which exist to bypass content inspection. Blanket rejection of this kind has low false positive rates and high payoff, provided the business confirms no legitimate workflow depends on macros.

6. The Identity and Process Layer

6.1 Anomalous Sign-In and Internal Sending Baselines

Anomalous sign-in is the layer existing identity systems already cover: impossible travel, a new device, a new region, an unusual hour. The action that matters after detection is not sending an alert but being able to revoke sessions. Many incidents stall exactly here: the password is reset while the token the attacker holds remains valid.

Internal sending baselines are the signal unique to mail. An account that normally writes to a handful of colleagues suddenly writes to the whole company. An account that never sends externally suddenly sends attachments to many external addresses. An account creates forwarding rules in bulk at three in the morning. None of these looks abnormal in a single-account view; all of them are glaring in a baseline view.

6.2 The Three Backdoors Most Often Missed

1. Automatic forwarding rules. The first thing an attacker often does after taking an account is create a rule forwarding everything to an external address, and from then on the mailbox keeps leaking while the user notices nothing. The audit is to export every account forwarding rule periodically and compare it against an approved list.
2. Abuse of OAuth application grants. A user is persuaded to authorize a third-party application to read mail, and afterwards the attacker reads the mailbox through the application interface without a password and without triggering a sign-in alert. The audit is to review third-party grants on the enterprise side and treat mail access by an unapproved application as high risk.
3. Hidden inbox rules. Subtler than forwarding: messages from specific senders, such as a bank, a security alert, or HR, are moved automatically to deleted items or an unread folder. No outbound traffic is generated, so traffic monitoring never sees it, and only rule auditing finds it.

All three share persistence. They do not depend on a second successful phish; one success stays resident. The response procedure must therefore include clearing persistence, or the password reset leaves the account still compromised. For the security boundary of OAuth grants themselves, see API security with OAuth and JWT.

6.3 Hard Rules for the Payment Process

The only approach proven against BEC moves the judgment away from reading email and toward verifying through a known channel. Three rules are implementable. A change of payee account must be confirmed by calling back the number already recorded in the contract, never a new number supplied by email or chat. Payments above a threshold require two approvals by people who are not in the same reporting line. First payments to a new supplier require a cooling-off period.

Policy only takes effect when it is technical: make the callback a mandatory field in the workflow system and make dual approval a hard system constraint rather than a line in a document that depends on goodwill. A rule that can be bypassed once will fail on the attempt that matters.

7. Selection Criteria and a Response Checklist

7.1 Selection Criteria and the Limits of Tools

Four dimensions matter: depth of integration with the identity provider, because sign-in risk and mail risk need to be correlated; whether an API exists beyond inline gateway deployment, since many response actions need to be scripted; the false positive workflow, meaning whether quarantine release is self-service and the approval chain is clear; and visibility into internal-to-internal mail, because many gateways look only at inbound external messages and ignore internal traffic entirely.

The limits need stating as well. Tools do not replace human judgment, and training and process are not optional. A gateway is also no substitute for DMARC, because they work on different layers: DMARC protects your brand from being used by someone else to send, and the gateway protects your employees from being deceived by messages someone else sends. Treating them as two implementations of one thing is the most common procurement error. A selection decision is usually followed by a mailbox migration, and the risk control for MX cutover day is covered in business email migration practice.

7.2 Six-Hour Incident Response Checklist

1. Minutes 0 to 30: stop the bleeding. Revoke all sessions and refresh tokens for the account, force a password reset and re-enrollment of the second factor, and disable suspicious third-party grants.
2. Minutes 30 to 60: clear persistence. Export and inspect forwarding rules, inbox rules, signatures, auto-replies, and delegated permissions; delete external forwarding and record what was there.
3. Hours 1 to 2: establish blast radius. Pull the last 30 days of sign-in logs, sent messages, and rule change history for the account; confirm whether anything left the organization.
4. Hours 2 to 3: block lateral movement. Check whether a cohort of accounts was hit by the same phishing page, look for related suspicious sign-ins, and restrict the affected accounts if needed.
5. Hours 3 to 4: communicate. Notify customers and partners who may have received forged mail, especially any correspondence involving payment details, and settle on one consistent message.
6. Hours 4 to 6: review and harden. Identify the entry point, whether a phishing page, password reuse, or a third-party grant, add the corresponding detection rules and process constraints, and turn the detectable features of this incident into alerts.

The conclusion compresses to one sentence: the authentication layer defends against others impersonating you, the content layer against malicious carriers, the identity layer against abuse of legitimate accounts, and the process layer against fraud with no technical feature at all; remove any one layer and attacks aimed at that layer get through. For infrastructure hardening, see the server hardening checklist; for parsing inbound mail safely, see inbound email processing automation; and for reputation governance on the sending side, see transactional and marketing email separation.

Source: 16IDC original. Protocol specifications, platform behavior, and compliance criteria follow the relevant IETF RFCs, regulatory requirements, and provider documentation.