Every guide to email security starts with SPF, DKIM and DMARC. Those three answer the question "is this message genuinely from the domain it claims?" None of them answer a different and equally important question: "was this message readable by anyone in between?"
That question belongs to transport encryption, and by default SMTP handles it badly. This article covers the three DNS-published mechanisms that fix it — MTA-STS, DANE and TLS-RPT — what each one actually enforces, and how to publish them without breaking inbound mail.
Why opportunistic TLS is not enough
When one mail server delivers to another, the sending server issues EHLO, reads the capability list, and if it sees STARTTLS it upgrades the connection. This is opportunistic TLS, defined in RFC 3207, and it is how the overwhelming majority of inter-server mail is encrypted today.
It has two structural weaknesses.
The capability advertisement is unauthenticated. An attacker positioned between the two servers can strip STARTTLS from the capability list. The sending server sees no encryption on offer, shrugs, and delivers in plaintext. Neither end logs an error, because as far as both are concerned nothing went wrong.
The certificate is usually not validated. Even when TLS is negotiated, most sending MTAs do not verify that the certificate matches the hostname or chains to a trusted root. Postfix’s default smtp_tls_security_level = may explicitly does not validate. A self-signed certificate presented by an attacker is accepted without complaint.
The result is that "we use TLS" is a statement about the happy path, not a guarantee.
MTA-STS: publishing a policy over HTTPS
MTA-STS (RFC 8461) lets a domain publish a policy stating: my mail servers are these hostnames, they support TLS, and their certificates are valid — so refuse to deliver to me any other way.
It has two moving parts.
A TXT record tells senders a policy exists and gives it a version identifier:
_mta-sts.example.com. IN TXT "v=STSv1; id=20260824120000"
The id is an opaque string that you change whenever you change the policy. Sending servers cache the policy and use the id to detect that a refresh is needed, so if you do not bump it, senders will keep using the cached copy until max_age expires.
The policy itself is served over HTTPS from a dedicated hostname:
https://mta-sts.example.com/.well-known/mta-sts.txt
with content type text/plain and a body like this:
version: STSv1
mode: enforce
mx: mx1.example.com
mx: mx2.example.com
max_age: 604800
Three things matter here.
mode takes testing, enforce or none. Start at testing — in that mode, a sender that cannot establish validated TLS delivers anyway and reports the failure via TLS-RPT. Only move to enforce once your reports are clean.
mx must list every hostname that appears in your MX records. Wildcards are permitted for a single label (*.example.com matches mx1.example.com but not mx1.mail.example.com). Miss one and mail to that MX will be refused once you are in enforce.
max_age is in seconds and is how long senders may cache the policy. The specification permits up to 31557600 (a year). A week (604800) is a sensible production value — long enough to be useful, short enough that a mistake ages out.
The security model rests on the HTTPS certificate for mta-sts.example.com. That certificate must be valid and publicly trusted, and the policy must be fetched without following a redirect to a different host. If an attacker cannot forge that certificate, they cannot forge the policy.
The practical trap
The most common MTA-STS failure we see is an expired certificate on the mta-sts hostname. Nobody visits it in a browser, so nobody notices. Whether senders then fall back to unvalidated TLS or keep using the cached policy depends on the sender’s implementation and where you are in the cache window — either way, you have quietly lost the protection you thought you had. Put that hostname in the same certificate monitoring as your main site.
DANE: pinning the certificate in DNS
DANE (RFC 7672 for SMTP) takes a different approach. Rather than publishing a policy over HTTPS, it publishes the expected certificate — or a hash of it — directly in DNS, and relies on DNSSEC to make that publication tamper-evident.
A TLSA record for a mail server on port 25 looks like this:
_25._tcp.mx1.example.com. IN TLSA 3 1 1 (
a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90 )
The three numbers are usage, selector and matching type:
- Usage 3 — "DANE-EE": the certificate presented must match this record directly. No chain to a public CA is required. This is the usual choice for SMTP.
- Selector 1 — the record covers the SubjectPublicKeyInfo, not the whole certificate. This means renewing the certificate with the same key pair does not require changing the TLSA record.
- Matching type 1 — the value is a SHA-256 hash.
DANE has one hard prerequisite: DNSSEC must be signed and valid for the zone containing the TLSA records and for the zone containing the MX records. Without DNSSEC, the TLSA record can be forged in transit and the whole mechanism is worthless. Sending servers check this, and will treat an unsigned zone as "no DANE".
The corresponding trap is the rollover. If you replace a certificate key without publishing the new TLSA record first, every DANE-validating sender starts refusing your mail immediately. The standard practice is to publish the new TLSA record alongside the old one, wait longer than the record’s TTL, deploy the new certificate, and only then remove the old record.
MTA-STS or DANE?
They solve the same problem with different trust anchors, and they are not mutually exclusive. A domain can publish both, and several large providers do.
DANE is cryptographically stronger — it does not depend on the public CA system — but it requires DNSSEC, which means your registrar, your DNS provider and your zone all have to support it correctly. Microsoft 365 supports DANE for outbound and inbound mail; Google has been notably slower.
MTA-STS requires no DNSSEC but does require you to operate an HTTPS endpoint reliably. Gmail honours MTA-STS.
Our general recommendation for a UK business domain: publish MTA-STS first, because it works with the largest senders today and has fewer prerequisites. Add DANE if your zone is already DNSSEC-signed, because at that point the marginal effort is small.
TLS-RPT: finding out when it breaks
Both mechanisms are fail-closed. In enforce mode, a misconfiguration does not degrade to plaintext — it stops mail. That makes reporting essential rather than optional.
TLS-RPT (RFC 8460) is a single TXT record:
_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@example.com"
Participating senders will then post you a daily JSON report summarising every connection attempt to your domain: which MX, which policy they applied, how many sessions succeeded, and — the useful part — a breakdown of failures by type, with the failing IP addresses.
The failure types you will actually see are starttls-not-supported, certificate-expired, certificate-host-mismatch, validation-failure and sts-policy-fetch-error. Each maps directly to a fixable problem. sts-policy-fetch-error, for instance, almost always means the HTTPS endpoint is down or its certificate has lapsed.
Reports arrive gzipped and are not pleasant to read by hand. Point rua at a mailbox you actually monitor, or at a service that parses them into a dashboard.
A sane rollout order
- Make sure every hostname in your MX records has a valid, publicly trusted certificate that matches the hostname. This is the prerequisite for everything else, and it is where most domains fail.
- Publish TLS-RPT. Do this first, so you have visibility before you have enforcement.
- Stand up
mta-sts.example.comover HTTPS and publish the policy atmode: testing, with the_mta-stsTXT record. - Watch the TLS-RPT reports for two to four weeks. Fix anything that appears.
- Change the policy to
mode: enforceand bump theid. - If your zone is DNSSEC-signed, add TLSA records for each MX, using selector 1 so that certificate renewals with a stable key do not require DNS changes.
The order matters. Publishing enforce before you have reporting is how you find out about a certificate mismatch from a customer rather than from a report.
FXRM’s email hosting ships with validated TLS on every MX, MTA-STS and TLS-RPT configured as standard, and DNSSEC available on our nameservers. See our email hosting →
