"TLS handshake failed"

When sending mail to a remote MTA from our sendmail STARTTLS-enabled servers, we receive the following errors:

Jan  2 19:58:51 MTA sm-mta[29534]: [ID 702911 mail.notice] ruleset=tls_server, arg1=SOFTWARE, relay=MTA, reject=403 4.7.0 TLS handshake failed.

Jan  2 19:58:51 MTA sm-mta[29534]: [ID 801593 mail.info] hBT7rwVP015583: to=<recipient>, delay=4+12:04:53, xdelay=00:00:02, mailer=esmtp, pri=5884931, relay=MTA. [IP_address], dsn=4.0.0, stat=Deferred: 403 4.7.0 TLS handshake failed.

From cf/README:

${verify} holds the result of the verification of the presented cert.
        Possible values are:
        SOFTWARE STARTTLS handshake failed.

If no access map is in use, the connection will be accepted unless ${verify} is SOFTWARE, in which case the connection is always aborted.

Disabling STARTTLS And Setting SMTP Server Features
---------------------------------------------------

By default STARTTLS is used whenever possible.  However, there are
some broken MTAs that don't properly implement STARTTLS.  To be able
to send to (or receive from) those MTAs, the ruleset try_tls
(srv_features) can be used that work together with the access map.
Entries for the access map must be tagged with Try_TLS (Srv_Features)
and refer to the hostname or IP address of the connecting system.
A default case can be specified by using just the tag.  For example,
the following entries in the access map:

        Try_TLS:broken.server   NO
        Srv_Features:my.domain  v
        Srv_Features:           V

will turn off STARTTLS when sending to broken.server (or any host
in that domain), and request a client certificate during the TLS
handshake only for hosts in my.domain.  The valid entries on the RHS
for Srv_Features are listed in the Sendmail Installation and
Operations Guide.

Therefore, to disable STARTTLS support when sending mail to a broken MTA/domain, add the following entry to the access database:

Try_TLS:recipient.domain    NO

In addition, to disable STARTTLS support when receiving mail from a broken MTA/domain, add the following entry to the access database:

Srv_Features:recipient.domain    S

IP addresses do not have to be placed in brackets in the above Try_TLS or Srv_Features entries.

Note: When specifying a specific hostname (instead of IP address) in the above Try_TLS and Srv_Features entries, make sure to use the value that corresponds to the DNS PTR record for the host's IP address. For example:

host1.example.com is a DNS A record for 10.0.0.1
host2.example.com is a DNS A record for 10.0.0.1
The DNS PTR record for 10.0.0.1 is host2.example.com.

In this example, you would use host2.example.com or the IP address 10.0.0.1 in the Try_TLS and Srv_Features entries.

We have also seen a Novell GroupWise 6.0.1 server return "450 TLS negotation failure" messages when communicating with our sendmail STARTTLS-enabled relays. Disabling STARTTLS support with this MTA fixed the problem.

Back to brandonhutchinson.com.
Last modified: 2006/06/15