Using STARTTLS with sendmail

The following instructions were primarily obtained from the excellent article Securing Sendmail with TLS and the page SMTP/STARTTLS in sendmail/Secure Switch. Another excellent page regarding certificate generation is Creating and Using SSL Certificates.

The steps below show how to create a Certificate Authority (CA) using OpenSSL, create server public and private keys, and enable STARTTLS support in sendmail.

1. Modify the OpenSSL CA.pl accessory program for TLS certificate generation. This modification allows sendmail to start automatically without being prompted for private key passphrases.

Fedore Core users do not need this step, as the CA.pl program provided with the openssl-perl RPM has a newreq-nodes flag for creating an unencrypted server private key.

$ cat /etc/fedora-release
Fedora Core release 4 (Stentz)
$ rpm -q openssl-perl
openssl-perl-0.9.7f-7
$ /etc/pki/tls/misc/CA.pl -?
usage: CA -newcert|-newreq|-newreq-nodes|-newca|-sign|-verify



Red Hat Linux instructions:

1. Install the openssl-perl RPM (version 0.9.7a-23 used in these instructions).

2. Create the CA.pl.diff file:
--- CA.pl       2003-09-30 16:59:10.000000000 -0500
+++ CA1.pl      2003-11-07 09:06:54.000000000 -0600
@@ -58,12 +58,12 @@
            exit 0;
        } elsif (/^-newcert$/) {
            # create a certificate
-           system ("$REQ -new -x509 -keyout newreq.pem -out newreq.pem $DAYS");
+           system ("$REQ -new -x509 -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Certificate (and private key) is in newreq.pem\n"
        } elsif (/^-newreq$/) {
            # create a certificate request
-           system ("$REQ -new -keyout newreq.pem -out newreq.pem $DAYS");
+           system ("$REQ -new -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Request (and private key) is in newreq.pem\n";
        } elsif (/^-newreq-nodes$/) {

3. Copy /usr/share/ssl/misc/CA.pl into the same directory as CA.pl.diff.

4. Patch CA.pl. The output file will be CA1.pl.
# patch -o CA1.pl < CA.pl.diff

Solaris instructions:

1. Install the OpenSSL package (version 0.9.7c used in these instructions) from Sunfreeware.

2. Create the CA.pl.diff file:
*** CA.pl       Mon Jul 19 09:11:14 2004
--- CA1.pl      Mon Jul 19 09:58:17 2004
***************
*** 58,69 ****
            exit 0;
        } elsif (/^-newcert$/) {
            # create a certificate
!           system ("$REQ -new -x509 -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Certificate (and private key) is in newreq.pem\n"
        } elsif (/^-newreq$/) {
            # create a certificate request
!           system ("$REQ -new -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Request (and private key) is in newreq.pem\n";
        } elsif (/^-newreq-nodes$/) {
--- 58,69 ----
            exit 0;
        } elsif (/^-newcert$/) {
            # create a certificate
!           system ("$REQ -new -x509 -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Certificate (and private key) is in newreq.pem\n"
        } elsif (/^-newreq$/) {
            # create a certificate request
!           system ("$REQ -new -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            $RET=$?;
            print "Request (and private key) is in newreq.pem\n";
        } elsif (/^-newreq-nodes$/) {

3. Copy /usr/local/ssl/misc/CA.pl into the same directory as CA.pl.diff.

4. Patch CA.pl. The output file will be CA1.pl.
# patch -o CA1.pl < CA.pl.diff



2. Create a Certificate Authority (CA).
# CA1.pl -newca (Red Hat Linux or Solaris)
# CA.pl -newca (Fedora Core)

Note: You may want to change $DAYS="-days 365"; in CA[1].pl to a larger value than one year when creating a CA. When your CA certificate expires, you will have to distribute your new CA certificate to communication partners if you are using TLS authentication/verification (i.e. if communication partners are "trusting" your CA). It is common to see CA certificates valid for 10 years or longer.

/etc/pki/CA/cacert.pem (Certificate Authority certificate, you may exchange this with communication partners for TLS authentication/verification)

/etc/pki/CA/private/cakey.pem (passphrase-protected Certificate Authority private key used to sign/revoke certificates. Do not exchange this with communication partners!)

3. Create a private key and certificate signing request (CSR).
# CA1.pl -newreq
(Red Hat Linux or Solaris)
# CA.pl -newreq-nodes (Fedora Core)

Note that this step places both the private key and CSR in a file named /etc/pki/tls/misc/newreq.pem. If you are having another CA sign your key, only send them the CSR information within the -----BEGIN CERTIFICATE REQUEST----- and -----END CERTIFICATE REQUEST----- lines.

4. Sign the CSR with your CA.
# CA1.pl -sign
(Red Hat Linux or Solaris)
# CA.pl -sign (Fedora Core)

This step creates a file named /etc/pki/tls/misc/newcert.pem; this file is your server's certificate, signed by your CA.

If you want to generate multiple certificates for different mail servers, rename newreq.pem and newcert.pem (to something like hostname_public_key.pem and hostname_private_key.pem, respectively), and repeat steps 3 and 4.

5. Create a certificates directory.
# mkdir /etc/mail/certs

6. Copy the CA certificate and server public and private keys to the certificates directory.
# cp /etc/pki/CA/cacert.pem /etc/mail/certs/CAcert.pem
# cp /etc/pki/tls/misc/newreq.pem /etc/mail/certs/MYkey.pem
# cp cp /etc/pki/tls/misc/newcert.pem /etc/mail/certs/MYcert.pem

7. Configure permissions on the certificates directory.
# chmod -R 700 /etc/mail/certs
# chown -R root:sys /etc/mail/certs

8. Configure the appropriate sendmail macro configuration (.mc) file for STARTTLS.

Add the following entries:
define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl


For Red Hat Linux/Fedora Core, add the above entries in
/etc/mail/sendmail.mc.

The version of Sendmail distributed with Red Hat Linux/Fedora Core is compiled with STARTTLS support. If you are using Solaris, please read http://brandonhutchinson.com/sendmail_solaris.html for instructions on compling sendmail with STARTTLS support.

To test if Sendmail is compiled with STARTTLS support, run the following command:
$ sendmail -bt -d0.8 < /dev/null

 Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX
                MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
                NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
                USERDB USE_LDAP_INIT


9. Build and install the sendmail configuration file. Send the sendmail daemon a SIGHUP for the configuration changes to take effect.

Red Hat Linux/Fedora instructions:
# cd /etc/mail
# make sendmail.cf
# /sbin/service sendmail restart

10. Test sendmail for STARTTLS support.
$ telnet your_mail_server 25
EHLO your_hostname

You should see:
250-STARTTLS

If you do not see this returned by the EHLO command, check your sendmail log for errors.

Note: When installing new certificates (e.g. the previous certificate is at or near expiration), sendmail must be sent a SIGHUP to read the newly-installed certificates.

You may use the following command to display a remote MTA's certificate if STARTTLS is supported:
$ openssl s_client -starttls smtp -showcerts -connect MTA:25

Using STARTTLS for authentication

STARTTLS-enabled MTAs provide confidentiality of communication by encrypting messages exchanged between the servers. In addition, STARTTLS can provide authentication (verification of the communication partner) by obtaining and "trusting" the communication partner's Certificate Authority (CA) certificate.

Without obtaining and trusting the communication partner's CA certificate, messages between the MTAs will be encrypted, but verification of the communication partner will "fail." Note that "failing" authentication is not a cause for alarm, especially with the large number of sites using self-signed certificates.

Nov 10 08:54:26 relay sm-mta[27997]: STARTTLS=server, relay=mail_server [192.168.1.100], version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256

To authenticate the communication partner:

1. Copy the communication partner's CA certificate into /etc/mail/certs.
2. Create a symbolic link of the communication partner's CA certificate hash.
# C=FileName_of_CA_Certificate
# ln -s $C `openssl x509 -noout -hash < $C`.0

Nov 11 08:53:29 relay sm-mta[24582]: STARTTLS=server, relay=mail_server [192.168.1.100], version=TLSv1/SSLv3, verify=OK, cipher=DHE-RSA-AES256-SHA, bits=256/256

Using the sendmail access database, it is possible to "force" STARTTLS communication with a communication partner. The following entry will reject mail with a 5.7.0 authentication required error if the communication_partner_MTA is not successfully authenticated, or if the communication partner is not using at least a 112-bit encryption key.

TLS_Clt:communication_partner_MTA                           PERM+VERIFY:112

Nov 11 08:38:59 mail_server sm-mta[24279]: hABEcx2B024279: ruleset=check_mail, arg1=<user@example.com>, relay=hutch [192.168.1.100], reject=503 5.7.0 authentication required

To enforce verification and encryption when sending mail to and from a domain, use the following entries in the access database:

TLS_Clt:
domain                                              PERM+VERIFY:112
TLS_Srv:domain                                              PERM+VERIFY:112
TLS_Rcpt:domain                                             PERM+VERIFY:112

The TLS_Clt ruleset is used when sendmail acts as a server.
The TLS_Srv ruleset is used when sendmail acts as a client.
The TLS_Rcpt ruleset ensures that STARTTLS access database rules are enforced for domain even if domain's MX records contain hosts that do not support STARTTLS.

"failed to update database
TXT_DB error number 2"

You may receive this error message when attempting to re-sign a CSR for an expired server certificate. The certificate must first be revoked and then recreated (if the CSR no longer exists) or re-signed (if the CSR exists).

# cat /etc/pki/CA/index.txt
V       060629140650Z           EAC5B40539715928        unknown /C=/ST=/L=/O=/CN=

# cd /etc/pki/tls/misc
# openssl ca -revoke /etc/pki/CA/newcerts/EAC5B40539715928.pem


Back to brandonhutchinson.com.
Last modified: 2006/06/15