Programming Python (119 page)

Here is the
popmail
script of
Example 13-18
in action,
displaying two messages in my account’s mailbox on machine
pop.secureserver.net
—the domain name of the mail
server machine used by the ISP hosting my
learning-python.com
domain name, as
configured
in the module
mailconfig
. To keep this output reasonably
sized, I’ve omitted or truncated a few irrelevant message header lines
here, including most of the
Received:
headers that chronicle an email’s journey; run this on your own to see
all the gory details of raw email text:

C:\...\PP4E\Internet\Email>
popmail.py
Password for pop.secureserver.net?
Connecting...
b'+OK <[email protected]>'
There are 2 mail messages in 3268 bytes
(b'+OK ', [b'1 1860', b'2 1408'], 16)
--------------------------------------------------------------------------------
[Press Enter key]
Received: (qmail 7690 invoked from network); 5 May 2010 15:29:43 −0000
X-IronPort-Anti-Spam-Result: AskCAG4r4UvRVllAlGdsb2JhbACDF44FjCkVAQEBAQkLCAkRAx+
Received: from 72.236.109.185 by webmail.earthlink.net with HTTP; Wed, 5 May 201
Message-ID: <27293081.1273073376592.JavaMail.root@mswamui-thinleaf.atl.sa.earthl
Date: Wed, 5 May 2010 11:29:36 −0400 (EDT)
From: [email protected]
Reply-To: [email protected]
To: [email protected]
Subject: I'm a Lumberjack, and I'm Okay
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Mailer: EarthLink Zoo Mail 1.0
X-ELNK-Trace: 309f369105a89a174e761f5d55cab8bca866e5da7af650083cf64d888edc8b5a35
X-Originating-IP: 209.86.224.51
X-Nonspam: None
I cut down trees, I skip and jump,
I like to press wild flowers...
--------------------------------------------------------------------------------
[Press Enter key]
Received: (qmail 17482 invoked from network); 5 May 2010 15:33:47 −0000
X-IronPort-Anti-Spam-Result: AlIBAIss4UthSoc7mWdsb2JhbACDF44FjD4BAQEBAQYNCgcRIq1
Received: (qmail 4009 invoked by uid 99); 5 May 2010 15:33:47 −0000
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"
X-Originating-IP: 72.236.109.185
User-Agent: Web-Based Email 5.2.13
Message-Id: <20100505083347.deec9532fd532622acfef00cad639f45.0371a89d29.wbe@emai
From: [email protected]
To: [email protected]
Cc: [email protected]
Subject: testing
Date: Wed, 05 May 2010 08:33:47 −0700
Mime-Version: 1.0
X-Nonspam: None
Testing Python mail tools.
--------------------------------------------------------------------------------
Bye.

This user interface is about as simple as it could be—after
connecting to the server, it prints the complete and raw full text of
one message at a time, pausing between each until you press the Enter
key. The
input
built-in is called to
wait for the key press between message displays. The pause keeps
messages from scrolling off the screen too fast; to make them visually
distinct, emails are also separated by lines of dashes.

We could make the display fancier (e.g., we can use the
email
package to parse headers, bodies, and
attachments—watch for examples in this and later chapters), but here we
simply display the whole message that was sent. This works well for
simple mails like these two, but it can be inconvenient for larger
messages with attachments; we’ll improve on this in later
clients.

This book won’t cover the full of set of headers that may appear
in emails, but we’ll make use of some along the way. For example, the
X-Mailer
header line, if present, typically
identifies the sending program; we’ll use it later to identify
Python-coded email senders we write. The more common headers such as
From
and
Subject
are more
crucial to a message. In fact, a variety of extra header lines can be
sent in a message’s text. The
Received
headers, for
example, trace the machines that a message passed through on its way to
the target mailbox.

Because
popmail
prints the
entire raw text of a message, you see all headers here, but you usually
see only a few by default in end-user-oriented mail GUIs such as Outlook
and webmail pages. The raw text here also makes apparent the email
structure we noted earlier: an email in general consists of a set of
headers like those here, followed by a blank line, which is followed by
the mail’s main text, though as we’ll see later, they can be more
complex if there are alternative parts or attachments.

The script in
Example 13-18
never deletes mail from
the server. Mail is simply retrieved and printed and will be shown again
the next time you run the script (barring deletion in another tool, of
course). To really remove mail permanently, we need to call other
methods (e.g.,
server.dele(msgnum)
),
but such a capability is best deferred until we develop more interactive
mail tools.

Notice how the reader script decodes each mail content line with
line.decode
into a
str
string for display; as mentioned earlier,
poplib
returns content as
bytes
strings in 3.X. In fact, if we change
the script to not decode, this becomes more obvious in its
output:

[Press Enter key]
...assorted lines omitted...
b'Date: Wed, 5 May 2010 11:29:36 −0400 (EDT)'
b'From: [email protected]'
b'Reply-To: [email protected]'
b'To: [email protected]'
b"Subject: I'm a Lumberjack, and I'm Okay"
b'Mime-Version: 1.0'
b'Content-Type: text/plain; charset=UTF-8'
b'Content-Transfer-Encoding: 7bit'
b'X-Mailer: EarthLink Zoo Mail 1.0'
b''
b'I cut down trees, I skip and jump,'
b'I like to press wild flowers...'
b''

As we’ll see later, we’ll need to decode similarly in order to
parse this text with email tools. The next section exposes the
bytes-based interface as
well.

If you don’t
mind typing code and reading POP server messages, it’s
possible to use the Python interactive prompt as a simple email client,
too. The following session uses two additional interfaces we’ll apply in
later examples:

The
top
call also returns a
tuple that includes the list of line strings sent back; its second
argument tells the server how many additional lines after the headers to
send, if any. If all you need are header details,
top
can be much quicker than the full text
fetch of
retr
, provided your mail
server implements the TOP command (most do):

C:\...\PP4E\Internet\Email>
python
>>>
from poplib import POP3
>>>
conn = POP3('pop.secureserver.net')
# connect to server
>>>
conn.user('[email protected]')
# log in to account
b'+OK '
>>>
conn.pass_('xxxxxxxx')
b'+OK '
>>>
conn.stat()
# num mails, num bytes
(2, 3268)
>>>
conn.list()
(b'+OK ', [b'1 1860', b'2 1408'], 16)
>>>
conn.top(1, 0)
(b'+OK 1860 octets ', [b'Received: (qmail 7690 invoked from network); 5 May 2010
...lines omitted...
b'X-Originating-IP: 209.86.224.51', b'X-Nonspam: None', b'', b''], 1827)
>>>
conn.retr(1)
(b'+OK 1860 octets ', [b'Received: (qmail 7690 invoked from network); 5 May 2010
...lines omitted...
b'X-Originating-IP: 209.86.224.51', b'X-Nonspam: None', b'',
b'I cut down trees, I skip and jump,', b'I like to press wild flowers...',
b'', b''], 1898)
>>>
conn.quit()
b'+OK '

Printing the full text of a message at the interactive prompt is
easy once it’s fetched: simply decode each line to a normal string as it
is printed, like our pop mail script did, or concatenate the line
strings returned by
retr
or
top
adding a newline between; any of the
following will suffice for an open POP server object:

>>>
info, msg, oct = connection.retr(1)
# fetch first email in mailbox
>>>
for x in msg: print(x.decode())
# four ways to display message lines
>>>
print(b'\n'.join(msg).decode())
>>>
x = [print(x.decode()) for x in msg]
>>>
x = list(map(print, map(bytes.decode, msg)))

Parsing email text to extract headers and components is more
complex, especially for mails with attached and possibly encoded parts,
such as images. As we’ll see later in this chapter, the standard
library’s
email
package can parse the
mail’s full or headers text after it has been fetched with
poplib
(or
imaplib
).

See the Python library manual for details on other POP module
tools. As of Python 2.4, there is also a
POP3_SSL
class in the
poplib
module that connects to the server over
an SSL-encrypted socket on port 995 by default (the standard port for
POP over SSL). It provides an identical interface, but it uses secure
sockets for the conversation where supported
by servers.

Calling
os.popen
to launch a command-line
mail program

On some systems,
you can send email from a script with a call of the
form:

os.popen('mail -s "xxx" [email protected]', 'w').write(text)

As we saw earlier in the book, the
popen
tool runs the command-line string
passed to its first argument, and returns a file-like object
connected to it. If we use an open mode of
w
, we are connected to the command’s
standard input stream—here, we write the text of the new mail
message to the standard Unix
mail
command-line program. The net effect is as if we had run
mail
interactively, but it happens inside
a running Python script.

Running the
sendmail
program

The open source
sendmail
program
offers another way to initiate mail from a program.
Assuming it is installed and configured on your system, you can
launch it using Python tools like the
os.popen
call of the previous
paragraph.

Using the standard
smtplib
Python
module

Python’s standard library comes
with support for the client-side interface to
SMTP—
the Simple Mail Transfer
Protocol—a higher-level Internet standard for sending mail over
sockets. Like the
poplib
module
we met in the previous section,
smtplib
hides all the socket and protocol
details and can be used to send mail on any machine with Python and
a suitable socket-based Internet link.

Fetching and using third-party packages and
tools

Other tools in the open source library provide higher-level
mail handling packages for Python; most build upon one of the prior
three techniques.

Since SMTP is arguably
the best option for sending mail from a Python script,
let’s explore a simple mailing program that illustrates its interfaces.
The Python script shown in
Example 13-19
is intended to be used
from an interactive command line; it reads a new mail message from the
user and sends the new mail by SMTP using Python’s
smtplib
module.

#!/usr/local/bin/python
"""
###########################################################################
use the Python SMTP mail interface module to send email messages; this
is just a simple one-shot send script--see pymail, PyMailGUI, and
PyMailCGI for clients with more user interaction features; also see
popmail.py for a script that retrieves mail, and the mailtools pkg
for attachments and formatting with the standard library email package;
###########################################################################
"""
import smtplib, sys, email.utils, mailconfig
mailserver = mailconfig.smtpservername         # ex: smtp.rmi.net
From = input('From? ').strip()                 # or import from mailconfig
To   = input('To?   ').strip()                 # ex: [email protected]
Tos  = To.split(';')                           # allow a list of recipients
Subj = input('Subj? ').strip()
Date = email.utils.formatdate()                # curr datetime, rfc2822
# standard headers, followed by blank line, followed by text
text = ('From: %s\nTo: %s\nDate: %s\nSubject: %s\n\n' % (From, To, Date, Subj))
print('Type message text, end with line=[Ctrl+d (Unix), Ctrl+z (Windows)]')
while True:
line = sys.stdin.readline()
if not line:
break                        # exit on ctrl-d/z
#if line[:4] == 'From':
#    line = '>' + line            # servers may escape
text += line
print('Connecting...')
server = smtplib.SMTP(mailserver)              # connect, no log-in step
failed = server.sendmail(From, Tos, text)
server.quit()
if failed:                                     # smtplib may raise exceptions
print('Failed recipients:', failed)        # too, but let them pass here
else:
print('No errors.')
print('Bye.')

Most of this script is user interface—it inputs the sender’s
address (From), one or more recipient addresses (To, separated by “;” if
more than one), and a subject line. The sending date is picked up from
Python’s standard
time
module,
standard header lines are formatted, and the
while
loop reads message lines until the user
types the end-of-file character (Ctrl-Z on Windows, Ctrl-D on
Linux).

To be robust, be sure to add a
blank line
between the header lines and the body in the message’s text; it’s
required by the SMTP protocol and some SMTP servers enforce this. Our
script conforms by inserting an empty line with
\n\n
at the end of the string format
expression—one
\n
to terminate the
current line and another for a blank line;
smtplib
expands
\n
to Internet-style
\r\n
internally prior to transmission, so the
short form is fine here. Later in this chapter, we’ll format our
messages with the Python
email
package, which handles such details for us automatically.

The rest of the script is where all the SMTP magic occurs: to send
a mail by SMTP, simply run these two sorts of calls:

When you’re done, be sure to call the object’s
quit
method to disconnect from the server and
finalize the transaction. Notice that, on failure, the
sendmail
method may either raise an exception
or return a list of the recipient addresses that failed; the script
handles the latter case itself but lets exceptions kill the script with
a Python error
message
.

Subtly, calling the server object’s
quit
method after
sendmail
raises an exception may or may not
work as expected—
quit
can actually
hang until a server timeout if the send fails internally and leaves the
interface in an unexpected state. For instance, this can occur on
Unicode encoding errors when translating the outgoing mail to bytes per
the ASCII scheme (the
rset
reset
request hangs in this case, too). An alternative
close
method simply closes the client’s
sockets without attempting to send a quit command to the server;
quit
calls close internally as a last
step (assuming the quit command can be sent!).

For advanced usage, SMTP objects provide additional calls not used
in this example:

See the Python library manual for more on these and other calls
not covered here.

Let’s ship a few messages across the world. The
smtpmail
script is a one-shot tool: each run
allows you to send a single new mail message. Like most of the
client-side tools in this chapter, it can be run from any computer with
Python and an Internet link that supports SMTP (most do, though some
public access machines may restrict users to HTTP [Web] access only or
require special server SMTP configuration). Here it is running on
Windows:

C:\...\PP4E\Internet\Email>
smtpmail.py
From?
[email protected]
To?
[email protected]
Subj?
A B C D E F G
Type message text, end with line=[Ctrl+d (Unix), Ctrl+z (Windows)]
Fiddle de dum, Fiddle de dee,
Eric the half a bee.
^Z
Connecting...
No errors.
Bye.

This mail is sent to the book’s email account address
(
[email protected]
), so it ultimately shows up in
the inbox at my ISP, but only after being routed through an arbitrary
number of machines on the Net, and across arbitrarily distant network
links. It’s complex at the bottom, but usually, the Internet “just
works.”

Notice the From address, though—it’s completely fictitious (as far
as I know, at least). It turns out that we can usually provide any From
address we like because SMTP doesn’t check its validity (only its
general format is checked). Furthermore, unlike POP, there is usually no
notion of a username or password in SMTP, so the sender is more
difficult to determine. We need only pass email to any machine with a
server listening on the SMTP port, and we don’t need an account or login
on that machine. Here,
the name
[email protected]
works
just
fine as the sender;
Marketing
.
Geek
.
[email protected]
might work just as well.

In fact, I didn’t import a From email address from the
mailconfig.py
module on purpose, because I
wanted to be able to demonstrate this behavior; it’s the basis
of some of those annoying junk emails that show up in your
mailbox without a real sender’s address.
^[
52
]
Marketers infected with e-millionaire mania will email
advertising to all addresses on a list without providing a real From
address, to cover their tracks.

Normally, of course, you should use the same To address in the
message and the SMTP call and provide your real email address as the
From value (that’s the only way people will be able to reply to your
message). Moreover, apart from teasing your significant other, sending
phony addresses is often just plain bad Internet citizenship. Let’s run
the script again to ship off another mail with more politically correct
coordinates:

C:\...\PP4E\Internet\Email>
smtpmail.py
From?
[email protected]
To?
[email protected]
Subj?
testing smtpmail
Type message text, end with line=[Ctrl+d (Unix), Ctrl+z (Windows)]
Lovely Spam! Wonderful Spam!
^Z
Connecting...
No errors.
Bye.

C:\...\PP4E\Internet\Email>
popmail.py
Password for pop.secureserver.net?
Connecting...
b'+OK <[email protected]>'
There are 4 mail messages in 5326 bytes
(b'+OK ', [b'1 1860', b'2 1408', b'3 1049', b'4 1009'], 32)
--------------------------------------------------------------------------------
[Press Enter key]
...first two mails omitted...
Received: (qmail 25683 invoked from network); 6 May 2010 14:12:07 −0000
Received: from unknown (HELO p3pismtp01-018.prod.phx3.secureserver.net) ([10.6.1
(envelope-sender )
by p3plsmtp06-04.prod.phx3.secureserver.net (qmail-1.03) with SMTP
for ; 6 May 2010 14:12:07 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
for ; Thu, 06 May 2010 10:12:12 −0400
From: [email protected]
To: [email protected]
Date: Thu, 06 May 2010 14:11:07 −0000
Subject: A B C D E F G
Message-ID: 
X-FromIP: 66.194.109.3
X-Nonspam: None
Fiddle de dum, Fiddle de dee,
Eric the half a bee.
--------------------------------------------------------------------------------
[Press Enter key]
Received: (qmail 4634 invoked from network); 6 May 2010 14:16:57 −0000
Received: from unknown (HELO p3pismtp01-025.prod.phx3.secureserver.net) ([10.6.1
(envelope-sender )
by p3plsmtp06-05.prod.phx3.secureserver.net (qmail-1.03) with SMTP
for ; 6 May 2010 14:16:57 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
for ; Thu, 06 May 2010 10:17:03 −0400
From: [email protected]
To: [email protected]
Date: Thu, 06 May 2010 14:16:31 −0000
Subject: testing smtpmail
Message-ID: <8fad1n462667fik006052010101703@SMTP>
X-FromIP: 66.194.109.3
X-Nonspam: None
Lovely Spam! Wonderful Spam!
--------------------------------------------------------------------------------
Bye.

Manipulating both From and To

The first mail listed at the end of the preceding section was
the one we sent with a fictitious sender address; the second was the
more legitimate message. Like sender addresses, header lines are a bit
arbitrary under SMTP. Our
smtpmail
script automatically adds From and To header lines in the message’s
text with the same addresses that are passed to the SMTP interface,
but only as a polite convention. Sometimes, though, you can’t tell who
a mail was sent to, either—to obscure the target audience or to
support legitimate email lists, senders may manipulate the contents of
both these headers in the message’s text.

For example, if we change
smtpmail
to not automatically generate a
“To:” header line with the same address(es) sent to the SMTP interface
call:

text = ('From: %s\nDate: %s\nSubject: %s\n' % (From, Date, Subj))

we can then manually type a “To:” header that differs from the
address we’re really sending to—the “To” address list passed into the
smtplib
send call gives the true
recipients, but the “To:” header line in the text of the message is
what most mail clients will display (see
smtpmail-noTo.py
in the examples package
for the code needed to support such anonymous behavior, and be sure to
type a blank line after “To:”):

C:\...\PP4E\Internet\Email>
smtpmail-noTo.py
From?
[email protected]
To?
[email protected]
Subj?
a b c d e f g
Type message text, end with line=(ctrl + D or Z)
To: [email protected]
Spam; Spam and eggs; Spam, spam, and spam.
^Z
Connecting...
No errors.
Bye.

In some ways, the From and To addresses in send method calls and
message header lines are similar to addresses on envelopes and letters
in envelopes, respectively. The former is used for routing, but the
latter is what the reader sees. Here, From is fictitious in both
places. Moreover, I gave the real To address for the account on the
server, but then gave a fictitious name in the manually typed “To:”
header line—the first address is where it really goes and the second
appears in mail clients. If your mail tool picks out the “To:” line,
such mails will look odd when viewed.

For instance, when the mail we just sent shows up in my mailbox
at
learning-
python.com
, it’s difficult to tell
much about its origin or destination in the webmail
interface
my ISP provides, as captured
in
Figure 13-5
.

Figure 13-5. Anonymous mail in a web-mail client (see also ahead:
PyMailGUI)

Furthermore, this email’s raw text won’t help unless we look
closely at the “Received:” headers added by the machines it has been
routed through:

C:\...\PP4E\Internet\Email>
popmail.py
Password for pop.secureserver.net?
Connecting...
b'+OK <[email protected]>'
There are 5 mail messages in 6364 bytes
(b'+OK ', [b'1 1860', b'2 1408', b'3 1049', b'4 1009', b'5 1038'], 40)
--------------------------------------------------------------------------------
[Press Enter key]
...first three mails omitted...
Received: (qmail 30325 invoked from network); 6 May 2010 14:33:45 −0000
Received: from unknown (HELO p3pismtp01-004.prod.phx3.secureserver.net) ([10.6.1
(envelope-sender )
by p3plsmtp06-03.prod.phx3.secureserver.net (qmail-1.03) with SMTP
for ; 6 May 2010 14:33:45 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
for ; Thu, 06 May 2010 10:33:16 −0400
From: [email protected]
Date: Thu, 06 May 2010 14:32:32 −0000
Subject: a b c d e f g
To: [email protected]
Message-ID: <66koqg66e0q1c8hl06052010103316@SMTP>
X-FromIP: 66.194.109.3
X-Nonspam: None
Spam; Spam and eggs; Spam, spam, and spam.
--------------------------------------------------------------------------------
Bye.

Once again, though, don’t do this unless you have good cause.
This demonstration is intended only to help you understand how mail
headers factor into email processing. To write an automatic spam
filter that deletes incoming junk mail, for instance, you need to know
some of the telltale signs to look for in a message’s text. Spamming
techniques have grown much more sophisticated than simply forging
sender and recipient names, of course (you’ll find much more on the
subject on the Web at large and in the
SpamBayes
mail filter written in Python), but it’s one common trick.

On the other hand, such To address juggling may also be useful
in the context of legitimate
mailing lists
—the
name of the list appears in the “To:” header when the message is
viewed, not the potentially many individual recipients named in the
send-mail call. As the next section’s example demonstrates, a mail
client can simply send a mail to all on the list but insert the
general list name in the “To:” header.

But in other contexts, sending email with bogus “From:” and
“To:” lines is equivalent to making anonymous phone calls. Most
mailers won’t even let you change the From line, and they don’t
distinguish between the To address and header line. When you program
mail scripts of your own, though, SMTP is wide open in this regard. So
be good out there, OK?

Does Anybody Really Know What Time It Is?

In the prior version of the
smtpmail
script of
Example 13-19
, a simple date
format was used for the Date email header that didn’t quite follow
the SMTP
date formatting standard:

>>>
import time
>>>
time.asctime()
'Wed May 05 17:52:05 2010'

Most servers don’t care and will let any sort of date text
appear in date header lines, or even add one if needed. Clients are
often similarly forgiving, but not always; one of my ISP webmail
programs shows dates correctly anyhow, but another leaves such
ill-formed dates blank in mail displays. If you want to be more in
line with the standard, you could format the date header with code
like this (the result can be parsed with standard tools such as the
time.strptime
call):

import time
gmt = time.gmtime(time.time())
fmt = '%a, %d %b %Y %H:%M:%S GMT'
str = time.strftime(fmt, gmt)
hdr = 'Date: ' + str
print(hdr)

The
hdr
variable’s value
looks like this when this code is run:

Date: Wed, 05 May 2010 21:49:32 GMT

The
time.strftime
call
allows arbitrary date and time formatting;
time.asctime
is just one standard format.
Better yet, do what
smtpmail
does
now—in the newer
email
package
(described in this chapter), an
email.utils
call can be used to properly
format date and time automatically. The
smtpmail
script uses the first of the
following format
alternatives
:

>>>
import email.utils
>>>
email.utils.formatdate()
'Wed, 05 May 2010 21:54:28 −0000'
>>>
email.utils.formatdate(localtime=True)
'Wed, 05 May 2010 17:54:52 −0400'
>>>
email.utils.formatdate(usegmt=True)
'Wed, 05 May 2010 21:55:22 GMT'

See the
pymail
and
mailtools
examples in this chapter for
additional usage examples; the latter is reused by the larger
PyMailGUI and PyMailCGI email clients later in this book.