Programming Python (20 page)

In fact, by
passing in the desired mode flag, we redirect either a
spawned program’s output
or
input streams to a
file in the calling scripts, and we can obtain the spawned program’s
exit status code from the
close
method (
None
means “no error”
here). To illustrate, consider the following two scripts:

C:\...\PP4E\System\Streams>
type hello-out.py
print('Hello shell world')
C:\...\PP4E\System\Streams>
type hello-in.py
inp = input()
open('hello-in.txt', 'w').write('Hello ' + inp + '\n')

These scripts can be run from a system shell window as
usual:

C:\...\PP4E\System\Streams>
python hello-out.py
Hello shell world
C:\...\PP4E\System\Streams>
python hello-in.py
Brian
C:\...\PP4E\System\Streams>
type hello-in.txt
Hello Brian

As we saw in the prior chapter, Python scripts can read
output
from other programs and scripts like
these, too, using code like the following:

C:\...\PP4E\System\Streams>
python
>>>
import os
>>>
pipe = os.popen('python hello-out.py')
# 'r' is default--read stdout
>>>
pipe.read()
'Hello shell world\n'
>>>
print(pipe.close())
# exit status: None is good
None

But Python scripts can also provide
input
to spawned programs’ standard input
streams—
passing a “w” mode argument,
instead of the default “r”, connects the returned object to the
spawned program’s input stream. What we write on the spawning end
shows up as input in the program started:

>>>
pipe = os.popen('python hello-in.py', 'w')
# 'w'--write to program stdin
>>>
pipe.write('Gumby\n')
6
>>>
pipe.close()
# \n at end is optional
>>>
open('hello-in.txt').read()
# output sent to a file
'Hello Gumby\n'

The
popen
call is also smart
enough to run the command string as an independent process on
platforms that support such a notion. It accepts an optional third
argument that can be used to control buffering of written text, which
we’ll finesse here.

For even more
control over the streams of spawned programs, we can
employ the
subprocess
module we
introduced in the preceding chapter. As we learned earlier, this
module can emulate
os.popen
functionality, but it can also achieve feats such as
bidirectional
stream communication
(accessing both a program’s input and output) and tying the output of
one program to the input of another.

For instance, this module provides multiple ways to spawn a
program and get both its standard output text and exit status. Here
are three common ways to leverage this module to start a program and
redirect its
output
stream (recall from
Chapter 2
that you may need to pass a
shell=True
argument to
Popen
and
call
to make this section’s examples work on
Unix-like platforms as they are coded here):

C:\...\PP4E\System\Streams>
python
>>>
from subprocess import Popen, PIPE, call
>>>
X = call('python hello-out.py')
# convenience
Hello shell world
>>>
X
0
>>>
pipe = Popen('python hello-out.py', stdout=PIPE)
>>>
pipe.communicate()[0]
# (stdout, stderr)
b'Hello shell world\r\n'
>>>
pipe.returncode
# exit status
0
>>>
pipe = Popen('python hello-out.py', stdout=PIPE)
>>>
pipe.stdout.read()
b'Hello shell world\r\n'
>>>
pipe.wait()
# exit status
0

The
call
in the first of
these three techniques is just a convenience function (there are more
of these which you can look up in the Python library manual), and the
communicate
in
the second is roughly a convenience for the third (it sends data to
stdin, reads data from stdout until end-of-file, and waits for the
process to end):

Redirecting and connecting to the spawned program’s
input
stream is just as simple, though a bit more
complex than the
os.popen
approach
with
'w'
file mode shown in the
preceding section (as mentioned in the last chapter,
os.popen
is implemented with
subprocess
, and is thus itself just
something of a convenience function today):

>>>
pipe = Popen('python hello-in.py', stdin=PIPE)
>>>
pipe.stdin.write(b'Pokey\n')
6
>>>
pipe.stdin.close()
>>>
pipe.wait()
0
>>>
open('hello-in.txt').read()
# output sent to a file
'Hello Pokey\n'

In fact, we can use obtain
both the input and
output
streams of a spawned program with this module. Let’s
reuse the simple writer and reader scripts we wrote earlier to
demonstrate
:

C:\...\PP4E\System\Streams>
type writer.py
print("Help! Help! I'm being repressed!")
print(42)
C:\...\PP4E\System\Streams>
type reader.py
print('Got this: "%s"' % input())
import sys
data = sys.stdin.readline()[:-1]
print('The meaning of life is', data, int(data) * 2)

Code like the following can both read from and write to the
reader script—the pipe object has two file-like objects available as
attached attributes, one connecting to the input stream, and one to
the output (Python 2.X users might recognize these as equivalent to
the tuple returned by the now-defunct
os.popen2
):

>>>
pipe = Popen('python reader.py', stdin=PIPE, stdout=PIPE)
>>>
pipe.stdin.write(b'Lumberjack\n')
11
>>>
pipe.stdin.write(b'12\n')
3
>>>
pipe.stdin.close()
>>>
output = pipe.stdout.read()
>>>
pipe.wait()
0
>>>
output
b'Got this: "Lumberjack"\r\nThe meaning of life is 12 24\r\n'

As we’ll learn in
Chapter 5
, we
have to be cautious when talking back and forth to a program like
this; buffered output streams can lead to deadlock if writes and reads
are interleaved, and we may eventually need to consider tools like the
Pexpect utility
as a workaround (more on this later).

Finally, even more exotic stream control is possible—the
following
connects two programs
, by piping the
output of one Python script into another, first with shell syntax, and
then with the
subprocess
module:

C:\...\PP4E\System\Streams>
python writer.py | python reader.py
Got this: "Help! Help! I'm being repressed!"
The meaning of life is 42 84
C:\...\PP4E\System\Streams>
python
>>>
from subprocess import Popen, PIPE
>>>
p1 = Popen('python writer.py', stdout=PIPE)
>>>
p2 = Popen('python reader.py', stdin=p1.stdout, stdout=PIPE)
>>>
output = p2.communicate()[0]
>>>
output
b'Got this: "Help! Help! I\'m being repressed!"\r\nThe meaning of life is 42 84\r\n'
>>>
p2.returncode
0

We can get close to this with
os.popen
, but that the fact that its pipes
are read or write (and not both) prevents us from catching the second
script’s output in our
code:

>>>
import os
>>>
p1 = os.popen('python writer.py', 'r')
>>>
p2 = os.popen('python reader.py', 'w')
>>>
p2.write( p1.read() )
36
>>>
X = p2.close()
Got this: "Help! Help! I'm being repressed!"
The meaning of life is 42 84
>>>
print(X)
None

From the broader perspective, the
os.popen
call and
subprocess
module are Python’s portable
equivalents of Unix-like shell syntax for redirecting the streams of
spawned programs. The Python versions also work on Windows, though,
and are the most
platform
-
neutral
way to launch another program
from a Python script. The command-line strings you pass to them may
vary per platform (e.g., a directory listing requires an
ls
on Unix but a
dir
on Windows), but the call itself works
on all major Python
platforms
.

On Unix-like platforms, the combination of the calls
os.fork
,
os.pipe
,
os.dup
, and some
os.exec
variants can also be used to start a
new independent program with streams connected to the parent program’s
streams. As such, it’s yet another way to redirect streams and a
low-level equivalent to tools such as
os.popen
(
os.fork
is available in Cygwin’s Python on
Windows).

Since these are all more advanced parallel processing tools,
though, we’ll defer further details on this front until
Chapter 5
, especially its coverage of pipes
and exit status codes. And we’ll resurrect
subprocess
again in
Chapter 6
, to code a regression tester
that intercepts all
three
standard streams of
spawned test scripts—inputs, outputs, and errors.

But first,
Chapter 4
continues
our survey of Python system interfaces by exploring the tools
available for processing files and directories. Although we’ll be
shifting focus somewhat, we’ll find that some of what we’ve learned
here will already begin to come in handy as general system-related
tools. Spawning shell commands, for instance, provides ways to inspect
directories, and the file interface we will expand on in the next
chapter is at the heart of the stream processing techniques we have
studied
here.

#!/bin/csh
foreach x (*.py)
echo $x
mail [email protected] -s $x < $x
end

#!/usr/bin/python
import os, glob
for x in glob.glob('*.py'):
print(x)
os.system('mail [email protected] -s %s < %s' % (x, x))