Programming Python (15 page)

os.system


Runs a
shell command from a Python script

os.popen


Runs a shell
command and connects to its input or output
streams

To understand the
scope of the calls listed above, we first need to define
a few terms. In this text, the term
shell
means
the system that reads and runs command-line strings on your computer,
and
shell command
means a command-line string
that you would normally enter at your computer’s shell prompt.

For example, on Windows, you can start an MS-DOS console window
(a.k.a. “Command Prompt”) and type DOS commands there—commands such as
dir
to get a directory listing,
type
to view a file, names of
programs you wish to start, and so on. DOS is the system shell, and
commands such as
dir
and
type
are shell commands. On Linux and Mac OS
X, you can start a new shell session by opening an xterm or terminal
window and typing shell commands there too—
ls
to list directories,
cat
to view files, and so on. A variety of
shells are available on Unix (e.g., csh, ksh), but they all read and
run command lines. Here are two shell commands typed and run in an
MS-DOS console box on Windows:

C:\...\PP4E\System>
dir /B
...type a shell command line
helloshell.py
...its output shows up here
more.py
...DOS is the shell on Windows
more.pyc
spam.txt
__init__.py
C:\...\PP4E\System>
type helloshell.py
# a Python program
print('The Meaning of Life')

None of this
is directly related to Python, of course (despite the
fact that Python
command
-line
scripts are sometimes confusingly called “shell tools”). But because
the
os
module’s
system
and
popen
calls let Python scripts run any sort
of command that the underlying system shell understands, our scripts
can make use of every command-line tool available on the computer,
whether it’s coded in Python or not. For example, here is some Python
code that runs the two DOS shell commands typed at the shell prompt
shown previously:

C:\...\PP4E\System>
python
>>>
import os
>>>
os.system('dir /B')
helloshell.py
more.py
more.pyc
spam.txt
__init__.py
0
>>>
os.system('type helloshell.py')
# a Python program
print('The Meaning of Life')
0
>>>
os.system('type hellshell.py')
The system cannot find the file specified.
1

The
0
s at the end of the
first two commands here are just the return values of the system call
itself (its exit status; zero generally means success). The system
call can be used to run any command line that we could type at the
shell’s prompt (here,
C:\...\PP4E\System>
). The command’s
output normally shows up in the Python session’s or program’s standard
output stream.

But what if we want to
grab a command’s output within a script? The
os.system
call simply runs a shell command line, but
os.popen
also
connects to the standard input or output streams of the command; we
get back a file-like object connected to the command’s output by
default (if we pass a
w
mode flag
to
popen
, we connect to the
command’s input stream instead). By using this object to read the
output of a command spawned with
popen
, we can intercept the text that would
normally appear in the
console
window where a command line is typed:

>>>
open('helloshell.py').read()
"# a Python program\nprint('The Meaning of Life')\n"
>>>
text = os.popen('type helloshell.py').read()
>>>
text
"# a Python program\nprint('The Meaning of Life')\n"
>>>
listing = os.popen('dir /B').readlines()
>>>
listing
['helloshell.py\n', 'more.py\n', 'more.pyc\n', 'spam.txt\n', '__init__.py\n']

Here, we first fetch a file’s content the usual way (using
Python files), then as the output of a shell
type
command. Reading the output of a
dir
command lets us get a listing
of files in a directory that we can then process in a loop. We’ll
learn other ways to obtain such a list in
Chapter 4
; there we’ll also learn how file
iterators
make the
readlines
call in the
os.popen
example above unnecessary in most
programs, except to display the
list
interactively as we did
here (see also
subprocess, os.popen, and Iterators
for more on the
subject).

So far, we’ve run basic DOS commands; because these calls can
run any command line that we can type at a shell prompt, they can also
be used to launch other Python scripts. Assuming your system search
path is set to locate your Python (so that you can use the shorter
“python” in the following instead of the longer
“C:\Python31\python”):

>>>
os.system('python helloshell.py')
# run a Python program
The Meaning of Life
0
>>>
output = os.popen('python helloshell.py').read()
>>>
output
'The Meaning of Life\n'

In all of these examples, the command-line strings sent to
system
and
popen
are hardcoded, but there’s no reason
Python programs could not construct such strings at
runtime
using normal string operations
(+, %, etc.). Given that commands can be dynamically built and run
this way,
system
and
popen
turn Python scripts into flexible and
portable tools for launching and orchestrating other programs. For
example, a Python test “driver” script can be used to run programs
coded in any language (e.g., C++, Java, Python) and analyze their
output. We’ll explore such a script in
Chapter 6
. We’ll also revisit
os.popen
in the next chapter in conjunction
with stream redirection; as we’ll find, this call can also send
input
to programs.

As mentioned,
in recent releases of Python the
subprocess
module can achieve the same
effect as
os.system
and
os.popen
; it generally requires extra code
but gives more control over how streams are connected and used. This
becomes especially useful when streams are tied in more complex
ways.

For example, to run a simple shell command like we did with
os.system
earlier, this new
module’s
call
function works
roughly the same (running commands like “type” that are built into the
shell on Windows requires extra protocol, though normal executables
like “python” do not):

>>>
import subprocess
>>>
subprocess.call('python helloshell.py')
# roughly like os.system()
The Meaning of Life
0
>>>
subprocess.call('cmd /C "type helloshell.py"')
# built-in shell cmd
# a Python program
print('The Meaning of Life')
0
>>>
subprocess.call('type helloshell.py', shell=True)
# alternative for built-ins
# a Python program
print('The Meaning of Life')
0

Notice the
shell=True
in the
last command here. This is a subtle and
platform
-
dependent
requirement:

More on some of this later; for now, it’s enough to note that
you may need to pass
shell=True
to
run some of the examples in this section and book in Unix-like
environments, if they rely on shell features like program path lookup.
Since I’m running code on Windows, this argument will often be omitted
here.

Besides imitating
os.system
,
we can similarly use this module to emulate the
os.popen
call used earlier, to run a shell
command and obtain its standard output text in our script:

>>>
pipe = subprocess.Popen('python helloshell.py', stdout=subprocess.PIPE)
>>>
pipe.communicate()
(b'The Meaning of Life\r\n', None)
>>>
pipe.returncode
0

Here, we connect the stdout stream to a pipe, and communicate to
run the command to completion and receive its standard output and
error streams’ text; the command’s exit status is available in an
attribute after it completes. Alternatively, we can use other
interfaces to read the command’s standard output directly and wait for
it to exit (which returns the exit status):

>>>
pipe = subprocess.Popen('python helloshell.py', stdout=subprocess.PIPE)
>>>
pipe.stdout.read()
b'The Meaning of Life\r\n'
>>>
pipe.wait()
0

In fact, there are direct mappings from
os.popen
calls to
subprocess.Popen
objects:

>>>
from subprocess import Popen, PIPE
>>>
Popen('python helloshell.py', stdout=PIPE).communicate()[0]
b'The Meaning of Life\r\n'
>>>
>>>
import os
>>>
os.popen('python helloshell.py').read()
'The Meaning of Life\n'

As you can probably tell
,
subprocess
is extra work in these relatively simple cases.
It starts to look better, though, when we need to control additional
streams in flexible ways. In fact, because it also allows us to
process a command’s error and input streams in similar ways, in Python
3.X
subprocess
replaces the
original
os.popen2
,
os.popen3
, and
os.popen4
calls which were available in
Python 2.X; these are now just use cases for
subprocess
object interfaces. Because more
advanced use cases for this module deal with standard streams, we’ll
postpone additional details about this module until we study stream
redirection in the next
chapter.

Before we move on,
you should keep in mind two limitations of
system
and
popen
. First, although these two functions
themselves are fairly portable, their use is really only as portable
as the commands that they run. The preceding examples that run DOS
dir
and
type
shell commands, for instance, work only
on
Windows, and would have to be changed in order to
run
ls
and
cat
commands on Unix-like platforms.

Second, it is important to remember that running Python files as
programs this way is very different and generally much slower than
importing program files and calling functions they define. When
os.system
and
os.popen
are called, they must start a
brand-new, independent program running on your operating system (they
generally run the command in a new process). When importing a program
file as a module, the Python interpreter simply loads and runs the
file’s code in the same process in order to generate a module object.
No other program is spawned along the way.
^[
6
]

There are good reasons to build systems as separate programs,
too, and in the next chapter we’ll explore things such as command-line
arguments and streams that allow programs to pass information back and
forth. But in many cases, imported modules are a faster and more
direct way to compose systems.

If you plan to use these calls in earnest, you should also know
that the
os.system
call normally
blocks—that is, pauses—its caller until the spawned command line
exits. On Linux and Unix-like platforms, the spawned command can
generally be made to run independently and in parallel with the caller
by adding an
&
shell background
operator at the end of the command line:

os.system("python program.py arg arg &")

On Windows, spawning with a DOS
start
command
will usually launch the command in parallel too:

os.system("start program.py arg arg")

In fact, this is so useful that an
os.startfile
call was added in recent Python
releases. This call opens a file with whatever program is listed in
the Windows registry for the file’s type—as though its icon has been
clicked with the mouse cursor:

os.startfile("webpage.html")    # open file in your web browser
os.startfile("document.doc")    # open file in Microsoft Word
os.startfile("myscript.py")     # run file with Python

The
os.popen
call does not
generally block its caller (by definition, the caller must be able to
read or write the file object returned) but callers may still
occasionally become blocked under both Windows and Linux if the pipe
object is closed—e.g., when garbage is collected—before the spawned
program exits or the pipe is read exhaustively (e.g., with its
read()
method). As we will see
later in this part of the book, the Unix
os.fork/exec
and Windows
os.spawnv
calls can also be used to run
parallel programs without blocking.

Because the
os
module’s
system
and
popen
calls, as well as the
subprocess
module, also fall under the
category of program launchers, stream redirectors, and cross-process
communication devices, they will show up again in the following
chapters, so we’ll defer further details for the time being. If you’re
looking for more details right away, be sure to see the stream
redirection section in the next chapter and the directory listings
section in
Chapter 4
.