Programming Python (200 page)

The last two
sections dealt with running strings of code, but it’s easy
for C programs to deal in terms of Python objects, too.
Example 20-27
accomplishes the same
task as Examples
20-23
and
20-26
, but it uses other API tools to
interact with objects in the Python module directly:

We used both of the data conversion functions earlier in this
chapter in extension modules. The
PyEval_CallObject
call in this version of the
example is the key point here: it runs the imported function with a
tuple of arguments, much like the Python
func(*args)
call syntax. The Python function’s
return value comes back to C as a
PyObject*
, a generic Python object
pointer.

/* fetch and call objects in modules */
#include 
main() {
char *cstr;
PyObject *pstr, *pmod, *pfunc, *pargs;
printf("embed-object\n");
Py_Initialize();
/* get usermod.message */
pmod = PyImport_ImportModule("usermod");
pstr = PyObject_GetAttrString(pmod, "message");
/* convert string to C */
PyArg_Parse(pstr, "s", &cstr);
printf("%s\n", cstr);
Py_DECREF(pstr);
/* call usermod.transform(usermod.message) */
pfunc = PyObject_GetAttrString(pmod, "transform");
pargs = Py_BuildValue("(s)", cstr);
pstr  = PyEval_CallObject(pfunc, pargs);
PyArg_Parse(pstr, "s", &cstr);
printf("%s\n", cstr);
/* free owned objects */
Py_DECREF(pmod);
Py_DECREF(pstr);
Py_DECREF(pfunc);        /* not really needed in main() */
Py_DECREF(pargs);        /* since all memory goes away  */
Py_Finalize();
}

When compiled and run, the result is the same again:

.../PP4E/Integrate/Embed/Basics$
./embed-object
embed-object
The meaning of life...
THE MEANING OF PYTHON...

However, this output is generated by C this time—first, by
fetching the Python module’s
message
attribute value, and then by fetching and calling the module’s
transform
function object directly and
printing its return value that is sent back to C. Input to the
transform
function is a function argument
here, not a preset global variable. Notice that
message
is fetched as a module attribute this
time, instead of by running its name as a code string; as this shows,
there is often more than one way to accomplish the same goals with
different API calls.

Running functions in modules like this is a simple way to
structure embedding; code in the module file can be changed arbitrarily
without having to recompile the C program that runs it. It also provides
a direct communication model: inputs and outputs to Python code can take
the form of function arguments and return
values.

When we
used
PyRun_String
earlier to run expressions with results, code was executed in the
namespace of an existing Python module. Sometimes, though, it’s more
convenient to
create a brand-new namespace for running code strings that
is independent of any existing module files. The C file in
Example 20-28
shows how; the new
namespace is created as a new Python dictionary object, and a handful of
new API calls are employed in the process:

The main trick here is the new dictionary. Inputs and outputs for
the embedded code strings are mapped to this dictionary by passing it as
the code’s namespace dictionaries in the
PyRun_String
call. The net effect is that the
C program in
Example 20-28
works just like this Python code:

>>>
d = {}
>>>
d['Y'] = 2
>>>
exec('X = 99', d, d)
>>>
exec('X = X + Y', d, d)
>>>
print(d['X'])
101

But here, each Python operation is replaced by a C API
call.

/* make a new dictionary for code string namespace */
#include 
main() {
int cval;
PyObject *pdict, *pval;
printf("embed-dict\n");
Py_Initialize();
/* make a new namespace */
pdict = PyDict_New();
PyDict_SetItemString(pdict, "__builtins__", PyEval_GetBuiltins());
PyDict_SetItemString(pdict, "Y", PyLong_FromLong(2));  /* dict['Y'] = 2   */
PyRun_String("X = 99",  Py_file_input, pdict, pdict);  /* run statements  */
PyRun_String("X = X+Y", Py_file_input, pdict, pdict);  /* same X and Y    */
pval = PyDict_GetItemString(pdict, "X");               /* fetch dict['X'] */
PyArg_Parse(pval, "i", &cval);                         /* convert to C */
printf("%d\n", cval);                                  /* result=101 */
Py_DECREF(pdict);
Py_Finalize();
}

When compiled and run, this C program creates this sort of output,
tailored for this use case:

.../PP4E/Integrate/Embed/Basics$
./embed-dict
embed-dict
101

The output is different this time: it reflects the value of the
Python variable
X
assigned by the
embedded Python code strings and fetched by C. In general, C can fetch
module attributes either by calling
PyObject_GetAttrString
with the module or by using
PyDict_GetItemString
to index the module’s
attribute dictionary (expression strings work, too, but they are less
direct). Here, there is no module at all, so dictionary indexing is used
to access the code’s namespace in C.

Besides allowing you to partition code string namespaces
independent of any Python module files on the underlying system, this
scheme provides a natural communication mechanism. Values that are
stored in the new dictionary before code is run serve as inputs, and
names assigned by the embedded code can later be fetched out of the
dictionary to serve as code outputs. For instance, the variable
Y
in the second string run refers to a name
set to 2 by C;
X
is assigned by the
Python code and fetched later by C code as the printed result.

There is one subtlety in this embedding
mode: dictionaries that serve as namespaces for running code are
generally required to have a
__builtins__
link to the built-in scope
searched last for name lookups, set with code of this form:

PyDict_SetItemString(pdict, "__builtins__", PyEval_GetBuiltins());

This is esoteric, and it is normally handled by Python internally
for modules and built-ins like the
exec
function. For raw dictionaries used as
namespaces, though, we are responsible for setting the link manually if
we expect to reference built-in names. This still holds true in Python
3.X.

Finally, when you
call Python function objects from C, you are actually
running the already compiled bytecode
associated with the object (e.g., a function body),
normally created when the enclosing module is imported. When running
strings, Python must compile the string before running it. Because
compilation is a slow process, this can be a substantial overhead if you
run a code string more than once. Instead, precompile the string to a
bytecode object to be run later, using the API calls illustrated in
Example 20-29
:

The first of these takes the mode flag that is normally passed to
PyRun_String
, as well as a second
string argument that is used only in error messages. The second takes
two namespace dictionaries. These two API calls are used in
Example 20-29
to compile and execute
three strings of Python code in turn.

/* precompile code strings to bytecode objects */
#include 
#include 
#include 
main() {
int i;
char *cval;
PyObject *pcode1, *pcode2, *pcode3, *presult, *pdict;
char *codestr1, *codestr2, *codestr3;
printf("embed-bytecode\n");
Py_Initialize();
codestr1 = "import usermod\nprint(usermod.message)";    /* statements */
codestr2 = "usermod.transform(usermod.message)";        /* expression */
codestr3 = "print('%d:%d' % (X, X ** 2), end=' ')";     /* use input X */
/* make new namespace dictionary */
pdict = PyDict_New();
if (pdict == NULL) return −1;
PyDict_SetItemString(pdict, "__builtins__", PyEval_GetBuiltins());
/* precompile strings of code to bytecode objects */
pcode1 = Py_CompileString(codestr1, "", Py_file_input);
pcode2 = Py_CompileString(codestr2, "", Py_eval_input);
pcode3 = Py_CompileString(codestr3, "", Py_file_input);
/* run compiled bytecode in namespace dict */
if (pcode1 && pcode2 && pcode3) {
(void)    PyEval_EvalCode((PyCodeObject *)pcode1, pdict, pdict);
presult = PyEval_EvalCode((PyCodeObject *)pcode2, pdict, pdict);
PyArg_Parse(presult, "s", &cval);
printf("%s\n", cval);
Py_DECREF(presult);
/* rerun code object repeatedly */
for (i = 0; i <= 10; i++) {
PyDict_SetItemString(pdict, "X", PyLong_FromLong(i));
(void) PyEval_EvalCode((PyCodeObject *)pcode3, pdict, pdict);
}
printf("\n");
}
/* free referenced objects */
Py_XDECREF(pdict);
Py_XDECREF(pcode1);
Py_XDECREF(pcode2);
Py_XDECREF(pcode3);
Py_Finalize();
}

This program combines a variety of techniques that we’ve already
seen. The namespace in which the compiled code strings run, for
instance, is a newly created dictionary (not an existing module object),
and inputs for code strings are passed as preset variables in the
namespace. When built and executed, the first part of the output is
similar to previous examples in this section, but the last line
represents running the same precompiled code string 11 times:

.../PP4E/Integrate/Embed/Basics$
embed-bytecode
embed-bytecode
The meaning of life...
THE MEANING OF PYTHON...
0:0 1:1 2:4 3:9 4:16 5:25 6:36 7:49 8:64 9:81 10:100

If your system executes Python code strings multiple times, it is
a major speedup to precompile to bytecode in this fashion. This step is
not required in other contexts that invoke callable Python
objects—including the common embedding use case presented in the
next section.

• Fetch and call
  functions
  by event name from
  one or more
  module
  files
  

• Fetch and run code
  strings
  associated with
  event names in a
  database
  

• Extract and run code associated with event
  tags
  in HTML or XML
  

• Run Python code that calls back to C to tell it what should be
  run
  

This section’s C and Python files demonstrate the coding
techniques used to implement explicitly registered callback handlers.
First, the C file in
Example 20-30
implements interfaces
for registering Python handlers, as well as code to run those handlers
in response to later events:

In other words, this example uses both the embedding and the
extending interfaces we’ve already met to register and invoke Python
event handler code. Study
Example 20-30
for more on its
operation.

#include 
#include 
/***********************************************/
/* 1) code to route events to Python object    */
/* note that we could run strings here instead */
/***********************************************/
static PyObject *Handler = NULL;     /* keep Python object in C */
void Route_Event(char *label, int count)
{
char *cres;
PyObject *args, *pres;
/* call Python handler */
args = Py_BuildValue("(si)", label, count);   /* make arg-list */
pres = PyEval_CallObject(Handler, args);      /* apply: run a call */
Py_DECREF(args);                              /* add error checks */
if (pres != NULL) {
/* use and decref handler result */
PyArg_Parse(pres, "s", &cres);
printf("%s\n", cres);
Py_DECREF(pres);
}
}
/*****************************************************/
/* 2) python extension module to register handlers   */
/* python imports this module to set handler objects */
/*****************************************************/
static PyObject *
Register_Handler(PyObject *self, PyObject *args)
{
/* save Python callable object */
Py_XDECREF(Handler);                 /* called before? */
PyArg_Parse(args, "(O)", &Handler);  /* one argument */
Py_XINCREF(Handler);                 /* add a reference */
Py_INCREF(Py_None);                  /* return 'None': success */
return Py_None;
}
static PyObject *
Trigger_Event(PyObject *self, PyObject *args)
{
/* let Python simulate event caught by C */
static count = 0;
Route_Event("spam", count++);
Py_INCREF(Py_None);
return Py_None;
}
static PyMethodDef cregister_methods[] = {
{"setHandler",    Register_Handler, METH_VARARGS, ""},  /* name, &func,... */
{"triggerEvent",  Trigger_Event,    METH_VARARGS, ""},
{NULL, NULL, 0, NULL}                                   /* end of table */
};
static struct PyModuleDef cregistermodule = {
PyModuleDef_HEAD_INIT,
"cregister",       /* name of module */
"cregister mod",   /* module documentation, may be NULL */
−1,                /* size of per-interpreter module state, −1=in global vars */
cregister_methods  /* link to methods table */
};
PyMODINIT_FUNC
PyInit_cregister()                      /* called on first import */
{
return PyModule_Create(&cregistermodule);
}

Ultimately, this C file is an extension module for Python, not a
standalone C program that embeds Python (though C could just as well be
on top). To compile it into a dynamically loaded module file, run the
makefile in
Example 20-31
on
Cygwin (and use something similar on other platforms). As we learned
earlier in this chapter, the resulting
cregister.dll
file will be loaded when first
imported by a Python script if it is placed in a directory on Python’s
module search path (e.g., in
.
or
PYTHONPATH
settings).

######################################################################
# Cygwin makefile that builds cregister.dll. a dynamically loaded
# C extension module (shareable), which is imported by register.py
######################################################################
PYLIB = /usr/local/bin
PYINC = /usr/local/include/python3.1
CMODS = cregister.dll
all: $(CMODS)
cregister.dll: cregister.c
gcc cregister.c -g -I$(PYINC) -shared -L$(PYLIB) -lpython3.1 -o $@
clean:
rm -f *.pyc $(CMODS)

Now that we have a C extension module set to register and dispatch
Python handlers, all we need are some Python handlers. The Python module
shown in
Example 20-32
defines
two callback handler functions and imports the C extension module to
register handlers and trigger events.

"""
#########################################################################
in Python, register for and handle event callbacks from the C language;
compile and link the C code, and launch this with 'python register.py'
#########################################################################
"""
####################################
# C calls these Python functions;
# handle an event, return a result
####################################
def callback1(label, count):
return 'callback1 => %s number %i' % (label, count)
def callback2(label, count):
return 'callback2 => ' +  label * count
#######################################
# Python calls a C extension module
# to register handlers, trigger events
#######################################
import cregister
print('\nTest1:')
cregister.setHandler(callback1)      # register callback function
for i in range(3):
cregister.triggerEvent()         # simulate events caught by C layer
print('\nTest2:')
cregister.setHandler(callback2)
for i in range(3):
cregister.triggerEvent()         # routes these events to callback2

That’s it—the Python/C callback integration is set to go. To kick
off the system, run the Python script; it registers one handler
function, forces three events to be triggered, and then changes the
event handler and does it again:

.../PP4E/Integrate/Embed/Regist$
make -f makefile.regist
gcc cregister.c -g -I/usr/local/include/python3.1 -shared -L/usr/local/bin
-lpython3.1 -o cregister.dll
.../PP4E/Integrate/Embed/Regist$
python register.py
Test1:
callback1 => spam number 0
callback1 => spam number 1
callback1 => spam number 2
Test2:
callback2 => spamspamspam
callback2 => spamspamspamspam
callback2 => spamspamspamspamspam

This output is printed by the C event router function, but its
content is the return values of the handler functions in the Python
module. Actually, something pretty wild is going on under the hood. When
Python forces an event to trigger, control flows between languages like
this:

That is, we jump from Python to C to Python and back again. Along
the way, control passes through both extending and embedding interfaces.
When the Python callback handler is running, two Python levels are
active, and one C level in the middle. Luckily, this just works;
Python’s API is reentrant, so you don’t need to be concerned about
having multiple Python interpreter levels active at the same time. Each
level runs different code and operates independently.

Trace through this example’s output and code for more
illumination. Here, we’re moving on to the last quick example we have
time and space to explore—in the name of symmetry, using Python classes
from C.