Howto Clinic

Argument Clinic How-ToRelease 3.4.3

Guido van Rossumand the Python development team

March 30, 2015

Python Software FoundationEmail: [email protected]

Contents

1 The Goals Of Argument Clinic 2

2 Basic Concepts And Usage 2

3 Converting Your First Function 3

4 Advanced Topics 84.1 Symbolic default values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2 Renaming the C functions and variables generated by Argument Clinic . . . . . . . . . . . . . . . 94.3 Converting functions using PyArg_UnpackTuple . . . . . . . . . . . . . . . . . . . . . . . . . . 94.4 Optional Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.5 Using real Argument Clinic converters, instead of legacy converters . . . . . . . . . . . . . . . 114.6 Py_buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.7 Advanced converters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.8 Parameter default values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.9 The NULL default value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.10 Expressions specified as default values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.11 Using a return converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.12 Cloning existing functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.13 Calling Python code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.14 Using a self converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.15 Writing a custom converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.16 Writing a custom return converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.17 METH_O and METH_NOARGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.18 tp_new and tp_init functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.19 Changing and redirecting Clinics output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.20 The #ifdef trick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.21 Using Argument Clinic in Python files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

author Larry Hastings

Abstract

Argument Clinic is a preprocessor for CPython C files. Its purpose is to automate all the boilerplate involvedwith writing argument parsing code for builtins. This document shows you how to convert your first Cfunction to work with Argument Clinic, and then introduces some advanced topics on Argument Clinicusage.Currently Argument Clinic is considered internal-only for CPython. Its use is not supported for files outsideCPython, and no guarantees are made regarding backwards compatibility for future versions. In other words:if you maintain an external C extension for CPython, youre welcome to experiment with Argument Clinic inyour own code. But the version of Argument Clinic that ships with CPython 3.5 could be totally incompatibleand break all your code.

1 The Goals Of Argument Clinic

Argument Clinics primary goal is to take over responsibility for all argument parsing code inside CPython. Thismeans that, when you convert a function to work with Argument Clinic, that function should no longer do any ofits own argument parsingthe code generated by Argument Clinic should be a black box to you, where CPythoncalls in at the top, and your code gets called at the bottom, with PyObject *args (and maybe PyObject*kwargs) magically converted into the C variables and types you need.

In order for Argument Clinic to accomplish its primary goal, it must be easy to use. Currently, working withCPythons argument parsing library is a chore, requiring maintaining redundant information in a surprising numberof places. When you use Argument Clinic, you dont have to repeat yourself.

Obviously, no one would want to use Argument Clinic unless its solving their problemand without creating newproblems of its own. So its paramount that Argument Clinic generate correct code. Itd be nice if the code wasfaster, too, but at the very least it should not introduce a major speed regression. (Eventually Argument Clinicshould make a major speedup possiblewe could rewrite its code generator to produce tailor-made argumentparsing code, rather than calling the general-purpose CPython argument parsing library. That would make for thefastest argument parsing possible!)

Additionally, Argument Clinic must be flexible enough to work with any approach to argument parsing. Pythonhas some functions with some very strange parsing behaviors; Argument Clinics goal is to support all of them.

Finally, the original motivation for Argument Clinic was to provide introspection signatures for CPythonbuiltins. It used to be, the introspection query functions would throw an exception if you passed in a builtin.With Argument Clinic, thats a thing of the past!

One idea you should keep in mind, as you work with Argument Clinic: the more information you give it, the betterjob itll be able to do. Argument Clinic is admittedly relatively simple right now. But as it evolves it will get moresophisticated, and it should be able to do many interesting and smart things with all the information you give it.

2 Basic Concepts And Usage

Argument Clinic ships with CPython; youll find it in Tools/clinic/clinic.py. If you run that script,specifying a C file as an argument:

% python3 Tools/clinic/clinic.py foo.c

Argument Clinic will scan over the file looking for lines that look exactly like this:

/*[clinic input]

When it finds one, it reads everything up to a line that looks exactly like this:

[clinic start generated code]*/

Everything in between these two lines is input for Argument Clinic. All of these lines, including the beginningand ending comment lines, are collectively called an Argument Clinic block.

When Argument Clinic parses one of these blocks, it generates output. This output is rewritten into the C fileimmediately after the block, followed by a comment containing a checksum. The Argument Clinic block nowlooks like this:

/*[clinic input]... clinic input goes here ...[clinic start generated code]*/... clinic output goes here .../*[clinic end generated code: checksum=...]*/

If you run Argument Clinic on the same file a second time, Argument Clinic will discard the old output and writeout the new output with a fresh checksum line. However, if the input hasnt changed, the output wont changeeither.

You should never modify the output portion of an Argument Clinic block. Instead, change the input until itproduces the output you want. (Thats the purpose of the checksumto detect if someone changed the output, asthese edits would be lost the next time Argument Clinic writes out fresh output.)

For the sake of clarity, heres the terminology well use with Argument Clinic:

The first line of the comment (/*[clinic input]) is the start line.

The last line of the initial comment ([clinic start generated code]*/) is the end line.

The last line (/*[clinic end generated code: checksum=...]*/) is the checksum line.

In between the start line and the end line is the input.

In between the end line and the checksum line is the output.

All the text collectively, from the start line to the checksum line inclusively, is the block. (A block thathasnt been successfully processed by Argument Clinic yet doesnt have output or a checksum line, but itsstill considered a block.)

3 Converting Your First Function

The best way to get a sense of how Argument Clinic works is to convert a function to work with it. Here, then, arethe bare minimum steps youd need to follow to convert a function to work with Argument Clinic. Note that forcode you plan to check in to CPython, you really should take the conversion farther, using some of the advancedconcepts youll see later on in the document (like return converters and self converters). But well keep itsimple for this walkthrough so you can learn.

Lets dive in!

0. Make sure youre working with a freshly updated checkout of the CPython trunk.

1. Find a Python builtin that calls either PyArg_ParseTuple() orPyArg_ParseTupleAndKeywords(), and hasnt been converted to work with Argument Clinic yet.For my example Im using _pickle.Pickler.dump().

2. If the call to the PyArg_Parse function uses any of the following format units:

O&O!eses#etet#

or if it has multiple calls to PyArg_ParseTuple(), you should choose a different function. ArgumentClinic does support all of these scenarios. But these are advanced topicslets do something simpler foryour first function.

Also, if the function has multiple calls to PyArg_ParseTuple() orPyArg_ParseTupleAndKeywords() where it supports different types for the same argument,

or if the function uses something besides PyArg_Parse functions to parse its arguments, it probablyisnt suitable for conversion to Argument Clinic. Argument Clinic doesnt support generic functions orpolymorphic parameters.

3. Add the following boilerplate above the function, creating our block:

/*[clinic input][clinic start generated code]*/

4. Cut the docstring and paste it in between the [clinic] lines, removing all the junk that makes it a properlyquoted C string. When youre done you should have just the text, based at the left margin, with no line widerthan 80 characters. (Argument Clinic will preserve indents inside the docstring.)

If the old docstring had a first line that looked like a function signature, throw that line away. (The docstringdoesnt need it anymorewhen you use help() on your builtin in the future, the first line will be builtautomatically based on the functions signature.)

Sample:

/*[clinic input]Write a pickled representation of obj to the open file.[clinic start generated code]*/

5. If your docstring doesnt have a summary line, Argument Clinic will complain. So lets make sure it hasone. The summary line should be a paragraph consisting of a single 80-column line at the beginning ofthe docstring.

(Our example docstring consists solely of a summary line, so the sample code doesnt have to change forthis step.)

6. Above the docstring, enter the name of the function, followed by a blank line. This should be the Pythonname of the function, and should be the full dotted path to the functionit should start with the name of themodule, include any sub-modules, and if the function is a method on a class it should include the class nametoo.

Sample:

/*[clinic input]_pickle.Pickler.dump

Write a pickled representation of obj to the open file.[clinic start generated code]*/

7. If this is the first time that module or class has been used with Argument Clinic in this C file, you mustdeclare the module and/or class. Proper Argument Clinic hygiene prefers declaring these in a separateblock somewhere near the top of the C file, in the same way that include files and statics go at the top. (Inour sample code well just show the two blocks next to each other.)

The name of the class and module should be the same as the one seen by Python. Check the name definedin the PyModuleDef or PyTypeObject as appropriate.

When you declare a class, you must also specify two aspects of its type in C: the type declaration youd usefor a pointer to an instance of this class, and a pointer to the PyTypeObject for this class.

Sample:

/*[clinic input]module _pickleclass _pickle.Pickler "PicklerObject *" "&Pickler_Type"[clinic start generated code]*/



8. Declare each of the parameters to the function. Each parameter should get its own line. All the parameterlines should be indented from the function name and the docstring.

The general form of these parameter lines is as follows:

name_of_parameter: converter

If the parameter has a default value, add that after the converter:

name_of_parameter: converter = default_value

Argument Clinics support for default values is quite sophisticated; please see the section below on defaultvalues for more information.

Add a blank line below the parameters.

Whats a converter? It establishes both the type of the variable used in C, and the method to convert thePython value into a C value at runtime. For now youre going to use whats called a legacy converteraconvenience syntax intended to make porting old code into Argument Clinic easier.

For each parameter, copy the format unit for that parameter from the PyArg_Parse() format argumentand specify that as its converter, as a quoted string. (format unit is the formal name for the one-to-threecharacter substring of the format parameter that tells the argument parsing function what the type of thevariable is and how to convert it. For more on format units please see arg-parsing.)

For multicharacter format units like z#, use the entire two-or-three character string.

Sample:



obj: 'O'


9. If your function has | in the format string, meaning some parameters have default values, you can ignore it.Argument Clinic infers which parameters are optional based on whether or not they have default values.

If your function has $ in the format string, meaning it takes keyword-only arguments, specify * on a line byitself before the first keyword-only argument, indented the same as the parameter lines.

(_pickle.Pickler.dump has neither, so our sample is unchanged.)

10. If the existing C function calls PyArg_ParseTuple() (as opposed toPyArg_ParseTupleAndKeywords()), then all its arguments are positional-only.

To mark all parameters as positional-only in Argument Clinic, add a / on a line by itself after the lastparameter, indented the same as the parameter lines.

Currently this is all-or-nothing; either all parameters are positional-only, or none of them are. (In the futureArgument Clinic may relax this restriction.)

Sample:



obj: 'O'/


11. Its helpful to write a per-parameter docstring for each parameter. But per-parameter docstrings are optional;you can skip this step if you prefer.

Heres how to add a per-parameter docstring. The first line of the per-parameter docstring must be indentedfurther than the parameter definition. The left margin of this first line establishes the left margin for thewhole per-parameter docstring; all the text you write will be outdented by this amount. You can write asmuch text as you like, across multiple lines if you wish.

Sample:



obj: 'O'The object to be pickled.

/


12. Save and close the file, then run Tools/clinic/clinic.py on it. With luck everything worked andyour block now has output! Reopen the file in your text editor to see:

/*[clinic input]module _pickleclass _pickle.Pickler "PicklerObject *" "&Pickler_Type"[clinic start generated code]*//*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/



/


PyDoc_STRVAR(_pickle_Pickler_dump__doc__,"Write a pickled representation of obj to the open file.\n""\n"...static PyObject *_pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj)/*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/

Obviously, if Argument Clinic didnt produce any output, its because it found an error in your input. Keepfixing your errors and retrying until Argument Clinic processes your file without complaint.

13. Double-check that the argument-parsing code Argument Clinic generated looks basically the same as theexisting code.

First, ensure both places use the same argument-parsing function. The existing code must call eitherPyArg_ParseTuple() or PyArg_ParseTupleAndKeywords(); ensure that the code generatedby Argument Clinic calls the exact same function.

Second, the format string passed in to PyArg_ParseTuple() orPyArg_ParseTupleAndKeywords() should be exactly the same as the hand-written one in theexisting function, up to the colon or semi-colon.

(Argument Clinic always generates its format strings with a : followed by the name of the function. If theexisting codes format string ends with ;, to provide usage help, this change is harmlessdont worry aboutit.)

Third, for parameters whose format units require two arguments (like a length variable, or an encodingstring, or a pointer to a conversion function), ensure that the second argument is exactly the same betweenthe two invocations.

Fourth, inside the output portion of the block youll find a preprocessor macro defining the appropriate staticPyMethodDef structure for this builtin:

#define __PICKLE_PICKLER_DUMP_METHODDEF \{"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__},

This static structure should be exactly the same as the existing static PyMethodDef structure for thisbuiltin.

If any of these items differ in any way, adjust your Argument Clinic function specification and rerunTools/clinic/clinic.py until they are the same.

14. Notice that the last line of its output is the declaration of your impl function. This is where the builtinsimplementation goes. Delete the existing prototype of the function youre modifying, but leave the openingcurly brace. Now delete its argument parsing code and the declarations of all the variables it dumps thearguments into. Notice how the Python arguments are now arguments to this impl function; if the imple-mentation used different names for these variables, fix it.

Lets reiterate, just because its kind of weird. Your code should now look like this:

static return_typeyour_function_impl(...)/*[clinic end generated code: checksum=...]*/{...

Argument Clinic generated the checksum line and the function prototype just above it. You should write theopening (and closing) curly braces for the function, and the implementation inside.

Sample:

/*[clinic input]module _pickleclass _pickle.Pickler "PicklerObject *" "&Pickler_Type"[clinic start generated code]*//*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/



/


PyDoc_STRVAR(__pickle_Pickler_dump__doc__,"Write a pickled representation of obj to the open file.\n""\n"...static PyObject *_pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj)/*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/{

/* Check whether the Pickler was initialized correctly (issue3664).Developers often forget to call __init__() in their subclasses, whichwould trigger a segfault without this check. */

if (self->write == NULL) {PyErr_Format(PicklingError,

"Pickler.__init__() was not called by %s.__init__()",Py_TYPE(self)->tp_name);

return NULL;}

if (_Pickler_ClearBuffer(self) < 0)return NULL;

...

15. Remember the macro with the PyMethodDef structure for this function? Find the existingPyMethodDef structure for this function and replace it with a reference to the macro. (If the builtinis at module scope, this will probably be very near the end of the file; if the builtin is a class method, thiswill probably be below but relatively near to the implementation.)

Note that the body of the macro contains a trailing comma. So when you replace the existing staticPyMethodDef structure with the macro, dont add a comma to the end.

Sample:

static struct PyMethodDef Pickler_methods[] = {__PICKLE_PICKLER_DUMP_METHODDEF__PICKLE_PICKLER_CLEAR_MEMO_METHODDEF{NULL, NULL} /* sentinel */

};

16. Compile, then run the relevant portions of the regression-test suite. This change should not introduce anynew compile-time warnings or errors, and there should be no externally-visible change to Pythons behavior.

Well, except for one difference: inspect.signature() run on your function should now provide avalid signature!

Congratulations, youve ported your first function to work with Argument Clinic!

4 Advanced Topics

Now that youve had some experience working with Argument Clinic, its time for some advanced topics.

4.1 Symbolic default values

The default value you provide for a parameter cant be any arbitrary expression. Currently the following areexplicitly supported:

Numeric constants (integer and float)

String constants

True, False, and None

Simple symbolic constants like sys.maxsize, which must start with the name of the module

In case youre curious, this is implemented in from_builtin() in Lib/inspect.py.

(In the future, this may need to get even more elaborate, to allow full expressions like CONSTANT - 1.)

4.2 Renaming the C functions and variables generated by Argument Clinic

Argument Clinic automatically names the functions it generates for you. Occasionally this may cause a problem,if the generated name collides with the name of an existing C function. Theres an easy solution: override thenames used for the C functions. Just add the keyword "as" to your function declaration line, followed by thefunction name you wish to use. Argument Clinic will use that function name for the base (generated) function,then add "_impl" to the end and use that for the name of the impl function.

For example, if we wanted to rename the C function names generated for pickle.Pickler.dump, itd looklike this:

/*[clinic input]pickle.Pickler.dump as pickler_dumper

...

The base function would now be named pickler_dumper(), and the impl function would now be namedpickler_dumper_impl().

Similarly, you may have a problem where you want to give a parameter a specific Python name, but that namemay be inconvenient in C. Argument Clinic allows you to give a parameter different names in Python and in C,using the same "as" syntax:

/*[clinic input]pickle.Pickler.dump

obj: objectfile as file_obj: objectprotocol: object = NULL

*fix_imports: bool = True

Here, the name used in Python (in the signature and the keywords array) would be file, but the C variablewould be named file_obj.

You can use this to rename the self parameter too!

4.3 Converting functions using PyArg_UnpackTuple

To convert a function parsing its arguments with PyArg_UnpackTuple(), simply write out all the arguments,specifying each as an object. You may specify the type argument to cast the type as appropriate. All argumentsshould be marked positional-only (add a / on a line by itself after the last argument).

Currently the generated code will use PyArg_ParseTuple(), but this will change soon.

4.4 Optional Groups

Some legacy functions have a tricky approach to parsing their arguments: they count the number of positionalarguments, then use a switch statement to call one of several different PyArg_ParseTuple() calls depend-ing on how many positional arguments there are. (These functions cannot accept keyword-only arguments.) Thisapproach was used to simulate optional arguments back before PyArg_ParseTupleAndKeywords() wascreated.

While functions using this approach can often be converted to use PyArg_ParseTupleAndKeywords(),optional arguments, and default values, its not always possible. Some of these legacy functions have behaviorsPyArg_ParseTupleAndKeywords() doesnt directly support. The most obvious example is the builtinfunction range(), which has an optional argument on the left side of its required argument! Another example iscurses.window.addch(), which has a group of two arguments that must always be specified together. (Thearguments are called x and y; if you call the function passing in x, you must also pass in yand if you dont passin x you may not pass in y either.)

In any case, the goal of Argument Clinic is to support argument parsing for all existing CPython builtins withoutchanging their semantics. Therefore Argument Clinic supports this alternate approach to parsing, using what arecalled optional groups. Optional groups are groups of arguments that must all be passed in together. They can beto the left or the right of the required arguments. They can only be used with positional-only parameters.

Note: Optional groups are only intended for use when converting functions that make multiple calls toPyArg_ParseTuple()! Functions that use any other approach for parsing arguments should almost neverbe converted to Argument Clinic using optional groups. Functions using optional groups currently cannot haveaccurate sigantures in Python, because Python just doesnt understand the concept. Please avoid using optionalgroups wherever possible.

To specify an optional group, add a [ on a line by itself before the parameters you wish to group together, and a] on a line by itself after these parameters. As an example, heres how curses.window.addch uses optionalgroups to make the first two parameters and the last parameter optional:

/*[clinic input]

curses.window.addch

[x: int

X-coordinate.y: int

Y-coordinate.]

ch: objectCharacter to add.

[attr: long

Attributes for the character.]/

...

Notes:

For every optional group, one additional parameter will be passed into the impl function representing thegroup. The parameter will be an int named group_{direction}_{number}, where {direction}is either right or left depending on whether the group is before or after the required parameters, and{number} is a monotonically increasing number (starting at 1) indicating how far away the group is fromthe required parameters. When the impl is called, this parameter will be set to zero if this group was unused,and set to non-zero if this group was used. (By used or unused, I mean whether or not the parametersreceived arguments in this invocation.)

If there are no required arguments, the optional groups will behave as if theyre to the right of the requiredarguments.

In the case of ambiguity, the argument parsing code favors parameters on the left (before the requiredparameters).

Optional groups can only contain positional-only parameters.

Optional groups are only intended for legacy code. Please do not use optional groups for new code.

4.5 Using real Argument Clinic converters, instead of legacy converters

To save time, and to minimize how much you need to learn to achieve your first port to Argument Clinic, the walk-through above tells you to use legacy converters. Legacy converters are a convenience, designed explicitly tomake porting existing code to Argument Clinic easier. And to be clear, their use is acceptable when porting codefor Python 3.4.

However, in the long term we probably want all our blocks to use Argument Clinics real syntax for converters.Why? A couple reasons:

The proper converters are far easier to read and clearer in their intent.

There are some format units that are unsupported as legacy converters, because they require arguments,and the legacy converter syntax doesnt support specifying arguments.

In the future we may have a new argument parsing library that isnt restricted to whatPyArg_ParseTuple() supports; this flexibility wont be available to parameters using legacy convert-ers.

Therefore, if you dont mind a little extra effort, please use the normal converters instead of legacy converters.

In a nutshell, the syntax for Argument Clinic (non-legacy) converters looks like a Python function call. How-ever, if there are no explicit arguments to the function (all functions take their default values), you may omit theparentheses. Thus bool and bool() are exactly the same converters.

All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the fol-lowing arguments:

c_default The default value for this parameter when defined in C. Specifically, this will be theinitializer for the variable declared in the parse function. See the section on default values forhow to use this. Specified as a string.

annotation The annotation value for this parameter. Not currently supported, because PEP 8mandates that the Python library may not use annotations.

In addition, some converters accept additional arguments. Here is a list of these arguments, along with theirmeanings:

bitwise Only supported for unsigned integers. The native integer value of this Python argumentwill be written to the parameter without any range checking, even for negative values.

converter Only supported by the object converter. Specifies the name of a C converter func-tion to use to convert this object to a native type.

encoding Only supported for strings. Specifies the encoding to use when converting this stringfrom a Python str (Unicode) value into a C char * value.

length Only supported for strings. If true, requests that the length of the string bepassed in to the impl function, just after the string parameter, in a parameter named_length.

nullable Only supported for strings. If true, this parameter may also be set to None, in whichcase the C parameter will be set to NULL.

subclass_of Only supported for the object converter. Requires that the Python value be asubclass of a Python type, as expressed in C.

types Only supported for the object (and self) converter. Specifies the C type that will be usedto declare the variable. Default value is "PyObject *".

types A string containing a list of Python types (and possibly pseudo-types); this restricts the al-lowable Python argument to values of these types. (This is not a general-purpose facility; as arule it only supports specific lists of types as shown in the legacy converter table.)

zeroes Only supported for strings. If true, embedded NUL bytes (\\0) are permitted inside thevalue.

Please note, not every possible combination of arguments will work. Often these arguments are implementedinternally by specific PyArg_ParseTuple format units, with specific behavior. For example, currently youcannot call str and pass in zeroes=True without also specifying an encoding; although its perfectly rea-sonable to think this would work, these semantics dont map to any existing format unit. So Argument Clinicdoesnt support it. (Or, at least, not yet.)

Below is a table showing the mapping of legacy converters into real Argument Clinic converters. On the left is thelegacy converter, on the right is the text youd replace it with.

B unsigned_char(bitwise=True)b unsigned_charc charC int(types=str)d doubleD Py_complexes# str(encoding=name_of_encoding, length=True, zeroes=True)es str(encoding=name_of_encoding)et# str(encoding=name_of_encoding, types=bytes bytearray str, length=True)et str(encoding=name_of_encoding, types=bytes bytearray str)f floath shortH unsigned_short(bitwise=True)i intI unsigned_int(bitwise=True)k unsigned_long(bitwise=True)K unsigned_PY_LONG_LONG(bitwise=True)L PY_LONG_LONGn Py_ssize_tO! object(subclass_of=&PySomething_Type)O& object(converter=name_of_c_function)O objectp bools# str(length=True)S PyBytesObjects strs* Py_buffer(types=str bytes bytearray buffer)u# Py_UNICODE(length=True)u Py_UNICODEU unicodew* Py_buffer(types=bytearray rwbuffer)y# str(types=bytes, length=True)Y PyByteArrayObjecty str(types=bytes)y* Py_bufferZ# Py_UNICODE(nullable=True, length=True)z# str(nullable=True, length=True)Z Py_UNICODE(nullable=True)z str(nullable=True)z* Py_buffer(types=str bytes bytearray buffer, nullable=True)

As an example, heres our sample pickle.Pickler.dump using the proper converter:

/*[clinic input]pickle.Pickler.dump

obj: object

The object to be pickled./


Argument Clinic will show you all the converters it has available. For each converter itll show you all theparameters it accepts, along with the default value for each parameter. Just run Tools/clinic/clinic.py--converters to see the full list.

4.6 Py_buffer

When using the Py_buffer converter (or the s*, w*, *y, or z* legacy converters), you must notcall PyBuffer_Release() on the provided buffer. Argument Clinic generates code that does it for you (inthe parsing function).

4.7 Advanced converters

Remeber those format units you skipped for your first time because they were advanced? Heres how to handlethose too.

The trick is, all those format units take argumentseither conversion functions, or types, or strings specifyingan encoding. (But legacy converters dont support arguments. Thats why we skipped them for your firstfunction.) The argument you specified to the format unit is now an argument to the converter; this argument iseither converter (for O&), subclass_of (for O!), or encoding (for all the format units that start with e).

When using subclass_of, you may also want to use the other custom argument for object(): type, whichlets you set the type actually used for the parameter. For example, if you want to ensure that the object is a subclassof PyUnicode_Type, you probably want to use the converter object(type=PyUnicodeObject *,subclass_of=&PyUnicode_Type).

One possible problem with using Argument Clinic: it takes away some possible flexibility for the format unitsstarting with e. When writing a PyArg_Parse call by hand, you could theoretically decide at runtime whatencoding string to pass in to PyArg_ParseTuple(). But now this string must be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; it made supporting this format unit much easier, and mayallow for future optimizations. This restriction doesnt seem unreasonable; CPython itself always passes in statichard-coded encoding strings for parameters whose format units start with e.

4.8 Parameter default values

Default values for parameters can be any of a number of values. At their simplest, they can be string, int, or floatliterals:

foo: str = "abc"bar: int = 123bat: float = 45.6

They can also use any of Pythons built-in constants:

yep: bool = Truenope: bool = Falsenada: object = None

Theres also special support for a default value of NULL, and for simple expressions, documented in the followingsections.

4.9 The NULL default value

For string and object parameters, you can set them to None to indicate that theres no default. However, thatmeans the C variable will be initialized to Py_None. For conveniences sakes, theres a special value calledNULL for just this reason: from Pythons perspective it behaves like a default value of None, but the C variable isinitialized with NULL.

4.10 Expressions specified as default values

The default value for a parameter can be more than just a literal value. It can be an entire expression, usingmath operators and looking up attributes on objects. However, this support isnt exactly simple, because of somenon-obvious semantics.

Consider the following example:

foo: Py_ssize_t = sys.maxsize - 1

sys.maxsize can have different values on different platforms. Therefore Argument Clinic cant simply evaluatethat expression locally and hard-code it in C. So it stores the default in such a way that it will get evaluated atruntime, when the user asks for the functions signature.

What namespace is available when the expression is evaluated? Its evaluated in the context of the module thebuiltin came from. So, if your module has an attribute called max_widgets, you may simply use it:

foo: Py_ssize_t = max_widgets

If the symbol isnt found in the current module, it fails over to looking in sys.modules. Thats how it canfind sys.maxsize for example. (Since you dont know in advance what modules the user will load into theirinterpreter, its best to restrict yourself to modules that are preloaded by Python itself.)

Evaluating default values only at runtime means Argument Clinic cant compute the correct equivalent C defaultvalue. So you need to tell it explicitly. When you use an expression, you must also specify the equivalentexpression in C, using the c_default parameter to the converter:

foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1

Another complication: Argument Clinic cant know in advance whether or not the expression you supply is valid.It parses it to make sure it looks legal, but it cant actually know. You must be very careful when using expressionsto specify values that are guaranteed to be valid at runtime!

Finally, because expressions must be representable as static C values, there are many restrictions on legal expres-sions. Heres a list of Python features youre not permitted to use:

Function calls.

Inline if statements (3 if foo else 5).

Automatic sequence unpacking (*[1, 2, 3]).

List/set/dict comprehensions and generator expressions.

Tuple/list/set/dict literals.

4.11 Using a return converter

By default the impl function Argument Clinic generates for you returns PyObject *. But your C function oftencomputes some C type, then converts it into the PyObject * at the last moment. Argument Clinic handlesconverting your inputs from Python types into native C typeswhy not have it convert your return value from anative C type into a Python type too?

Thats what a return converter does. It changes your impl function to return some C type, then adds code to thegenerated (non-impl) function to handle converting that value into the appropriate PyObject *.

The syntax for return converters is similar to that of parameter converters. You specify the return converter like itwas a return annotation on the function itself. Return converters behave much the same as parameter converters;

they take arguments, the arguments are all keyword-only, and if youre not changing any of the default argumentsyou can omit the parentheses.

(If you use both "as" and a return converter for your function, the "as" should come before the return converter.)

Theres one additional complication when using return converters: how do you indicate an error has occured?Normally, a function returns a valid (non-NULL) pointer for success, and NULL for failure. But if you use aninteger return converter, all integers are valid. How can Argument Clinic detect an error? Its solution: each returnconverter implicitly looks for a special value that indicates an error. If you return that value, and an error has beenset (PyErr_Occurred() returns a true value), then the generated code will propogate the error. Otherwise itwill encode the value you return like normal.

Currently Argument Clinic supports only a few return converters:

boolintunsigned intlongunsigned intsize_tPy_ssize_tfloatdoubleDecodeFSDefault

None of these take parameters. For the first three, return -1 to indicate error. For DecodeFSDefault, the returntype is char *; return a NULL pointer to indicate an error.

(Theres also an experimental NoneType converter, which lets you return Py_None on success or NULL onfailure, without having to increment the reference count on Py_None. Im not sure it adds enough clarity to beworth using.)

To see all the return converters Argument Clinic supports, along with their parameters (if any), just runTools/clinic/clinic.py --converters for the full list.

4.12 Cloning existing functions

If you have a number of functions that look similar, you may be able to use Clinics clone feature. When youclone an existing function, you reuse:

its parameters, including

their names,

their converters, with all parameters,

their default values,

their per-parameter docstrings,

their kind (whether theyre positional only, positional or keyword, or keyword only), and

its return converter.

The only thing not copied from the original function is its docstring; the syntax allows you to specify a newdocstring.

Heres the syntax for cloning a function:

/*[clinic input]module.class.new_function [as c_basename] = module.class.existing_function

Docstring for new_function goes here.[clinic start generated code]*/

(The functions can be in different modules or classes. I wrote module.class in the sample just to illustratethat you must use the full path to both functions.)

Sorry, theres no syntax for partially-cloning a function, or cloning a function then modifying it. Cloning is anall-or nothing proposition.

Also, the function you are cloning from must have been previously defined in the current file.

4.13 Calling Python code

The rest of the advanced topics require you to write Python code which lives inside your C file and modifiesArgument Clinics runtime state. This is simple: you simply define a Python block.

A Python block uses different delimiter lines than an Argument Clinic function block. It looks like this:

/*[python input]# python code goes here[python start generated code]*/

All the code inside the Python block is executed at the time its parsed. All text written to stdout inside the blockis redirected into the output after the block.

As an example, heres a Python block that adds a static integer variable to the C code:

/*[python input]print('static int __ignored_unused_variable__ = 0;')[python start generated code]*/static int __ignored_unused_variable__ = 0;/*[python checksum:...]*/

4.14 Using a self converter

Argument Clinic automatically adds a self parameter for you using a default converter. It automatically sets thetype of this parameter to the pointer to an instance you specified when you declared the type. However, youcan override Argument Clinics converter and specify one yourself. Just add your own self parameter as the firstparameter in a block, and ensure that its converter is an instance of self_converter or a subclass thereof.

Whats the point? This lets you override the type of self, or give it a different default name.

How do you specify the custom type you want to cast self to? If you only have one or two functions with thesame type for self, you can directly use Argument Clinics existing self converter, passing in the type youwant to use as the type parameter:

/*[clinic input]

_pickle.Pickler.dump

self: self(type="PicklerObject *")obj: object/

Write a pickled representation of the given object to the open file.[clinic start generated code]*/

On the other hand, if you have a lot of functions that will use the same type for self, its best to create your ownconverter, subclassing self_converter but overwriting the type member:

/*[python input]class PicklerObject_converter(self_converter):

type = "PicklerObject *"[python start generated code]*/

/*[clinic input]

_pickle.Pickler.dump

self: PicklerObjectobj: object/

Write a pickled representation of the given object to the open file.[clinic start generated code]*/

4.15 Writing a custom converter

As we hinted at in the previous section... you can write your own converters! A converter is simply a Python classthat inherits from CConverter. The main purpose of a custom converter is if you have a parameter using theO& format unitparsing this parameter means calling a PyArg_ParseTuple() converter function.

Your converter class should be named *something*_converter. If the name follows this convention, thenyour converter class will be automatically registered with Argument Clinic; its name will be the name of yourclass with the _converter suffix stripped off. (This is accomplished with a metaclass.)

You shouldnt subclass CConverter.__init__. Instead, you should write a converter_init() func-tion. converter_init() always accepts a self parameter; after that, all additional parameters must bekeyword-only. Any arguments passed in to the converter in Argument Clinic will be passed along to yourconverter_init().

There are some additional members of CConverter you may wish to specify in your subclass. Heres the currentlist:

type The C type to use for this variable. type should be a Python string specifying the type, e.g. int. If thisis a pointer type, the type string should end with *.

default The Python default value for this parameter, as a Python value. Or the magic value unspecified ifthere is no default.

py_default default as it should appear in Python code, as a string. Or None if there is no default.

c_default default as it should appear in C code, as a string. Or None if there is no default.

c_ignored_default The default value used to initialize the C variable when there is no default, but notspecifying a default may result in an uninitialized variable warning. This can easily happen when usingoption groupsalthough properly-written code will never actually use this value, the variable does get passedin to the impl, and the C compiler will complain about the use of the uninitialized value. This value shouldalways be a non-empty string.

converter The name of the C converter function, as a string.

impl_by_reference A boolean value. If true, Argument Clinic will add a & in front of the name of thevariable when passing it into the impl function.

parse_by_reference A boolean value. If true, Argument Clinic will add a & in front of the name of thevariable when passing it into PyArg_ParseTuple().

Heres the simplest example of a custom converter, from Modules/zlibmodule.c:

/*[python input]

class uint_converter(CConverter):type = 'unsigned int'converter = 'uint_converter'

[python start generated code]*//*[python end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/

This block adds a converter to Argument Clinic named uint. Parameters declared as uint will be declaredas type unsigned int, and will be parsed by the O& format unit, which will call the uint_converterconverter function. uint variables automatically support default values.

More sophisticated custom converters can insert custom C code to handle initialization and cleanup. You can seemore examples of custom converters in the CPython source tree; grep the C files for the string CConverter.

4.16 Writing a custom return converter

Writing a custom return converter is much like writing a custom converter. Except its somewhat simpler, becausereturn converters are themselves much simpler.

Return converters must subclass CReturnConverter. There are no examples yet of custom return con-verters, because they are not widely used yet. If you wish to write your own return converter, please readTools/clinic/clinic.py, specifically the implementation of CReturnConverter and all its sub-classes.

4.17 METH_O and METH_NOARGS

To convert a function using METH_O, make sure the functions single argument is using the object converter,and mark the arguments as positional-only:

/*[clinic input]meth_o_sample

argument: object/

[clinic start generated code]*/

To convert a function using METH_NOARGS, just dont specify any arguments.

You can still use a self converter, a return converter, and specify a type argument to the object converter forMETH_O.

4.18 tp_new and tp_init functions

You can convert tp_new and tp_init functions. Just name them __new__ or __init__ as appropriate.Notes:

The function name generated for __new__ doesnt end in __new__ like it would by default. Its just thename of the class, converted into a valid C identifier.

No PyMethodDef #define is generated for these functions.

__init__ functions return int, not PyObject *.

Use the docstring as the class docstring.

Although __new__ and __init__ functions must always accept both the args and kwargs objects,when converting you may specify any signature for these functions that you like. (If your function doesntsupport keywords, the parsing function generated will throw an exception if it receives any.)

4.19 Changing and redirecting Clinics output

It can be inconvenient to have Clinics output interspersed with your conventional hand-edited C code. Luckily,Clinic is configurable: you can buffer up its output for printing later (or earlier!), or write its output to a separatefile. You can also add a prefix or suffix to every line of Clinics generated output.

While changing Clinics output in this manner can be a boon to readability, it may result in Clinic code usingtypes before they are defined, or your code attempting to use Clinic-generated code befire it is defined. These

problems can be easily solved by rearranging the declarations in your file, or moving where Clinics generatedcode goes. (This is why the default behavior of Clinic is to output everything into the current block; while manypeople consider this hampers readability, it will never require rearranging your code to fix definition-before-useproblems.)

Lets start with defining some terminology:

field A field, in this context, is a subsection of Clinics output. For example, the #define for thePyMethodDef structure is a field, called methoddef_define. Clinic has seven different fields it canoutput per function definition:

docstring_prototypedocstring_definitionmethoddef_defineimpl_prototypeparser_prototypeparser_definitionimpl_definition

All the names are of the form "_", where "" is the semantic object represented (the parsingfunction, the impl function, the docstring, or the methoddef structure) and "" represents what kindof statement the field is. Field names that end in "_prototype" represent forward declarations of thatthing, without the actual body/data of the thing; field names that end in "_definition" represent theactual definition of the thing, with the body/data of the thing. ("methoddef" is special, its the only onethat ends with "_define", representing that its a preprocessor #define.)

destination A destination is a place Clinic can write output to. There are five built-in destinations:

block The default destination: printed in the output section of the current Clinic block.

buffer A text buffer where you can save text for later. Text sent here is appended to the end of anyexsiting text. Its an error to have any text left in the buffer when Clinic finishes processing a file.

file A separate clinic file that will be created automatically by Clinic. The filename chosen for the fileis {basename}.clinic{extension}, where basename and extension were assigned theoutput from os.path.splitext() run on the current file. (Example: the file destination for_pickle.c would be written to _pickle.clinic.c.)

Important: When using a file destination, you must check in the generated file!

two-pass A buffer like buffer. However, a two-pass buffer can only be written once, and it prints outall text sent to it during all of processing, even from Clinic blocks after the

suppress The text is suppressedthrown away.

Clinic defines five new directives that let you reconfigure its output.

The first new directive is dump:

dump

This dumps the current contents of the named destination into the output of the current block, and empties it. Thisonly works with buffer and two-pass destinations.

The second new directive is output. The most basic form of output is like this:

output

This tells Clinic to output field to destination. output also supports a special meta-destination, calledeverything, which tells Clinic to output all fields to that destination.

output has a number of other functions:

output pushoutput popoutput preset

output push and output pop allow you to push and pop configurations on an internal configuration stack,so that you can temporarily modify the output configuration, then easily restore the previous configuration. Sim-ply push before your change to save the current configuration, then pop when you wish to restore the previousconfiguration.

output preset sets Clinics output to one of several built-in preset configurations, as follows:

block Clinics original starting configuration. Writes everything immediately after the input block.

Suppress the parser_prototype and docstring_prototype, write everything else toblock.

file Designed to write everything to the clinic file that it can. You then #include this filenear the top of your file. You may need to rearrange your file to make this work, though usu-ally this just means creating forward declarations for various typedef and PyTypeObjectdefinitions.

Suppress the parser_prototype and docstring_prototype, write theimpl_definition to block, and write everything else to file.

The default filename is "{dirname}/clinic/{basename}.h".

buffer Save up all most of the output from Clinic, to be written into your file near the end. ForPython files implementing modules or builtin types, its recommended that you dump the bufferjust above the static structures for your module or builtin type; these are normally very nearthe end. Using buffer may require even more editing than file, if your file has staticPyMethodDef arrays defined in the middle of the file.

Suppress the parser_prototype, impl_prototype, and docstring_prototype,write the impl_definition to block, and write everything else to file.

two-pass Similar to the buffer preset, but writes forward declarations to the two-pass buffer,and definitions to the buffer. This is similar to the buffer preset, but may require lessediting than buffer. Dump the two-pass buffer near the top of your file, and dump thebuffer near the end just like you would when using the buffer preset.

Suppresses the impl_prototype, write the impl_definition to block, writedocstring_prototype, methoddef_define, and parser_prototype totwo-pass, write everything else to buffer.

partial-buffer Similar to the buffer preset, but writes more things to block, only writingthe really big chunks of generated code to buffer. This avoids the definition-before-use prob-lem of buffer completely, at the small cost of having slightly more stuff in the blocks output.Dump the buffer near the end, just like you would when using the buffer preset.

Suppresses the impl_prototype, write the docstring_definition andparser_defintion to buffer, write everything else to block.

The third new directive is destination:

destination [...]

This performs an operation on the destination named name.

There are two defined subcommands: new and clear.

The new subcommand works like this:

destination new

This creates a new destination with name and type .

There are five destination types:

suppress Throws the text away.

block Writes the text to the current block. This is what Clinic originally did.

buffer A simple text buffer, like the buffer builtin destination above.

file A text file. The file destination takes an extra argument, a template to use for building thefilename, like so:

destination new

The template can use three strings internally that will be replaced by bits of the filename:

{path} The full path to the file, including directory and full filename.

{dirname} The name of the directory the file is in.

{basename} Just the name of the file, not including the directory.

{basename_root} Basename with the extension clipped off (everything up to but notincluding the last .).

{basename_extension} The last . and everything after it. If the basename does notcontain a period, this will be the empty string.

If there are no periods in the filename, {basename} and {filename} are the same, and {exten-sion} is empty. {basename}{extension} is always exactly the same as {filename}.

two-pass A two-pass buffer, like the two-pass builtin destination above.

The clear subcommand works like this:

destination clear

It removes all the accumulated text up to this point in the destination. (I dont know what youd need this for, butI thought maybe itd be useful while someones experimenting.)

The fourth new directive is set:

set line_prefix "string"set line_suffix "string"

set lets you set two internal variables in Clinic. line_prefix is a string that will be prepended to every lineof Clinics output; line_suffix is a string that will be appended to every line of Clinics output.

Both of these suport two format strings:

{block comment start} Turns into the string /*, the start-comment text sequence for C files.

{block comment end} Turns into the string */, the end-comment text sequence for C files.

The final new directive is one you shouldnt need to use directly, called preserve:

preserve

This tells Clinic that the current contents of the output should be kept, unmodifed. This is used internally byClinic when dumping output into file files; wrapping it in a Clinic block lets Clinic use its existing checksumfunctionality to ensure the file was not modified by hand before it gets overwritten.

4.20 The #ifdef trick

If youre converting a function that isnt available on all platforms, theres a trick you can use to make life a littleeasier. The existing code probably looks like this:

#ifdef HAVE_FUNCTIONNAMEstatic module_functionname(...){...}#endif /* HAVE_FUNCTIONNAME */

And then in the PyMethodDef structure at the bottom the existing code will have:

#ifdef HAVE_FUNCTIONNAME{'functionname', ... },#endif /* HAVE_FUNCTIONNAME */

In this scenario, you should enclose the body of your impl function inside the #ifdef, like so:

#ifdef HAVE_FUNCTIONNAME/*[clinic input]module.functionname...[clinic start generated code]*/static module_functionname(...){...}#endif /* HAVE_FUNCTIONNAME */

Then, remove those three lines from the PyMethodDef structure, replacing them with the macro ArgumentClinic generated:

MODULE_FUNCTIONNAME_METHODDEF

(You can find the real name for this macro inside the generated code. Or you can calculate it yourself: its the nameof your function as defined on the first line of your block, but with periods changed to underscores, uppercased,and "_METHODDEF" added to the end.)

Perhaps youre wondering: what if HAVE_FUNCTIONNAME isnt defined? TheMODULE_FUNCTIONNAME_METHODDEF macro wont be defined either!

Heres where Argument Clinic gets very clever. It actually detects that the Argument Clinic block might bedeactivated by the #ifdef. When that happens, it generates a little extra code that looks like this:

#ifndef MODULE_FUNCTIONNAME_METHODDEF#define MODULE_FUNCTIONNAME_METHODDEF

#endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */

That means the macro always works. If the function is defined, this turns into the correct structure, including thetrailing comma. If the function is undefined, this turns into nothing.

However, this causes one ticklish problem: where should Argument Clinic put this extra code when using theblock output preset? It cant go in the output block, because that could be decativated by the #ifdef. (Thatsthe whole point!)

In this situation, Argument Clinic writes the extra code to the buffer destination. This may mean that you get acomplaint from Argument Clinic:

Warning in file "Modules/posixmodule.c" on line 12357:Destination buffer 'buffer' not empty at end of file, emptying.

When this happens, just open your file, find the dump buffer block that Argument Clinic added to your file(itll be at the very bottom), then move it above the PyMethodDef structure where that macro is used.

4.21 Using Argument Clinic in Python files

Its actually possible to use Argument Clinic to preprocess Python files. Theres no point to using Argument Clinicblocks, of course, as the output wouldnt make any sense to the Python interpreter. But using Argument Clinic torun Python blocks lets you use Python as a Python preprocessor!

Since Python comments are different from C comments, Argument Clinic blocks embedded in Python files lookslightly different. They look like this:

#/*[python input]#print("def foo(): pass")#[python start generated code]*/

def foo(): pass#/*[python checksum:...]*/

The Goals Of Argument ClinicBasic Concepts And UsageConverting Your First FunctionAdvanced TopicsSymbolic default valuesRenaming the C functions and variables generated by Argument ClinicConverting functions using PyArg_UnpackTupleOptional GroupsUsing real Argument Clinic converters, instead of ``legacy converters''Py_bufferAdvanced convertersParameter default valuesThe NULL default valueExpressions specified as default valuesUsing a return converterCloning existing functionsCalling Python codeUsing a ``self converter''Writing a custom converterWriting a custom return converterMETH_O and METH_NOARGStp_new and tp_init functionsChanging and redirecting Clinic's outputThe #ifdef trickUsing Argument Clinic in Python files

Documents

Howto Clinic