Python 3.8.0b1 Positional-Only Arguments and the Walrus Operator

2019-06-06

On June 4th 2019, Python 3.8.0b1 was released. The official changelog is here.

There are two interesting syntactic changes/features that were added which I believe are useful to explore in some depth. Specifically, the new “walrus”‘ := operator and the new Positional-Only function parameter features.

Walrus

First, the “walrus” expression operator (:=) defined in PEP-572

…naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few name := markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer.

A (contrived) example using Python 3.8.0b1 built from source “3.8.0b1+ (heads/3.8:23f41a64ea)”

1
2
3
4
5
>>> xs = list(range(0, 10))
>>> xs
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [(x, y) for x in xs if (y := x * 2) < 5]
[(0, 0), (1, 2), (2, 4)]

The idea is to use an expression based approach to remove unnecessary chatter and potential bugs of storing local state.

Another simple example:

1
2
3
4
5
6
7
8
>>> # Python 3.8
>>> import re
>>> rx = re.compile(r'([A-z]*)\.([A-z]*)')
>>> def g(first, last): return f"First: {first} Last: {last}"
>>> names = ['ralph', 'steve.smith']
>>> for name in names:
...     if (match := rx.match(name)): print(g(*match.groups()))
First: steve Last: smith

As a side note, many of these “None-ish” based examples in the PEP (somewhat mechanically) look like a map, flatMap, 'foreach' on Option[T] cases in Scala.

Python doesn’t really do this well due to its inside-out nature of composing maps/filter/generators (versus a left to right model). Nevertheless, here’s the example to demonstrate the general idea using a functional centric approach.

1
2
3
4
5
>>> def processor(sx): return rx.match(sx)
>>> def not_none(x): return x is not None
>>> def printer(x): print(g(*x.groups()))
>>> _ = list(map(printer, filter(not_none, map(processor, names))))
First: steve Last: smith

The “Exceptional cases” described in the PEP are worth investigating in more detail. There’s several cases where “Valid, though probably confusing” is used.

For example:

1
2
y := f(x)  # INVALID
(y := f(x))  # Valid, though not recommended

Note that the “walrus” operator can also be used in function definitions. 

1
2
def foo(answer = p := 42): return "" # INVALID
def foo(answer=(p := 42)):  return "" # Valid, though not great style

Positional Only Args

The other interesting feature added to Python 3.8 is Positional-Only arguments in function definitions.

For as long as I can recall, Python has had this fundamental feature (or bug) on how functions or methods are called.

For example, 

1
2
3
4
5
6
7
8
9
def f(x, y=1234): return x + y
>>> f(1)
1235
>>> f(1, 2)
3
>>> f(x=1, y=2)
3
>>> f(y=1, x=2)
3

Often this fundamental ambiguity of function call “style” isn’t really a big deal. However, it can leak local variable names as part of the public interface.As a result, minor variable renaming can potentially break interfaces. It’s also not clear what should be a keyword only argument or a positional only argument with a default. For example, simply changing f(x, y=1234) to f(n, y=1234) can potentially break the interface depending on the call “style”.

I’ve worked with a few developers over the years that viewed this as a feature and thought that this style made the API calls more explicit. For example:

1
2
3
4
def compute(alpha, beta, gamma):
    return 0

compute(alpha=90, gamma=80, beta=120)

I never really liked this looseness of positional vs keyword and would (if possible) try to discourage its use. Regardless, it can be argued this is a feature of the language (at least in Python <= 3.7). I would guess that many Python developers are also leveraging the unpacking dict style as well to call functions.

1
2
d = dict(alpha=90, gamma=70, beta=120)
compute(**d)

In Python 3.0, function definitions using Keyword-Only arguments was added (see PEP-3102 from 2006) using the * delimiter. All arguments to the right of the * are Keyword-Only arguments. 

1
2
def f(a, b, *, c=1234):
    return a + b + c

Unfortunately, this still leaves a fundamental issue with clearly defining function arguments. There are three cases: Positional-Only, Positional or Keyword, and Keyword-Only. PEP-3102 only solves the Keyword-Only case and doesn’t address the other two cases.

Hence in Python < 3.8, there’s still a fundamental ambiguity when defining a function and how it can be called.

For example:

1
2
3
4
5
6
7

def f(a, b=1, *, c=1234):
    return a + b + c

f(1, 2, c=3)
f(a=1, b=2, c=3)

Starting with Python 3.8.0, a Positional-Only parameters mechanism was added. The details are captured in PEP-570

Similar to the * delimiter in Python 3.0.0 for Keyword-Only args), a / delimiter was added to clearly delineate Positional-Only (or conversely Keyword-Only args) in function or method definitions. This makes the three cases of function arguments unambigious in how they should be called.

Here’s a few examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def f0(a, b, c=1234, /):
    # a, b, c are only positional args. 
    return a + b + c

def f1(a, b, /, c=1234):
    # a, b must be a positional,
    # b can be positional or keyword
    return a + b + c

def f2(a, b, *, c=1234):
    # a, b can be keyword or positional
    # but c MUST be keyword
    return a + b + c

def f3(a, b, /, *, c=1234):
    # a, b only positional
    # c only keyword
    # e.g., # f3(1, 2, c=3)
    return a + b + c

Combining the / and * with the type annotations yields:

1
2
3
4
5
def f4(a:int, b:int, /, *, c:int=1234):
    # this can only be called as
    # f4(1, 2, c=3)
    return a + b + c

We can dive a bit deeper and inspect the function signature via inspect.

1
2
3
4
import inspect
def extractor(f):
    sx = inspect.signature(f)
    return [(v.name, v.kind, v.default) for v in sx.parameters.values()]

Let’s inspect each example:

1
2
3
def pf(f):
    print(f"Function {f.__name__} with type annotations {f.__annotations__}")
    print(extractor(f))

Running in the REPL yeilds:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
>>> funcs = (f0, f1, f2, f2, f2, f4)
>>> _ = list(map(pf, funcs))
Function f0 with type annotations {}
[('a', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('c', <_ParameterKind.POSITIONAL_ONLY: 0>, 1234)]
Function f1 with type annotations {}
[('a', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('c', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, 1234)]
Function f2 with type annotations {}
[('a', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('c', <_ParameterKind.KEYWORD_ONLY: 3>, 1234)]
Function f2 with type annotations {}
[('a', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('c', <_ParameterKind.KEYWORD_ONLY: 3>, 1234)]
Function f2 with type annotations {}
[('a', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <class 'inspect._empty'>), ('c', <_ParameterKind.KEYWORD_ONLY: 3>, 1234)]
Function f4 with type annotations {'a': <class 'int'>, 'b': <class 'int'>, 'c': <class 'int'>}
[('a', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('b', <_ParameterKind.POSITIONAL_ONLY: 0>, <class 'inspect._empty'>), ('c', <_ParameterKind.KEYWORD_ONLY: 3>, 1234)]

Note, you can use bind to call the func (this must also adhere to the correct function definition of arg and keywords in the function of interest).

Also, it’s worth noting that both scipy and numpy have been using this / style in the docs for some time.

When Should I Start Adopting these Features?

If you’re a library developer that has packages on PyPi, it might not be clear when it’s “safe” to start leveraging these features. I was only able to find one source of Python 3 adoption and as a result, I’m only able outline a very crude model.

On December 23, 2016, Python 3.6 was officially released. In the Fall of 2018, JetBrains release the Python Developer Survey which contains the Python 2/3 breakdown, as well as the breakdown of different versions within Python 3. As of the Fall 2018, 54% of Python 3 developers were using Python 3.6.x. Therefore, using this very crude model, if you assume that the rate of adoption of 3.6 and 3.8 are the same and if the minimum threshold of adoption of 3.8 is 54%, then you’ll need to wait approximately 2 years before starting to leverage these 3.8 features.

Jetbrains Python Survey

When you do plan to leverage these 3.8 specific features and pushing a package to the Python Package Index, then I would suggest clearly defining the Python version in the setup.py. For more details, see the official packaging docs for more details.

1
2
# setup.py
setup(python_requires='>=3.8')

Summary and Conclusion

  • Python 3.8 added the “walrus” operator := that enables results of expressions to be used
  • It’s recommended reading the Exceptional Cases for better understanding of where to (and to not) use the := operator.
  • Python 3.8 added Positional-Only function definitions using the / delimiter
  • Defining functions with Positional-Only arguments will require a trailing / in the definition. E.g., def adder(n, m, /): return 0
  • There are changes in the standard lib to communicate. It’s not clear how locked down or backward compatible the interfaces were changes. Here’s a random example of the function signature of functools.partial being updated to use /.
  • Positional-Only arguments should improve consistency of API calls across Python runtimes (e.g., cpython and pypi)
  • The Positional-Only PEP-570 outlines improvements in performance, however, I wasn’t able to find any performance studies on this topic.
  • Migrating to 3.8 might involve potentially breaking API changes based on the usage of / in the Python 3.8 standard lib
  • For core library authors of libs on pypi, I would recommend using the crude approximation (described above) of approximately 2 years away from being able to adopt the new 3.8 features
  • For mypy users, you might want to make sure you investigate the supported versions of Python 3.8 (More Details on the compatiblity matrix)

I understand the general motivation to solve core friction points or ambiguities at the language level, however, the new syntatic changes are a little too noisy for my tastes. Specifically the Positional-Only / combined with the * and type annotations. Regardless, the (Python 3.8) ship has sailed long ago. I would encourage the Python community to periodically track and provide feedback on the current PEPs to help guide the evolution of the Python programming language. And finally, Python 3.8.0 (beta and future 3.8 RCs) bugs should be filed to https://bugs.python.org .

Best to you and your Python-ing!

Further Reading

P.S. A reminder that the PSF has a Q2 2019 fundraiser that ends June 30th.

© 2022 - Powered by Hexo - Theme Chan