I am trying to use asttokens for source-level transformations in a code editor. For example, I would like to position the cursor on a binary operation, press a key, and the operands would be transposed: (a+b) * c → c * (a+b).

The way I’m trying to do that is:

1. Parse a small portion of the file into an asttokens.ASTTokens. (The smaller the better because the whole file cannot be guaranteed to be syntactically correct all the time. For now, I operate at the level of function definitions.)
2. Find the target node by the cursor position. Check if it’s a BinOp.
3. Determine the character ranges corresponding to the left and right operands, and swap them. (This is to preserve as much of the original code formatting as possible. For instance, re-generating the whole function using astor will strip all comments and possibly mess up spacing and line structure.)

However, I find that at step 3, if either operand is enclosed in parentheses, they do not count towards the token range:

import asttokens

tokens = asttokens.ASTTokens('(a + b) * c', parse=True)
print('\n'.join(
    '# {} at {}: {!r}'.format(node.__class__.name,
                              tokens.get_text_range(node),
                              tokens.get_text(node))
    for node in asttokens.util.walk(tokens.tree))

# Module at (0, 11): '(a + b) * c'
# Expr at (0, 11): '(a + b) * c'
# BinOp at (0, 11): '(a + b) * c'
# BinOp at (1, 6): 'a + b'
# Name at (1, 2): 'a'
# Name at (5, 6): 'b'
# Name at (10, 11): 'c'

so if I swap the corresponding character ranges, I get (c) * a + b which is wrong.

It would be nice if any parentheses that are not part of the parent node’s syntax were included in the token range of the child nodes:

# Module at (0, 11): '(a + b) * c'
# Expr at (0, 11): '(a + b) * c'
# BinOp at (0, 11): '(a + b) * c'
# BinOp at (0, 7): '(a + b)'
# Name at (1, 2): 'a'
# Name at (5, 6): 'b'
# Name at (10, 11): 'c'

(Or am I overlooking some other Obvious Way to Do It?)