ast --- 抽象語法樹

源代碼： Lib/ast.py

---

ast 模塊幫助 Python 程序處理 Python 語法的抽象語法樹。抽象語法或許會隨著 Python 的更新發布而改變；該模塊能夠幫助理解當前語法在編程層面的樣貌。

抽象語法樹可通過將 ast.PyCF_ONLY_AST 作為旗標傳遞給 compile() 內置函數來生成，或是使用此模塊中提供的 parse() 輔助函數。返回結果將是一個對象樹，，其中的類都繼承自 ast.AST。抽象語法樹可被內置的 compile() 函數編譯為一個 Python 代碼對象。

節點類

class ast.AST

這是所有 AST 節點類的基類。實際上，這些節點類派生自 Parser/Python.asdl 文件，其中定義的語法樹示例 如下。它們在 C 語言模塊 _ast 中定義，并被導出至 ast 模塊。

抽象語法定義的每個左側符號(比方說， ast.stmt 或者 ast.expr)定義了一個類。另外，在抽象語法定義的右側，對每一個構造器也定義了一個類；這些類繼承自樹左側的類。比如，ast.BinOp 繼承自 ast.expr。對于多分支產生式(也就是"和規則")，樹右側的類是抽象的；只有特定構造器結點的實例能被構造。

_fields

每個具體類都有個屬性 _fields, 用來給出所有子節點的名字。

每個具體類的實例對它每個子節點都有一個屬性，對應類型如文法中所定義。比如，ast.BinOp 的實例有個屬性 left，類型是 ast.expr.

如果這些屬性在文法中標記為可選（使用問號），對應值可能會是 None。如果這些屬性有零或多個（用星號標記），對應值會用Python的列表來表示。所有可能的屬性必須在用 compile() 編譯得到AST時給出，且是有效的值。

lineno

col_offset

ast.expr 和 ast.stmt 子類的實例有 lineno 和 col_offset 屬性。lineno 是源代碼的行數(從1開始, 所以第一行行數是1)，而 col_offset 是該生成節點第一個token的UTF-8字節偏移量。記錄下UTF-8偏移量的原因是parser內部使用UTF-8。

一個類的構造器 ast.T 像下面這樣parse它的參數。

• 如果有位置參數，它們必須和 T._fields 中的元素一樣多；他們會像這些名字的屬性一樣被賦值。

• 如果有關鍵字參數，它們必須被設為和給定值同名的屬性。

比方說，要創建和填充節點 ast.UnaryOp，你得用

node = ast.UnaryOp()
node.op = ast.USub()
node.operand = ast.Num()
node.operand.n = 5
node.operand.lineno = 0
node.operand.col_offset = 0
node.lineno = 0
node.col_offset = 0

或者更緊湊點

node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
                   lineno=0, col_offset=0)

抽象文法

抽象文法目前定義如下

-- ASDL's 7 builtin types are:
-- identifier, int, string, bytes, object, singleton, constant
--
-- singleton: None, True or False
-- constant can be None, whereas None means "no value" for object.

module Python
{
    mod = Module(stmt* body)
        | Interactive(stmt* body)
        | Expression(expr body)

        -- not really an actual node but useful in Jython's typesystem.
        | Suite(stmt* body)

    stmt = FunctionDef(identifier name, arguments args,
                       stmt* body, expr* decorator_list, expr? returns)
          | AsyncFunctionDef(identifier name, arguments args,
                             stmt* body, expr* decorator_list, expr? returns)

          | ClassDef(identifier name,
             expr* bases,
             keyword* keywords,
             stmt* body,
             expr* decorator_list)
          | Return(expr? value)

          | Delete(expr* targets)
          | Assign(expr* targets, expr value)
          | AugAssign(expr target, operator op, expr value)
          -- 'simple' indicates that we annotate simple name without parens
          | AnnAssign(expr target, expr annotation, expr? value, int simple)

          -- use 'orelse' because else is a keyword in target languages
          | For(expr target, expr iter, stmt* body, stmt* orelse)
          | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse)
          | While(expr test, stmt* body, stmt* orelse)
          | If(expr test, stmt* body, stmt* orelse)
          | With(withitem* items, stmt* body)
          | AsyncWith(withitem* items, stmt* body)

          | Raise(expr? exc, expr? cause)
          | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
          | Assert(expr test, expr? msg)

          | Import(alias* names)
          | ImportFrom(identifier? module, alias* names, int? level)

          | Global(identifier* names)
          | Nonlocal(identifier* names)
          | Expr(expr value)
          | Pass | Break | Continue

          -- XXX Jython will be different
          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

          -- BoolOp() can use left & right?
    expr = BoolOp(boolop op, expr* values)
         | BinOp(expr left, operator op, expr right)
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
         | Dict(expr* keys, expr* values)
         | Set(expr* elts)
         | ListComp(expr elt, comprehension* generators)
         | SetComp(expr elt, comprehension* generators)
         | DictComp(expr key, expr value, comprehension* generators)
         | GeneratorExp(expr elt, comprehension* generators)
         -- the grammar constrains where yield expressions can occur
         | Await(expr value)
         | Yield(expr? value)
         | YieldFrom(expr value)
         -- need sequences for compare to distinguish between
         -- x < 4 < 3 and (x < 4) < 3
         | Compare(expr left, cmpop* ops, expr* comparators)
         | Call(expr func, expr* args, keyword* keywords)
         | Num(object n) -- a number as a PyObject.
         | Str(string s) -- need to specify raw, unicode, etc?
         | FormattedValue(expr value, int? conversion, expr? format_spec)
         | JoinedStr(expr* values)
         | Bytes(bytes s)
         | NameConstant(singleton value)
         | Ellipsis
         | Constant(constant value)

         -- the following expression can appear in assignment context
         | Attribute(expr value, identifier attr, expr_context ctx)
         | Subscript(expr value, slice slice, expr_context ctx)
         | Starred(expr value, expr_context ctx)
         | Name(identifier id, expr_context ctx)
         | List(expr* elts, expr_context ctx)
         | Tuple(expr* elts, expr_context ctx)

          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

    expr_context = Load | Store | Del | AugLoad | AugStore | Param

    slice = Slice(expr? lower, expr? upper, expr? step)
          | ExtSlice(slice* dims)
          | Index(expr value)

    boolop = And | Or

    operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift
                 | RShift | BitOr | BitXor | BitAnd | FloorDiv

    unaryop = Invert | Not | UAdd | USub

    cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

    comprehension = (expr target, expr iter, expr* ifs, int is_async)

    excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body)
                    attributes (int lineno, int col_offset)

    arguments = (arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults,
                 arg? kwarg, expr* defaults)

    arg = (identifier arg, expr? annotation)
           attributes (int lineno, int col_offset)

    -- keyword arguments supplied to call (NULL identifier for **kwargs)
    keyword = (identifier? arg, expr value)

    -- import name with optional 'as' alias.
    alias = (identifier name, identifier? asname)

    withitem = (expr context_expr, expr? optional_vars)
}

ast 中的輔助函數

除了節點類， ast 模塊里為遍歷抽象語法樹定義了這些工具函數和類:

ast.parse(source, filename='<unknown>', mode='exec')

把源碼解析為AST節點。和 compile(source, filename, mode,ast.PyCF_ONLY_AST) 等價。

警告

足夠復雜或是巨大的字符串可能導致Python解釋器的崩潰，因為Python的AST編譯器是有棧深限制的。

ast.literal_eval(node_or_string)

對表達式節點以及包含Python字面量或容器的字符串進行安全的求值。傳入的字符串或者節點里可能只包含下列的Python字面量結構: 字符串，字節對象(bytes)，數值，元組，列表，字典，集合，布爾值和 None。

This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.

警告

足夠復雜或是巨大的字符串可能導致Python解釋器的崩潰，因為Python的AST編譯器是有棧深限制的。

在 3.2 版更改: 目前支持字節和集合。

ast.get_docstring(node, clean=True)

Return the docstring of the given node (which must be a FunctionDef, AsyncFunctionDef, ClassDef, or Module node), or None if it has no docstring. If clean is true, clean up the docstring's indentation with inspect.cleandoc().

在 3.5 版更改: 目前支持 AsyncFunctionDef

ast.fix_missing_locations(node)

When you compile a node tree with compile(), the compiler expects lineno and col_offset attributes for every node that supports them. This is rather tedious to fill in for generated nodes, so this helper adds these attributes recursively where not already set, by setting them to the values of the parent node. It works recursively starting at node.

ast.increment_lineno(node, n=1)

Increment the line number of each node in the tree starting at node by n. This is useful to "move code" to a different location in a file.

ast.copy_location(new_node, old_node)

Copy source location (lineno and col_offset) from old_node to new_node if possible, and return new_node.

ast.iter_fields(node)

Yield a tuple of (fieldname, value) for each field in node._fields that is present on node.

ast.iter_child_nodes(node)

Yield all direct child nodes of node, that is, all fields that are nodes and all items of fields that are lists of nodes.

ast.walk(node)

Recursively yield all descendant nodes in the tree starting at node (including node itself), in no specified order. This is useful if you only want to modify nodes in place and don't care about the context.

class ast.NodeVisitor

A node visitor base class that walks the abstract syntax tree and calls a visitor function for every node found. This function may return a value which is forwarded by the visit() method.

This class is meant to be subclassed, with the subclass adding visitor methods.

visit(node)

Visit a node. The default implementation calls the method called self.visit_classname where classname is the name of the node class, or generic_visit() if that method doesn't exist.

generic_visit(node)

This visitor calls visit() on all children of the node.

Note that child nodes of nodes that have a custom visitor method won't be visited unless the visitor calls generic_visit() or visits them itself.

Don't use the NodeVisitor if you want to apply changes to nodes during traversal. For this a special visitor exists (NodeTransformer) that allows modifications.

class ast.NodeTransformer

子類 NodeVisitor 用于遍歷抽象語法樹，并允許修改節點。

NodeTransformer 將遍歷抽象語法樹并使用visitor方法的返回值去替換或移除舊節點。如果visitor方法的返回值為 None , 則該節點將從其位置移除，否則將替換為返回值。當返回值是原始節點時，無需替換。

如下是一個轉換器示例，它將所有出現的名稱 (foo) 重寫為 data['foo']:

class RewriteName(NodeTransformer):

    def visit_Name(self, node):
        return Subscript(
            value=Name(id='data', ctx=Load()),
            slice=Index(value=Str(s=node.id)),
            ctx=node.ctx
        )

請記住，如果您正在操作的節點具有子節點，則必須先轉換其子節點或為該節點調用 generic_visit() 方法。

對于屬于語句集合（適用于所有語句節點）的節點，訪問者還可以返回節點列表而不僅僅是單個節點。

If NodeTransformer introduces new nodes (that weren't part of original tree) without giving them location information (such as lineno), fix_missing_locations() should be called with the new sub-tree to recalculate the location information:

tree = ast.parse('foo', mode='eval')
new_tree = fix_missing_locations(RewriteName().visit(tree))

通常你可以像這樣使用轉換器:

node = YourTransformer().visit(node)

ast.dump(node, annotate_fields=True, include_attributes=False)

Return a formatted dump of the tree in node. This is mainly useful for debugging purposes. If annotate_fields is true (by default), the returned string will show the names and the values for fields. If annotate_fields is false, the result string will be more compact by omitting unambiguous field names. Attributes such as line numbers and column offsets are not dumped by default. If this is wanted, include_attributes can be set to true.

參見

Green Tree Snakes, an external documentation resource, has good details on working with Python ASTs.