Test stub files against real code

[gaborbernat]

There are a few use cases when stubs may be compelling over type annotations or type comments:

• annotate code that needs to be Python 2 compatible, ideally by using type annotations (so no type comments),
• often times users prefer to have type information outside of the source code (this is to avoid modifying legacy code change - and in practice lowers resistance from maintainers),
• the maintainer of the library does not want to take on the in-line type annotations, someone else trying to provide it as a separate package.

However at the moment this has some limitations: stub files are used only to check clients of the stub, not the code itself.

# a.py
def add_int(a, b):
    return a + b

add_int("a", "b") # mypy does not complain, actually ignores the entire file

# a.pyi
def add_int(a:int, b:int) -> int: ...

# client.py
from a import add_int
add_int(1, 2)
add_int(1, "2") # here mypy is going to complain

In case of the above files mypy will warn about bad usage of inside client.py, but not in a.py (that is, mypy does not validate against the source files being annotated by the stub). This has serious drawbacks:

• maintaining the stub files is hard, now users need to manually make sure whatever is in the stub file is an accurate representation of the matching source files (both interface - aka number of arguments, names - and type wise),
• the stub creator does not know if the source code itself is type correct or not.

Here I propose to implement a way to support testing stub files against the real code:

• build the syntax tree for the source file,
• build the syntax tree for the stub file,
• merge stub tree into the source tree
  • source file only tree available: dynamically typed - noop
  • abstract syntax tree only: raise error - missing annotated sources
  • both stub and source AST exists:
    • for all source elements for what there is a matching stub element
      • copy inject over type annotations
      • complain if function interface missmatch
• now just run the existing type checking mechanism on the enriched source code ast

Merging the AST definitely is not trivial. The plan is to start with a POC implementation, that works for the most cases, and then see the viability of this approach. Note this would help a lot both stub maintainers I think (typeshed) and projects that need Python 2 support.

[gvanrossum]

This is not a new idea. Do you also have a plan for implementing this, or are you just requesting that the mypy core team takes this on?

[gaborbernat]

@gvanrossum I'm aware of this. I've chatted earlier today with some mypy core team members. Actually the plan described here is something we mostly came up together with @JukkaL. I do plan to implement this, or at least help with this (hopefully get the ball rolling during the sprints next week). That being said created the issue to discuss the general approach before anyone starts coding. I did not found a ticket dedicated existing, so created a new one.

[gaborbernat]

I've got an initial proof of concept version of this working (https://github.com/gaborbernat/mypy/commit/a56f007386daab9229af957ac252b3397146b8dc). Will give it a test run trying to use it to type hint the https://github.com/tox-dev/tox code base, which contains a lot more edge cases I'm sure it's not handling yet. And fix the issues coming up. Once manage to do that I'll create the PR for us to review/discuss it (probably will take a week or so).

[mchlnix]

Are there any updates on this? As someone using stub files exclusively, this is very unintuitive behavior (and hard to realize/easy to forget). It also makes type checking if __name__ == "__main__":" sections complicated (maybe impossible?).