Automatically dedent docstring constants by default

[gpshead]

BPO	37102
Nosy	@gpshead, @methane, @Carreau, @pablogsal, @remilapeyre

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-30.18:00:00.547>
labels = ['interpreter-core', 'type-feature', '3.9']
title = 'Automatically dedent docstring constants by default'
updated_at = <Date 2019-06-06.14:36:45.161>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2019-06-06.14:36:45.161>
actor = 'remi.lapeyre'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2019-05-30.18:00:00.547>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 37102
keywords = []
message_count = 4.0
messages = ['343990', '344128', '344801', '344819']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'methane', 'mbussonn', 'pablogsal', 'remi.lapeyre']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37102'
versions = ['Python 3.9']

Linked PRs

• gh-81283: compiler: remove indent from docstring #106411

[gpshead]

I'm spawning this issue of as a separate feature from https://bugs.python.org/issue36906 (adding string dedent method and an optimization to do it at compile timer on constants).

It'd be great if docstrings were given a similar treatment. Right now we carry the whitespace burden within the str constants for __doc__ stored in all code objects in the process. This adds up.

This is not _quite_ the same as calling textwrap.dedent() or the upcoming str.dedent method on them at compile time. We need to special case the first line of docstrings. inspect.getdoc() is our library function that does this for us today.

Chance of breaking things with this change? not impossible, but extremely minor. Something using docstrings as data _and_ depending on leading indentation whitespace preservation would not like this. I am not aware of anything that would ever do that.

[remilapeyre]

Hi, I'm working on a PR. It should be ready in a couple of days. It's more involved than what I thought as to avoid importing inspect during compilation I will probably need to port cleandoc() in C.

[methane]

How about do inspect.cleandoc() instead of just a dedent?
Some method has docstring like this:

"""First line

blah blah blah blah

    example code here
...
"""

In such docstring, dedent can not strip indent well.

There is existing attempt (in Japanese):

https://qiita.com/hhatto/items/3da6c6820817395f2c39#%E6%84%9A%E7%9B%B4%E3%81%AB%E5%AE%9F%E8%A3%85%E3%81%97%E3%81%9F%E3%82%B3%E3%83%BC%E3%83%89

[remilapeyre]

This is the function I inlined and as far as I can tell, my approach as been similar to the one you linked.

I'm still need to fix some issues as doctest was expecting to find the string before dedenting though.

[methane]

cleandoc is not idempotent. If we cleandoc on compile time, pydoc and inspect.getdoc() shouldn't cleandoc(doc).

And if user write to obj.__doc__, user need to cleandoc. Otherwise, pydoc will have broken layout.
This is backward incompatibile change.

import inspect

s = """
  first line
    second line
      third line
"""

while True:
    print('---')
    print(s)
    t = inspect.cleandoc(s)
    if t == s:
        break
    s = t

print('---')

output:

---

  first line
    second line
      third line

---
first line
  second line
    third line
---
first line
second line
  third line
---

[methane]

In the PR, compile time cleandoc doesn't remove leading and trailing newlines.

code:

import inspect

def foo():
    """
    first line
      second
        third
    """

print('---')
print(foo.__doc__)
print('---')
print(inspect.getdoc(foo))
print('---')

current behavior:

---

    first line
      second
        third

---
first line
  second
    third
---

c-cleandoc (#106066)

---

first line
  second
    third

---
first line
  second
    third
---

By this way, I think the PR is minimize incompatibility.
Only one small change to docstring is needed to pass test suite.

https://github.com/python/cpython/pull/106411/files#diff-2752e009dea2759fb0a5a0866446e8bbd3f5ed30c030b26d252d7a1178b413e7R1290