Sphinx Autodoc fails in can_document_member with Django

[arcivanov]

Subject: Sphinx Autodoc fails in can_document_member with Django

Problem

can_document_member should handle errors and return False (as in "No, I cannot") if import produces an error of any kind. In other words, if any error is produced in can_document_member, it should exclude the member from documentation, possibly with a warning being issued.

Procedure to reproduce the problem

N/A

Error logs / results

Traceback (most recent call last):
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/application.py", line 333, in build
    self.builder.build_update()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 265, in build
    self.doctreedir, self.app))
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 556, in update
    self._read_serial(docnames, app)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 576, in _read_serial
    self.read_doc(docname, app)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 684, in read_doc
    pub.publish()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/io.py", line 55, in read
    self.parse()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/__init__.py", line 185, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2745, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 326, in section
    self.new_subsection(title, lineno, messages)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 394, in new_subsection
    node=section_node, match_titles=True)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2318, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2330, in explicit_construct
    return method(self, expmatch)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2073, in directive
    directive_class, match, type_name, option_presets)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2122, in run_directive
    result = directive_instance.run()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 1647, in run
    documenter.generate(more_content=self.content)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 992, in generate
    self.document_members(all_members)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 883, in document_members
    classes = [cls for cls in itervalues(AutoDirective._registry)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 884, in <listcomp>
    if cls.can_document_member(member, mname, isattr, self)]
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 1253, in can_document_member
    return isinstance(member, class_types)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/utils/functional.py", line 234, in inner
    self._setup()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Expected results

Skip the member and issue a warning

Reproducible project / your project

N/A

Environment info

N/A

[arcivanov]

For 1.6 tree the exception is as follows:

Traceback (most recent call last):
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/cmdline.py", line 305, in main
    app.build(opts.force_all, filenames)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/application.py", line 350, in build
    self.builder.build_update()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 275, in build_update
    'out of date' % len(to_build))
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 290, in build
    self.doctreedir, self.app))
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 608, in update
    self._read_serial(docnames, app)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 629, in _read_serial
    self.read_doc(docname, app)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 743, in read_doc
    pub.publish()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/io.py", line 69, in read
    self.parse()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/__init__.py", line 185, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2745, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 326, in section
    self.new_subsection(title, lineno, messages)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 394, in new_subsection
    node=section_node, match_titles=True)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2318, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2330, in explicit_construct
    return method(self, expmatch)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2073, in directive
    directive_class, match, type_name, option_presets)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2122, in run_directive
    result = directive_instance.run()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 1759, in run
    documenter.generate(more_content=self.content)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 1062, in generate
    self.document_members(all_members)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 952, in document_members
    classes = [cls for cls in itervalues(AutoDirective._registry)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 953, in <listcomp>
    if cls.can_document_member(member, mname, isattr, self)]
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 1338, in can_document_member
    return isinstance(member, class_types)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/utils/functional.py", line 234, in inner
    self._setup()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

[arcivanov]

@tk0miya thoughts?

[arcivanov]

Based on the latest sphinx master the error is gone and module loading warnings are emitted instead. This behavior seems appropriate.

WARNING: /home/arcivanov/Documents/src/karellen/karellen-core/docs/apidoc/karellen.core.app.serializers.rst:4: (WARNING/2) autodoc: failed to import module 'karellen.core.app.serializers'; the following exception was raised:
Traceback (most recent call last):
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 597, in import_object
    __import__(self.modname)
  File "/home/arcivanov/Documents/src/karellen/karellen-core/src/main/python/karellen/core/app/serializers.py", line 19, in <module>
    from django.contrib.auth.models import User, Group
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/contrib/auth/models.py", line 4, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/contrib/auth/base_user.py", line 52, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/db/models/base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/apps/registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "/home/arcivanov/.pyenv/versions/3.6.0/envs/karellen-env/lib/python3.6/site-packages/django/apps/registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

[arcivanov]

I believe this is related to #1061, #2336, #3235