IMPR: Upgrade plugin architecture

[dimitri-yatsenko]

Improvement Request

Problem

The datajoint-python project currently relies on pkg_resources for plugin discovery and management. The pkg_resources module, which is part of setuptools, is considered a legacy component in the Python packaging ecosystem. It is known for its slower performance and a more complex API compared to modern alternatives. The official Python packaging guidance now recommends using importlib.metadata (for Python 3.8 and later) or directly defining entrypoints in pyproject.toml for creating and discovering plugins.

Continuing to use pkg_resources manifests in several ways:

• Performance Overhead: pkg_resources can introduce a noticeable performance impact, particularly during application startup or when discovering a large number of plugins.
• Dependency Management Complexity: Relying on an older mechanism can complicate integration with modern build backends and dependency resolution tools that favor the newer standards.
• Maintenance Burden: As pkg_resources is considered a legacy component, continued reliance on it may divert development effort from adopting more modern, streamlined approaches.

Requirements

To transition datajoint-python to the new plugin architecture, the following changes are required:

• Migrate Plugin Entry Points:
  • Identify all existing plugin entry points currently defined using pkg_resources (these are often found in setup.py or older setup.cfg configurations).
  • Redefine these entry points in the pyproject.toml file under the [project.entry-points] table, adhering to the importlib.metadata convention. For example, existing console script entry points are already defined in this manner. New, custom plugin groups would follow a similar pattern.
• Update Plugin Discovery Logic:
  • Replace any pkg_resources calls (e.g., pkg_resources.iter_entry_points, pkg_resources.load_entry_point) with equivalent functionalities provided by importlib.metadata.
  • This typically involves using importlib.metadata.entry_points() to retrieve entry points for a specific group and then loading them.
• Update Build System: Ensure setuptools (or the chosen build backend) is configured to correctly parse and expose the pyproject.toml-defined entry points. Modern versions of setuptools with build-backend = "setuptools.build_meta" usually handle this automatically.
• Testing: Conduct comprehensive testing of all plugin functionalities to ensure they are correctly discovered and loaded after the migration.

Justification

Migrating to the new plugin architecture (importlib.metadata and pyproject.toml entry points) would provide several key benefits:

• Improved Performance: importlib.metadata offers a faster and more efficient way to discover and load plugins, which can reduce application startup times and improve overall responsiveness.
• Modernization: This change aligns datajoint-python with current best practices in Python packaging, making the project more maintainable and easier to integrate with other modern tools and libraries.
• Simplified API: importlib.metadata provides a cleaner and more intuitive API for plugin discovery compared to pkg_resources.
• Reduced Dependencies: Less reliance on the broader setuptools package at runtime could potentially lead to smaller dependency trees for deployed applications.

Alternative Considerations

• Staying with pkg_resources: This is the current state. The primary drawback is the continued use of a legacy system that is less performant and potentially harder to maintain. While it doesn't cause immediate functional breakage, it represents technical debt.

Related Errors

No direct errors are typically observed as a direct result of using pkg_resources as it generally functions correctly. However, the manifestation is often in less optimal performance or increased complexity in packaging and dependency resolution, which are harder to capture as specific error outputs.

Additional Research and Context

• Python Packaging User Guide - Creating and Discovering Plugins: This is the primary reference outlining the recommended approach for plugins using importlib.metadata and pyproject.toml.
• pyproject.toml: The current datajoint-python's pyproject.toml already utilizes the [project.entry-points."console_scripts"] table for dj and datajoint CLI commands, demonstrating existing familiarity with the new entry point syntax. This should make the migration for other plugin groups more straightforward.
• importlib.metadata: This module provides direct access to metadata defined by installed distributions, including entry points.