Просмотр кода страницы Management of Python dependencies sources

<span id="management-of-dependencies-sources"></span>
= Management of dependencies sources =

Collect [https://peps.python.org/pep-0508/ PEP508] requirements from different sources, store and evaluate them in Python environment.

See [https://github.com/stanislavlevin/pyproject_installer/issues/40 RFE] for details.

<tt><span style="color:green">%pyproject_deps</span></tt>-based RPM macros are shipped in [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-pyproject/ rpm-build-pyproject] RPM package, that in turn depends on [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-python3/ rpm-build-python3]. Hence, <tt><span style="color:green">rpm-build-python3</span></tt> should be replaced with <tt><span style="color:green">rpm-build-pyproject</span></tt> in RPM specfile.

<span id="build"></span>
== BUILD ==

Build dependencies format is standardized by [https://peps.python.org/pep-0518/ PEP518] and [https://peps.python.org/pep-0517/ PEP517].

<ul>
<li><p>add <tt><span style="color:green">%pyproject_builddeps_build</span></tt> macro into RPM specfile that:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">PEP518</span></tt> and <tt><span style="color:green">PEP517</span></tt> sources (they will be defined later) in Python environment</li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul>
</li>
<li><p>append <tt><span style="color:green">%pyproject_deps_resync_build</span></tt> macro to RPM <tt><span style="color:green">%prep</span></tt> section. This macro:</p>
<ul>
<li>configure <tt><span style="color:green">PEP518</span></tt> source of dependencies if it doesn’t exist yet</li>
<li>verify <tt><span style="color:green">PEP518</span></tt> source and resync it if needed</li>
<li>configure <tt><span style="color:green">PEP517</span></tt> source of dependencies if it doesn’t exist yet</li>
<li>verify <tt><span style="color:green">PEP517</span></tt> source and resync it if needed</li></ul>
</li>
<li><p>run build. The RPM build may fail on verification of <tt><span style="color:green">PEP518</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li>
<li><p><tt><span style="color:green">%pyproject_deps</span></tt>-based RPM macros expect to find its configuration in RPM sources directory by default. Hence,</p>
<ul>
<li>add <tt><span style="color:green">copy: .gear/pyproject_deps.json</span></tt> to .gear/rules</li>
<li>add <tt><span style="color:green">SourceXXXX: pyproject_deps.json</span></tt> to RPM specfile (the source number doesn’t matter)</li></ul>
</li>
<li><p>exclude filters can be applied on evaluation of dependencies. For example, filter out dependencies having normalized names that match <tt><span style="color:green">types-</span></tt>:</p>
<pre class="rpmspec">%set_pyproject_deps_build_filter types-</pre>
<p>or</p>
<pre class="rpmspec">%add_pyproject_deps_build_filter types-</pre></li>
<li><p>rerun build. The RPM build may fail on verification of PEP517 dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li></ul>

This is enough to manage ''Python'' build dependencies of vast of majority of Python projects.

<span id="runtime"></span>
== RUNTIME ==

By default, runtime dependencies of Python projects in ALTLinux are generated automatically by parsing source code and fetching of all of external imports (this feature is known as <tt><span style="color:green">autoreq</span></tt>). This may or may not work well. For example:

* conditional or optional dependencies can’t be automatically accounted
* installation tools like pip may expect all installed dependencies of projects

Downstream maintainers have to manually handle such cases.

Runtime dependencies format is standardized by [https://packaging.python.org/en/latest/specifications/core-metadata/ core metadata] and it’s usually generated by build backend from some tool-specific format. Produced list of dependencies is not ''necessary and sufficient''. But it may complement auto-generated dependencies to cover unhandled cases.

<ul>
<li><p>add <tt><span style="color:green">%pyproject_runtimedeps_metadata</span></tt> macro into RPM specfile that:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">metadata</span></tt> source (it will be defined later) in Python environment</li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">Requires</span></tt> tags</li></ul>
</li>
<li><p>append <tt><span style="color:green">%pyproject_deps_resync_metadata</span></tt> macro to RPM <tt><span style="color:green">%prep</span></tt> section. This macro:</p>
<ul>
<li>configure <tt><span style="color:green">core metadata</span></tt> source of dependencies if it doesn’t exist yet</li>
<li>verify <tt><span style="color:green">core metadata</span></tt> source and resync it if needed</li></ul>
</li>
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">metadata</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li>
<li><p>exclude filters can be applied on evaluation of dependencies: For example, filter out dependencies having normalized names that match <tt><span style="color:green">types-</span></tt>:</p>
<pre class="rpmspec">%set_pyproject_deps_runtime_filter types-</pre>
<p>or</p>
<pre class="rpmspec">%add_pyproject_deps_runtime_filter types-</pre></li></ul>

<span id="check"></span>
== CHECK ==

Tests dependencies format is not ''standardized'', but <tt><span style="color:green">pyproject_installer</span></tt> supports most ''common'' of them like:

* [[#core-metadata|core metadata]]
* [[#pip-reqfile|pip’s reqfile]]
* [[#poetry-group|poetry group]]
* [[#tox-env|tox env]]

Usually tests require all [[#core-metadata|runtime]] dependencies of project ''and'' additional packages for test environment (e.g. pytest). Thus, configuration of several sources may be required (e.g. metadata + pip_reqfile).

<span id="check-filter"></span>
=== check filter ===

There can be many of redundant dependencies like:

* <tt><span style="color:red">code style</span></tt> checkers that are really helpful in development but they often report different results from version to version.
* <tt><span style="color:red">coverage</span></tt> measurement, it’s pointless overhead to do this for RPM packaging.

Hence, there is a predefined filter of <tt><span style="color:green">check</span></tt> dependencies <tt><span style="color:green">%pyproject_deps_check_filter</span></tt>. This filter is applied automatically on <tt><span style="color:green">%pyproject_builddeps_check</span></tt> or <tt><span style="color:green">%pyproject_builddeps_metadata</span></tt> macro usage.

Today’s default filter includes:

* <tt><span style="color:green">{?pypi_name:%{pep503_name %pypi_name}}</span></tt>
* flake8
* pytest-flake8
* isort
* black
* pytest-black
* coverage
* pytest-cov
* covdefaults
* mypy
* pytest-mypy
* pytest-checkdocs
* pre-commit

This filter can be changed like:

<pre class="rpmspec">%set_pyproject_deps_check_filter foo bar</pre>
or

<pre class="rpmspec">%add_pyproject_deps_check_filter foo bar</pre>
Note: this is exposed as ''deps'' <tt><span style="color:green">–exclude</span></tt> option that accepts regex patterns and excludes requirement having [https://peps.python.org/pep-0503/#normalized-names PEP503-normalized name] which matches one of these patterns. Thus, if only the name should be excluded then this name should be normalized first:

<pre class="rpmspec">%add_pyproject_deps_check_filter %{pep503_name Foo-Project.my}</pre>
<span id="core-metadata"></span>
=== core metadata ===

Specification:

* https://packaging.python.org/en/latest/specifications/core-metadata/
* https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use

For example, upstream use core metadata as source of tests dependencies and mark them with <tt><span style="color:green">testing</span></tt> extra (this means that all runtime dependencies of current package and any dependencies that are listed in the <tt><span style="color:green">testing</span></tt> extra of current package will be installed, see [https://peps.python.org/pep-0508/#extras extras]). Usually this is exposed as <tt><span style="color:green">extras = testing</span></tt> in <tt><span style="color:green">tox.ini</span></tt> or <tt><span style="color:green">pip install .[testing]</span></tt> in CI config.

<ul>
<li><p>add macro to RPM specfile</p>
<pre class="rpmspec">%if_with check
%pyproject_builddeps_metadata -- --extra testing
%endif</pre>
<p>This macro:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">metadata</span></tt> source (it’s already defined as source of [[#runtime|runtime]] dependencies, otherwise it should be defined manually) in Python environment with <tt><span style="color:green">testing</span></tt> extra</li>
<li>filter out dependencies according to <tt><span style="color:green">%pyproject_deps_check_filter</span></tt></li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul>
</li>
<li><p>filter out extra dependencies with [[#check-filter|check-filter]]</p></li></ul>

<span id="pip-reqfile"></span>
=== pip reqfile ===

Specification:

* https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format

Limitations:

* supported only [https://peps.python.org/pep-0508/ PEP508] requirements
* line continuations are not supported
* pip’s inline options are not supported

For example, upstream use pip’s reqfile as source of tests dependencies and it’s saved as <tt><span style="color:green">test-requirements.txt</span></tt>.

<ul>
<li><p>add macro to RPM specfile</p>
<pre class="rpmspec">%if_with check
%pyproject_builddeps_check
%endif</pre>
<p>This macro:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">check</span></tt> source (it will be defined later) in Python environment</li>
<li>filter out dependencies according to <tt><span style="color:green">%pyproject_deps_check_filter</span></tt></li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul>
</li>
<li><p>configure <tt><span style="color:green">check</span></tt> source of dependencies by appending to RPM <tt><span style="color:green">%prep</span></tt> section:</p>
<pre class="rpmspec">%if_with check
%pyproject_deps_resync check pip_reqfile test-requirements.txt
%endif</pre></li>
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li>
<li><p>filter out extra dependencies with [[#check-filter|check-filter]]</p></li></ul>

<span id="poetry-group"></span>
=== poetry group ===

Specification:

* https://python-poetry.org/docs/dependency-specification/
* https://python-poetry.org/docs/managing-dependencies/

Limitations:

* versions are not supported (poetry version format is not compatible with [https://peps.python.org/pep-0440/ PEP440])
* <tt><span style="color:red">python</span></tt> field is not supported

For example, upstream uses poetry dependencies group <tt><span style="color:green">dev</span></tt> as source of tests dependencies. By default, poetry installs dependencies of all non-optional groups including dependencies of project. Hence, [[#runtime|runtime]] should be configured first.

<ul>
<li><p>add macro to RPM specfile</p>
<pre class="rpmspec">%if_with check
%pyproject_builddeps_metadata
%pyproject_builddeps_check
%endif</pre>
<p>These macros:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">metadata</span></tt> and <tt><span style="color:green">check</span></tt> sources (<tt><span style="color:green">metadata</span></tt> is already defined as source of [[#runtime|runtime]] dependencies, otherwise it should be defined manually) in Python environment</li>
<li>filter out dependencies according to <tt><span style="color:green">%pyproject_deps_check_filter</span></tt></li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul>
</li>
<li><p>configure <tt><span style="color:green">check</span></tt> source of dependencies by appending to RPM <tt><span style="color:green">%prep</span></tt> section:</p>
<pre class="rpmspec">%if_with check
%pyproject_deps_resync check poetry dev
%endif</pre></li>
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li>
<li><p>filter out extra dependencies with [[#check-filter|check-filter]]</p></li></ul>

<span id="tox-env"></span>
=== tox env ===

Specification:

* https://tox.wiki/en/latest/config.html#deps
* config formats: https://tox.wiki/en/latest/config.html#discovery-and-file-types

Limitations:

* supported only [https://peps.python.org/pep-0508/ PEP508] requirements
* config substitutions are not supported
* <tt><span style="color:green">testenv</span></tt> inheritance is not supported
* pip’s req files or options are not supported

Note: if <tt><span style="color:green">deps</span></tt> includes pip’s requirements file then use [[#pip-reqfile|pip’s reqfile]] instead. If <tt><span style="color:green">testenv</span></tt> has <tt><span style="color:green">extras</span></tt> then use [[#core-metadata|core metadata]] instead.

For example, upstream uses tox’ <tt><span style="color:green">deps</span></tt> of <tt><span style="color:green">testenv</span></tt> section of <tt><span style="color:green">tox.ini</span></tt> config as source of tests dependencies. By default, <tt><span style="color:green">tox</span></tt> installs these dependencies prior to package installation and then installs package and its runtime dependencies. Hence, [[#runtime|runtime]] should be configured first.

<ul>
<li><p>add macro to RPM specfile</p>
<pre class="rpmspec">%if_with check
%pyproject_builddeps_metadata
%pyproject_builddeps_check
%endif</pre>
<p>These macros:</p>
<ul>
<li>eval dependencies of <tt><span style="color:green">metadata</span></tt> and <tt><span style="color:green">check</span></tt> sources (<tt><span style="color:green">metadata</span></tt> is already defined as source of [[#runtime|runtime]] dependencies, otherwise it should be defined manually) in Python environment</li>
<li>filter out dependencies according to <tt><span style="color:green">%pyproject_deps_check_filter</span></tt></li>
<li>map dependencies to distro names</li>
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul>
</li>
<li><p>configure <tt><span style="color:green">check</span></tt> source of dependencies by appending to RPM <tt><span style="color:green">%prep</span></tt> section:</p>
<pre class="rpmspec">%if_with check
%pyproject_deps_resync check tox tox.ini testenv
%endif</pre></li>
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p>
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li>
<li><p>filter out extra dependencies with [[#check-filter|check-filter]]</p></li></ul>

<span id="mapping-project-names-to-distro-names"></span>
== Mapping project names to distro names ==

Python ecosystem uses PEP503 normalization rules for names comparison:

* https://packaging.python.org/en/latest/specifications/name-normalization/#package-name-normalization
* https://peps.python.org/pep-0503/#normalized-names

Mapping rule in ALTLinux:

<pre>project_name =&gt; python3-module-%{pep503_name project_name}</pre>
This almost always works out of the box, but distro packages having unnormalized names ''must'' provide such the name:

<pre>Provides: python3-module-%{pep503_name %pypi_name} = %EVR</pre>
The following naming rule for new Python distro packages is strongly encouraged:

<pre>Name: python3-module-%{pep503_name %pypi_name}</pre>