From 4ae370793b97d03f733d542bd74bdf63a2e3c00b Mon Sep 17 00:00:00 2001 From: Malcolm Smith Date: Tue, 24 Sep 2024 19:38:35 +0100 Subject: [PATCH 01/23] Add Android instructions to the devguide (GH-1402) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- getting-started/setup-building.rst | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/getting-started/setup-building.rst b/getting-started/setup-building.rst index 8d432f0c0f..0c0e286916 100644 --- a/getting-started/setup-building.rst +++ b/getting-started/setup-building.rst @@ -458,6 +458,12 @@ used in ``python.sh``: .. _wasmtime: https://wasmtime.dev .. _WebAssembly: https://webassembly.org +Android +------- + +Build and test instructions for Android are maintained in the CPython repository +at :cpy-file:`Android/README.md`. + iOS --- @@ -608,8 +614,8 @@ for details. Install dependencies ==================== -This section explains how to install additional extensions (for example, ``zlib``) -on Linux, macOS and iOS. +This section explains how to install libraries which are needed to compile +some of CPython's modules (for example, ``zlib``). .. tab:: Linux @@ -775,6 +781,16 @@ on Linux, macOS and iOS. On Windows, extensions are already included and built automatically. +.. tab:: Android + + The BeeWare project maintains `scripts for building Android dependencies`_, + and distributes `pre-compiled binaries`_ for each of them. + These binaries are automatically downloaded and used by the CPython + build script at :cpy-file:`Android/android.py`. + + .. _scripts for building Android dependencies: https://github.com/beeware/cpython-android-source-deps + .. _pre-compiled binaries: https://github.com/beeware/cpython-android-source-deps/releases + .. tab:: iOS As with CPython itself, the dependencies for CPython must be compiled for From 009574812e266f870a360e694215d8c67008908a Mon Sep 17 00:00:00 2001 From: Mariatta Date: Tue, 24 Sep 2024 13:14:39 -0700 Subject: [PATCH 02/23] Remove coverity from the DevGuide (#1411) * Remove coverity * Update conf.py * Link to valgrind doc on CPython repo. * Update development-tools/clang.rst Co-authored-by: Jelle Zijlstra --- conf.py | 2 - core-developers/experts.rst | 1 - development-tools/clang.rst | 4 +- development-tools/coverity.rst | 141 --------------------------------- development-tools/index.rst | 1 - 5 files changed, 3 insertions(+), 146 deletions(-) delete mode 100644 development-tools/coverity.rst diff --git a/conf.py b/conf.py index 44e625a83d..64d7c68b90 100644 --- a/conf.py +++ b/conf.py @@ -114,11 +114,9 @@ rediraffe_redirects = { # Development Tools "clang.rst": "development-tools/clang.rst", - "coverity.rst": "development-tools/coverity.rst", "gdb.rst": "development-tools/gdb.rst", # Advanced Tools was renamed Development Tools in gh-1149 "advanced-tools/clang.rst": "development-tools/clang.rst", - "advanced-tools/coverity.rst": "development-tools/coverity.rst", "advanced-tools/gdb.rst": "development-tools/gdb.rst", # Core Developers "coredev.rst": "core-developers/become-core-developer.rst", diff --git a/core-developers/experts.rst b/core-developers/experts.rst index 61ad86b8eb..950efc6564 100644 --- a/core-developers/experts.rst +++ b/core-developers/experts.rst @@ -319,7 +319,6 @@ buildbots zware, pablogsal bytecode benjaminp, 1st1, markshannon, brandtbucher, carljm, iritkatriel context managers ncoghlan core workflow Mariatta, ezio-melotti, hugovk, AA-Turner -coverity scan tiran, Yhg1s cryptography gpshead, dstufft data formats database malemburg diff --git a/development-tools/clang.rst b/development-tools/clang.rst index 3f62f69f39..f06834731a 100644 --- a/development-tools/clang.rst +++ b/development-tools/clang.rst @@ -54,7 +54,7 @@ A complete list of sanitizers can be found at `Controlling Code Generation Clang and its sanitizers have strengths (and weaknesses). Its just one tool in the war chest to uncovering bugs and improving code quality. Clang should be -used to complement other methods, including Code Reviews, Valgrind, Coverity, +used to complement other methods, including Code Reviews, `Valgrind`_, etc. Clang/LLVM setup @@ -281,3 +281,5 @@ Unfortunately, you won't know what to ignorelist until you run the sanitizer. The documentation is available at `Sanitizer special case list `_. + +.. _Valgrind: https://github.com/python/cpython/blob/main/Misc/README.valgrind diff --git a/development-tools/coverity.rst b/development-tools/coverity.rst deleted file mode 100644 index 7c165a3126..0000000000 --- a/development-tools/coverity.rst +++ /dev/null @@ -1,141 +0,0 @@ -.. _coverity: - -============= -Coverity Scan -============= - -Coverity Scan is a free service for static code analysis of Open Source -projects. It is based on Coverity's commercial product and is able to analyze -C, C++ and Java code. - -Coverity's static code analysis doesn't run the code. Instead of that it uses -abstract interpretation to gain information about the code's control flow and -data flow. It's able to follow all possible code paths that a program may -take. For example the analyzer understands that ``malloc()`` returns a memory -that must be freed with ``free()`` later. It follows all branches and function -calls to see if all possible combinations free the memory. The analyzer is -able to detect all sorts of issues like resource leaks (memory, file -descriptors), NULL dereferencing, use after free, unchecked return values, -dead code, buffer overflows, integer overflows, uninitialized variables, and -many more. - - -Access to analysis reports -========================== - -The results are available on the `Coverity Scan`_ website. In order to -access the results you have to create an account yourself. Then go to -*Projects using Scan* and add yourself to the Python project. New members must -be approved by an admin (see `Contact`_). - -Access is restricted to Python core developers only. Other individuals may be -given access at our own discretion, too. Every now and then Coverity detects a -critical issue in Python's code -- new analyzers may even find new bugs in -mature code. We don't want to disclose issues prematurely. - - -Building and uploading analysis -=============================== - -The process is automated. A script checks out the code, runs -``cov-build`` and uploads the latest analysis to Coverity. Since Coverity has -limited the maximum number of builds per week Python is analyzed every second -day. The build runs on a dedicated virtual machine on PSF's infrastructure at -OSU Open Source Labs. The process is maintained by Christian Heimes (see -`Contact`_). At present only the tip is analyzed with the 64bit Linux tools. - - -Known limitations -================= - -Some aspects of Python's C code are not yet understood by Coverity. - -False positives ---------------- - -``Py_BuildValue("N", PyObject*)`` - Coverity doesn't understand that ``N`` format char passes the object along - without touching its reference count. On this ground the analyzer detects - a resource leak. CID 719685 - -``PyLong_FromLong()`` for negative values - Coverity claims that ``PyLong_FromLong()`` and other ``PyLong_From*()`` - functions cannot handle a negative value because the value might be used as - an array index in ``get_small_int()``. CID 486783 - -``PyLong_FromLong()`` for n in [-5 ... +255] - For integers in the range of Python's small int cache the ``PyLong_From*()`` - function can never fail and never returns NULL. CID 1058291 - -``PyArg_ParseTupleAndKeywords(args, kwargs, "s#", &data, &length)`` - Some functions use the format char combination such as ``s#``, ``u#`` or - ``z#`` to get data and length of a character array. Coverity doesn't - recognize the relation between data and length. Sometimes it detects a buffer - overflow if data is written to a fixed size buffer although - ``length <= sizeof(buffer)``. CID 486613 - -``path_converter()`` dereferencing after null check - The ``path_converter()`` function in ``posixmodule.c`` makes sure that - either ``path_t.narrow`` or ``path_t.wide`` is filled unless - ``path_t.nullable`` is explicitly enabled. CID 719648 - - -Modeling -======== - -Modeling is explained in the *Coverity Help Center* which is available in -the help menu of `Coverity Connect`_. `coverity_model.c`_ contains a copy of -Python's modeling file for Coverity. Please keep the copy in sync with the -model file in *Analysis Settings* of `Coverity Scan`_. - - -Workflow -======== - -False positive and intentional issues -------------------------------------- - -If the problem is listed under `Known limitations`_ then please set the -classification to either "False positive" or "Intentional", the action to -"Ignore", owner to your own account and add a comment why the issue -is considered false positive or intentional. - -If you think it's a new false positive or intentional then please contact an -admin. The first step should be an updated to Python's `Modeling`_ file. - - -Positive issues ---------------- - -You should always create an issue unless it's really a trivial case. Please -add the full url to the ticket under *Ext. Reference* and add the CID -(Coverity ID) to both the ticket and the checkin message. It makes it much -easier to understand the relation between tickets, fixes and Coverity issues. - - -Contact -======= - -Please include both Brett and Christian in any mail regarding Coverity. Mails -to Coverity should go through Brett or Christian, too. - -Christian Heimes - admin, maintainer of build machine, intermediary between Python and Coverity - -Brett Cannon - co-admin - -Dakshesh Vyas - Technical Manager - Coverity Scan - - -.. seealso:: - - `Coverity Scan FAQ `_ - - -.. _Coverity Scan: https://scan.coverity.com/ - -.. _Coverity Connect: https://scan.coverity.com/projects/python - -.. _coverity_model.c: https://github.com/python/cpython/blob/main/Misc/coverity_model.c diff --git a/development-tools/index.rst b/development-tools/index.rst index d62a847691..610d8d8ea8 100644 --- a/development-tools/index.rst +++ b/development-tools/index.rst @@ -8,5 +8,4 @@ Development tools clinic gdb clang - coverity warnings From 539703be4f095c7bbeee54e95a4f8759c8585d75 Mon Sep 17 00:00:00 2001 From: Mariatta Date: Tue, 24 Sep 2024 14:14:37 -0700 Subject: [PATCH 03/23] Remove the inactionable TODO (#1412) The TODO has been there for 14 years. --- getting-started/fixing-issues.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/getting-started/fixing-issues.rst b/getting-started/fixing-issues.rst index 6161f4aa60..6b752d79d9 100644 --- a/getting-started/fixing-issues.rst +++ b/getting-started/fixing-issues.rst @@ -23,5 +23,3 @@ you can also always provide useful comments if you do attempt a fix, successful or not. .. _"easy" issues: https://github.com/python/cpython/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy - -.. TODO: add something about no active core developer for the area? From 0f959e02e0a2c8eb4ee547411a310c70605c6c31 Mon Sep 17 00:00:00 2001 From: Ayappan Perumal Date: Fri, 27 Sep 2024 21:36:36 +0530 Subject: [PATCH 04/23] Update AIX Maintainers list (#1417) --- core-developers/experts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core-developers/experts.rst b/core-developers/experts.rst index 950efc6564..10f6b5727f 100644 --- a/core-developers/experts.rst +++ b/core-developers/experts.rst @@ -286,7 +286,7 @@ for “their” platform as a third-party project. =================== =========== Platform Maintainers =================== =========== -AIX edelsohn +AIX edelsohn, ayappanec Android mhsmith Cygwin jlt63^, stutzbach^ Emscripten hoodmane, pmp-p, rdb, rth, ryanking13 From f7b933cf50ba03fcdba3d181614b328694ccedd9 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Sat, 28 Sep 2024 00:16:18 +0200 Subject: [PATCH 05/23] Document ``next`` in versionadded & similar directives (GH-1413) --- documentation/markup.rst | 43 +++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/documentation/markup.rst b/documentation/markup.rst index 58b2bdac06..a7a9bcb767 100644 --- a/documentation/markup.rst +++ b/documentation/markup.rst @@ -1083,8 +1083,11 @@ units as well as normal text: (``class``, ``attribute``, ``function``, ``method``, ``c:type``, etc), a ``versionadded`` should be included at the end of its description block. - The first argument must be given and is the version in question. The second - argument is optional and can be used to describe the details of the feature. + The first argument must be given and is the version in question. + Instead of a specific version number, you can---and should---use + the word ``next``, indicating that the API will first appear in the + upcoming release. + The second argument is optional and can be used to describe the details of the feature. Example:: @@ -1092,7 +1095,30 @@ units as well as normal text: Return foo and bar. - .. versionadded:: 3.5 + .. versionadded:: next + + When a release is made, the release manager will change the ``next`` to + the just-released version. For example, if ``func`` in the above example is + released in 3.14, the snippet will be changed to:: + + .. function:: func() + + Return foo and bar. + + .. versionadded:: 3.14 + + The tool to do this replacement is `update_version_next.py`_ + in the release-tools repository. + + .. _update_version_next.py: https://github.com/python/release-tools/blob/master/update_version_next.py + + When backporting to versions before 3.14, check if ``Doc/tools/extensions/pyspecific.py`` + contains the function ``expand_version_arg``. If it's not there, + use a specific version instead of ``next``. + + When adding documentation for a function that existed in a past version, + but wasn't documented yet, use the version number where the function was + added instead of ``next``. .. describe:: versionchanged @@ -1106,7 +1132,7 @@ units as well as normal text: Return foo and bar, optionally with *spam* applied. - .. versionchanged:: 3.6 + .. versionchanged:: next Added the *spam* parameter. Note that there should be no blank line between the directive head and the @@ -1118,10 +1144,12 @@ units as well as normal text: There is one required argument: the version from which the feature is deprecated. + Similarly to ``versionadded``, you should use the word ``next`` to indicate + the API will be first deprecated in the upcoming release. Example:: - .. deprecated:: 3.8 + .. deprecated:: next .. describe:: deprecated-removed @@ -1129,11 +1157,12 @@ units as well as normal text: removed. There are two required arguments: the version from which the feature is - deprecated, and the version in which the feature is removed. + deprecated (usually ``next``), and the version in which the feature + is removed, which must be a specific version number (*not* ``next``). Example:: - .. deprecated-removed:: 3.8 4.0 + .. deprecated-removed:: next 4.0 .. describe:: impl-detail From 1f2722ee56ef2604fed7f7a109c9a79be808a986 Mon Sep 17 00:00:00 2001 From: Itamar Oren Date: Fri, 27 Sep 2024 16:53:52 -0700 Subject: [PATCH 06/23] Update the link to the buildbot docs about setting up the worker as a Windows service (#1422) --- testing/new-buildbot-worker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testing/new-buildbot-worker.rst b/testing/new-buildbot-worker.rst index debd8df2f9..2ddbfaeac5 100644 --- a/testing/new-buildbot-worker.rst +++ b/testing/new-buildbot-worker.rst @@ -179,7 +179,7 @@ For Windows: * Alternatively (note: don't do both!), set up the worker service as described in the `buildbot documentation - `_. + `_. To start the worker running for your initial testing, you can do:: From 87d74cee97ca59cc06c5b066d955a0de571cf376 Mon Sep 17 00:00:00 2001 From: Itamar Oren Date: Fri, 27 Sep 2024 17:06:06 -0700 Subject: [PATCH 07/23] gh-1420: Add a note about long paths support to Windows buildbot setup (#1421) --- testing/new-buildbot-worker.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/testing/new-buildbot-worker.rst b/testing/new-buildbot-worker.rst index 2ddbfaeac5..5da61b937b 100644 --- a/testing/new-buildbot-worker.rst +++ b/testing/new-buildbot-worker.rst @@ -102,6 +102,18 @@ can put the ``buildarea`` wherever you want to):: (Note that on Windows, the ``buildbot-worker`` command will be in the :file:`Scripts` directory of your Python installation.) +On Windows, `the maximum length for a path is limited +`_. +This might cause some tests to fail, unless long paths support is enabled. + +Use this PowerShell command to check whether long paths are enabled:: + + Get-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" + +If the value is not "1", you can enable long paths using this PowerShell command:: + + New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force + Once this initial worker setup completes, you should edit the files ``buildarea/info/admin`` and ``buildarea/info/host`` to provide your contact info and information on the host configuration, respectively. This information From 3defe53bdafc253a2a602d2c6f1d66ca654296be Mon Sep 17 00:00:00 2001 From: Itamar Oren Date: Sat, 28 Sep 2024 06:24:57 -0700 Subject: [PATCH 08/23] Add more packages to CentOS setup instructions (#1423) --- getting-started/setup-building.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/getting-started/setup-building.rst b/getting-started/setup-building.rst index 0c0e286916..021916e4d4 100644 --- a/getting-started/setup-building.rst +++ b/getting-started/setup-building.rst @@ -627,9 +627,19 @@ some of CPython's modules (for example, ``zlib``). On **Fedora**, **RHEL**, **CentOS** and other ``dnf``-based systems:: + $ sudo dnf install git pkg-config $ sudo dnf install dnf-plugins-core # install this to use 'dnf builddep' $ sudo dnf builddep python3 + Some optional development dependencies are not included in the above. + To install some additional dependencies for optional build and test components:: + + $ sudo dnf install \ + gcc gcc-c++ gdb lzma glibc-devel libstdc++-devel openssl-devel \ + readline-devel zlib-devel libffi-devel bzip2-devel xz-devel \ + sqlite sqlite-devel sqlite-libs libuuid-devel gdbm-libs perf \ + expat expat-devel mpdecimal python3-pip + On **Debian**, **Ubuntu**, and other ``apt``-based systems, try to get the dependencies for the Python you're working on by using the ``apt`` command. From 91a1d1dfc9232f0cc4052fec9eaa3d9066646858 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 1 Oct 2024 06:56:12 +0300 Subject: [PATCH 09/23] Bump sphinx-lint from 0.9.1 to 1.0.0 (#1425) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index fd04efd8cc..ea5fdb74d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,7 +2,7 @@ furo>=2022.6.4 jinja2 sphinx-autobuild>=2024.9.19 sphinx-inline-tabs>=2023.4.21 -sphinx-lint==0.9.1 +sphinx-lint==1.0.0 sphinx-notfound-page>=1.0.0 sphinx_copybutton>=0.3.3 sphinxext-opengraph>=0.7.1 From e8a01957825ef8455431d5c80a2591011e40a067 Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Thu, 3 Oct 2024 13:18:52 -0500 Subject: [PATCH 10/23] Update 3.13.0 expected release date (#1427) --- include/release-cycle.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/release-cycle.json b/include/release-cycle.json index fdb574b119..5070eace2e 100644 --- a/include/release-cycle.json +++ b/include/release-cycle.json @@ -11,7 +11,7 @@ "branch": "3.13", "pep": 719, "status": "prerelease", - "first_release": "2024-10-01", + "first_release": "2024-10-07", "end_of_life": "2029-10", "release_manager": "Thomas Wouters" }, From 26de21433345d26262bec340ea0952a81c20150f Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Sat, 5 Oct 2024 00:50:46 +0100 Subject: [PATCH 11/23] gh-124989: remove exception related details which have moved to cpython/InternalsDoc (#1428) --- internals/interpreter.rst | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/internals/interpreter.rst b/internals/interpreter.rst index 50332ac5e8..477a688d92 100644 --- a/internals/interpreter.rst +++ b/internals/interpreter.rst @@ -171,15 +171,7 @@ Do not confuse the evaluation stack with the call stack, which is used to implem Error handling ============== -When an instruction like ``BINARY_OP`` encounters an error, an exception is raised. -At this point, a traceback entry is added to the exception (by ``PyTraceBack_Here()``) and cleanup is performed. -In the simplest case (absent any ``try`` blocks), this results in the remaining objects being popped off the evaluation stack and their reference count decremented (if not ``NULL``) . -Then the interpreter function (``_PyEval_EvalFrameDefault()``) returns ``NULL``. - -However, if an exception is raised in a ``try`` block, the interpreter must jump to the corresponding ``except`` or ``finally`` block. -In 3.10 and before, there was a separate "block stack" which was used to keep track of nesting ``try`` blocks. -In 3.11, this mechanism has been replaced by a statically generated table, ``code->co_exceptiontable``, -which is described in detail in the `internals documentation +See the `internals documentation `_. The locations table @@ -206,18 +198,6 @@ From C code, you have to call :c:func:`PyCode_Addr2Location`. Fortunately, the locations table is only consulted by exception handling (to set ``tb_lineno``) and by tracing (to pass the line number to the tracing function). In order to reduce the overhead during tracing, the mapping from instruction offset to line number is cached in the ``_co_linearray`` field. -Exception chaining ------------------- - -When an exception is raised during exception handling, the new exception is chained to the old one. -This is done by making the ``__context__`` field of the new exception point to the old one. -This is the responsibility of ``_PyErr_SetObject()`` in :cpy-file:`Python/errors.c` (which is ultimately called by all ``PyErr_Set*()`` functions). -Separately, if a statement of the form :samp:`raise {X} from {Y}` is executed, the ``__cause__`` field of the raised exception (:samp:`{X}`) is set to :samp:`{Y}`. -This is done by :c:func:`PyException_SetCause`, called in response to all ``RAISE_VARARGS`` instructions. -A special case is :samp:`raise {X} from None`, which sets the ``__cause__`` field to ``None`` (at the C level, it sets ``cause`` to ``NULL``). - -(TODO: Other exception details.) - Python-to-Python calls ====================== From 9433f72238cebfd5014e1febb72d30baf65cf500 Mon Sep 17 00:00:00 2001 From: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Date: Mon, 7 Oct 2024 16:34:38 +0100 Subject: [PATCH 12/23] Latest and greatest (3.13.0) (#1429) --- include/release-cycle.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/release-cycle.json b/include/release-cycle.json index 5070eace2e..ede7900693 100644 --- a/include/release-cycle.json +++ b/include/release-cycle.json @@ -10,7 +10,7 @@ "3.13": { "branch": "3.13", "pep": 719, - "status": "prerelease", + "status": "bugfix", "first_release": "2024-10-07", "end_of_life": "2029-10", "release_manager": "Thomas Wouters" From f7ce17f936921690419d4e618a2197acb7e3f756 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 7 Oct 2024 20:10:39 +0200 Subject: [PATCH 13/23] So long, Python 3.8 (#1430) --- include/release-cycle.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/release-cycle.json b/include/release-cycle.json index ede7900693..b77e904879 100644 --- a/include/release-cycle.json +++ b/include/release-cycle.json @@ -50,9 +50,9 @@ "3.8": { "branch": "3.8", "pep": 569, - "status": "security", + "status": "end-of-life", "first_release": "2019-10-14", - "end_of_life": "2024-10", + "end_of_life": "2024-10-07", "release_manager": "Łukasz Langa" }, "3.7": { From bd3f9fd7f58d0cafd999a45e869c8542fca76ae1 Mon Sep 17 00:00:00 2001 From: Jacob Coffee Date: Tue, 8 Oct 2024 17:37:16 -0500 Subject: [PATCH 14/23] feat: add `infra` label (#1432) * feat: add `infra` label Closes: #1418 * chore: hierarchical sort grouped by levels * chore: apply review Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- triage/labels.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/triage/labels.rst b/triage/labels.rst index 783bb555d7..b75f757213 100644 --- a/triage/labels.rst +++ b/triage/labels.rst @@ -109,21 +109,23 @@ for a list of active branches. Other labels ============ -* :gh-label:`triaged`: for issue has been accepted as valid by a triager. -* :gh-label:`easy`: for issues that are considered easy. * :gh-label:`build`/:gh-label:`performance`: for issues related to the build process or performance, respectively. +* :gh-label:`easy`: for issues that are considered easy. +* :gh-label:`infra`: for issues related to the infrastructure of the + project (for example, GitHub Actions, dependabot, the buildbots). +* :gh-label:`pending`: for issues/PRs that will be closed unless further + feedback is provided. * :gh-label:`release-blocker`/:gh-label:`deferred-blocker`: for issues/PRs + and the :ref:`branch's release manager ` + removing or retaining the label as appropriate. that, unless fixed, will hold the current or next release respectively. Triagers may set these labels for issues that must be fixed before a release, - and the :ref:`branch's release manager ` will review them and determine if they indeed qualify, - removing or retaining the label as appropriate. -* :gh-label:`pending`: for issues/PRs that will be closed unless further - feedback is provided. -* :gh-label:`stale`: for issues/PRs that have been inactive for a while. * :gh-label:`sprint`: for easier filtering of issues/PRs being worked on during official sprints. +* :gh-label:`stale`: for issues/PRs that have been inactive for a while. +* :gh-label:`triaged`: for issue has been accepted as valid by a triager. .. _GitHub Labels for PRs: From 9fb3c00caa608f91040546b278f40e6cdbc8ce7c Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 9 Oct 2024 15:28:46 +0200 Subject: [PATCH 15/23] Use --with-dbmliborder or macOS+Homebrew on 3.10 and below (#1433) --- getting-started/setup-building.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/getting-started/setup-building.rst b/getting-started/setup-building.rst index 021916e4d4..4f7478dbce 100644 --- a/getting-started/setup-building.rst +++ b/getting-started/setup-building.rst @@ -739,7 +739,12 @@ some of CPython's modules (for example, ``zlib``). ./configure --with-pydebug \ --with-openssl="$(brew --prefix openssl@3)" \ --with-tcltk-libs="$(pkg-config --libs tcl tk)" \ - --with-tcltk-includes="$(pkg-config --cflags tcl tk)" + --with-tcltk-includes="$(pkg-config --cflags tcl tk)" \ + --with-dbmliborder=gdbm:ndbm + + (``--with-dbmliborder`` is a workaround for a Homebrew-specific change + to ``gdbm``; see `#89452 `_ + for details.) .. tab:: MacPorts From f4524c425db5918c14a693b2e97ba1de90e1c268 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Wed, 9 Oct 2024 18:23:31 +0100 Subject: [PATCH 16/23] gh-119786: Move parser doc from devguide to InternalDocs (#1431) --- internals/parser.rst | 920 +------------------------------------------ 1 file changed, 2 insertions(+), 918 deletions(-) diff --git a/internals/parser.rst b/internals/parser.rst index ad97950ef8..688ad61e77 100644 --- a/internals/parser.rst +++ b/internals/parser.rst @@ -6,921 +6,5 @@ Guide to the parser .. highlight:: none -Abstract -======== - -The Parser in CPython is currently a `PEG (Parser Expression Grammar) -`_ parser. The first -version of the parser used to be an `LL(1) -`_ based parser that was one of the -oldest parts of CPython implemented before it was replaced by :pep:`617`. In -particular, both the current parser and the old LL(1) parser are the output of a -`parser generator `_. This -means that the way the parser is written is by feeding a description of the -Grammar of the Python language to a special program (the parser generator) which -outputs the parser. The way the Python language is changed is therefore by -modifying the grammar file and developers rarely need to interact with the -parser generator itself other than use it to generate the parser. - -How PEG parsers work -==================== - -.. _how-peg-parsers-work: - -A PEG (Parsing Expression Grammar) grammar (like the current one) differs from a -context-free grammar in that the way it is written more closely -reflects how the parser will operate when parsing it. The fundamental technical -difference is that the choice operator is ordered. This means that when writing:: - - rule: A | B | C - -a context-free-grammar parser (like an LL(1) parser) will generate constructions -that given an input string will *deduce* which alternative (``A``, ``B`` or ``C``) -must be expanded, while a PEG parser will check if the first alternative succeeds -and only if it fails, will it continue with the second or the third one in the -order in which they are written. This makes the choice operator not commutative. - -Unlike LL(1) parsers, PEG-based parsers cannot be ambiguous: if a string parses, -it has exactly one valid parse tree. This means that a PEG-based parser cannot -suffer from the ambiguity problems that can arise with LL(1) parsers and with -context-free grammars in general. - -PEG parsers are usually constructed as a recursive descent parser in which every -rule in the grammar corresponds to a function in the program implementing the -parser and the parsing expression (the "expansion" or "definition" of the rule) -represents the "code" in said function. Each parsing function conceptually takes -an input string as its argument, and yields one of the following results: - -* A "success" result. This result indicates that the expression can be parsed by - that rule and the function may optionally move forward or consume one or more - characters of the input string supplied to it. -* A "failure" result, in which case no input is consumed. - -Notice that "failure" results do not imply that the program is incorrect, nor do -they necessarily mean that the parsing has failed. Since the choice operator is -ordered, a failure very often merely indicates "try the following option". A -direct implementation of a PEG parser as a recursive descent parser will present -exponential time performance in the worst case, because PEG parsers have -infinite lookahead (this means that they can consider an arbitrary number of -tokens before deciding for a rule). Usually, PEG parsers avoid this exponential -time complexity with a technique called "packrat parsing" [1]_ which not only -loads the entire program in memory before parsing it but also allows the parser -to backtrack arbitrarily. This is made efficient by memoizing the rules already -matched for each position. The cost of the memoization cache is that the parser -will naturally use more memory than a simple LL(1) parser, which normally are -table-based. - - -Key ideas ---------- - -.. important:: - Don't try to reason about a PEG grammar in the same way you would to with an EBNF - or context free grammar. PEG is optimized to describe **how** input strings will - be parsed, while context-free grammars are optimized to generate strings of the - language they describe (in EBNF, to know if a given string is in the language, you need - to do work to find out as it is not immediately obvious from the grammar). - -* Alternatives are ordered ( ``A | B`` is not the same as ``B | A`` ). -* If a rule returns a failure, it doesn't mean that the parsing has failed, - it just means "try something else". -* By default PEG parsers run in exponential time, which can be optimized to linear by - using memoization. -* If parsing fails completely (no rule succeeds in parsing all the input text), the - PEG parser doesn't have a concept of "where the :exc:`SyntaxError` is". - - -.. _consequences-of-ordered-choice: - -Consequences of the ordered choice operator -------------------------------------------- - -Although PEG may look like EBNF, its meaning is quite different. The fact -that in PEG parsers alternatives are ordered (which is at the core of how PEG -parsers work) has deep consequences, other than removing ambiguity. - -If a rule has two alternatives and the first of them succeeds, the second one is -**not** attempted even if the caller rule fails to parse the rest of the input. -Thus the parser is said to be "eager". To illustrate this, consider -the following two rules (in these examples, a token is an individual character): :: - - first_rule: ( 'a' | 'aa' ) 'a' - second_rule: ('aa' | 'a' ) 'a' - -In a regular EBNF grammar, both rules specify the language ``{aa, aaa}`` but -in PEG, one of these two rules accepts the string ``aaa`` but not the string -``aa``. The other does the opposite -- it accepts the string ``aa`` -but not the string ``aaa``. The rule ``('a'|'aa')'a'`` does -not accept ``aaa`` because ``'a'|'aa'`` consumes the first ``a``, letting the -final ``a`` in the rule consume the second, and leaving out the third ``a``. -As the rule has succeeded, no attempt is ever made to go back and let -``'a'|'aa'`` try the second alternative. The expression ``('aa'|'a')'a'`` does -not accept ``aa`` because ``'aa'|'a'`` accepts all of ``aa``, leaving nothing -for the final ``a``. Again, the second alternative of ``'aa'|'a'`` is not -tried. - -.. caution:: - - The effects of ordered choice, such as the ones illustrated above, may be hidden by many levels of rules. - -For this reason, writing rules where an alternative is contained in the next one is in almost all cases a mistake, -for example: :: - - my_rule: - | 'if' expression 'then' block - | 'if' expression 'then' block 'else' block - -In this example, the second alternative will never be tried because the first one will -succeed first (even if the input string has an ``'else' block`` that follows). To correctly -write this rule you can simply alter the order: :: - - my_rule: - | 'if' expression 'then' block 'else' block - | 'if' expression 'then' block - -In this case, if the input string doesn't have an ``'else' block``, the first alternative -will fail and the second will be attempted without said part. - -Syntax -====== - -The grammar consists of a sequence of rules of the form: :: - - rule_name: expression - -Optionally, a type can be included right after the rule name, which -specifies the return type of the C or Python function corresponding to -the rule: :: - - rule_name[return_type]: expression - -If the return type is omitted, then a ``void *`` is returned in C and an -``Any`` in Python. - -Grammar expressions -------------------- - -``# comment`` -^^^^^^^^^^^^^ - -Python-style comments. - -``e1 e2`` -^^^^^^^^^ - -Match ``e1``, then match ``e2``. - -:: - - rule_name: first_rule second_rule - -``e1 | e2`` -^^^^^^^^^^^ - -Match ``e1`` or ``e2``. - -The first alternative can also appear on the line after the rule name -for formatting purposes. In that case, a \| must be used before the -first alternative, like so: - -:: - - rule_name[return_type]: - | first_alt - | second_alt - -``( e )`` -^^^^^^^^^ - -Match ``e``. - -:: - - rule_name: (e) - -A slightly more complex and useful example includes using the grouping -operator together with the repeat operators: - -:: - - rule_name: (e1 e2)* - -``[ e ] or e?`` -^^^^^^^^^^^^^^^ - -Optionally match ``e``. - -:: - - rule_name: [e] - -A more useful example includes defining that a trailing comma is -optional: - -:: - - rule_name: e (',' e)* [','] - -``e*`` -^^^^^^ - -Match zero or more occurrences of ``e``. - -:: - - rule_name: (e1 e2)* - -``e+`` -^^^^^^ - -Match one or more occurrences of ``e``. - -:: - - rule_name: (e1 e2)+ - -``s.e+`` -^^^^^^^^ - -Match one or more occurrences of ``e``, separated by ``s``. The generated parse -tree does not include the separator. This is otherwise identical to -``(e (s e)*)``. - -:: - - rule_name: ','.e+ - -``&e`` -^^^^^^ - -.. _peg-positive-lookahead: - -Succeed if ``e`` can be parsed, without consuming any input. - -``!e`` -^^^^^^ - -.. _peg-negative-lookahead: - -Fail if ``e`` can be parsed, without consuming any input. - -An example taken from the Python grammar specifies that a primary -consists of an atom, which is not followed by a ``.`` or a ``(`` or a -``[``: - -:: - - primary: atom !'.' !'(' !'[' - -``~`` -^^^^^ - -Commit to the current alternative, even if it fails to parse (this is called -the "cut"). - -:: - - rule_name: '(' ~ some_rule ')' | some_alt - -In this example, if a left parenthesis is parsed, then the other -alternative won’t be considered, even if some_rule or ``)`` fail to be -parsed. - -Left recursion --------------- - -PEG parsers normally do not support left recursion but CPython's parser -generator implements a technique similar to the one described in Medeiros et al. -[2]_ but using the memoization cache instead of static variables. This approach -is closer to the one described in Warth et al. [3]_. This allows us to write not -only simple left-recursive rules but also more complicated rules that involve -indirect left-recursion like:: - - rule1: rule2 | 'a' - rule2: rule3 | 'b' - rule3: rule1 | 'c' - -and "hidden left-recursion" like:: - - rule: 'optional'? rule '@' some_other_rule - -Variables in the grammar ------------------------- - -A sub-expression can be named by preceding it with an identifier and an -``=`` sign. The name can then be used in the action (see below), like this: :: - - rule_name[return_type]: '(' a=some_other_rule ')' { a } - -Grammar actions ---------------- - -.. _peg-grammar-actions: - -To avoid the intermediate steps that obscure the relationship between the -grammar and the AST generation the PEG parser allows directly generating AST -nodes for a rule via grammar actions. Grammar actions are language-specific -expressions that are evaluated when a grammar rule is successfully parsed. These -expressions can be written in Python or C depending on the desired output of the -parser generator. This means that if one would want to generate a parser in -Python and another in C, two grammar files should be written, each one with a -different set of actions, keeping everything else apart from said actions -identical in both files. As an example of a grammar with Python actions, the -piece of the parser generator that parses grammar files is bootstrapped from a -meta-grammar file with Python actions that generate the grammar tree as a result -of the parsing. - -In the specific case of the PEG grammar for Python, having actions allows -directly describing how the AST is composed in the grammar itself, making it -more clear and maintainable. This AST generation process is supported by the use -of some helper functions that factor out common AST object manipulations and -some other required operations that are not directly related to the grammar. - -To indicate these actions each alternative can be followed by the action code -inside curly-braces, which specifies the return value of the alternative:: - - rule_name[return_type]: - | first_alt1 first_alt2 { first_alt1 } - | second_alt1 second_alt2 { second_alt1 } - -If the action is omitted, a default action is generated: - -* If there's a single name in the rule, it gets returned. - -* If there is more than one name in the rule, a collection with all parsed - expressions gets returned (the type of the collection will be different - in C and Python). - -This default behaviour is primarily made for very simple situations and for -debugging purposes. - -.. warning:: - - It's important that the actions don't mutate any AST nodes that are passed - into them via variables referring to other rules. The reason for mutation - being not allowed is that the AST nodes are cached by memoization and could - potentially be reused in a different context, where the mutation would be - invalid. If an action needs to change an AST node, it should instead make a - new copy of the node and change that. - -The full meta-grammar for the grammars supported by the PEG generator is: - -:: - - start[Grammar]: grammar ENDMARKER { grammar } - - grammar[Grammar]: - | metas rules { Grammar(rules, metas) } - | rules { Grammar(rules, []) } - - metas[MetaList]: - | meta metas { [meta] + metas } - | meta { [meta] } - - meta[MetaTuple]: - | "@" NAME NEWLINE { (name.string, None) } - | "@" a=NAME b=NAME NEWLINE { (a.string, b.string) } - | "@" NAME STRING NEWLINE { (name.string, literal_eval(string.string)) } - - rules[RuleList]: - | rule rules { [rule] + rules } - | rule { [rule] } - - rule[Rule]: - | rulename ":" alts NEWLINE INDENT more_alts DEDENT { - Rule(rulename[0], rulename[1], Rhs(alts.alts + more_alts.alts)) } - | rulename ":" NEWLINE INDENT more_alts DEDENT { Rule(rulename[0], rulename[1], more_alts) } - | rulename ":" alts NEWLINE { Rule(rulename[0], rulename[1], alts) } - - rulename[RuleName]: - | NAME '[' type=NAME '*' ']' {(name.string, type.string+"*")} - | NAME '[' type=NAME ']' {(name.string, type.string)} - | NAME {(name.string, None)} - - alts[Rhs]: - | alt "|" alts { Rhs([alt] + alts.alts)} - | alt { Rhs([alt]) } - - more_alts[Rhs]: - | "|" alts NEWLINE more_alts { Rhs(alts.alts + more_alts.alts) } - | "|" alts NEWLINE { Rhs(alts.alts) } - - alt[Alt]: - | items '$' action { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=action) } - | items '$' { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=None) } - | items action { Alt(items, action=action) } - | items { Alt(items, action=None) } - - items[NamedItemList]: - | named_item items { [named_item] + items } - | named_item { [named_item] } - - named_item[NamedItem]: - | NAME '=' ~ item {NamedItem(name.string, item)} - | item {NamedItem(None, item)} - | it=lookahead {NamedItem(None, it)} - - lookahead[LookaheadOrCut]: - | '&' ~ atom {PositiveLookahead(atom)} - | '!' ~ atom {NegativeLookahead(atom)} - | '~' {Cut()} - - item[Item]: - | '[' ~ alts ']' {Opt(alts)} - | atom '?' {Opt(atom)} - | atom '*' {Repeat0(atom)} - | atom '+' {Repeat1(atom)} - | sep=atom '.' node=atom '+' {Gather(sep, node)} - | atom {atom} - - atom[Plain]: - | '(' ~ alts ')' {Group(alts)} - | NAME {NameLeaf(name.string) } - | STRING {StringLeaf(string.string)} - - # Mini-grammar for the actions - - action[str]: "{" ~ target_atoms "}" { target_atoms } - - target_atoms[str]: - | target_atom target_atoms { target_atom + " " + target_atoms } - | target_atom { target_atom } - - target_atom[str]: - | "{" ~ target_atoms "}" { "{" + target_atoms + "}" } - | NAME { name.string } - | NUMBER { number.string } - | STRING { string.string } - | "?" { "?" } - | ":" { ":" } - -As an illustrative example this simple grammar file allows directly -generating a full parser that can parse simple arithmetic expressions and that -returns a valid C-based Python AST: - -:: - - start[mod_ty]: a=expr_stmt* ENDMARKER { _PyAST_Module(a, NULL, p->arena) } - expr_stmt[stmt_ty]: a=expr NEWLINE { _PyAST_Expr(a, EXTRA) } - - expr[expr_ty]: - | l=expr '+' r=term { _PyAST_BinOp(l, Add, r, EXTRA) } - | l=expr '-' r=term { _PyAST_BinOp(l, Sub, r, EXTRA) } - | term - - term[expr_ty]: - | l=term '*' r=factor { _PyAST_BinOp(l, Mult, r, EXTRA) } - | l=term '/' r=factor { _PyAST_BinOp(l, Div, r, EXTRA) } - | factor - - factor[expr_ty]: - | '(' e=expr ')' { e } - | atom - - atom[expr_ty]: - | NAME - | NUMBER - -Here ``EXTRA`` is a macro that expands to ``start_lineno, start_col_offset, -end_lineno, end_col_offset, p->arena``, those being variables automatically -injected by the parser; ``p`` points to an object that holds on to all state -for the parser. - -A similar grammar written to target Python AST objects: - -:: - - start[ast.Module]: a=expr_stmt* ENDMARKER { ast.Module(body=a or [] } - expr_stmt: a=expr NEWLINE { ast.Expr(value=a, EXTRA) } - - expr: - | l=expr '+' r=term { ast.BinOp(left=l, op=ast.Add(), right=r, EXTRA) } - | l=expr '-' r=term { ast.BinOp(left=l, op=ast.Sub(), right=r, EXTRA) } - | term - - term: - | l=term '*' r=factor { ast.BinOp(left=l, op=ast.Mult(), right=r, EXTRA) } - | l=term '/' r=factor { ast.BinOp(left=l, op=ast.Div(), right=r, EXTRA) } - | factor - - factor: - | '(' e=expr ')' { e } - | atom - - atom: - | NAME - | NUMBER - - -Pegen -===== - -Pegen is the parser generator used in CPython to produce the final PEG parser used by the interpreter. It is the -program that can be used to read the python grammar located in :cpy-file:`Grammar/python.gram` and produce the final C -parser. It contains the following pieces: - -* A parser generator that can read a grammar file and produce a PEG parser written in Python or C that can parse - said grammar. The generator is located at :cpy-file:`Tools/peg_generator/pegen`. -* A PEG meta-grammar that automatically generates a Python parser that is used for the parser generator itself - (this means that there are no manually-written parsers). The meta-grammar is - located at :cpy-file:`Tools/peg_generator/pegen/metagrammar.gram`. -* A generated parser (using the parser generator) that can directly produce C and Python AST objects. - -The source code for Pegen lives at :cpy-file:`Tools/peg_generator/pegen` but normally all typical commands to interact -with the parser generator are executed from the main makefile. - -How to regenerate the parser ----------------------------- - -Once you have made the changes to the grammar files, to regenerate the ``C`` -parser (the one used by the interpreter) just execute: :: - - make regen-pegen - -using the ``Makefile`` in the main directory. If you are on Windows you can -use the Visual Studio project files to regenerate the parser or to execute: :: - - ./PCbuild/build.bat --regen - -The generated parser file is located at :cpy-file:`Parser/parser.c`. - -How to regenerate the meta-parser ---------------------------------- - -The meta-grammar (the grammar that describes the grammar for the grammar files -themselves) is located at :cpy-file:`Tools/peg_generator/pegen/metagrammar.gram`. -Although it is very unlikely that you will ever need to modify it, if you make any modifications -to this file (in order to implement new Pegen features) you will need to regenerate -the meta-parser (the parser that parses the grammar files). To do so just execute: :: - - make regen-pegen-metaparser - -If you are on Windows you can use the Visual Studio project files -to regenerate the parser or to execute: :: - - ./PCbuild/build.bat --regen - - -Grammatical elements and rules ------------------------------- - -Pegen has some special grammatical elements and rules: - -* Strings with single quotes (') (for example, ``'class'``) denote KEYWORDS. -* Strings with double quotes (") (for example, ``"match"``) denote SOFT KEYWORDS. -* Uppercase names (for example, ``NAME``) denote tokens in the :cpy-file:`Grammar/Tokens` file. -* Rule names starting with ``invalid_`` are used for specialized syntax errors. - - - These rules are NOT used in the first pass of the parser. - - Only if the first pass fails to parse, a second pass including the invalid - rules will be executed. - - If the parser fails in the second phase with a generic syntax error, the - location of the generic failure of the first pass will be used (this avoids - reporting incorrect locations due to the invalid rules). - - The order of the alternatives involving invalid rules matter - (like any rule in PEG). - -Tokenization ------------- - -It is common among PEG parser frameworks that the parser does both the parsing and the tokenization, -but this does not happen in Pegen. The reason is that the Python language needs a custom tokenizer -to handle things like indentation boundaries, some special keywords like ``ASYNC`` and ``AWAIT`` -(for compatibility purposes), backtracking errors (such as unclosed parenthesis), dealing with encoding, -interactive mode and much more. Some of these reasons are also there for historical purposes, and some -others are useful even today. - -The list of tokens (all uppercase names in the grammar) that you can use can be found in the :cpy-file:`Grammar/Tokens` -file. If you change this file to add new tokens, make sure to regenerate the files by executing: :: - - make regen-token - -If you are on Windows you can use the Visual Studio project files to regenerate the tokens or to execute: :: - - ./PCbuild/build.bat --regen - -How tokens are generated and the rules governing this are completely up to the tokenizer -(:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`); -the parser just receives tokens from it. - -Memoization ------------ - -As described previously, to avoid exponential time complexity in the parser, memoization is used. - -The C parser used by Python is highly optimized and memoization can be expensive both in memory and time. Although -the memory cost is obvious (the parser needs memory for storing previous results in the cache) the execution time -cost comes for continuously checking if the given rule has a cache hit or not. In many situations, just parsing it -again can be faster. Pegen **disables memoization by default** except for rules with the special marker ``memo`` after -the rule name (and type, if present): :: - - rule_name[typr] (memo): - ... - -By selectively turning on memoization for a handful of rules, the parser becomes faster and uses less memory. - -.. note:: - Left-recursive rules always use memoization, since the implementation of left-recursion depends on it. - -To know if a new rule needs memoization or not, benchmarking is required -(comparing execution times and memory usage of some considerably big files with -and without memoization). There is a very simple instrumentation API available -in the generated C parse code that allows to measure how much each rule uses -memoization (check the :cpy-file:`Parser/pegen.c` file for more information) but it -needs to be manually activated. - -Automatic variables -------------------- - -To make writing actions easier, Pegen injects some automatic variables in the namespace available -when writing actions. In the C parser, some of these automatic variable names are: - -* ``p``: The parser structure. -* ``EXTRA``: This is a macro that expands to ``(_start_lineno, _start_col_offset, _end_lineno, _end_col_offset, p->arena)``, - which is normally used to create AST nodes as almost all constructors need these attributes to be provided. All of the - location variables are taken from the location information of the current token. - -Hard and soft keywords ----------------------- - -.. note:: - In the grammar files, keywords are defined using **single quotes** (for example, ``'class'``) while soft - keywords are defined using **double quotes** (for example, ``"match"``). - -There are two kinds of keywords allowed in pegen grammars: *hard* and *soft* -keywords. The difference between hard and soft keywords is that hard keywords -are always reserved words, even in positions where they make no sense (for example, ``x = class + 1``), -while soft keywords only get a special meaning in context. Trying to use a hard -keyword as a variable will always fail: - -.. code-block:: - - >>> class = 3 - File "", line 1 - class = 3 - ^ - SyntaxError: invalid syntax - >>> foo(class=3) - File "", line 1 - foo(class=3) - ^^^^^ - SyntaxError: invalid syntax - -While soft keywords don't have this limitation if used in a context other the one where they -are defined as keywords: - -.. code-block:: python - - >>> match = 45 - >>> foo(match="Yeah!") - -The ``match`` and ``case`` keywords are soft keywords, so that they are recognized as -keywords at the beginning of a match statement or case block respectively, but are -allowed to be used in other places as variable or argument names. - -You can get a list of all keywords defined in the grammar from Python: - -.. code-block:: python - - >>> import keyword - >>> keyword.kwlist - ['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', - 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', - 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', - 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield'] - -as well as soft keywords: - -.. code-block:: python - - >>> import keyword - >>> keyword.softkwlist - ['_', 'case', 'match'] - -.. caution:: - Soft keywords can be a bit challenging to manage as they can be accepted in - places you don't intend to, given how the order alternatives behave in PEG - parsers (see :ref:`consequences of ordered choice section - ` for some background on this). In general, - try to define them in places where there is not a lot of alternatives. - -Error handling --------------- - -When a pegen-generated parser detects that an exception is raised, it will -**automatically stop parsing**, no matter what the current state of the parser -is and it will unwind the stack and report the exception. This means that if a -:ref:`rule action ` raises an exception all parsing will -stop at that exact point. This is done to allow to correctly propagate any -exception set by calling Python's C API functions. This also includes :exc:`SyntaxError` -exceptions and this is the main mechanism the parser uses to report custom syntax -error messages. - -.. note:: - Tokenizer errors are normally reported by raising exceptions but some special - tokenizer errors such as unclosed parenthesis will be reported only after the - parser finishes without returning anything. - -How syntax errors are reported ------------------------------- - -As described previously in the :ref:`how PEG parsers work section -`, PEG parsers don't have a defined concept of where -errors happened in the grammar, because a rule failure doesn't imply a -parsing failure like in context free grammars. This means that some heuristic -has to be used to report generic errors unless something is explicitly declared -as an error in the grammar. - -To report generic syntax errors, pegen uses a common heuristic in PEG parsers: -the location of *generic* syntax errors is reported in the furthest token that -was attempted to be matched but failed. This is only done if parsing has failed -(the parser returns ``NULL`` in C or ``None`` in Python) but no exception has -been raised. - -.. caution:: - Positive and negative lookaheads will try to match a token so they will affect - the location of generic syntax errors. Use them carefully at boundaries - between rules. - -As the Python grammar was primordially written as an LL(1) grammar, this heuristic -has an extremely high success rate, but some PEG features can have small effects, -such as :ref:`positive lookaheads ` and -:ref:`negative lookaheads `. - -To generate more precise syntax errors, custom rules are used. This is a common practice -also in context free grammars: the parser will try to accept some construct that is known -to be incorrect just to report a specific syntax error for that construct. In pegen grammars, -these rules start with the ``invalid_`` prefix. This is because trying to match these rules -normally has a performance impact on parsing (and can also affect the 'correct' grammar itself -in some tricky cases, depending on the ordering of the rules) so the generated parser acts in -two phases: - -1. The first phase will try to parse the input stream without taking into account rules that - start with the ``invalid_`` prefix. If the parsing succeeds it will return the generated AST - and the second phase will not be attempted. - -2. If the first phase failed, a second parsing attempt is done including the rules that start - with an ``invalid_`` prefix. By design this attempt **cannot succeed** and is only executed - to give to the invalid rules a chance to detect specific situations where custom, more precise, - syntax errors can be raised. This also allows to trade a bit of performance for precision reporting - errors: given that we know that the input text is invalid, there is no need to be fast because - the interpreter is going to stop anyway. - -.. important:: - When defining invalid rules: - - * Make sure all custom invalid rules raise :exc:`SyntaxError` exceptions (or a subclass of it). - * Make sure **all** invalid rules start with the ``invalid_`` prefix to not - impact performance of parsing correct Python code. - * Make sure the parser doesn't behave differently for regular rules when you introduce invalid rules - (see the :ref:`how PEG parsers work section ` for more information). - -You can find a collection of macros to raise specialized syntax errors in the -:cpy-file:`Parser/pegen.h` header file. These macros allow also to report ranges for -the custom errors that will be highlighted in the tracebacks that will be -displayed when the error is reported. - -.. tip:: - A good way to test if an invalid rule will be triggered when you expect is to test if introducing - a syntax error **after** valid code triggers the rule or not. For example: :: - - $ 42 - - Should trigger the syntax error in the ``$`` character. If your rule is not correctly defined this - won't happen. For example, if you try to define a rule to match Python 2 style ``print`` statements - to make a better error message and you define it as: :: - - invalid_print: "print" expression - - This will **seem** to work because the parser will correctly parse ``print(something)`` because it is valid - code and the second phase will never execute but if you try to parse ``print(something) $ 3`` the first pass - of the parser will fail (because of the ``$``) and in the second phase, the rule will match the - ``print(something)`` as ``print`` followed by the variable ``something`` between parentheses and the error - will be reported there instead of the ``$`` character. - -Generating AST objects ----------------------- - -The output of the C parser used by CPython that is generated by the -:cpy-file:`Grammar/python.gram` grammar file is a Python AST object (using C -structures). This means that the actions in the grammar file generate AST objects -when they succeed. Constructing these objects can be quite cumbersome (see the -`AST compiler section `_ -for more information on how these objects are constructed and how they are used -by the compiler) so special helper functions are used. These functions are declared in the -:cpy-file:`Parser/pegen.h` header file and defined in the :cpy-file:`Parser/action_helpers.c` -file. These functions allow you to join AST sequences, get specific elements -from them or to do extra processing on the generated tree. - -.. caution:: - Actions must **never** be used to accept or reject rules. It may be tempting - in some situations to write a very generic rule and then check the generated - AST to decide if is valid or not but this will render the `official grammar - `_ partially incorrect - (because actions are not included) and will make it more difficult for other - Python implementations to adapt the grammar to their own needs. - -As a general rule, if an action spawns multiple lines or requires something more -complicated than a single expression of C code, is normally better to create a -custom helper in :cpy-file:`Parser/action_helpers.c` and expose it in the -:cpy-file:`Parser/pegen.h` header file so it can be used from the grammar. - -If the parsing succeeds, the parser **must** return a **valid** AST object. - -Testing -======= - -There are three files that contain tests for the grammar and the parser: - -* :cpy-file:`Lib/test/test_grammar.py` -* :cpy-file:`Lib/test/test_syntax.py` -* :cpy-file:`Lib/test/test_exceptions.py` - -Check the contents of these files to know which is the best place to place new tests depending -on the nature of the new feature you are adding. - -Tests for the parser generator itself can be found in the :cpy-file:`Lib/test/test_peg_generator` directory. - - -Debugging generated parsers -=========================== - -Making experiments ------------------- - -As the generated C parser is the one used by Python, this means that if something goes wrong when adding some -new rules to the grammar you cannot correctly compile and execute Python anymore. This makes it a bit challenging -to debug when something goes wrong, especially when making experiments. - -For this reason it is a good idea to experiment first by generating a Python parser. To do this, you can go to the -:cpy-file:`Tools/peg_generator/` directory on the CPython repository and manually call the parser generator by executing: - -.. code-block:: shell - - $ python -m pegen python - -This will generate a file called :file:`parse.py` in the same directory that you can use to parse some input: - -.. code-block:: shell - - $ python parse.py file_with_source_code_to_test.py - -As the generated :file:`parse.py` file is just Python code, you can modify it and add breakpoints to debug or -better understand some complex situations. - - -Verbose mode ------------- - -When Python is compiled in debug mode (by adding ``--with-pydebug`` when running the configure step in Linux or by -adding ``-d`` when calling the :cpy-file:`PCbuild/build.bat` script in Windows), it is possible to activate a **very** verbose -mode in the generated parser. This is very useful to debug the generated parser and to understand how it works, but it -can be a bit hard to understand at first. - -.. note:: - - When activating verbose mode in the Python parser, it is better to not use interactive mode as it can be much harder to - understand, because interactive mode involves some special steps compared to regular parsing. - -To activate verbose mode you can add the ``-d`` flag when executing Python: - -.. code-block:: shell - - $ python -d file_to_test.py - -This will print **a lot** of output to ``stderr`` so is probably better to dump it to a file for further analysis. The output -consists of trace lines with the following structure:: - - ('>'|'-'|'+'|'!') []: ... - -Every line is indented by a different amount (````) depending on how deep the call stack is. The next -character marks the type of the trace: - -* ``>`` indicates that a rule is going to be attempted to be parsed. -* ``-`` indicates that a rule has failed to be parsed. -* ``+`` indicates that a rule has been parsed correctly. -* ``!`` indicates that an exception or an error has been detected and the parser is unwinding. - -The ```` part indicates the current index in the token array, -the ```` part indicates what rule is being parsed and -the ```` part indicates what alternative within that rule -is being attempted. - - -References -========== - -.. [1] Ford, Bryan - https://pdos.csail.mit.edu/~baford/packrat/thesis/ - -.. [2] Medeiros et al. - https://arxiv.org/pdf/1207.0443 - -.. [3] Warth et al. - http://web.cs.ucla.edu/~todd/research/pepm08.pdf - - -.. admonition:: Document history - :class: note - - Pablo Galindo Salgado - Original author +This document is now part of the +`CPython Internals Docs `_. From 3743f23debbf9ee0ea1761a99350aea4e591e302 Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Wed, 9 Oct 2024 13:34:30 -0500 Subject: [PATCH 17/23] Remove obsolete 'custom builders' section (#1435) --- testing/buildbots.rst | 32 -------------------------------- 1 file changed, 32 deletions(-) diff --git a/testing/buildbots.rst b/testing/buildbots.rst index 38e6063647..22962faaf6 100644 --- a/testing/buildbots.rst +++ b/testing/buildbots.rst @@ -223,37 +223,5 @@ and unpredictable, the issue should be reported on the bug tracker; even better if it can be diagnosed and suppressed by fixing the test's implementation, or by making its parameters - such as a timeout - more robust. - -Custom builders -=============== - -.. highlight:: console - -When working on a platform-specific issue, you may want to test your changes on -the buildbot fleet rather than just on GitHub Actions and Azure Pipelines. To do so, you can -make use of the `custom builders -`_. -These builders track the ``buildbot-custom`` short-lived branch of the -``python/cpython`` repository, which is only accessible to core developers. - -To start a build on the custom builders, push the commit you want to test to -the ``buildbot-custom`` branch:: - - $ git push upstream :buildbot-custom - -You may run into conflicts if another developer is currently using the custom -builders or forgot to delete the branch when they finished. In that case, make -sure the other developer is finished and either delete the branch or force-push -(add the ``-f`` option) over it. - -When you have gotten the results of your tests, delete the branch:: - - $ git push upstream :buildbot-custom # or use the GitHub UI - -If you are interested in the results of a specific test file only, we -recommend you change (temporarily, of course) the contents of the -``buildbottest`` clause in ``Makefile.pre.in``; or, for Windows builders, -the ``Tools/buildbot/test.bat`` script. - .. seealso:: :ref:`buildworker` From 356fea2853505515f5da8899e2f8c1c4a988f665 Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Wed, 9 Oct 2024 13:56:47 -0500 Subject: [PATCH 18/23] Restore support for Python 3.10 to build (#1436) `/usr/bin/python3` is still 3.10 in some widely-used distros and the support requirement is not onerous, needing only to use a longer-lived alias for the same object (`assert datetime.UTC is datetime.timezone.utc` holds where both exist). --- .ruff.toml | 2 +- _tools/generate_release_cycle.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.ruff.toml b/.ruff.toml index 550f27e614..af448e5b6e 100644 --- a/.ruff.toml +++ b/.ruff.toml @@ -1,4 +1,4 @@ -target-version = "py311" +target-version = "py310" fix = true output-format = "full" line-length = 88 diff --git a/_tools/generate_release_cycle.py b/_tools/generate_release_cycle.py index ed27424ddb..3a8fefec02 100644 --- a/_tools/generate_release_cycle.py +++ b/_tools/generate_release_cycle.py @@ -45,7 +45,7 @@ def __init__(self) -> None: def write_csv(self) -> None: """Output CSV files.""" - now_str = str(dt.datetime.now(dt.UTC)) + now_str = str(dt.datetime.now(dt.timezone.utc)) versions_by_category = {"branches": {}, "end-of-life": {}} headers = None From 97ad6943bb33b082b175f819c9c84b540d5e4354 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Thu, 10 Oct 2024 08:47:42 +0300 Subject: [PATCH 19/23] Remove mentions of EOL Python 3.8 (#1434) --- developer-workflow/development-cycle.rst | 24 +++++++++++------------- getting-started/setup-building.rst | 8 ++++---- internals/garbage-collector.rst | 2 +- 3 files changed, 16 insertions(+), 18 deletions(-) diff --git a/developer-workflow/development-cycle.rst b/developer-workflow/development-cycle.rst index 9c156abbd5..c7cf6a8b2b 100644 --- a/developer-workflow/development-cycle.rst +++ b/developer-workflow/development-cycle.rst @@ -37,7 +37,7 @@ Branches -------- There is a branch for each *feature version*, whether released or not (for -example, 3.7, 3.8). +example, 3.12, 3.13). .. _indevbranch: @@ -51,13 +51,11 @@ changes, performance improvements, bug fixes. At some point during the life-cycle of a release, a new :ref:`maintenance branch ` is created to host all bug fixing -activity for further micro versions in a feature version (3.8.1, 3.8.2, etc.). +activity for further micro versions in a feature version (3.12.1, 3.12.2, and so +on). -For versions 3.4 and before, this was conventionally done when the final -release was cut (for example, 3.4.0 final). - -Starting with the 3.5 release, we create the release maintenance branch -(``3.5``) at the time we enter beta (3.5.0 beta 1). This allows +We create the release maintenance branch +(``3.14``) at the time we enter beta (3.14.0 beta 1). This allows feature development for the release 3.n+1 to occur within the main branch alongside the beta and release candidate stabilization periods for release 3.n. @@ -79,7 +77,7 @@ releases; the terms are used interchangeably. These releases have a The only changes allowed to occur in a maintenance branch without debate are bug fixes, test improvements, and edits to the documentation. Also, a general rule for maintenance branches is that compatibility -must not be broken at any point between sibling micro releases (3.5.1, 3.5.2, +must not be broken at any point between sibling micro releases (3.12.1, 3.12.2, etc.). For both rules, only rare exceptions are accepted and **must** be discussed first. @@ -97,9 +95,9 @@ that maintenance branch. Sometime following the final release (3.x.0), the maintenance branch for the previous minor version will go into :ref:`security mode `, usually after at least one more bugfix release at the discretion of the -release manager. For example, the 3.4 maintenance branch was put into -:ref:`security mode ` after the 3.4.4 bugfix release -which followed the release of 3.5.1. +release manager. For example, the 3.11 maintenance branch was put into +:ref:`security mode ` after the 3.11.9 bugfix release +which followed the release of 3.12.2. .. _secbranch: @@ -131,7 +129,7 @@ End-of-life branches The code base for a release cycle which has reached end-of-life status is frozen and no longer has a branch in the repo. The final state of the end-of-lifed branch is recorded as a tag with the same name as the -former branch, for example, ``3.3`` or ``2.6``. +former branch, for example, ``3.8`` or ``2.7``. The :ref:`versions` page contains list of active and end-of-life branches. @@ -347,7 +345,7 @@ Current administrators | Pablo Galindo | Python 3.10 and 3.11 Release Manager, | pablogsal | | | Maintainer of buildbot.python.org | | +-------------------+----------------------------------------------------------+-----------------+ -| Łukasz Langa | Python 3.8 and 3.9 Release Manager, | ambv | +| Łukasz Langa | Python 3.9 Release Manager, | ambv | | | PSF CPython Developer in Residence 2021-present | | +-------------------+----------------------------------------------------------+-----------------+ | Brett Cannon | | brettcannon | diff --git a/getting-started/setup-building.rst b/getting-started/setup-building.rst index 4f7478dbce..dc5634c2fa 100644 --- a/getting-started/setup-building.rst +++ b/getting-started/setup-building.rst @@ -116,8 +116,8 @@ in the ``cpython`` directory and two remotes that refer to your own GitHub fork If you want a working copy of an already-released version of Python, that is, a version in :ref:`maintenance mode `, you can checkout -a release branch. For instance, to checkout a working copy of Python 3.8, -do ``git switch 3.8``. +a release branch. For instance, to checkout a working copy of Python 3.13, +do ``git switch 3.13``. You will need to re-compile CPython when you do such an update. @@ -730,9 +730,9 @@ some of CPython's modules (for example, ``zlib``). ./configure --with-pydebug \ --with-openssl="$(brew --prefix openssl@3)" - .. tab:: Python 3.8-3.10 + .. tab:: Python 3.9-3.10 - For Python 3.8, 3.9, and 3.10:: + For Python 3.9 and 3.10:: $ CPPFLAGS="-I$(brew --prefix gdbm)/include -I$(brew --prefix xz)/include" \ LDFLAGS="-L$(brew --prefix gdbm)/lib -L$(brew --prefix xz)/lib" \ diff --git a/internals/garbage-collector.rst b/internals/garbage-collector.rst index 5b220bfe50..e84a53c229 100644 --- a/internals/garbage-collector.rst +++ b/internals/garbage-collector.rst @@ -167,7 +167,7 @@ C APIs Specific APIs are offered to allocate, deallocate, initialize, track, and untrack objects with GC support. These APIs can be found in the `Garbage Collector C API -documentation `_. +documentation `_. Apart from this object structure, the type object for objects supporting garbage collection must include the ``Py_TPFLAGS_HAVE_GC`` in its ``tp_flags`` slot and From e661e33bfe2fcdf39f81a9d93cd6e1fabb00cd5b Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Thu, 10 Oct 2024 03:46:22 -0500 Subject: [PATCH 20/23] Fix linkcheck issues (#1437) * Update linkcheck configuration for unavoidable link issues * Update the home of the python-checkins webhook * Fix broken extlink usage (The extlink extension does not handle the `!` prefix.) * Fix renamed PEP-0 section link * Update redirected links * Fix links to labels with spaces * Remove dead link --- conf.py | 17 +++++++++++------ core-developers/motivations.rst | 2 +- developer-workflow/communication-channels.rst | 4 ++-- developer-workflow/extension-modules.rst | 2 +- developer-workflow/lang-changes.rst | 2 +- getting-started/git-boot-camp.rst | 2 +- getting-started/setup-building.rst | 2 +- triage/issue-tracker.rst | 2 +- triage/labels.rst | 5 +++-- 9 files changed, 22 insertions(+), 16 deletions(-) diff --git a/conf.py b/conf.py index 64d7c68b90..aed6592c38 100644 --- a/conf.py +++ b/conf.py @@ -67,6 +67,7 @@ # Login page r"https://github.com/python/buildmaster-config/issues/new.*": r"https://github.com/login.*", # noqa: E501 r"https://github.com/python/core-workflow/issues/new.*": r"https://github.com/login.*", # noqa: E501 + r"https://github.com/orgs/python/teams.*": r"https://github.com/login.*", # noqa: E501 # Archive redirect r"https://github.com/python/cpython/archive/main.zip": r"https://codeload.github.com/python/cpython/zip/refs/heads/main", # noqa: E501 # Blob to tree @@ -89,6 +90,13 @@ r'\/.*', ] +# Check the link itself, but ignore anchors that are added by JS +# https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-linkcheck_anchors_ignore_for_url +linkcheck_anchors_ignore_for_url = [ + # GitHub + r'https://github.com/.*', +] + linkcheck_ignore = [ # The voters repo is private and appears as a 404 'https://github.com/python/voters', @@ -98,17 +106,14 @@ 'https://discuss.python.org/groups/staff', 'https://discuss.python.org/groups/moderators', 'https://discuss.python.org/groups/admins', - # The crawler gets "Anchor not found" for GitHub anchors - r'https://github.com.+?#L\d+', - r'https://github.com/cli/cli#installation', - r'https://github.com/github/renaming#renaming-existing-branches', - r'https://github.com/python/bedevere/#pr-state-machine', # "Anchor not found": r'https://packaging.python.org/.*#', # "-rate limited-", causing a timeout r'https://stackoverflow.com/.*', # Discord doesn't allow robot crawlers: "403 Client Error: Forbidden" - 'https://support.discord.com/hc/en-us/articles/219070107-Server-Nicknames', + r'https://support.discord.com/hc/en-us/articles/219070107-Server-Nicknames', + # Patreon also gives 403 to the GHA linkcheck runner + r'https://www.patreon.com/.*', ] rediraffe_redirects = { diff --git a/core-developers/motivations.rst b/core-developers/motivations.rst index dfe41d5aeb..b805a7c67f 100644 --- a/core-developers/motivations.rst +++ b/core-developers/motivations.rst @@ -261,7 +261,7 @@ participating in the CPython core development process: .. topic:: Carol Willing (United States) - * Noteable: ``__ (VP Engineering) + * Noteable (VP Engineering) * Personal site: `Willing Consulting `_ * `Extended bio `__ * Project Jupyter (Software Council, Core Team for JupyterHub/Binder) diff --git a/developer-workflow/communication-channels.rst b/developer-workflow/communication-channels.rst index 60ba90339b..499ca9b964 100644 --- a/developer-workflow/communication-channels.rst +++ b/developer-workflow/communication-channels.rst @@ -263,7 +263,7 @@ Other core workflow tools are: * `blurb_it`_ * `miss-islington`_ * `cla-bot`_ -* `cpython-emailer-webhook`_ +* `webhook-mailer`_ Python `Performance Benchmark`_ project is intended to be an authoritative source of benchmarks for all Python implementations. @@ -274,5 +274,5 @@ source of benchmarks for all Python implementations. .. _blurb_it: https://github.com/python/blurb_it .. _miss-islington: https://github.com/python/miss-islington .. _cla-bot: https://github.com/ambv/cla-bot -.. _cpython-emailer-webhook: https://github.com/berkerpeksag/cpython-emailer-webhook +.. _webhook-mailer: https://github.com/python/webhook-mailer .. _Performance Benchmark: https://github.com/python/pyperformance diff --git a/developer-workflow/extension-modules.rst b/developer-workflow/extension-modules.rst index 118fe02d23..6150bb7c40 100644 --- a/developer-workflow/extension-modules.rst +++ b/developer-workflow/extension-modules.rst @@ -75,7 +75,7 @@ with the following :func:`!foo.greet` function: Instead of using the Python implementation of :func:`!foo.greet`, we want to use its corresponding C extension implementation exposed in the :mod:`!_foo` -module. Ideally, we want to modify :cpy-file:`!Lib/foo.py` as follows: +module. Ideally, we want to modify ``Lib/foo.py`` as follows: .. code-block:: python :caption: Lib/foo.py diff --git a/developer-workflow/lang-changes.rst b/developer-workflow/lang-changes.rst index 70ecd679d9..52aabb15dd 100644 --- a/developer-workflow/lang-changes.rst +++ b/developer-workflow/lang-changes.rst @@ -45,7 +45,7 @@ The `Ideas Discourse category`_ is specifically intended for discussion of new features and language changes. Please don't be disappointed if your idea isn't met with universal approval: as the :pep:`long list of Withdrawn and Rejected PEPs -<0#abandoned-withdrawn-and-rejected-peps>` +<0#rejected-superseded-and-withdrawn-peps>` in the :pep:`PEP Index <0>` attests, and as befits a reasonably mature programming language, getting significant changes into Python isn't a simple task. diff --git a/getting-started/git-boot-camp.rst b/getting-started/git-boot-camp.rst index f28ebefbce..15952d6bd2 100644 --- a/getting-started/git-boot-camp.rst +++ b/getting-started/git-boot-camp.rst @@ -341,7 +341,7 @@ example, backports to other branches), so this features is only useful if you know for sure that a single PR is enough to address and close the issue. .. _bedevere: https://github.com/python/bedevere -.. _special keywords: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword +.. _special keywords: https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword Updating your CPython fork -------------------------- diff --git a/getting-started/setup-building.rst b/getting-started/setup-building.rst index dc5634c2fa..8c55bc625a 100644 --- a/getting-started/setup-building.rst +++ b/getting-started/setup-building.rst @@ -47,7 +47,7 @@ itself. Git is easily available for all common operating systems. or the `Git project instructions `_ for step-by-step installation directions. You may also want to consider a graphical client such as `TortoiseGit `_ or - `GitHub Desktop `_. + `GitHub Desktop `_. - **Configure** diff --git a/triage/issue-tracker.rst b/triage/issue-tracker.rst index a38e8d9e01..63076e7a19 100644 --- a/triage/issue-tracker.rst +++ b/triage/issue-tracker.rst @@ -159,6 +159,6 @@ reason either as ``complete`` or ``not planned``. .. _Python Discourse: https://discuss.python.org/ .. _autolinks: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls .. _checklists: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/about-task-lists -.. _duplicates: https://docs.github.com/en/issues/tracking-your-work-with-issues/marking-issues-or-pull-requests-as-a-duplicate +.. _duplicates: https://docs.github.com/en/issues/tracking-your-work-with-issues/administering-issues/marking-issues-or-pull-requests-as-a-duplicate .. _Core Development Discourse category: https://discuss.python.org/c/core-dev/23 .. _old bug tracker: https://bugs.python.org/ diff --git a/triage/labels.rst b/triage/labels.rst index b75f757213..c2981f666d 100644 --- a/triage/labels.rst +++ b/triage/labels.rst @@ -147,9 +147,10 @@ to trigger specific bot behaviors. by these labels. See also :ref:`the status of the Python branches ` for a list of branches and the type of PRs that can be backported to them. -* :gh-label:`skip issue`: for trivial changes (such as typo fixes, comment +* :gh-label:`skip issue `: for trivial changes (such as + typo fixes, comment changes, and section rephrases) that don't require a corresponding issue. -* :gh-label:`skip news`: for PRs that don't need a NEWS entry. +* :gh-label:`skip news `: for PRs that don't need a NEWS entry. The :ref:`news-entry` section covers in details in which cases the NEWS entry can be skipped. * :gh-label:`test-with-buildbots`: used to test the latest commit with From 64b5b0e30870a7e1cfa8aac62dbaf3093558b926 Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Thu, 10 Oct 2024 10:43:13 -0500 Subject: [PATCH 21/23] Fix reference warnings caused by removal of __file__ from cpython docs (#1438) --- developer-workflow/extension-modules.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/developer-workflow/extension-modules.rst b/developer-workflow/extension-modules.rst index 6150bb7c40..7e32283c2e 100644 --- a/developer-workflow/extension-modules.rst +++ b/developer-workflow/extension-modules.rst @@ -31,7 +31,7 @@ Extension modules can be classified into two categories: * A *built-in* extension module is a module built and shipped with the Python interpreter. A built-in module is *statically* linked - into the interpreter, thereby lacking a :attr:`__file__` attribute. + into the interpreter, thereby lacking a :attr:`!__file__` attribute. .. seealso:: :data:`sys.builtin_module_names` --- names of built-in modules. @@ -41,7 +41,7 @@ Extension modules can be classified into two categories: * A *shared* (or *dynamic*) extension module is built as a shared library (``.so`` or ``.dll`` file) and is *dynamically* linked into the interpreter. - In particular, the module's :attr:`__file__` attribute contains the path + In particular, the module's :attr:`!__file__` attribute contains the path to the ``.so`` or ``.dll`` file. Shared modules are built with the :c:macro:`!Py_BUILD_CORE_MODULE` From 9c777785ee1c47dff8606f1be7f07f31fe4576e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Thu, 10 Oct 2024 20:03:55 +0200 Subject: [PATCH 22/23] Add Matt Page to core developers (#1439) --- core-developers/developers.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/core-developers/developers.csv b/core-developers/developers.csv index 3808e76c92..52bbf00dd0 100644 --- a/core-developers/developers.csv +++ b/core-developers/developers.csv @@ -1,3 +1,4 @@ +Matt Page,mpage,2024-10-10,, Kirill Podoprigora,Eclips4,2024-09-20,, Ned Batchelder,nedbat,2024-07-16,, Tian Gao,gaogaotiantian,2024-06-06,, From 8ec03f998290751ba7414de4580dc59232a3104f Mon Sep 17 00:00:00 2001 From: Jacob Coffee Date: Thu, 10 Oct 2024 14:42:48 -0500 Subject: [PATCH 23/23] Infra: Use GitHub issue forms (#1414) Co-authored-by: Ezio Melotti Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.md | 25 -------------- .github/ISSUE_TEMPLATE/bug_report.yml | 39 ++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 14 ++++++++ .github/ISSUE_TEMPLATE/feature_request.md | 22 ------------ .github/ISSUE_TEMPLATE/feature_request.yml | 39 ++++++++++++++++++++++ 5 files changed, 92 insertions(+), 47 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml delete mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md deleted file mode 100644 index 78aa34d6bb..0000000000 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -name: Bug report -about: Create a report to help us improve -title: '' -labels: '' -assignees: '' - ---- - -> Note: This repo is for the Python devguide. If you are requesting an -enhancement for the Python language or CPython interpreter, -then the CPython issue tracker is better -suited for this report: https://github.com/python/cpython/issues - -**Describe the bug** -A clear and concise description of what the bug is. - -**Expected behavior** -A clear and concise description of what you expected to happen. - -**Screenshots** -If applicable, add screenshots to help explain your problem. - -**Additional context** -Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000000..b160c6ea11 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,39 @@ +name: "Bug report" +description: Create a report to help us improve the Python devguide +title: "Bug: " +labels: ["bug"] +assignees: [] + +body: + - type: markdown + attributes: + value: | + > [!NOTE] + > This repo is for the [Python developer's guide](https://devguide.python.org/). + > If you are reporting a bug for the Python language or + > CPython interpreter, then use the + > [CPython issue tracker](https://github.com/python/cpython/issues) instead. + + - type: textarea + id: bug_description + attributes: + label: "Describe the bug" + description: A clear and concise description of what the bug is and, optionally, what you expected to happen. + validations: + required: true + + - type: textarea + id: screenshots + attributes: + label: "Screenshots" + description: If applicable, add screenshots to help explain your problem. + validations: + required: false + + - type: textarea + id: additional_context + attributes: + label: "Additional context" + description: Add any other context about the problem here. + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000..cd8c31d2a9 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,14 @@ +blank_issues_enabled: false +contact_links: + - name: CPython Documentation + url: https://docs.python.org/ + about: Official CPython documentation - please check here before opening an issue. + - name: Python Website + url: https://python.org/ + about: For all things Python + - name: PyPI Issues / Support + url: https://github.com/pypi/support + about: For issues with PyPI itself, PyPI accounts, or with packages hosted on PyPI. + - name: CPython Issues + url: https://github.com/python/cpython/issues + about: For issues with the CPython interpreter itself. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md deleted file mode 100644 index eff8cb8f7a..0000000000 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -name: Feature request -about: Suggest an idea for this project -title: '' -labels: '' -assignees: '' - ---- - -> Note: This repo is for the Python devguide. If you are requesting an -enhancement for the Python language or CPython interpreter, -then the CPython issue tracker is better -suited for this report: https://github.com/python/cpython/issues - -**Describe the enhancement or feature you'd like** -A clear and concise description of what you want to happen. - -**Describe alternatives you've considered** -A clear and concise description of any alternative solutions or features you've considered. - -**Additional context** -Add any other context or screenshots about the feature request here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000000..a4413c137a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,39 @@ +name: "Feature request" +description: Suggest an idea for the Python devguide +title: "Feature: <title>" +labels: ["enhancement"] +assignees: [] + +body: + - type: markdown + attributes: + value: | + > [!NOTE] + > This repo is for the [Python developer's guide](https://devguide.python.org/). + > If you are requesting an enhancement for the Python language or + > CPython interpreter, then use the + > [CPython issue tracker](https://github.com/python/cpython/issues) instead. + + - type: textarea + id: feature_description + attributes: + label: "Describe the enhancement or feature you would like" + description: A clear and concise description of what you want to happen. + validations: + required: true + + - type: textarea + id: alternatives + attributes: + label: "Describe alternatives you have considered" + description: A clear and concise description of any alternative solutions or features you have considered. + validations: + required: false + + - type: textarea + id: additional_context + attributes: + label: "Additional context" + description: Add any other context or screenshots about the feature request here. + validations: + required: false