Merge branch 'main' into parameters

Author: msullivan

peps/pep-0820.rst

@@ -100,6 +100,33 @@ Limited forward compatibility
     versions.
 
 
+Endorsements
+------------
+
+This PEP is unusual in that it doesn't immediately help end users -- it needs
+to be in place well before it starts being helpful.
+In the words of a Cython maintainer,
+`Da Woods <https://discuss.python.org/t/105552/2>`__:
+
+  I suspect Cython wouldn’t immediately use it (at least for classes… obviously
+  if it goes into PEP 793 then we would for that).
+  Just because it’s mostly targeted at future improvements rather than an
+  immediate new feature. It looks usable though.
+
+Instead, support comes from core devs who'll need change the C API, and want
+do so in backwards-compatible ways:
+
+- `Mark Shannon <https://discuss.python.org/t/105552/4>`__:
+
+    This seems like a worthwhile improvement. Count me as a +1
+
+- `Victor Stinner <https://discuss.python.org/t/105552/19>`__:
+
+    I changed my mind and I’m now a supporter of PEP 820 :)
+    It took me a while but I now see the PEP 820 advantages: it enhances the
+    backward compatibility and the stable ABI.
+
+
 Example
 =======
 

peps/pep-0828.rst

@@ -28,9 +28,8 @@ For example, the following code is valid under this PEP:
         yield from generator()
 
 
-
-In addition, this PEP introduces a new ``async yield from`` syntax to use
-existing ``yield from`` semantics on an asynchronous generator:
+In addition, this PEP introduces a new ``async yield from`` construct to
+delegate to an asynchronous generator:
 
 .. code-block:: python
 
@@ -67,6 +66,10 @@ as an :term:`asynchronous generator`, sometimes suffixed with "function".
 In contrast, the object returned by an asynchronous generator is referred to
 as an :term:`asynchronous generator iterator` in this PEP.
 
+This PEP also uses the term "subgenerator" to refer to a generator, synchronous
+or asynchronous, that is used inside of a ``yield from`` or ``async yield from``
+expression.
+
 
 Motivation
 ==========
@@ -105,8 +108,8 @@ in asynchronous generators:
 2. https://discuss.python.org/t/47050
 3. https://discuss.python.org/t/66886
 
-Additionally, this design decision has `come up
-<https://stackoverflow.com/questions/47376408>`__ on Stack Overflow.
+Additionally, users have `questioned <https://stackoverflow.com/questions/47376408>`__
+this design decision on Stack Overflow.
 
 
 Subgenerator delegation is useful for asynchronous generators
@@ -119,16 +122,17 @@ item. This comes with a few drawbacks:
 1. It obscures the intent of the code and increases the amount of effort
    necessary to work with asynchronous generators, because each delegation
    point becomes a loop. This damages the power of asynchronous generators.
-2. :meth:`~agen.asend`, :meth:`~agen.athrow`, and :meth:`~agen.aclose`,
+2. :meth:`~agen.asend`, :meth:`~agen.athrow`, and :meth:`~agen.aclose`
    do not interact properly with the caller. This is the primary reason that
    ``yield from`` was added in the first place.
 3. Return values are not natively supported with asynchronous generators. The
-   workaround for this it to raise an exception, which increases boilerplate.
+   workaround for this is to raise an exception, which increases boilerplate.
 
 
 Specification
 =============
 
+
 Syntax
 ------
 
@@ -163,7 +167,7 @@ This PEP retains all existing ``yield from`` semantics; the only detail is
 that asynchronous generators may now use it.
 
 Because the existing ``yield from`` behavior may only yield from a synchronous
-generator, this is true for asynchronous generators as well.
+subgenerator, this is true for asynchronous generators as well.
 
 For example:
 
@@ -294,6 +298,7 @@ knowledge of ``yield from`` in synchronous generators.
 Potential footguns
 ------------------
 
+
 Forgetting to ``await`` a future
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -314,18 +319,45 @@ For example:
         await asyncio.sleep(0.25)
         return [1, 2, 3]
 
-    async def generator():
+    async def agenerator():
         # Forgot to await!
         yield from asyncio.ensure_future(steps())
 
     async def run():
         total = 0
-        async for i in generator():
+        async for i in agenerator():
             # TypeError?!
             total += i
         print(total)
 
 
+Attempting to use ``yield from`` on an asynchronous subgenerator
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A common intuition among developers is that ``yield from`` inside an
+asynchronous generator will also delegate to another asynchronous generator.
+As such, many users were surprised to see that, in this proposal, the following
+code is invalid:
+
+.. code-block:: python
+
+    async def asubgenerator():
+        yield 1
+        yield 2
+
+    async def agenerator():
+        yield from asubgenerator()
+
+
+As a solution, when ``yield from`` is given an object that is not iterable,
+the implementation can detect if that object is asynchronously iterable.
+If it is, ``async yield from`` can be suggested in the exception message.
+
+This is done in the reference implementation of this proposal; the example
+above raises a :exc:`TypeError` that reads ``async_generator object is not
+iterable. Did you mean 'async yield from'?``
+
+
 Reference Implementation
 ========================
 
@@ -336,17 +368,87 @@ A reference implementation of this PEP can be found at
 Rejected Ideas
 ==============
 
-TBD.
+
+Using ``yield from`` to delegate to asynchronous generators
+-----------------------------------------------------------
+
+It has been argued that many developers may intuitively believe that using a
+plain ``yield from`` inside an asynchronous generator would also delegate to
+an asynchronous subgenerator rather than a synchronous subgenerator. As such,
+it was proposed to make ``yield from`` always delegate to an asynchronous
+subgenerator.
+
+For example:
+
+.. code-block:: python
+
+    async def asubgenerator():
+        yield 1
+        yield 2
+
+    async def agenerator():
+        yield from asubgenerator()
+
+
+This was rejected, primarily because it felt very wrong for ``yield from x`` to
+be valid or invalid depending on the type of generator it was used in.
+
+In addition, there is no precedent for this kind of behavior in Python; inherently
+synchronous constructs always have an asynchronous counterpart for use in
+asynchronous functions, instead of implicitly switching protocols depending on
+the type of function it is used in. For example, :keyword:`with` always means that the
+:term:`synchronous context management protocol <context management protocol>` will
+be invoked, even when used in an ``async def`` function.
+
+Finally, this would leave a gap in asynchronous generators, because there would be
+no mechanism for delegating to a synchronous subgenerator. Even if this is not a
+common pattern today, this may become common in the future, in which case it would
+be very difficult to change the meaning of ``yield from`` in an asynchronous
+generator.
+
+
+Letting ``yield from`` determine which protocol to use
+------------------------------------------------------
+
+As a solution to the above rejected idea, it was proposed to allow ``yield from x``
+to invoke the synchronous or asynchronous generator protocol depending on the type
+of ``x``. In turn, this would allow developers to delegate to both synchronous
+and asynchronous subgenerators while continuing to use the familiar ``yield from``
+syntax.
+
+For example:
+
+.. code-block:: python
+
+    async def asubgenerator():
+        yield 1
+        yield 2
+
+    async def agenerator():
+        yield from asubgenerator()
+        yield from range(3, 5)
+
+
+Mechanically, this is possible, but the exact behavior will likely be counterintuitive
+and ambigious. In particular:
+
+1. If an object implements both :meth:`~object.__iter__` and :meth:`~object.__aiter__`,
+   it's not clear which protocol Python should choose.
+2. If the chosen protocol raises an exception, should the exception be propagated, or
+   should Python try to use the other protocol first?
+
+Additionally, this approach is inherently slower, because of the additional overhead
+of detecting which generator protocol to use.
 
 
 Acknowledgements
 ================
 
 Thanks to Bartosz Sławecki for aiding in the development of the reference
 implementation of this PEP. In addition, the :exc:`StopAsyncIteration`
-changes in addition to the support for non-``None`` return values inside
+changes alongside the support for non-``None`` return values inside
 asynchronous generators were largely based on Alex Dixon's design from
-`python/cpython#125401 <https://github.com/python/cpython/pull/125401>`__
+`python/cpython#125401 <https://github.com/python/cpython/pull/125401>`__.
 
 
 Change History