diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 7486a378dbb06f..84ec535ca98e97 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -1,7 +1,7 @@ .. _regex-howto: **************************** - Regular Expression HOWTO + Regular expression HOWTO **************************** :Author: A.M. Kuchling @@ -47,7 +47,7 @@ Python code to do the processing; while Python code will be slower than an elaborate regular expression, it will also probably be more understandable. -Simple Patterns +Simple patterns =============== We'll start by learning about the simplest possible regular expressions. Since @@ -59,7 +59,7 @@ expressions (deterministic and non-deterministic finite automata), you can refer to almost any textbook on writing compilers. -Matching Characters +Matching characters ------------------- Most letters and characters will simply match themselves. For example, the @@ -159,7 +159,7 @@ match even a newline. ``.`` is often used where you want to match "any character". -Repeating Things +Repeating things ---------------- Being able to match varying sets of characters is the first thing regular @@ -210,7 +210,7 @@ this RE against the string ``'abcbd'``. | | | ``[bcd]*`` is only matching | | | | ``bc``. | +------+-----------+---------------------------------+ -| 6 | ``abcb`` | Try ``b`` again. This time | +| 7 | ``abcb`` | Try ``b`` again. This time | | | | the character at the | | | | current position is ``'b'``, so | | | | it succeeds. | @@ -255,7 +255,7 @@ is equivalent to ``+``, and ``{0,1}`` is the same as ``?``. It's better to use to read. -Using Regular Expressions +Using regular expressions ========================= Now that we've looked at some simple regular expressions, how do we actually use @@ -264,7 +264,7 @@ expression engine, allowing you to compile REs into objects and then perform matches with them. -Compiling Regular Expressions +Compiling regular expressions ----------------------------- Regular expressions are compiled into pattern objects, which have @@ -295,7 +295,7 @@ disadvantage which is the topic of the next section. .. _the-backslash-plague: -The Backslash Plague +The backslash plague -------------------- As stated earlier, regular expressions use the backslash character (``'\'``) to @@ -335,7 +335,7 @@ expressions will often be written in Python code using this raw string notation. In addition, special escape sequences that are valid in regular expressions, but not valid as Python string literals, now result in a -:exc:`DeprecationWarning` and will eventually become a :exc:`SyntaxError`, +:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`, which means the sequences will be invalid if raw string notation or escaping the backslashes isn't used. @@ -351,7 +351,7 @@ the backslashes isn't used. +-------------------+------------------+ -Performing Matches +Performing matches ------------------ Once you have an object representing a compiled regular expression, what do you @@ -369,10 +369,10 @@ for a complete listing. | | location where this RE matches. | +------------------+-----------------------------------------------+ | ``findall()`` | Find all substrings where the RE matches, and | -| | returns them as a list. | +| | return them as a list. | +------------------+-----------------------------------------------+ | ``finditer()`` | Find all substrings where the RE matches, and | -| | returns them as an :term:`iterator`. | +| | return them as an :term:`iterator`. | +------------------+-----------------------------------------------+ :meth:`~re.Pattern.match` and :meth:`~re.Pattern.search` return ``None`` if no match can be found. If @@ -473,7 +473,7 @@ Two pattern methods return all of the matches for a pattern. The ``r`` prefix, making the literal a raw string literal, is needed in this example because escape sequences in a normal "cooked" string literal that are not recognized by Python, as opposed to regular expressions, now result in a -:exc:`DeprecationWarning` and will eventually become a :exc:`SyntaxError`. See +:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`. See :ref:`the-backslash-plague`. :meth:`~re.Pattern.findall` has to create the entire list before it can be returned as the @@ -491,7 +491,7 @@ result. The :meth:`~re.Pattern.finditer` method returns a sequence of (29, 31) -Module-Level Functions +Module-level functions ---------------------- You don't have to create a pattern object and call its methods; the @@ -518,7 +518,7 @@ Outside of loops, there's not much difference thanks to the internal cache. -Compilation Flags +Compilation flags ----------------- .. currentmodule:: re @@ -642,7 +642,7 @@ of each one. whitespace is in a character class or preceded by an unescaped backslash; this lets you organize and indent the RE more clearly. This flag also lets you put comments within a RE that will be ignored by the engine; comments are marked by - a ``'#'`` that's neither in a character class or preceded by an unescaped + a ``'#'`` that's neither in a character class nor preceded by an unescaped backslash. For example, here's a RE that uses :const:`re.VERBOSE`; see how much easier it @@ -669,7 +669,7 @@ of each one. to understand than the version using :const:`re.VERBOSE`. -More Pattern Power +More pattern power ================== So far we've only covered a part of the features of regular expressions. In @@ -679,7 +679,7 @@ retrieve portions of the text that was matched. .. _more-metacharacters: -More Metacharacters +More metacharacters ------------------- There are some metacharacters that we haven't covered yet. Most of them will be @@ -875,7 +875,7 @@ Backreferences like this aren't often useful for just searching through a string find out that they're *very* useful when performing string substitutions. -Non-capturing and Named Groups +Non-capturing and named groups ------------------------------ Elaborate REs may use many groups, both to capture substrings of interest, and @@ -979,7 +979,7 @@ current point. The regular expression for finding doubled words, 'the the' -Lookahead Assertions +Lookahead assertions -------------------- Another zero-width assertion is the lookahead assertion. Lookahead assertions @@ -1061,7 +1061,7 @@ end in either ``bat`` or ``exe``: ``.*[.](?!bat$|exe$)[^.]*$`` -Modifying Strings +Modifying strings ================= Up to this point, we've simply performed searches against a static string. @@ -1083,7 +1083,7 @@ using the following pattern methods: +------------------+-----------------------------------------------+ -Splitting Strings +Splitting strings ----------------- The :meth:`~re.Pattern.split` method of a pattern splits a string apart @@ -1137,7 +1137,7 @@ argument, but is otherwise the same. :: ['Words', 'words, words.'] -Search and Replace +Search and replace ------------------ Another common task is to find all the matches for a pattern, and replace them @@ -1236,7 +1236,7 @@ pattern object as the first parameter, or use embedded modifiers in the pattern string, e.g. ``sub("(?i)b+", "x", "bbbb BBBB")`` returns ``'x x'``. -Common Problems +Common problems =============== Regular expressions are a powerful tool for some applications, but in some ways @@ -1244,7 +1244,7 @@ their behaviour isn't intuitive and at times they don't behave the way you may expect them to. This section will point out some of the most common pitfalls. -Use String Methods +Use string methods ------------------ Sometimes using the :mod:`re` module is a mistake. If you're matching a fixed @@ -1310,7 +1310,7 @@ string and then backtracking to find a match for the rest of the RE. Use :func:`re.search` instead. -Greedy versus Non-Greedy +Greedy versus non-greedy ------------------------ When repeating a regular expression, as in ``a*``, the resulting action is to @@ -1388,9 +1388,9 @@ Feedback ======== Regular expressions are a complicated topic. Did this document help you -understand them? Were there parts that were unclear, or Problems you +understand them? Were there parts that were unclear, or problems you encountered that weren't covered here? If so, please send suggestions for -improvements to the author. +improvements to the :ref:`issue tracker `. The most complete book on regular expressions is almost certainly Jeffrey Friedl's Mastering Regular Expressions, published by O'Reilly. Unfortunately, diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index ae541680c534d6..f3ed1539493b6a 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -560,7 +560,7 @@ start with a character in the "letter-like" set ``xid_start``, and the remaining characters must be in the "letter- and digit-like" set ``xid_continue``. -These sets based on the *XID_Start* and *XID_Continue* sets as defined by the +These sets are based on the *XID_Start* and *XID_Continue* sets as defined by the Unicode standard annex `UAX-31`_. Python's ``xid_start`` additionally includes the underscore (``_``). Note that Python does not necessarily conform to `UAX-31`_. diff --git a/Include/cpython/pystats.h b/Include/cpython/pystats.h index 5d1f44988a6df1..e473110eca7415 100644 --- a/Include/cpython/pystats.h +++ b/Include/cpython/pystats.h @@ -144,7 +144,6 @@ typedef struct _optimization_stats { uint64_t unknown_callee; uint64_t trace_immediately_deopts; uint64_t executors_invalidated; - uint64_t fitness_terminated_traces; UOpStats opcode[PYSTATS_MAX_UOP_ID + 1]; uint64_t unsupported_opcode[256]; uint64_t trace_length_hist[_Py_UOP_HIST_SIZE]; diff --git a/Include/internal/pycore_interp_structs.h b/Include/internal/pycore_interp_structs.h index 0cebe1b4b9e995..f76d4f41c55119 100644 --- a/Include/internal/pycore_interp_structs.h +++ b/Include/internal/pycore_interp_structs.h @@ -449,10 +449,6 @@ typedef struct _PyOptimizationConfig { uint16_t side_exit_initial_value; uint16_t side_exit_initial_backoff; - // Trace fitness thresholds - uint16_t fitness_initial; - uint16_t fitness_initial_side; - // Optimization flags bool specialization_enabled; bool uops_optimize_enabled; diff --git a/Include/internal/pycore_optimizer.h b/Include/internal/pycore_optimizer.h index 820ee32201c1f8..cf01c620476ff7 100644 --- a/Include/internal/pycore_optimizer.h +++ b/Include/internal/pycore_optimizer.h @@ -15,23 +15,6 @@ extern "C" { #include "pycore_optimizer_types.h" #include -/* Default fitness configuration values for trace quality control. - * FITNESS_INITIAL and FITNESS_INITIAL_SIDE can be overridden via - * PYTHON_JIT_FITNESS_INITIAL and PYTHON_JIT_FITNESS_INITIAL_SIDE */ -#define FITNESS_PER_INSTRUCTION 2 -#define FITNESS_BRANCH_BASE 5 -#define FITNESS_INITIAL (FITNESS_PER_INSTRUCTION * 1000) -#define FITNESS_INITIAL_SIDE (FITNESS_INITIAL / 2) -#define FITNESS_BACKWARD_EDGE (FITNESS_INITIAL / 10) - -/* Exit quality constants for fitness-based trace termination. - * Higher values mean better places to stop the trace. */ - -#define EXIT_QUALITY_DEFAULT 200 -#define EXIT_QUALITY_CLOSE_LOOP (4 * EXIT_QUALITY_DEFAULT) -#define EXIT_QUALITY_ENTER_EXECUTOR (2 * EXIT_QUALITY_DEFAULT + 100) -#define EXIT_QUALITY_SPECIALIZABLE (EXIT_QUALITY_DEFAULT / 4) - typedef struct _PyJitUopBuffer { _PyUOpInstruction *start; @@ -118,8 +101,7 @@ typedef struct _PyJitTracerPreviousState { } _PyJitTracerPreviousState; typedef struct _PyJitTracerTranslatorState { - int32_t fitness; // Current trace fitness, starts high, decrements - int frame_depth; // Current inline depth (0 = root frame) + int jump_backward_seen; } _PyJitTracerTranslatorState; typedef struct _PyJitTracerState { @@ -412,6 +394,7 @@ extern JitOptRef _Py_uop_sym_new_type( extern JitOptRef _Py_uop_sym_new_const(JitOptContext *ctx, PyObject *const_val); extern JitOptRef _Py_uop_sym_new_const_steal(JitOptContext *ctx, PyObject *const_val); bool _Py_uop_sym_is_safe_const(JitOptContext *ctx, JitOptRef sym); +bool _Py_uop_sym_is_not_container(JitOptRef sym); _PyStackRef _Py_uop_sym_get_const_as_stackref(JitOptContext *ctx, JitOptRef sym); extern JitOptRef _Py_uop_sym_new_null(JitOptContext *ctx); extern bool _Py_uop_sym_has_type(JitOptRef sym); diff --git a/Lib/test/test_capi/test_opt.py b/Lib/test/test_capi/test_opt.py index 022f05bbe37fa4..56f90194b480a1 100644 --- a/Lib/test/test_capi/test_opt.py +++ b/Lib/test/test_capi/test_opt.py @@ -23,6 +23,9 @@ # For frozendict JIT tests FROZEN_DICT_CONST = frozendict(x=1, y=2) +# For frozenset JIT tests +FROZEN_SET_CONST = frozenset({1, 2, 3}) + class _GenericKey: pass @@ -2169,7 +2172,8 @@ def f(n): self.assertIsNotNone(ex) uops = get_opnames(ex) self.assertNotIn("_GUARD_TOS_ANY_SET", uops) - self.assertIn("_CONTAINS_OP_SET", uops) + # _CONTAINS_OP_SET is constant-folded away for frozenset literals + self.assertIn("_INSERT_2_LOAD_CONST_INLINE_BORROW", uops) def test_remove_guard_for_known_type_tuple(self): def f(n): @@ -4399,6 +4403,20 @@ def testfunc(n): # lookup result is folded to constant 1, so comparison is optimized away self.assertNotIn("_COMPARE_OP_INT", uops) + def test_contains_op_frozenset_const_fold(self): + def testfunc(n): + x = 0 + for _ in range(n): + if 1 in FROZEN_SET_CONST: + x += 1 + return x + + res, ex = self._run_with_optimizer(testfunc, TIER2_THRESHOLD) + self.assertEqual(res, TIER2_THRESHOLD) + self.assertIsNotNone(ex) + uops = get_opnames(ex) + self.assertNotIn("_CONTAINS_OP_SET", uops) + def test_binary_subscr_list_slice(self): def testfunc(n): x = 0 diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index aaab4709464fd0..7502b120825fbc 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -741,6 +741,38 @@ def test_empty_zone(self): with self.assertRaises(ValueError): self.klass.from_file(zf) + def test_invalid_transition_index(self): + STD = ZoneOffset("STD", ZERO) + DST = ZoneOffset("DST", ONE_H, ONE_H) + + zf = self.construct_zone([ + ZoneTransition(datetime(2026, 3, 1, 2), STD, DST), + ZoneTransition(datetime(2026, 11, 1, 2), DST, STD), + ], after="", version=1) + + data = bytearray(zf.read()) + timecnt = struct.unpack_from(">l", data, 32)[0] + idx_offset = 44 + timecnt * 4 + data[idx_offset + 1] = 2 # typecnt is 2, so index 2 is OOB + f = io.BytesIO(bytes(data)) + + with self.assertRaises(ValueError): + self.klass.from_file(f) + + def test_transition_lookahead_out_of_bounds(self): + STD = ZoneOffset("STD", ZERO) + DST = ZoneOffset("DST", ONE_H, ONE_H) + EXT = ZoneOffset("EXT", ONE_H) + + zf = self.construct_zone([ + ZoneTransition(datetime(2026, 3, 1), STD, DST), + ZoneTransition(datetime(2026, 6, 1), DST, EXT), + ZoneTransition(datetime(2026, 9, 1), EXT, DST), + ], after="") + + zi = self.klass.from_file(zf) + self.assertIsNotNone(zi) + def test_zone_very_large_timestamp(self): """Test when a transition is in the far past or future. diff --git a/Lib/zoneinfo/_common.py b/Lib/zoneinfo/_common.py index 59f3f0ce853f74..98668c15d8bf94 100644 --- a/Lib/zoneinfo/_common.py +++ b/Lib/zoneinfo/_common.py @@ -67,6 +67,10 @@ def load_data(fobj): f">{timecnt}{time_type}", fobj.read(timecnt * time_size) ) trans_idx = struct.unpack(f">{timecnt}B", fobj.read(timecnt)) + + if max(trans_idx) >= typecnt: + raise ValueError("Invalid transition index found while reading TZif: " + f"{max(trans_idx)}") else: trans_list_utc = () trans_idx = () diff --git a/Lib/zoneinfo/_zoneinfo.py b/Lib/zoneinfo/_zoneinfo.py index bd3fefc6c9d959..7063eb6a9025ac 100644 --- a/Lib/zoneinfo/_zoneinfo.py +++ b/Lib/zoneinfo/_zoneinfo.py @@ -338,7 +338,7 @@ def _utcoff_to_dstoff(trans_idx, utcoffsets, isdsts): if not isdsts[comp_idx]: dstoff = utcoff - utcoffsets[comp_idx] - if not dstoff and idx < (typecnt - 1): + if not dstoff and idx < (typecnt - 1) and i + 1 < len(trans_idx): comp_idx = trans_idx[i + 1] # If the following transition is also DST and we couldn't diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-04-04-20-59-12.gh-issue-148083.9ZHNBN.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-04-04-20-59-12.gh-issue-148083.9ZHNBN.rst new file mode 100644 index 00000000000000..fea4659d0b9916 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-04-04-20-59-12.gh-issue-148083.9ZHNBN.rst @@ -0,0 +1 @@ +Constant-fold ``_CONTAINS_OP_SET`` for :class:`frozenset`. Patch by Donghee Na. diff --git a/Misc/NEWS.d/next/Library/2026-03-12-21-01-48.gh-issue-145883.lUvXcc.rst b/Misc/NEWS.d/next/Library/2026-03-12-21-01-48.gh-issue-145883.lUvXcc.rst new file mode 100644 index 00000000000000..2c17768c5189da --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-03-12-21-01-48.gh-issue-145883.lUvXcc.rst @@ -0,0 +1,2 @@ +:mod:`zoneinfo`: Fix heap buffer overflow reads from malformed TZif data. +Found by OSS Fuzz, issues :oss-fuzz:`492245058` and :oss-fuzz:`492230068`. diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index aa0b1302cb2fc6..eaffd020ed97c0 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -1075,7 +1075,7 @@ load_data(zoneinfo_state *state, PyZoneInfo_ZoneInfo *self, PyObject *file_obj) } trans_idx[i] = (size_t)cur_trans_idx; - if (trans_idx[i] > self->num_ttinfos) { + if (trans_idx[i] >= self->num_ttinfos) { PyErr_Format( PyExc_ValueError, "Invalid transition index found while reading TZif: %zd", @@ -2081,7 +2081,7 @@ utcoff_to_dstoff(size_t *trans_idx, long *utcoffs, long *dstoffs, dstoff = utcoff - utcoffs[comp_idx]; } - if (!dstoff && idx < (num_ttinfos - 1)) { + if (!dstoff && idx < (num_ttinfos - 1) && i + 1 < num_transitions) { comp_idx = trans_idx[i + 1]; // If the following transition is also DST and we couldn't find diff --git a/Objects/typeobject.c b/Objects/typeobject.c index b19aee6338dcc0..0ac5377d168812 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -9345,6 +9345,7 @@ type_ready_post_checks(PyTypeObject *type) PyErr_Format(PyExc_SystemError, "type %s has a tp_dictoffset that is too small", type->tp_name); + return -1; } } return 0; diff --git a/Python/optimizer.c b/Python/optimizer.c index c7a6b7e746545c..f09bf778587b12 100644 --- a/Python/optimizer.c +++ b/Python/optimizer.c @@ -549,6 +549,8 @@ dynamic_exit_uop[MAX_UOP_ID + 1] = { }; +#define CONFIDENCE_RANGE 1000 +#define CONFIDENCE_CUTOFF 333 #ifdef Py_DEBUG #define DPRINTF(level, ...) \ @@ -596,46 +598,6 @@ add_to_trace( ((uint32_t)((INSTR) - ((_Py_CODEUNIT *)(CODE)->co_code_adaptive))) -/* Compute branch fitness penalty based on how likely the traced path is. - * The penalty is small when the traced path is common, large when rare. - * A branch that historically goes the other way gets a heavy penalty. */ -static inline int -compute_branch_penalty(uint16_t history, bool branch_taken) -{ - int taken_count = _Py_popcount32((uint32_t)history); - int on_trace_count = branch_taken ? taken_count : 16 - taken_count; - int off_trace = 16 - on_trace_count; - /* Linear scaling: off_trace ranges from 0 (fully biased our way) - * to 16 (fully biased against us), so the penalty ranges from - * FITNESS_BRANCH_BASE to FITNESS_BRANCH_BASE + 32. */ - return FITNESS_BRANCH_BASE + off_trace * 2; -} - -/* Compute exit quality for the current trace position. - * Higher values mean better places to stop the trace. */ -static inline int32_t -compute_exit_quality(_Py_CODEUNIT *target_instr, int opcode, - const _PyJitTracerState *tracer) -{ - if (target_instr == tracer->initial_state.start_instr || - target_instr == tracer->initial_state.close_loop_instr) { - return EXIT_QUALITY_CLOSE_LOOP; - } - if (target_instr->op.code == ENTER_EXECUTOR) { - return EXIT_QUALITY_ENTER_EXECUTOR; - } - if (_PyOpcode_Caches[_PyOpcode_Deopt[opcode]] > 0) { - return EXIT_QUALITY_SPECIALIZABLE; - } - return EXIT_QUALITY_DEFAULT; -} - -static inline int32_t -compute_frame_penalty(const _PyOptimizationConfig *cfg) -{ - return (int32_t)cfg->fitness_initial / 10 + 1; -} - static int is_terminator(const _PyUOpInstruction *uop) { @@ -675,7 +637,6 @@ _PyJit_translate_single_bytecode_to_trace( _Py_CODEUNIT *this_instr = tracer->prev_state.instr; _Py_CODEUNIT *target_instr = this_instr; uint32_t target = 0; - int end_trace_opcode = _DEOPT; target = Py_IsNone((PyObject *)old_code) ? (uint32_t)(target_instr - _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR) @@ -773,14 +734,16 @@ _PyJit_translate_single_bytecode_to_trace( DPRINTF(2, "Unsupported: oparg too large\n"); unsupported: { + // Rewind to previous instruction and replace with _EXIT_TRACE. _PyUOpInstruction *curr = uop_buffer_last(trace); while (curr->opcode != _SET_IP && uop_buffer_length(trace) > 2) { trace->next--; curr = uop_buffer_last(trace); } + assert(curr->opcode == _SET_IP || uop_buffer_length(trace) == 2); if (curr->opcode == _SET_IP) { int32_t old_target = (int32_t)uop_get_target(curr); - curr->opcode = end_trace_opcode; + curr->opcode = _DEOPT; curr->format = UOP_FORMAT_TARGET; curr->target = old_target; } @@ -800,23 +763,6 @@ _PyJit_translate_single_bytecode_to_trace( return 1; } - // Fitness-based trace quality check (before reserving space for this instruction) - _PyJitTracerTranslatorState *ts = &tracer->translator_state; - int32_t eq = compute_exit_quality(target_instr, opcode, tracer); - DPRINTF(3, "Fitness check: %s(%d) fitness=%d, exit_quality=%d, depth=%d\n", - _PyOpcode_OpName[opcode], oparg, ts->fitness, eq, ts->frame_depth); - - // Check if fitness is depleted — should we stop the trace? - if (ts->fitness < eq) { - // This is a tracer heuristic rather than normal program control flow, - // so leave operand1 clear and let the resulting side exit increase chain_depth. - ADD_TO_TRACE(_EXIT_TRACE, 0, 0, target); - OPT_STAT_INC(fitness_terminated_traces); - DPRINTF(2, "Fitness terminated: %s(%d) fitness=%d < exit_quality=%d\n", - _PyOpcode_OpName[opcode], oparg, ts->fitness, eq); - goto done; - } - // One for possible _DEOPT, one because _CHECK_VALIDITY itself might _DEOPT trace->end -= 2; @@ -870,12 +816,6 @@ _PyJit_translate_single_bytecode_to_trace( assert(jump_happened ? (next_instr == computed_jump_instr) : (next_instr == computed_next_instr)); uint32_t uopcode = BRANCH_TO_GUARD[opcode - POP_JUMP_IF_FALSE][jump_happened]; ADD_TO_TRACE(uopcode, 0, 0, INSTR_IP(jump_happened ? computed_next_instr : computed_jump_instr, old_code)); - int bp = compute_branch_penalty(target_instr[1].cache, jump_happened); - tracer->translator_state.fitness -= bp; - DPRINTF(3, " branch penalty: -%d (history=0x%04x, taken=%d) -> fitness=%d\n", - bp, target_instr[1].cache, jump_happened, - tracer->translator_state.fitness); - break; } case JUMP_BACKWARD_JIT: @@ -883,9 +823,6 @@ _PyJit_translate_single_bytecode_to_trace( case JUMP_BACKWARD_NO_JIT: case JUMP_BACKWARD: ADD_TO_TRACE(_CHECK_PERIODIC, 0, 0, target); - tracer->translator_state.fitness -= FITNESS_BACKWARD_EDGE; - DPRINTF(3, " backward edge penalty: -%d -> fitness=%d\n", - FITNESS_BACKWARD_EDGE, tracer->translator_state.fitness); _Py_FALLTHROUGH; case JUMP_BACKWARD_NO_INTERRUPT: { @@ -1008,44 +945,6 @@ _PyJit_translate_single_bytecode_to_trace( assert(next->op.code == STORE_FAST); operand = next->op.arg; } - else if (uop == _PUSH_FRAME) { - _PyJitTracerTranslatorState *ts_depth = &tracer->translator_state; - ts_depth->frame_depth++; - if (ts_depth->frame_depth >= MAX_ABSTRACT_FRAME_DEPTH) { - // The optimizer can't handle frames this deep, - // so there's no point continuing the trace. - DPRINTF(2, "Unsupported: frame depth %d >= MAX_ABSTRACT_FRAME_DEPTH\n", - ts_depth->frame_depth); - end_trace_opcode = _EXIT_TRACE; - goto unsupported; - } - int32_t frame_penalty = compute_frame_penalty(&tstate->interp->opt_config); - int32_t cost = frame_penalty * ts_depth->frame_depth; - ts_depth->fitness -= cost; - DPRINTF(3, " _PUSH_FRAME: depth=%d, penalty=-%d (per_frame=%d) -> fitness=%d\n", - ts_depth->frame_depth, cost, frame_penalty, - ts_depth->fitness); - } - else if (uop == _RETURN_VALUE || uop == _RETURN_GENERATOR || uop == _YIELD_VALUE) { - _PyJitTracerTranslatorState *ts_depth = &tracer->translator_state; - int32_t frame_penalty = compute_frame_penalty(&tstate->interp->opt_config); - if (ts_depth->frame_depth <= 0) { - // Underflow: returning from a frame we didn't enter - ts_depth->fitness -= frame_penalty * 2; - DPRINTF(3, " %s: underflow penalty=-%d -> fitness=%d\n", - _PyOpcode_uop_name[uop], frame_penalty * 2, - ts_depth->fitness); - } - else { - // Reward returning: small inlined calls should be encouraged - ts_depth->fitness += frame_penalty; - DPRINTF(3, " %s: return reward=+%d, depth=%d -> fitness=%d\n", - _PyOpcode_uop_name[uop], frame_penalty, - ts_depth->frame_depth - 1, - ts_depth->fitness); - } - ts_depth->frame_depth = ts_depth->frame_depth <= 0 ? 0 : ts_depth->frame_depth - 1; - } else if (_PyUop_Flags[uop] & HAS_RECORDS_VALUE_FLAG) { PyObject *recorded_value = tracer->prev_state.recorded_value; tracer->prev_state.recorded_value = NULL; @@ -1087,13 +986,7 @@ _PyJit_translate_single_bytecode_to_trace( ADD_TO_TRACE(_JUMP_TO_TOP, 0, 0, 0); goto done; } - // Update fitness AFTER translation, BEFORE returning to continue tracing. - // This ensures the next iteration's fitness check reflects the cost of - // all instructions translated so far. - tracer->translator_state.fitness -= FITNESS_PER_INSTRUCTION; - DPRINTF(3, " per-insn cost: -%d -> fitness=%d\n", - FITNESS_PER_INSTRUCTION, tracer->translator_state.fitness); - DPRINTF(2, "Trace continuing (fitness=%d)\n", tracer->translator_state.fitness); + DPRINTF(2, "Trace continuing\n"); return 1; done: DPRINTF(2, "Trace done\n"); @@ -1176,17 +1069,6 @@ _PyJit_TryInitializeTracing( assert(curr_instr->op.code == JUMP_BACKWARD_JIT || curr_instr->op.code == RESUME_CHECK_JIT || (exit != NULL)); tracer->initial_state.jump_backward_instr = curr_instr; - // Initialize fitness tracking state - const _PyOptimizationConfig *cfg = &tstate->interp->opt_config; - _PyJitTracerTranslatorState *ts = &tracer->translator_state; - bool is_side_trace = (exit != NULL); - ts->fitness = is_side_trace - ? (int32_t)cfg->fitness_initial_side - : (int32_t)cfg->fitness_initial; - ts->frame_depth = 0; - DPRINTF(3, "Fitness init: %s trace, fitness=%d\n", - is_side_trace ? "side" : "root", ts->fitness); - tracer->is_tracing = true; return 1; } diff --git a/Python/optimizer_analysis.c b/Python/optimizer_analysis.c index 4672a272fc9203..92e1c081d524db 100644 --- a/Python/optimizer_analysis.c +++ b/Python/optimizer_analysis.c @@ -30,6 +30,7 @@ #include "pycore_unicodeobject.h" #include "pycore_ceval.h" #include "pycore_floatobject.h" +#include "pycore_setobject.h" #include #include @@ -251,6 +252,7 @@ add_op(JitOptContext *ctx, _PyUOpInstruction *this_instr, #define sym_is_not_null _Py_uop_sym_is_not_null #define sym_is_const _Py_uop_sym_is_const #define sym_is_safe_const _Py_uop_sym_is_safe_const +#define sym_is_not_container _Py_uop_sym_is_not_container #define sym_get_const _Py_uop_sym_get_const #define sym_new_const_steal _Py_uop_sym_new_const_steal #define sym_get_const_as_stackref _Py_uop_sym_get_const_as_stackref diff --git a/Python/optimizer_bytecodes.c b/Python/optimizer_bytecodes.c index dfb97625bf924f..f2645553513f3d 100644 --- a/Python/optimizer_bytecodes.c +++ b/Python/optimizer_bytecodes.c @@ -515,7 +515,8 @@ dummy_func(void) { res = sym_new_not_null(ctx); ds = dict_st; ss = sub_st; - if (sym_matches_type(dict_st, &PyFrozenDict_Type)) { + if (sym_is_not_container(sub_st) && + sym_matches_type(dict_st, &PyFrozenDict_Type)) { REPLACE_OPCODE_IF_EVALUATES_PURE(dict_st, sub_st, res); } } @@ -706,6 +707,10 @@ dummy_func(void) { b = sym_new_type(ctx, &PyBool_Type); l = left; r = right; + if (sym_is_not_container(left) && + sym_matches_type(right, &PyFrozenSet_Type)) { + REPLACE_OPCODE_IF_EVALUATES_PURE(left, right, b); + } } op(_CONTAINS_OP_DICT, (left, right -- b, l, r)) { diff --git a/Python/optimizer_cases.c.h b/Python/optimizer_cases.c.h index 4643a0ed0c5f9d..fb3ec39a42eabc 100644 --- a/Python/optimizer_cases.c.h +++ b/Python/optimizer_cases.c.h @@ -1462,7 +1462,8 @@ res = sym_new_not_null(ctx); ds = dict_st; ss = sub_st; - if (sym_matches_type(dict_st, &PyFrozenDict_Type)) { + if (sym_is_not_container(sub_st) && + sym_matches_type(dict_st, &PyFrozenDict_Type)) { if ( sym_is_safe_const(ctx, dict_st) && sym_is_safe_const(ctx, sub_st) @@ -2993,6 +2994,51 @@ b = sym_new_type(ctx, &PyBool_Type); l = left; r = right; + if (sym_is_not_container(left) && + sym_matches_type(right, &PyFrozenSet_Type)) { + if ( + sym_is_safe_const(ctx, left) && + sym_is_safe_const(ctx, right) + ) { + JitOptRef left_sym = left; + JitOptRef right_sym = right; + _PyStackRef left = sym_get_const_as_stackref(ctx, left_sym); + _PyStackRef right = sym_get_const_as_stackref(ctx, right_sym); + _PyStackRef b_stackref; + _PyStackRef l_stackref; + _PyStackRef r_stackref; + /* Start of uop copied from bytecodes for constant evaluation */ + PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); + PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyAnySet_CheckExact(right_o)); + STAT_INC(CONTAINS_OP, hit); + int res = _PySet_Contains((PySetObject *)right_o, left_o); + if (res < 0) { + JUMP_TO_LABEL(error); + } + b_stackref = (res ^ oparg) ? PyStackRef_True : PyStackRef_False; + l_stackref = left; + r_stackref = right; + /* End of uop copied from bytecodes for constant evaluation */ + (void)l_stackref; + (void)r_stackref; + b = sym_new_const_steal(ctx, PyStackRef_AsPyObjectSteal(b_stackref)); + if (sym_is_const(ctx, b)) { + PyObject *result = sym_get_const(ctx, b); + if (_Py_IsImmortal(result)) { + // Replace with _INSERT_2_LOAD_CONST_INLINE_BORROW since we have two inputs and an immortal result + ADD_OP(_INSERT_2_LOAD_CONST_INLINE_BORROW, 0, (uintptr_t)result); + } + } + CHECK_STACK_BOUNDS(1); + stack_pointer[-2] = b; + stack_pointer[-1] = l; + stack_pointer[0] = r; + stack_pointer += 1; + ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); + break; + } + } CHECK_STACK_BOUNDS(1); stack_pointer[-2] = b; stack_pointer[-1] = l; diff --git a/Python/optimizer_symbols.c b/Python/optimizer_symbols.c index a0ee175fd10c1a..6230b8948697e2 100644 --- a/Python/optimizer_symbols.c +++ b/Python/optimizer_symbols.c @@ -283,7 +283,22 @@ _Py_uop_sym_is_safe_const(JitOptContext *ctx, JitOptRef sym) (typ == &PyFloat_Type) || (typ == &_PyNone_Type) || (typ == &PyBool_Type) || - (typ == &PyFrozenDict_Type); + (typ == &PyFrozenDict_Type) || + (typ == &PyFrozenSet_Type); +} + +bool +_Py_uop_sym_is_not_container(JitOptRef sym) +{ + PyTypeObject *typ = _Py_uop_sym_get_type(sym); + if (typ == NULL) { + return false; + } + return (typ == &PyLong_Type) || + (typ == &PyFloat_Type) || + (typ == &PyUnicode_Type) || + (typ == &_PyNone_Type) || + (typ == &PyBool_Type); } void diff --git a/Python/pystate.c b/Python/pystate.c index 78eab7cc7d2459..143175da0f45c7 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -635,22 +635,6 @@ init_interpreter(PyInterpreterState *interp, "PYTHON_JIT_SIDE_EXIT_INITIAL_BACKOFF", SIDE_EXIT_INITIAL_BACKOFF, 0, MAX_BACKOFF); - // Trace fitness configuration - init_policy(&interp->opt_config.fitness_initial, - "PYTHON_JIT_FITNESS_INITIAL", - FITNESS_INITIAL, 100, 10000); - init_policy(&interp->opt_config.fitness_initial_side, - "PYTHON_JIT_FITNESS_INITIAL_SIDE", - FITNESS_INITIAL_SIDE, 50, 5000); - /* The tracer starts at start_instr, so initial fitness must not be below - * the close-loop exit quality or tracing will terminate immediately. */ - if (interp->opt_config.fitness_initial < EXIT_QUALITY_CLOSE_LOOP) { - interp->opt_config.fitness_initial = EXIT_QUALITY_CLOSE_LOOP; - } - if (interp->opt_config.fitness_initial_side < EXIT_QUALITY_CLOSE_LOOP) { - interp->opt_config.fitness_initial_side = EXIT_QUALITY_CLOSE_LOOP; - } - interp->opt_config.specialization_enabled = !is_env_enabled("PYTHON_SPECIALIZATION_OFF"); interp->opt_config.uops_optimize_enabled = !is_env_disabled("PYTHON_UOPS_OPTIMIZE"); if (interp != &runtime->_main_interpreter) { diff --git a/Python/pystats.c b/Python/pystats.c index 2fac2db1b738c7..a057ad884566d8 100644 --- a/Python/pystats.c +++ b/Python/pystats.c @@ -274,7 +274,6 @@ print_optimization_stats(FILE *out, OptimizationStats *stats) fprintf(out, "Optimization low confidence: %" PRIu64 "\n", stats->low_confidence); fprintf(out, "Optimization unknown callee: %" PRIu64 "\n", stats->unknown_callee); fprintf(out, "Executors invalidated: %" PRIu64 "\n", stats->executors_invalidated); - fprintf(out, "Optimization fitness terminated: %" PRIu64 "\n", stats->fitness_terminated_traces); print_histogram(out, "Trace length", stats->trace_length_hist); print_histogram(out, "Trace run length", stats->trace_run_length_hist); diff --git a/Tools/build/compute-changes.py b/Tools/build/compute-changes.py index c15dc599f993f3..4870388da0d8a5 100644 --- a/Tools/build/compute-changes.py +++ b/Tools/build/compute-changes.py @@ -99,6 +99,9 @@ Path("Modules/pyexpat.c"), # zipfile Path("Lib/zipfile/"), + # zoneinfo + Path("Lib/zoneinfo/"), + Path("Modules/_zoneinfo.c"), })