[Tooling, CI, & Tests] Upgrade to python 3.13 & fix related bugs (#60)

* First pass upgrading representer and CI to python 3.13 with tooling changes.

* Renamed run tests in docker job

* Renamed run tests in docker job

* Due to AST changes in Python 3.13, re-recorded stests for repo.  This adds new Python 3.13 representation files for all tests.

* Adding back run.sh - accidentially deleted.

* Added legacy test folders to gitignore.

* Added str type guard for md5 hash function to prevent it being passed the Ellipsis object.  Updated parse params from python 11.5 to 13.5.

* Added clean_up_normalization function and call to add pass to function bodies that bbecome empty when the docstring is removed.

* - Added fix_missing_bodies method to parse the AST and add [Pass()] anywhere there was an empty list for the function or class body.
  This corrects an error where the normalization that removes docstrings then produces invalid code where the docstring was
  the only valid expression for the class or function.
- Added documentation notes for upcomming AST object depriciations in Python 3.14.  Alsot commented specifically in the import list.
- Modified the AssAssign visitor to insert an assigned value of None where a node had an AnnAssign without a value.
  This covers the bug where a DataClass or other Class or Function has its tupehint removed, but had no default value when written.
  Dataclasses and other parsing steps at runtime assign these values, but the written code does not.  Our normalization process
  removes the typehint, so produces invalid code.  Assigning the value to None during parsing avoids this probblem.
  Since normalization is for grouping purposes only and does not produce executable code, this was considered a valid change.

* Modified and added test casees.  Regenerated golden files to accomodated code changes and new cases.

* Annotated test file and reran goldent results for Python 3.13.

* Added new test case for class normalization and created golden files for tests.

* Added new scenarios for datatclasses and recreated golden test file.

Author: BethanyG

.github/workflows/main.yml

@@ -16,7 +16,7 @@ on:
       - "**.md"
 jobs:
   test:
-    name: Test Representer
+    name: Test Python Representer
     runs-on: ubuntu-24.04
     steps:
       - name: Checkout code
@@ -37,5 +37,5 @@ jobs:
           cache-from: type=gha
           cache-to: type=gha,mode=max
 
-      - name: Run Tests in Docker
+      - name: Run Tests for Python Representer in Docker
         run: bin/run-tests-in-docker.sh

.gitignore

@@ -47,6 +47,10 @@ coverage.xml
 .hypothesis/
 .pytest_cache/
 
+# Legacy test directories
+test/*/OLD_output
+test/*/rerun
+
 # Translations
 *.mo
 *.pot
@@ -102,3 +106,8 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+
+# mac directory files
+.DS_Store
+test/.DS_Store
+*/.DS_Store

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.11.5-alpine3.18
+FROM python:3.13.5-alpine3.22
 
 COPY requirements.txt /requirements.txt
 COPY dev-requirements.txt /dev-requirements.txt

bin/run-tests-in-docker.sh

@@ -27,4 +27,4 @@ docker run \
     --mount type=tmpfs,dst=/tmp \
     --workdir /opt/representer \
     --entrypoint pytest \
-    exercism/python-representer -v  --disable-warnings
+    exercism/python-representer -vv  --disable-warnings

dev-requirements.txt

@@ -1,2 +1,3 @@
-pytest~=7.2.2
-pytest-subtests~=0.10.0
+black<=25.1.0
+pytest~=8.4.0
+pytest-subtests~=0.14.2

representer/__init__.py

@@ -26,6 +26,14 @@ def normalize(self) -> None:
         """
         self._tree = self._normalizer.visit(self._tree)
 
+    def clean_up_normalization(self) -> None:
+        """
+        Clean up the tree normalization, replacing empty ClassDef
+        and Function Def bodies with pass, so that black is happy
+         formatting them.
+        """
+        self._tree = self._normalizer.fix_empty_bodies(self._tree)
+
     def dump_tree(self) -> str:
         """
         Dump the current state of the tree for printing.
@@ -106,6 +114,9 @@ def represent(slug: utils.Slug, input: utils.Directory, output: utils.Directory)
     # normalize the tree
     representation.normalize()
 
+    # clean up missing function and class bodies with pass
+    representation.clean_up_normalization()
+
     # save dump of normalized code for debug (from un-parsing the normalized AST).
     out[0:0] = ['## BEGIN NORMALIZED CODE ##', representation.dump_code(), "## END NORMALIZED CODE ##", '']
 

representer/normalizer.py

@@ -1,5 +1,18 @@
 """
 Representer for Python.
+
+The following features have been deprecated in documentation since
+Python 3.8, now cause a DeprecationWarning to be emitted at runtime
+when they are accessed or used, and
+***will be removed in Python 3.14***:
+
+ast.Num
+ast.Str
+ast.Bytes
+ast.NameConstant
+ast.Ellipsis
+
+Use ast.Constant instead.
 """
 import builtins
 from itertools import count
@@ -20,6 +33,7 @@
     Eq,
     ExceptHandler,
     Expr,
+    Ellipsis, # <-- deprecated in 3.14
     FunctionDef,
     GeneratorExp,
     Global,
@@ -31,15 +45,18 @@
     Name,
     NodeTransformer,
     Nonlocal,
+    Pass,
     SetComp,
-    Str,
+    Str, # <-- deprecated in 3.14
     Store,
     UnaryOp,
     Yield,
     YieldFrom,
     alias,
     arg,
+    fix_missing_locations,
     get_docstring,
+    iter_child_nodes,
     keyword,
 )
 
@@ -82,11 +99,25 @@ def get_placeholders(self) -> Dict[str, str]:
         """
         return {value: key for key, value in self._placeholder_cache.items()}
 
+
+    def fix_empty_bodies(self, node):
+        def _fix(node):
+            if isinstance(getattr(node, 'body', None), list) and not node.body:
+                node.body = [Pass()]
+            for child in iter_child_nodes(node):
+                _fix(child)
+
+        _fix(node)
+        return node
+
+
     def register_docstring(self, node: AST) -> None:
         """
         Register the docstring for this node.
         """
+
         docstring = get_docstring(node, clean=False)
+
         if docstring:
             self._docstring_cache.add(utils.md5sum(docstring))
 
@@ -116,31 +147,49 @@ def _visit_definition(self, node):
         self.generic_visit(node)
         return node
 
+
+    def visit_AnnAssign(self, node: AnnAssign) -> Assign:
+        """
+        Any type-annotated assignment
+
+        Converts type-annotated assignments to regular assignments.
+
+        In a class decorated by a dataclass decorator or where there
+         is an unassigned but annotated class variable,
+        the "missing" value is assigned None for the purposes of
+         this representation.
+
+        This is to avoid ast parsing errors caused by annotation removal
+        pre dataclass decorator (the decorator fills values in at runtime).
+        Otherwise, the ast lib tosses an error because there are no node._fields
+        for it to iterate through.
+        """
+
+        new_assign = Assign(targets=[node.target],
+                            value=node.value if node.value else Constant(value=None),
+                            lineno=node.lineno)
+
+        self.generic_visit(new_assign)
+        return new_assign
+
+
     def visit_ClassDef(self, node: ClassDef) -> ClassDef:
         """
         Any `class name` definition.
         """
+
         return self._visit_definition(node)
 
+
     def visit_FunctionDef(self, node: FunctionDef) -> FunctionDef:
         """
         Any `def name` definition.
         """
+
         if node.returns:
             node.returns = None
-        return self._visit_definition(node)
-
-    def visit_AnnAssign(self, node: AnnAssign) -> Assign:
-        """
-        Any type-annotated assignment
 
-        Converts type-annotated assignments to regular assignments.
-        """
-        new_assign = Assign(targets=[node.target],
-                            value=node.value,
-                            lineno=node.lineno)
-        self.generic_visit(new_assign)
-        return new_assign
+        return self._visit_definition(node)
 
     def visit_AsyncFunctionDef(self, node: AsyncFunctionDef) -> AsyncFunctionDef:
         """
@@ -154,6 +203,7 @@ def visit_arg(self, node: arg) -> arg:
         Drops type annotations.
         """
         node.arg = self.add_placeholder(node.arg)
+
         if node.annotation:
             node.annotation = None
         self.generic_visit(node)
@@ -303,6 +353,7 @@ def visit_If(self, node: If) -> None:
         self.generic_visit(node)
         return node
 
+
     def visit_Expr(self, node: Expr) -> Optional[Expr]:
         """Expressions not assigned to an identifier.
 
@@ -316,10 +367,13 @@ def visit_Expr(self, node: Expr) -> Optional[Expr]:
             if node.value.func.id == 'print':
                 return None
 
+        # Eliminate registered docstrings
+        # Added guard to utils.py to pass on anything
+        # that's not a string so that ellipsis doesn't
+        # cause the md5sum function to barf.
         if isinstance(node.value, Constant) and not isinstance(node.value, Call):
-            # eliminate registered docstrings
-            if utils.md5sum(node.value.value) in self._docstring_cache:
-                return None
+           if utils.md5sum(node.value.value) in self._docstring_cache:
+               return None
 
         self.generic_visit(node)
         return node

representer/utils.py

@@ -53,8 +53,21 @@ def directory(string: str) -> Directory:
 def md5sum(data: str) -> str:
     """
     Return the md5 sum of the given string.
+
+    Somehow, the normalizer code is passing Ellipsis objects
+    to this function. This causes issues with encoding:
+    AttributeError: 'ellipsis' object has no attribute 'encode',
+    requiring the guard below until we can stop the normalizer from
+    passing the object.
+
+    See L360 in normalizer.py for the TODO.
     """
-    return md5(data.encode("utf-8")).hexdigest()
+
+    if not isinstance(data, str):
+        return
+
+    else:
+        return md5(data.encode("utf-8")).hexdigest()
 
 
 def single_space(text: str) -> str:
@@ -83,7 +96,7 @@ def parse(source: str) -> ast.AST:
     Wrapper around ast.parse.
     Preserves type annotations.
     """
-    return ast.parse(source, type_comments=False, feature_version=(3,11))
+    return ast.parse(source, type_comments=True, feature_version=(3,13))
 
 
 def dump_tree(tree: ast.AST) -> str:

requirements.txt

@@ -1 +1 @@
-black<=22.3.0
+black<=25.1.0