Dynamic references rework

plans/2026-01-types-and-forms.md

@@ -83,6 +83,8 @@ Honestly, it's really hard. Would need to think about what operations can be per
 
 Might come back to this at some point.
 
+Relevant todos are marked as `TODO[non-leaf-form]`
+
 ## What's next?
 
 Let's put the "concepts" to-leaf on indefinite pause. It's definitely resulted in a lot of good cleaning up, BUT it's also created a lot of mess.

plans/TODO.md

@@ -241,34 +241,87 @@ Moved to [2026-01-types-and-forms.md](./2026-01-types-and-forms.md).
     - [x] Separation of `ExecutionInterrupt` and `FunctionError` - incorporated below
     - [x] `Iterator` returns `Result` change -> replaced with below
   * To implement `.map()`, we have a few things we need first:
-    -[x] An iterator trait where Interpreter is passed at next time.
-    -[ ] Possibly - not require `Clone` on iterators:
-      - Make `TryClone -> Result<T, &T>`
-      - Make `to_string` for iterator return `Iterator[?]`
+    - [x] An iterator trait where Interpreter is passed at next time.
+    - [x] Make `to_string` for iterator return `Iterator[?]`
   - [x] Create new iterator trait `PreinterpretIterator` and `IntoPreinterpretIterator` with `.next(&mut Interpreter)`
     - [x] Blanket implement it for `Iterator<AnyValue> + Clone`
     - [x] Then replace e.g. for loop impl with it.
     - [x] Create `Map` and `Filter` types on top of it, to be able to implement `map` and `filter`
   - [ ] See if new iterators on iterator value can be fixed to be lazy
   - [ ] Salvage half-baked `FunctionValue` changes to allow invocation
-  - [ ] Consider if IteratorValue should have `Item = ReturnedValue`
   - [ ] Add `iterable.map`, `iterable.filter`, `iterable.flatten`, `iterable.flatmap`
   - [ ] Add tests for iterable methods
-- [ ] Add `array.sort`, `array.sort_by`
-- [ ] Look into if a closure like `|| { }()` can evade `attempt` block statue mutation checks. Maybe lean into it as a way to avoid htem, and mention it in the error message
+- [ ] Look into if a closure like `|| { }()` can evade `attempt` block statue mutation checks. Maybe lean into it as a way to avoid them, and mention it in the error message
 - [ ] Resolve all `TODO[functions]`
 
 Possible punted:
-- [ ] Allow variables to be marked owned / shared / mutable
-  - [ ] ... and maybe no marking accepts any?
-        ... so e.g. push_one_and_return_mut can have `let y = arr; y` without erroring
-  - [ ] Else - improve the errors so that the conversion error has some hint from the target explaining that the target can be changed to allow
-  shared/mutable instead.
+- [ ] Add `array.sort`, `array.sort_by`
 - [ ] Allow destructuring shared and mutable variables and arguments
 - [ ] Support optional arguments in closures
 - [ ] Support for `move()` expressions in closures.
   - `move(x)` / `move(a.b.as_ref())` / `move(a.b.as_mut())` => we hoist up the content into the previous frame (effectively temporarily change `current_frame_id` to be the parent in the `FlowAnalysisState` - pretty easy).
   - These can be an anonymous definition in the root frame of the closure, which is referenced inline.
+- [ ] Possibly - not require `Clone` on iterators:
+  - Make `TryClone -> Result<T, &T>`
+
+## Fix broken "Disabled" abstraction
+
+### Background
+
+Suddenly dawned on me - my Disabled arguments might not be safe.
+* Imagine if I get `my_arr = [[]]` a Shared `my_arr[0]` then do `my_arr.pop()`
+* Then we enable `my_arr[0]` and get a use after free(!!). e.g. `my_arr[0].push(my_arr.pop())`.
+
+### Task list
+
+- [x] Write up model in `dynamic_references/mod.rs`
+- [x] Add structure for `dynamic_references`
+- [x] Create Referenceable, and Reference types
+- [x] Create error messages for rule breaks in `referenceable.rs`
+- [x] Add the following to `ReferenceCore` and maybe others:
+  - [x] Emplacing
+  - [x] Map, Try map
+  - [x] Ability to map deeper. Should take a `PathExtension` and a new span.
+- [x] Try replacing existing RefCell based abstractions with new custom dynamic references, and report on what breaks:
+  - `pub(crate) type Referenceable<L> = Rc<RefCell<L>>;` becomes `pub(crate) type Referenceable<L> = dynamic_references::Referenceable`
+  - `Shared` / `SharedSubRcRefCell` - both become type aliases for `SharedReference`
+  - `DisabledShared` becomes `InactiveSharedReference`
+  - `Mutable` / `MutableSubRcRefCell` - becomes type alias for `MutableReference`
+  - `DisabledMutable` becomes `InactiveMutableReference`
+- [x] Replace all the aliases:
+  - [x] Remove `Shared`, `Mutable`, `SharedValue`, `AssigneeValue`
+  - [x] Move `type Shared<T>` and `type Mutable<T>` alongside `SharedReference` / `MutableReference`
+  - [x] Remove references to `MutableReference` and `SharedReference` outside these aliases
+  - [x] Rename `SharedReference -> Shared` and `MutableReference -> Mutable`
+  - [x] Rename `DisabledShared` -> `InactiveShared` and same for `Mutable`
+  - [x] Remove `type AssigneeValue` and `type SharedValue`
+  - [x] Move `Assignee` out of bindings. To e.g. `dynamic_references`
+  - [x] Rename `PathExtension::Tightening` to `PathExtension::TypeNarrowing`
+  - [x] Rename  `DisabledArgumentValue` and `DisabledCopyOnWrite` to `Inactive__` and their method from `enable` to `activate` and ditto with `disable -> deactivate`
+- [x] See what else can be deleted from `bindings.rs`
+  - [x] Unify LateBound into `LateBound`
+  - [x] Unify CopyOnWrite into `QqqCopyOnWrite`
+- [x] Add various tests:
+  - [x] Stretching different error messages
+  - [x] Showing I can do e.g. `x.a += x.b`
+  - [x] Show that `let my_arr = [[]]; my_arr[0].push(my_arr.pop())` gives a suitable error
+
+### Other ideas
+
+- [ ] Make it so that we disable the parent whilst resolving a property/index 
+
+We could imagine a world where we are more clever over our mutation:
+* Note that I can reference `x.a` and `x.b` separately, or `x[0]` and `x[1]` but in that case, can't mutate `x` itself to create new fields.
+  * Or we use a `ExpandableVec<N>` which allocates a `Vec<Box<[_; N]>>` and doesn't move the inner chunks; allowing us to add new fields without breaking pointers to existing ones.
+  Maybe some SmallVec or SegVec like library has this feature? Stores an initial chunk `[_; N]` and then a `Vec<Box<[_; N]>>`
+
+- [ ] Consider if IteratorValue, Object and Array should have `Item = DisabledReturnedValue`
+  - [ ] And add `iter()` and `iter_mut()` methods
+- [ ] Allow variables to be marked owned / shared / mutable
+  - [ ] ... and maybe no marking accepts any?
+        ... so e.g. push_one_and_return_mut can have `let y = arr; y` without erroring
+  - [ ] Else - improve the errors so that the conversion error has some hint from the target explaining that the target can be changed to allow
+  shared/mutable instead.
 
 ## Parser - Methods using closures
 
@@ -301,7 +354,7 @@ input.repeated(
 )
 ```
 - [ ] `input.any_group(|inner| { })`
-- [ ] `input.group('()', |inner| { })`
+- [ ] `input.group("()", |inner| { })`
 - [ ] `input.transparent_group(|inner| { })`
 
 ## Parser - Better Types for Tokens

src/expressions/closures.rs

@@ -33,7 +33,6 @@ impl ClosureExpression {
 pub(crate) struct ClosureValue {
     definition: Rc<ClosureDefinition>,
     closed_references: Vec<(VariableDefinitionId, VariableContent)>,
-    // TODO[functions]: Add closed_values from moves here
 }
 
 impl PartialEq for ClosureValue {
@@ -89,7 +88,7 @@ impl ClosureValue {
                 (Pattern::Variable(variable), ArgumentValue::Shared(shared)) => {
                     context.interpreter.define_variable(
                         variable.definition.id,
-                        VariableContent::Shared(shared.disable()),
+                        VariableContent::Shared(shared.deactivate()),
                     );
                 }
                 (_, ArgumentValue::Shared(_)) => {
@@ -100,7 +99,7 @@ impl ClosureValue {
                 (Pattern::Variable(variable), ArgumentValue::Mutable(mutable)) => {
                     context.interpreter.define_variable(
                         variable.definition.id,
-                        VariableContent::Mutable(mutable.disable()),
+                        VariableContent::Mutable(mutable.deactivate()),
                     );
                 }
                 (_, ArgumentValue::Mutable(_)) => {

src/expressions/concepts/content.rs

@@ -2,8 +2,7 @@ use super::*;
 
 /// Shorthand for representing the form F of a type T with a particular lifetime 'a.
 pub(crate) type Content<'a, T, F> = <T as IsHierarchicalType>::Content<'a, F>;
-pub(crate) type DynContent<'a, D, F> =
-    <F as IsDynCompatibleForm>::DynLeaf<'a, <D as IsDynType>::DynContent>;
+pub(crate) type DynContent<'a, D, F> = <F as IsDynCompatibleForm>::DynLeaf<'a, D>;
 
 /// For types which have an associated value (type and form)
 pub(crate) trait IsValueContent {
@@ -69,18 +68,6 @@ where
     {
         self.upcast()
     }
-
-    fn into_referenceable(self) -> Content<'a, Self::Type, BeReferenceable>
-    where
-        Self: IsValueContent<Form = BeOwned>,
-    {
-        map_via_leaf! {
-            input: (Content<'a, Self::Type, Self::Form>) = self.into_content(),
-            fn map_leaf<F = BeOwned, T>(leaf) -> (Content<'a, T, BeReferenceable>) {
-                Rc::new(RefCell::new(leaf))
-            }
-        }
-    }
 }
 
 impl<'a, C> Spanned<C>
@@ -193,7 +180,7 @@ where
     fn clone_to_owned_transparently<'r>(
         &'r self,
         span_range: SpanRange,
-    ) -> ExecutionResult<Content<'static, Self::Type, BeOwned>>
+    ) -> FunctionResult<Content<'static, Self::Type, BeOwned>>
     where
         'a: 'r,
         Self::Form: LeafAsRefForm,
@@ -202,7 +189,7 @@ where
         map_via_leaf! {
             input: &'r (Content<'a, Self::Type, Self::Form>) = self,
             state: SpanRange | let span_range = span_range,
-            fn map_leaf<F: LeafAsRefForm, T>(leaf) -> (ExecutionResult<Content<'static, T, BeOwned>>) {
+            fn map_leaf<F: LeafAsRefForm, T>(leaf) -> (FunctionResult<Content<'static, T, BeOwned>>) {
                 F::leaf_clone_to_owned_transparently(leaf, span_range)
             }
         }
@@ -226,7 +213,7 @@ impl<
     type ValueType = T;
     const OWNERSHIP: ArgumentOwnership = F::ARGUMENT_OWNERSHIP;
     fn from_argument(Spanned(value, span_range): Spanned<ArgumentValue>) -> FunctionResult<Self> {
-        let ownership_mapped = F::from_argument_value(value)?;
+        let ownership_mapped = F::from_argument_value(Spanned(value, span_range))?;
         let type_mapped = T::resolve(ownership_mapped, span_range, "This argument")?;
         Ok(X::from_content(type_mapped))
     }

src/expressions/concepts/form.rs

@@ -18,6 +18,13 @@ pub(crate) trait IsHierarchicalForm: IsForm {
     type Leaf<'a, T: IsLeafType>: IsValueContent<Type = T, Form = Self>
         + IntoValueContent<'a>
         + FromValueContent<'a>;
+
+    /// Proof of covariance: shrinks the lifetime of a leaf from `'a` to `'b`.
+    /// Implementors should use the identity function `{ leaf }` as the body.
+    /// This will only compile if the leaf type is genuinely covariant in `'a`.
+    fn covariant_leaf<'a, 'b, T: IsLeafType>(leaf: Self::Leaf<'a, T>) -> Self::Leaf<'b, T>
+    where
+        'a: 'b;
 }
 
 pub(crate) trait LeafAsRefForm: IsHierarchicalForm {
@@ -32,7 +39,7 @@ pub(crate) trait LeafAsRefForm: IsHierarchicalForm {
     fn leaf_clone_to_owned_transparently<'r, 'a: 'r, T: IsLeafType>(
         leaf: &'r Self::Leaf<'a, T>,
         error_span: SpanRange,
-    ) -> ExecutionResult<T::Leaf> {
+    ) -> FunctionResult<T::Leaf> {
         let type_kind = T::type_kind();
         if type_kind.supports_transparent_cloning() {
             Ok(Self::leaf_clone_to_owned_infallible(leaf))
@@ -53,20 +60,21 @@ pub(crate) trait IsDynCompatibleForm: IsHierarchicalForm {
     /// The container for a dyn Trait based type.
     /// The DynLeaf can be similar to the standard leaf, but must be
     /// able to support an unsized D.
-    type DynLeaf<'a, D: 'static + ?Sized>;
+    type DynLeaf<'a, D: IsDynType>;
 
-    fn leaf_to_dyn<'a, T: IsLeafType, D: ?Sized + 'static>(
-        leaf: Self::Leaf<'a, T>,
+    fn leaf_to_dyn<'a, T: IsLeafType, D: IsDynType>(
+        leaf: Spanned<Self::Leaf<'a, T>>,
     ) -> Result<Self::DynLeaf<'a, D>, Content<'a, T, Self>>
     where
-        T::Leaf: CastDyn<D>;
+        T::Leaf: CastDyn<D::DynContent>;
 }
 
 pub(crate) trait MapFromArgument: IsHierarchicalForm {
     const ARGUMENT_OWNERSHIP: ArgumentOwnership;
 
-    fn from_argument_value(value: ArgumentValue)
-        -> FunctionResult<Content<'static, AnyType, Self>>;
+    fn from_argument_value(
+        value: Spanned<ArgumentValue>,
+    ) -> FunctionResult<Content<'static, AnyType, Self>>;
 }
 
 pub(crate) trait MapIntoReturned: IsHierarchicalForm {

src/expressions/concepts/forms/any_mut.rs

@@ -22,20 +22,34 @@ pub(crate) struct BeAnyMut;
 impl IsForm for BeAnyMut {}
 impl IsHierarchicalForm for BeAnyMut {
     type Leaf<'a, T: IsLeafType> = AnyMut<'a, T::Leaf>;
+
+    #[inline]
+    fn covariant_leaf<'a, 'b, T: IsLeafType>(leaf: Self::Leaf<'a, T>) -> Self::Leaf<'b, T>
+    where
+        'a: 'b,
+    {
+        leaf
+    }
 }
 
 impl IsDynCompatibleForm for BeAnyMut {
-    type DynLeaf<'a, D: 'static + ?Sized> = AnyMut<'a, D>;
+    type DynLeaf<'a, D: IsDynType> = AnyMut<'a, D::DynContent>;
 
-    fn leaf_to_dyn<'a, T: IsLeafType, D: ?Sized + 'static>(
-        leaf: Self::Leaf<'a, T>,
+    fn leaf_to_dyn<'a, T: IsLeafType, D: IsDynType>(
+        Spanned(leaf, span): Spanned<Self::Leaf<'a, T>>,
     ) -> Result<Self::DynLeaf<'a, D>, Content<'a, T, Self>>
     where
-        T::Leaf: CastDyn<D>,
+        T::Leaf: CastDyn<D::DynContent>,
     {
-        leaf.replace(|content, emplacer| match <T::Leaf>::map_mut(content) {
-            Ok(mapped) => Ok(emplacer.emplace(mapped)),
-            Err(this) => Err(emplacer.emplace(this)),
+        leaf.emplace_map(|content, emplacer| match <T::Leaf>::map_mut(content) {
+            Ok(mapped) => {
+                // SAFETY: PathExtension is correct for mapping to a dyn type
+                let mapped_mut = unsafe {
+                    MappedMut::new(mapped, PathExtension::TypeNarrowing(D::type_kind()), span)
+                };
+                Ok(emplacer.emplace(mapped_mut))
+            }
+            Err(_this) => Err(emplacer.revert()),
         })
     }
 }
@@ -56,11 +70,12 @@ impl MapFromArgument for BeAnyMut {
     const ARGUMENT_OWNERSHIP: ArgumentOwnership = ArgumentOwnership::Mutable;
 
     fn from_argument_value(
-        value: ArgumentValue,
+        Spanned(value, span): Spanned<ArgumentValue>,
     ) -> FunctionResult<Content<'static, AnyType, Self>> {
-        Ok(value
-            .expect_mutable()
-            .0
-            .replace(|inner, emplacer| inner.as_mut_value().into_mutable_any_mut(emplacer)))
+        Ok(value.expect_mutable().emplace_map(|inner, emplacer| {
+            inner
+                .as_mut_value()
+                .into_mutable_any_mut(emplacer, Some(span))
+        }))
     }
 }

src/expressions/concepts/forms/any_ref.rs

@@ -23,20 +23,34 @@ impl IsForm for BeAnyRef {}
 
 impl IsHierarchicalForm for BeAnyRef {
     type Leaf<'a, T: IsLeafType> = crate::internal_prelude::AnyRef<'a, T::Leaf>;
+
+    #[inline]
+    fn covariant_leaf<'a, 'b, T: IsLeafType>(leaf: Self::Leaf<'a, T>) -> Self::Leaf<'b, T>
+    where
+        'a: 'b,
+    {
+        leaf
+    }
 }
 
 impl IsDynCompatibleForm for BeAnyRef {
-    type DynLeaf<'a, D: 'static + ?Sized> = crate::internal_prelude::AnyRef<'a, D>;
+    type DynLeaf<'a, D: IsDynType> = crate::internal_prelude::AnyRef<'a, D::DynContent>;
 
-    fn leaf_to_dyn<'a, T: IsLeafType, D: ?Sized + 'static>(
-        leaf: Self::Leaf<'a, T>,
+    fn leaf_to_dyn<'a, T: IsLeafType, D: IsDynType>(
+        Spanned(leaf, span): Spanned<Self::Leaf<'a, T>>,
     ) -> Result<Self::DynLeaf<'a, D>, Content<'a, T, Self>>
     where
-        T::Leaf: CastDyn<D>,
+        T::Leaf: CastDyn<D::DynContent>,
     {
-        leaf.replace(|content, emplacer| match <T::Leaf>::map_ref(content) {
-            Ok(mapped) => Ok(emplacer.emplace(mapped)),
-            Err(this) => Err(emplacer.emplace(this)),
+        leaf.emplace_map(|content, emplacer| match <T::Leaf>::map_ref(content) {
+            Ok(mapped) => {
+                // SAFETY: PathExtension is correct for mapping to a dyn type
+                let mapped_ref = unsafe {
+                    MappedRef::new(mapped, PathExtension::TypeNarrowing(D::type_kind()), span)
+                };
+                Ok(emplacer.emplace(mapped_ref))
+            }
+            Err(_this) => Err(emplacer.revert()),
         })
     }
 }
@@ -51,11 +65,12 @@ impl MapFromArgument for BeAnyRef {
     const ARGUMENT_OWNERSHIP: ArgumentOwnership = ArgumentOwnership::Shared;
 
     fn from_argument_value(
-        value: ArgumentValue,
+        Spanned(value, span): Spanned<ArgumentValue>,
     ) -> FunctionResult<Content<'static, AnyType, Self>> {
-        Ok(value
-            .expect_shared()
-            .0
-            .replace(|inner, emplacer| inner.as_ref_value().into_shared_any_ref(emplacer)))
+        Ok(value.expect_shared().emplace_map(|inner, emplacer| {
+            inner
+                .as_ref_value()
+                .into_shared_any_ref(emplacer, Some(span))
+        }))
     }
 }