Introduce OperandOwnership to classify OSSA uses.

Migrating to this classification was made easy by the recent rewrite
of the OSSA constraint model. It's also consistent with
instruction-level abstractions for working with different kinds of
OperandOwnership that are being designed.

This classification vastly simplifies OSSA passes that rewrite OSSA
live ranges, making it straightforward to reason about completeness
and correctness. It will allow a simple utility to canonicalize OSSA
live ranges on-the-fly.

This avoids the need for OSSA-based utilities and passes to hard-code
SIL opcodes. This will allow several of those unmaintainable pieces of
code to be replaced with a trivial OperandOwnership check.

It's extremely important for SIL maintainers to see a list of all SIL
opcodes associated with a simple OSSA classification and set of
well-specified rules for each opcode class, without needing to guess
or reverse-engineer the meaning from the implementation. This
classification does that while eliminating a pile of unreadable
macros.

This classification system is the model that CopyPropagation was
initially designed to use. Now, rather than relying on a separate
pass, a simple, lightweight utility will canonicalize OSSA
live ranges.

The major problem with writing optimizations based on OperandOwnership
is that some operations don't follow structural OSSA requirements,
such as project_box and unchecked_ownership_conversion. Those are
classified as PointerEscape which prevents the compiler from reasoning
about, or rewriting the OSSA live range.

Functional Changes:

As a side effect, this corrects many operand constraints that should
in fact require trivial operand values.

Author: atrick

include/swift/SIL/ApplySite.h

@@ -337,7 +337,7 @@ class ApplySite {
   }
 
   /// Return the SILArgumentConvention for the given applied argument operand.
-  SILArgumentConvention getArgumentConvention(Operand &oper) const {
+  SILArgumentConvention getArgumentConvention(const Operand &oper) const {
     unsigned calleeArgIdx =
         getCalleeArgIndexOfFirstAppliedArg() + getAppliedArgIndex(oper);
     return getSubstCalleeConv().getSILArgumentConvention(calleeArgIdx);
@@ -615,7 +615,8 @@ class FullApplySite : public ApplySite {
   /// Returns true if \p op is an operand that passes an indirect
   /// result argument to the apply site.
   bool isIndirectResultOperand(const Operand &op) const {
-    return getCalleeArgIndex(op) < getNumIndirectSILResults();
+    return isArgumentOperand(op)
+      && (getCalleeArgIndex(op) < getNumIndirectSILResults());
   }
 
   static FullApplySite getFromOpaqueValue(void *p) { return FullApplySite(p); }

include/swift/SIL/OwnershipUtils.h

@@ -32,29 +32,47 @@ class DeadEndBlocks;
 /// Returns true if v is an address or trivial.
 bool isValueAddressOrTrivial(SILValue v);
 
-/// Is this an operand that can forward both owned and guaranteed ownership into
-/// one of the operand's owner instruction's result.
-bool isOwnershipForwardingUse(Operand *op);
-
-/// Is this an operand that can forward guaranteed ownership into one of the
-/// operand's owner instruction's result.
-bool isGuaranteedForwardingUse(Operand *op);
-
-/// Is this an operand that can forward owned ownership into one of the
-/// operand's owner instruction's result.
-bool isOwnedForwardingUse(Operand *use);
+/// Is the opcode that produces \p value capable of forwarding guaranteed
+/// values?
+///
+/// This may be true even if the current instance of the instruction is not a
+/// ForwardingBorrow. If true, then the operation may be trivially rewritten
+/// with Guaranteed ownership.
+bool canOpcodeForwardGuaranteedValues(SILValue value);
 
-/// Is this a value that is the result of an instruction that forwards
-/// guaranteed ownership from one of its operands.
-bool isGuaranteedForwardingValue(SILValue value);
+/// Is the opcode that consumes \p use capable of forwarding guaranteed values?
+///
+/// This may be true even if \p use is not a ForwardingBorrow. If true, then the
+/// operation may be trivially rewritten with Guaranteed ownership.
+bool canOpcodeForwardGuaranteedValues(Operand *use);
+
+// This is the use-def equivalent of use->getOperandOwnership() ==
+// OperandOwnership::ForwardingBorrow.
+inline bool isForwardingBorrow(SILValue value) {
+  assert(value.getOwnershipKind() == OwnershipKind::Guaranteed);
+  return canOpcodeForwardGuaranteedValues(value);
+}
+
+/// Is the opcode that produces \p value capable of forwarding owned values?
+///
+/// This may be true even if the current instance of the instruction is not a
+/// ForwardingConsume. If true, then the operation may be trivially rewritten
+/// with Owned ownership.
+bool canOpcodeForwardOwnedValues(SILValue value);
 
-/// Is this value the result of an instruction that 'forward's owned ownership,
-/// but may not be able to forward guaranteed ownership.
+/// Is this opcode that consumes \p use capable of forwarding owned values?
 ///
-/// This will be either a multiple value instruction resuilt, a single value
-/// instruction that forwards or an argument that forwards the ownership from a
-/// previous terminator.
-bool isOwnedForwardingValue(SILValue value);
+/// This may be true even if the current instance of the instruction is not a
+/// ForwardingConsume. If true, then the operation may be trivially rewritten
+/// with Owned ownership.
+bool canOpcodeForwardOwnedValues(Operand *use);
+
+// This is the use-def equivalent of use->getOperandOwnership() ==
+// OperandOwnership::ForwardingConsume.
+inline bool isForwardingConsume(SILValue value) {
+  assert(value.getOwnershipKind() == OwnershipKind::Owned);
+  return canOpcodeForwardOwnedValues(value);
+}
 
 class ForwardingOperand {
   Operand *use;
@@ -225,25 +243,6 @@ struct BorrowingOperand {
     llvm_unreachable("Covered switch isn't covered?!");
   }
 
-  /// Is this a borrow scope operand that can open new borrow scopes
-  /// for owned values.
-  bool canAcceptOwnedValues() const {
-    switch (kind) {
-    // begin_borrow can take any parameter
-    case BorrowingOperandKind::BeginBorrow:
-    // Yield can implicit borrow owned values.
-    case BorrowingOperandKind::Yield:
-    // FullApplySites can implicit borrow owned values.
-    case BorrowingOperandKind::BeginApply:
-    case BorrowingOperandKind::Apply:
-    case BorrowingOperandKind::TryApply:
-      return true;
-    case BorrowingOperandKind::Branch:
-      return false;
-    }
-    llvm_unreachable("Covered switch isn't covered?!");
-  }
-
   /// Is the result of this instruction also a borrow introducer?
   ///
   /// TODO: This needs a better name.

include/swift/SIL/SILValue.h

@@ -226,6 +226,8 @@ struct OwnershipKind {
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &os, const OwnershipKind &kind);
 
+enum class OperandOwnership;
+
 /// A value representing the specific ownership semantics that a SILValue may
 /// have.
 struct ValueOwnershipKind {
@@ -286,6 +288,8 @@ struct ValueOwnershipKind {
     llvm_unreachable("covered switch");
   }
 
+  OperandOwnership getForwardingOperandOwnership() const;
+
   /// Returns true if \p Other can be merged successfully with this, implying
   /// that the two ownership kinds are "compatibile".
   ///
@@ -597,12 +601,6 @@ class OwnershipConstraint {
     return lifetimeConstraint;
   }
 
-  /// Return a constraint that is appropriate for an operand that can accept a
-  /// value with any ownership kind without ending said value's lifetime.
-  static OwnershipConstraint any() {
-    return {OwnershipKind::Any, UseLifetimeConstraint::NonLifetimeEnding};
-  }
-
   bool satisfiedBy(const Operand *use) const;
 
   bool satisfiesConstraint(ValueOwnershipKind testKind) const {
@@ -618,6 +616,124 @@ class OwnershipConstraint {
 llvm::raw_ostream &operator<<(llvm::raw_ostream &os,
                               OwnershipConstraint constraint);
 
+/// Categorize all uses in terms of their ownership effect.
+///
+/// Used to verify completeness of the ownership use model and exhaustively
+/// switch over any category of ownership use. Implies ownership constraints and
+/// lifetime constraints.
+enum class OperandOwnership {
+  /// Uses of ownership None. These uses are incompatible with values that have
+  /// ownership but are otherwise not verified.
+  None,
+
+  /// MARK: Uses of any ownership values:
+
+  /// Point-in-time use. Uses the value instantaneously.
+  /// (copy_value, single-instruction apply with @guaranteed argument)
+  InstantaneousUse,
+  // FIXME: The PointerEscape category should be eliminated. All pointer escapes
+  // should be InteriorPointer, guarded by a borrow scope.
+  PointerEscape,
+  /// Bitwise escape. Escapes the nontrivial contents of the value.
+  /// OSSA does not enforce the lifetime of the escaping bits.
+  /// The programmer must explicitly force lifetime extension.
+  /// (ref_to_unowned, unchecked_trivial_bitcast)
+  BitwiseEscape,
+
+  /// MARK: Uses of Unowned values:
+
+  /// Forwarding instruction with an Unowned result must have Unowned operands.
+  ForwardingUnowned,
+
+  /// MARK: Uses of Owned values:
+
+  /// Borrow. Propagates the owned value within a scope, without consuming it.
+  /// (begin_borrow, begin_apply with @guaranteed argument)
+  Borrow,
+  /// Destroying Consume. Destroys the owned value immediately.
+  /// (store, destroy, @owned destructure).
+  DestroyingConsume,
+  /// Forwarding Consume. Consumes the owned value indirectly via a move.
+  /// (br, destructure, tuple, struct, cast, switch).
+  ForwardingConsume,
+
+  /// MARK: Uses of Guaranteed values:
+
+  /// Nested Borrow. Propagates the guaranteed value within a nested borrow
+  /// scope, without ending the outer borrow scope, following stack discipline.
+  /// (begin_borrow, begin_apply with @guaranteed).
+  NestedBorrow,
+  /// Interior Pointer. Propagates an address into the guaranteed value within
+  /// the base's borrow scope.  (ref_element_addr, open_existential_box)
+  InteriorPointer,
+  /// Forwarded Borrow. Propagates the guaranteed value within the base's
+  /// borrow scope.
+  /// (tuple_extract, struct_extract, cast, switch)
+  ForwardingBorrow,
+  /// End Borrow. End the borrow scope opened directly by the operand.
+  /// The operand must be a begin_borrow, begin_apply, or function argument.
+  /// (end_borrow, end_apply)
+  EndBorrow,
+  // Reborrow. Ends the borrow scope opened directly by the operand and begins
+  // one or multiple disjoint borrow scopes. If a forwarded value is reborrowed,
+  // then its base must also be reborrowed at the same point.
+  // (br, FIXME: should also include destructure, tuple, struct)
+  Reborrow
+};
+
+llvm::raw_ostream &operator<<(llvm::raw_ostream &os, OperandOwnership operandOwnership);
+
+/// Return the OwnershipConstraint for a OperandOwnership.
+///
+/// Defined inline so the switch is eliminated for constant OperandOwnership.
+inline OwnershipConstraint
+getOwnershipConstraint(OperandOwnership operandOwnership) {
+  switch (operandOwnership) {
+  case OperandOwnership::None:
+    return {OwnershipKind::None, UseLifetimeConstraint::NonLifetimeEnding};
+  case OperandOwnership::InstantaneousUse:
+  case OperandOwnership::PointerEscape:
+  case OperandOwnership::BitwiseEscape:
+    return {OwnershipKind::Any, UseLifetimeConstraint::NonLifetimeEnding};
+  case OperandOwnership::ForwardingUnowned:
+    return {OwnershipKind::Unowned, UseLifetimeConstraint::NonLifetimeEnding};
+  case OperandOwnership::Borrow:
+    return {OwnershipKind::Owned, UseLifetimeConstraint::NonLifetimeEnding};
+  case OperandOwnership::DestroyingConsume:
+  case OperandOwnership::ForwardingConsume:
+    return {OwnershipKind::Owned, UseLifetimeConstraint::LifetimeEnding};
+  case OperandOwnership::NestedBorrow:
+  case OperandOwnership::InteriorPointer:
+  case OperandOwnership::ForwardingBorrow:
+    return {OwnershipKind::Guaranteed,
+            UseLifetimeConstraint::NonLifetimeEnding};
+  case OperandOwnership::EndBorrow:
+  case OperandOwnership::Reborrow:
+    return {OwnershipKind::Guaranteed, UseLifetimeConstraint::LifetimeEnding};
+  }
+}
+
+// Forwarding instructions have a dynamic ownership kind. Their forwarded
+// operand constraint depends on that dynamic result ownership. If the result is
+// owned, then the instruction moves owned operand to its result, ending its
+// lifetime. If the result is guaranteed value, then the instruction propagates
+// the lifetime of its borrows operand through its result.
+inline OperandOwnership
+ValueOwnershipKind::getForwardingOperandOwnership() const {
+  switch (value) {
+  case OwnershipKind::Any:
+    llvm_unreachable("invalid value ownership");
+  case OwnershipKind::None:
+    return OperandOwnership::None;
+  case OwnershipKind::Unowned:
+    return OperandOwnership::ForwardingUnowned;
+  case OwnershipKind::Guaranteed:
+    return OperandOwnership::ForwardingBorrow;
+  case OwnershipKind::Owned:
+    return OperandOwnership::ForwardingConsume;
+  }
+}
+
 /// A formal SIL reference to a value, suitable for use as a stored
 /// operand.
 class Operand {
@@ -695,12 +811,22 @@ class Operand {
   /// Return which operand this is in the operand list of the using instruction.
   unsigned getOperandNumber() const;
 
+  /// Return the use ownership of this operand. Returns none if the operand is a
+  /// type dependent operand.
+  ///
+  /// NOTE: This is implemented in OperandOwnership.cpp.
+  Optional<OperandOwnership> getOperandOwnership() const;
+
   /// Return the ownership constraint that restricts what types of values this
   /// Operand can contain. Returns none if the operand is a type dependent
   /// operand.
-  ///
-  /// NOTE: This is implemented in OperandOwnership.cpp.
-  Optional<OwnershipConstraint> getOwnershipConstraint() const;
+  Optional<OwnershipConstraint> getOwnershipConstraint() const {
+    auto operandOwnership = getOperandOwnership();
+    if (!operandOwnership) {
+      return None;
+    }
+    return swift::getOwnershipConstraint(operandOwnership.getValue());
+  }
 
   /// Returns true if changing the operand to use a value with the given
   /// ownership kind would not cause the operand to violate the operand's
@@ -711,24 +837,28 @@ class Operand {
   /// operand constraint.
   bool satisfiesConstraints() const;
 
-  /// Returns true if this operand acts as a use that consumes its associated
-  /// value.
+  /// Returns true if this operand acts as a use that ends the lifetime its
+  /// associated value, either by consuming the owned value or ending the
+  /// guaranteed scope.
   bool isLifetimeEnding() const;
 
   SILBasicBlock *getParentBlock() const;
   SILFunction *getParentFunction() const;
 
 private:
   void removeFromCurrent() {
-    if (!Back) return;
+    if (!Back)
+      return;
     *Back = NextUse;
-    if (NextUse) NextUse->Back = Back;
+    if (NextUse)
+      NextUse->Back = Back;
   }
 
   void insertIntoCurrent() {
     Back = &TheValue->FirstUse;
     NextUse = TheValue->FirstUse;
-    if (NextUse) NextUse->Back = &NextUse;
+    if (NextUse)
+      NextUse->Back = &NextUse;
     TheValue->FirstUse = this;
   }