[WIP] Entity miners V1

entitiy_management/README.txt

@@ -0,0 +1,55 @@
+propose a solution for a new feature "Entity miners"
+
+
+ One miner hotkey VANTA_ENTITY_HOTKEY will correspond to an entity.
+
+  We will track entities with an EntityManager which persists data to disk, offers getters and setters via a client,
+  and has a server class that delegates to a manager instance (just like challenge_period flow).
+
+  Each entity i,e VANTA_ENTITY_HOTKEY can have subaccounts (monotonically increasing id).
+  Subaccounts get their own synthetic hotkey which is f"{VANTA_ENTITY_HOTKEY}_{subaccount_id}"
+  If a subaccount gets eliminated, that id can never be assigned again. An entity can only have MAX_SUBACCOUNTS_PER_ENTITY
+  subaccounts at once. The limit is 500. Thus instead of tracking eliminated subaccount ids,
+  we can simply maintain the active subaccount ids as well as the next id to be assigned
+
+
+  We must support rest api requests of entity data using an EntityManagerClient in rest server.
+
+    1. `POST register_subaccount` → returns {success, subaccount_id, subaccount_uuid}
+        1. Verifies entity collateral and slot allowance
+    2. `GET subaccount_status/{subaccouunt_id}` → active/eliminated/unknown
+
+This is the approach we want to utilize for the subaccount registration process:
+VantaRestServer endpoint exposed which does the collateral operations (placeholder for now)
+ and then returns the newly-registered subaccount id to the caller.
+  The validator then send a synapse message to all other validators so they are synced with the new subaccount id.
+  Refer for the flow in broadcast_asset_selection_to_validators to see how we should do this.
+
+
+EntityManager (RPCServerBase) will have its own daemon that periodically assess elimination criteria for entitiy miners.
+Put a placeholder in this logic for now.
+
+
+Most endpoints in VantaRestServer will support subaccounts directly since the passed in hotkey can be synthetic and
+our existing code will be able to work with synthetic hotkeys as long as we adjust the metagraph logic to detect
+synthetic hotkeys (have an underscore) and then making the appropriate call to the EntityManagerClient to see if
+that subaccount is still active. and if the VANTA_ENTITY_HOTKEY hotkey is still in the raw metagraph. Our has_hotkey method
+with this update should allow to work smoothly but let me know if there are other key parts of our system that
+need to be updated to support synthetic hotkeys.
+
+
+1. The entity hotkey (VANTA_ENTITY_HOTKEY) cannot place orders itself. Only its subaccounts can. This will need
+to be enforced in validator.py.
+
+2. Account sizes for synthetic hotkeys is set to a fixed value using a ContractClient after a blackbox function
+transfers collateral from VANTA_ENTITY_HOTKEY. Leave placeholder functions for this. This account size init is done during
+the subaccount registration flow.
+
+3. debt based scoring will read debt ledgers for all miners including subaccounts. It needs to agrgeagte the debt
+ledgers for all subaccounts into a single debt ledger representing the sum of all subaccount performance.
+The key for this debt ledger will simply be the entity hotkey (VANTA_ENTITY_HOTKEY).
+
+4.  Sub-accounts challenge period is an instantaneous pass if they get 3% returns against 6% drawdown within 90 days. Just like how in mdd checker, we can get returns and drawdown in different intervals, we will implement this in our
+EntityManager daemon. A PerfLedgerClient is thus needed.
+
+- Each entity miner can host up to **500 active sub-accounts** 

entitiy_management/entity_client.py

@@ -0,0 +1,300 @@
+# developer: jbonilla
+# Copyright � 2024 Taoshi Inc
+"""
+EntityClient - Lightweight RPC client for entity miner management.
+
+This client connects to the EntityServer via RPC.
+Can be created in ANY process - just needs the server to be running.
+
+Usage:
+    from entitiy_management.entity_client import EntityClient
+
+    # Connect to server (uses ValiConfig.RPC_ENTITY_PORT by default)
+    client = EntityClient()
+
+    # Register an entity
+    success, message = client.register_entity("my_entity_hotkey")
+
+    # Create a subaccount
+    success, subaccount_info, message = client.create_subaccount("my_entity_hotkey")
+
+    # Check if hotkey is synthetic
+    if client.is_synthetic_hotkey(hotkey):
+        entity_hotkey, subaccount_id = client.parse_synthetic_hotkey(hotkey)
+"""
+from typing import Optional, Tuple, Dict
+
+import template.protocol
+from shared_objects.rpc.rpc_client_base import RPCClientBase
+from vali_objects.vali_config import ValiConfig, RPCConnectionMode
+
+
+class EntityClient(RPCClientBase):
+    """
+    Lightweight RPC client for EntityServer.
+
+    Can be created in ANY process. No server ownership.
+    Port is obtained from ValiConfig.RPC_ENTITY_PORT.
+
+    In LOCAL mode (connection_mode=RPCConnectionMode.LOCAL), the client won't connect via RPC.
+    Instead, use set_direct_server() to provide a direct EntityServer instance.
+    """
+
+    def __init__(
+        self,
+        port: int = None,
+        connection_mode: RPCConnectionMode = RPCConnectionMode.RPC,
+        running_unit_tests: bool = False,
+        connect_immediately: bool = False
+    ):
+        """
+        Initialize entity client.
+
+        Args:
+            port: Port number of the entity server (default: ValiConfig.RPC_ENTITY_PORT)
+            connection_mode: RPCConnectionMode.LOCAL for tests (use set_direct_server()), RPCConnectionMode.RPC for production
+            running_unit_tests: Whether running in test mode
+            connect_immediately: Whether to connect immediately (default: False for lazy connection)
+        """
+        self._direct_server = None
+        self.running_unit_tests = running_unit_tests
+
+        # In LOCAL mode, don't connect via RPC - tests will set direct server
+        super().__init__(
+            service_name=ValiConfig.RPC_ENTITY_SERVICE_NAME,
+            port=port or ValiConfig.RPC_ENTITY_PORT,
+            max_retries=5,
+            retry_delay_s=1.0,
+            connect_immediately=connect_immediately,
+            connection_mode=connection_mode
+        )
+
+    # ==================== Entity Registration Methods ====================
+
+    def register_entity(
+        self,
+        entity_hotkey: str,
+        collateral_amount: float = 0.0,
+        max_subaccounts: int = None
+    ) -> Tuple[bool, str]:
+        """
+        Register a new entity.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+            collateral_amount: Collateral amount (placeholder)
+            max_subaccounts: Maximum allowed subaccounts
+
+        Returns:
+            (success: bool, message: str)
+        """
+        return self._server.register_entity_rpc(entity_hotkey, collateral_amount, max_subaccounts)
+
+    def create_subaccount(self, entity_hotkey: str) -> Tuple[bool, Optional[dict], str]:
+        """
+        Create a new subaccount for an entity.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+
+        Returns:
+            (success: bool, subaccount_info_dict: Optional[dict], message: str)
+        """
+        return self._server.create_subaccount_rpc(entity_hotkey)
+
+    def eliminate_subaccount(
+        self,
+        entity_hotkey: str,
+        subaccount_id: int,
+        reason: str = "unknown"
+    ) -> Tuple[bool, str]:
+        """
+        Eliminate a subaccount.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+            subaccount_id: The subaccount ID to eliminate
+            reason: Elimination reason
+
+        Returns:
+            (success: bool, message: str)
+        """
+        return self._server.eliminate_subaccount_rpc(entity_hotkey, subaccount_id, reason)
+
+    def update_collateral(self, entity_hotkey: str, collateral_amount: float) -> Tuple[bool, str]:
+        """
+        Update collateral for an entity.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+            collateral_amount: New collateral amount
+
+        Returns:
+            (success: bool, message: str)
+        """
+        return self._server.update_collateral_rpc(entity_hotkey, collateral_amount)
+
+    # ==================== Query Methods ====================
+
+    def get_subaccount_status(self, synthetic_hotkey: str) -> Tuple[bool, Optional[str], str]:
+        """
+        Get the status of a subaccount by synthetic hotkey.
+
+        Args:
+            synthetic_hotkey: The synthetic hotkey ({entity_hotkey}_{subaccount_id})
+
+        Returns:
+            (found: bool, status: Optional[str], synthetic_hotkey: str)
+        """
+        return self._server.get_subaccount_status_rpc(synthetic_hotkey)
+
+    def get_entity_data(self, entity_hotkey: str) -> Optional[dict]:
+        """
+        Get full entity data.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+
+        Returns:
+            Entity data as dict or None
+        """
+        return self._server.get_entity_data_rpc(entity_hotkey)
+
+    def get_all_entities(self) -> Dict[str, dict]:
+        """
+        Get all entities.
+
+        Returns:
+            Dict mapping entity_hotkey -> entity_data_dict
+        """
+        return self._server.get_all_entities_rpc()
+
+    def is_synthetic_hotkey(self, hotkey: str) -> bool:
+        """
+        Check if a hotkey is synthetic (contains underscore with integer suffix).
+
+        Args:
+            hotkey: The hotkey to check
+
+        Returns:
+            True if synthetic, False otherwise
+        """
+        return self._server.is_synthetic_hotkey_rpc(hotkey)
+
+    def parse_synthetic_hotkey(self, synthetic_hotkey: str) -> Tuple[Optional[str], Optional[int]]:
+        """
+        Parse a synthetic hotkey into entity_hotkey and subaccount_id.
+
+        Args:
+            synthetic_hotkey: The synthetic hotkey ({entity_hotkey}_{subaccount_id})
+
+        Returns:
+            (entity_hotkey, subaccount_id) or (None, None) if invalid
+        """
+        return self._server.parse_synthetic_hotkey_rpc(synthetic_hotkey)
+
+    # ==================== Validator Broadcast Methods ====================
+
+    def broadcast_subaccount_registration(
+        self,
+        entity_hotkey: str,
+        subaccount_id: int,
+        subaccount_uuid: str,
+        synthetic_hotkey: str
+    ) -> None:
+        """
+        Broadcast subaccount registration to other validators.
+
+        Args:
+            entity_hotkey: The VANTA_ENTITY_HOTKEY
+            subaccount_id: The subaccount ID
+            subaccount_uuid: The subaccount UUID
+            synthetic_hotkey: The synthetic hotkey
+        """
+        return self._server.broadcast_subaccount_registration_rpc(
+            entity_hotkey, subaccount_id, subaccount_uuid, synthetic_hotkey
+        )
+
+    def receive_subaccount_registration_update(self, subaccount_data: dict) -> bool:
+        """
+        Process incoming subaccount registration from another validator.
+
+        Args:
+            subaccount_data: Dict containing entity_hotkey, subaccount_id, subaccount_uuid, synthetic_hotkey
+
+        Returns:
+            bool: True if successful, False otherwise
+        """
+        # This delegates to EntityManager.receive_subaccount_registration via RPC
+        # Returns True if successful, False otherwise
+        success = self._server.receive_subaccount_registration_rpc(
+            template.protocol.SubaccountRegistration(subaccount_data=subaccount_data)
+        ).successfully_processed
+        return success
+
+    def receive_subaccount_registration(
+        self,
+        synapse: 'template.protocol.SubaccountRegistration'
+    ) -> 'template.protocol.SubaccountRegistration':
+        """
+        Receive subaccount registration synapse (for axon attachment).
+
+        This delegates to the server's RPC handler. Used by validator_base.py for axon attachment.
+
+        Args:
+            synapse: SubaccountRegistration synapse from another validator
+
+        Returns:
+            Updated synapse with success/error status
+        """
+        return self._server.receive_subaccount_registration_rpc(synapse)
+
+    # ==================== Health Check Methods ====================
+
+    def health_check(self) -> dict:
+        """
+        Get health status from server.
+
+        Returns:
+            dict: Health status with 'status', 'service', 'timestamp_ms' and service-specific info
+        """
+        return self._server.health_check_rpc()
+
+    # ==================== Testing/Admin Methods ====================
+
+    def clear_all_entities(self) -> None:
+        """Clear all entity data (for testing only)."""
+        self._server.clear_all_entities_rpc()
+
+    def to_checkpoint_dict(self) -> dict:
+        """Get entity data as a checkpoint dict for serialization."""
+        return self._server.to_checkpoint_dict_rpc()
+
+    # ==================== Daemon Control Methods ====================
+
+    def start_daemon(self) -> bool:
+        """
+        Start the daemon thread remotely via RPC.
+
+        Returns:
+            bool: True if daemon was started, False if already running
+        """
+        return self._server.start_daemon_rpc()
+
+    def stop_daemon(self) -> bool:
+        """
+        Stop the daemon thread remotely via RPC.
+
+        Returns:
+            bool: True if daemon was stopped, False if not running
+        """
+        return self._server.stop_daemon_rpc()
+
+    def is_daemon_running(self) -> bool:
+        """
+        Check if daemon is running via RPC.
+
+        Returns:
+            bool: True if daemon is running, False otherwise
+        """
+        return self._server.is_daemon_running_rpc()