react-native-agentkit

Convert React Native app flows to CLI for AI agent automation.

What it does

react-native-agentkit lets AI agents and CLI tools control your React Native app programmatically. Wrap your app with <AgentKitProvider>, and the library automatically discovers all interactive elements (buttons, inputs, scrollviews, switches) and exposes them through a command interface.

Installation

npm install react-native-agentkit
# or
yarn add react-native-agentkit

Quick Start

1. Wrap your app

import { AgentKitProvider } from 'react-native-agentkit';

export default function App() {
  return (
    <AgentKitProvider debug={__DEV__} relayUrl="ws://localhost:8347">
      <YourApp />
    </AgentKitProvider>
  );
}

That's it! The library automatically:

• Scans your component tree for interactive elements
• Starts a command bridge on the device
• Accepts commands from CLI tools

2. Start the relay server

The CLI communicates with the app through a lightweight WebSocket relay:

npx react-native-agentkit-relay
# Listening on port 8347

3. Control via CLI

# Interactive REPL
npx react-native-agentkit connect --relay=ws://localhost:8347

# Execute single commands
npx react-native-agentkit exec list --relay=ws://localhost:8347
npx react-native-agentkit exec tap submit-btn --relay=ws://localhost:8347
npx react-native-agentkit exec type email-input "user@test.com" --relay=ws://localhost:8347

# Pipe mode for AI agents
echo '{"cmd":"list"}' | npx react-native-agentkit pipe --relay=ws://localhost:8347

Commands

Command	Description	Example
list	List all interactive elements	{"cmd": "list"}
tap	Tap an element	{"cmd": "tap", "target": "submit-btn"}
toggle	Toggle a switch/checkbox	{"cmd": "toggle", "target": "agree-switch"}
setValue	Set slider/numeric value	{"cmd": "setValue", "target": "brightness", "value": 75}
type	Type text into input	{"cmd": "type", "target": "email", "text": "hi@test.com"}
clear	Clear input text	{"cmd": "clear", "target": "email"}
scroll	Scroll a view	{"cmd": "scroll", "target": "main-scroll", "direction": "down"}
swipe	Swipe an element	{"cmd": "swipe", "target": "item-1", "direction": "left"}
read	Read element value	{"cmd": "read", "target": "header"}
state	Full screen state snapshot	{"cmd": "state"}
find	Search elements by text	{"cmd": "find", "filterText": "login"}
back	Navigate back	{"cmd": "back"}
wait	Wait for element to appear	{"cmd": "wait", "target": "success-msg", "timeout": 5000}
ping	Check connection	{"cmd": "ping"}

Response Format

All commands return structured JSON:

{
  "success": true,
  "command": "tap",
  "target": "submit-btn",
  "result": { "elementTapped": true },
  "screenState": { "elements": [...], "count": 5 },
  "timestamp": 1710000000000
}

Props

Prop	Type	Default	Description
port	number	8347	Server port
debug	boolean	false	Enable debug logging
scanInterval	number	1000	Element scan interval (ms)
includeNonInteractive	boolean	false	Include non-interactive elements
includeScreenState	boolean	true	Include screen state in responses
enabled	boolean	true	Enable/disable the bridge
relayUrl	string	—	Cloud relay WebSocket URL for production
channelId	string	"default"	Channel ID for relay pairing
channelSecret	string	—	Shared secret for relay authentication

Production Usage (Cloud Relay)

In production builds, Metro isn't available. The cloud relay enables AI agents to control the app over the network.

Architecture

┌──────────┐        ┌───────────────┐        ┌──────────────┐
│  CLI /   │◄──WS──►│  Relay Server │◄──WS──►│   RN App     │
│ AI Agent │        │  (Node.js)    │        │ (production) │
└──────────┘        └───────────────┘        └──────────────┘

1. Start the relay server

# Default port 8347
npx react-native-agentkit-relay

# Custom port
npx react-native-agentkit-relay --port=9000

2. Configure the app

<AgentKitProvider
  relayUrl="ws://your-relay-host:8347"
  channelId="my-app"
  channelSecret="your-shared-secret"
>
  <YourApp />
</AgentKitProvider>

3. Connect via CLI

# Interactive REPL
npx react-native-agentkit connect --relay=ws://your-relay-host:8347 --channel=my-app --secret=your-shared-secret

# Execute a command
npx react-native-agentkit exec list --relay=ws://your-relay-host:8347 --channel=my-app --secret=your-shared-secret

# Pipe mode (for AI agents)
echo '{"cmd":"list"}' | npx react-native-agentkit pipe --relay=ws://your-relay-host:8347 --secret=your-shared-secret

Authentication

The relay supports shared-secret authentication to prevent unauthorized access:

• Dev mode (default): Auth is optional. If no secret is set, anyone can connect to the channel.
• Production: Set channelSecret in the app and --secret in the CLI. The first client to join a channel with a secret "locks" that channel — subsequent clients must present matching secrets.
• Enforced auth: Start the relay with --require-auth to reject all unauthenticated connections.

# Start relay with enforced auth
npx react-native-agentkit-relay --require-auth

💡 Tip: Use a unique channelId + channelSecret per app/device for isolation and security.

Hooks

import {
  useAgentKit, // Full context access
  useAgentKitElement, // Manually register an element
  useAgentKitRescan, // Trigger a re-scan
  useAgentKitStatus, // Monitor bridge status
} from 'react-native-agentkit';

Manual Element Registration

For custom components that aren't auto-discovered:

function CustomButton({ onPress, label }) {
  const ref = useRef(null);

  useAgentKitElement(
    'my-btn',
    {
      type: 'button',
      label,
      state: { disabled: false, selected: false },
      position: { x: 0, y: 0, width: 0, height: 0 },
      children: [],
      actions: ['tap'],
    },
    ref,
    { onPress }
  );

  return (
    <Pressable ref={ref} onPress={onPress}>
      <Text>{label}</Text>
    </Pressable>
  );
}

Element Discovery

Elements are discovered via:

1. testID prop — Always included, highest priority
2. Accessibility props — accessibilityLabel, accessibilityRole
3. Interactive handlers — onPress, onChangeText, onValueChange
4. Component type — Pressable, TextInput, Switch, ScrollView, etc.

💡 Tip: Use testID on key elements for reliable AI agent targeting.

Native Dialog Handling

AgentKit automatically intercepts native Alert.alert(), PermissionsAndroid, and react-native-permissions dialogs:

• Conditional suppression — Native dialogs are only suppressed when an agent is connected. When no agent is connected, they display normally.
• Alert buttons: alert-{title}-{button-text}
• Permission buttons: permission-{name}-grant / permission-{name}-deny / permission-{name}-block
• Buttons auto-clean up after being tapped or after a timeout

# After an action triggers Alert.alert('Confirm', '...', [{text:'Cancel'}, {text:'Delete'}])
list                              # Shows: alert-confirm-cancel, alert-confirm-delete
tap alert-confirm-delete          # Executes the Delete button's onPress handler

# Permission request (via PermissionsAndroid or react-native-permissions)
list                              # Shows: permission-camera-grant, permission-camera-deny, permission-camera-block
tap permission-camera-grant       # Grants the permission

💡 Tip: Apps using react-native-permissions get automatic interception for both iOS and Android.

AI Agent Prompts

Copy-paste these prompts to give an AI agent (Claude Code, Cursor, etc.) the ability to control your app.

Local Development

Use this when your app and relay are running locally:

You can control a React Native app using `react-native-agentkit`. The app is running
on a local device/simulator with a relay server at ws://localhost:8347.

**Read the skill file first:**
Read the file at `node_modules/react-native-agentkit/SKILL.md` for the full command
reference and workflow patterns.

**To send commands, use pipe mode:**
echo '{"cmd":"state"}' | npx react-native-agentkit pipe --relay=ws://localhost:8347

**Workflow:** Always start with `state` to see what's on screen, then use commands
like `tap`, `type`, `scroll`, `swipe`, `toggle`, `setValue`, etc. to interact.
Each response includes the updated screen state.

Now please: [describe what you want the agent to do]

Production / Remote Device

Use this when connecting to a deployed relay with authentication:

You can control a React Native app using `react-native-agentkit`. The app is connected
to a relay server.

**Read the skill file first:**
Read the file at `node_modules/react-native-agentkit/SKILL.md` for the full command
reference and workflow patterns.

**To send commands, use pipe mode with authentication:**
echo '{"cmd":"state"}' | npx react-native-agentkit pipe --relay=ws://your-relay-host:8347 --channel=your-channel --secret=your-secret

**Workflow:** Always start with `state` to see what's on screen, then use commands
like `tap`, `type`, `scroll`, `swipe`, `toggle`, `setValue`, etc. to interact.
Each response includes the updated screen state.

Now please: [describe what you want the agent to do]

Tip: The key is having the agent read SKILL.md first — it contains all command patterns, element discovery strategies, and best practices the agent needs.

Running the Example

# 1. Install dependencies
yarn install

# 2. Start the relay server (in a separate terminal)
npx react-native-agentkit-relay

# 3. Start the example app on iOS simulator
cd example
npx expo start --ios

# 4. Connect the CLI (in a separate terminal)
npx react-native-agentkit connect --relay=ws://localhost:8347

# Or execute a single command
npx react-native-agentkit exec list --relay=ws://localhost:8347

License

MIT