diff --git a/helm-charts/mobile-sandbox/.helmignore b/helm-charts/mobile-sandbox/.helmignore new file mode 100644 index 0000000..97f74c2 --- /dev/null +++ b/helm-charts/mobile-sandbox/.helmignore @@ -0,0 +1,26 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ +# Scripts +*.sh +.docs/ \ No newline at end of file diff --git a/helm-charts/mobile-sandbox/Chart.yaml b/helm-charts/mobile-sandbox/Chart.yaml new file mode 100644 index 0000000..6d7f6b1 --- /dev/null +++ b/helm-charts/mobile-sandbox/Chart.yaml @@ -0,0 +1,24 @@ +apiVersion: v2 +name: mobile-sandbox +description: A Helm chart for deploying Redroid Android containers with kernel module support on Kubernetes + +# A chart can be either an 'application' or a 'library' chart. +# +# Application charts are a collection of templates that can be packaged into versioned archives +# to be deployed. +# +# Library charts provide useful utilities or functions for the chart developer. They're included as +# a dependency of application charts to inject those utilities and functions into the rendering +# pipeline. Library charts do not define any templates and therefore cannot be deployed. +type: application + +# This is the chart version. This version number should be incremented each time you make changes +# to the chart and its templates, including the app version. +# Versions are expected to follow Semantic Versioning (https://semver.org/) +version: 0.1.0 + +# This is the version number of the application being deployed. This version number should be +# incremented each time you make changes to the application. Versions are not expected to +# follow Semantic Versioning. They should reflect the version the application is using. +# It is recommended to use it with quotes. +appVersion: "11.0.0" diff --git a/helm-charts/mobile-sandbox/README.md b/helm-charts/mobile-sandbox/README.md new file mode 100644 index 0000000..167f9be --- /dev/null +++ b/helm-charts/mobile-sandbox/README.md @@ -0,0 +1,208 @@ +# mobile-sandbox + +Deploy Android containers on Tencent Kubernetes Engine (TKE) for mobile app testing and automation. + +## Quick Start + +### Prerequisites + +- TKE cluster v1.28+ +- Node pool configuration: + - Machine: **S6 series** + - OS: **Ubuntu Server 24.04 LTS 64-bit** (`img-mmytdhbn`) + - Kernel: **6.8.0-87** (upgraded via init script) + +### Installation + +**1. Create Node Pool** + +Select S6 series with Ubuntu 24.04: + +![Node Pool Configuration](docs/images/image1.png) + +Add kernel upgrade script in "Post Node Init": + +![Initialization Script](docs/images/image2.png) + +The script `upgrade_kernel_to_6.8.0-87.sh` is included in this project. + +**2. Deploy Chart** + +Modify values to specify node pool ID. +```bash +helm install mobile-sandbox ./helm-charts/mobile-sandbox +``` + +**3. Verify Deployment** + +```bash +kubectl get pods -l k8s-app=redroid +kubectl get daemonset -l app=redroid-ubuntu-module-loader +``` + +## Usage + +### Connect via ADB + +**Direct (via EIP):** +```bash +# Get EIP +kubectl get pod -o jsonpath='{.metadata.annotations.tke\.cloud\.tencent\.com/eip-public-ip}' + +# Connect device +adb connect :5555 +``` + +### Example: WeChat Testing + +**1. Connect to device** + +```bash +adb connect :5555 +``` + +![ADB Connection](docs/images/image3.png) + +**2. Install APK** + +```bash +adb push weixin.apk /sdcard/ +adb install /sdcard/weixin.apk +``` + +![Push APK](docs/images/image4.png) + +**3. Open UI** + +```bash +scrcpy -s +``` + +![Android UI](docs/images/image5.png) + +![WeChat Home](docs/images/image6.png) + +**4. Launch WeChat** + +![Launch WeChat](docs/images/image7.png) + +**5. Test Features** + +Login: + +![WeChat Login](docs/images/image8.png) + +Send messages: + +![Send Message](docs/images/image9.png) + +Mini-programs: + +![Mini Program](docs/images/image10.png) + +Screenshots: + +![Screenshot](docs/images/image11.png) + +## Configuration + +### Key Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `redroid.replicas` | Number of Android instances | `1` | +| `redroid.nodeAffinity.nodepoolIds` | Node pool IDs for scheduling | `[np-xxxxxxxx]` | +| `redroid.container.resources.requests.cpu` | CPU per instance | `6500m` | +| `redroid.container.resources.requests.memory` | Memory per instance | `10Gi` | +| `moduleLoader.nodeAffinity.nodepoolIds` | Node pools for module loader | `[np-xxxxxxxx]` | + +### Android Settings + +```yaml +redroid: + container: + args: + - androidboot.redroid_width=1080 # Screen width + - androidboot.redroid_height=1920 # Screen height + - androidboot.redroid_dpi=480 # Screen DPI + - androidboot.redroid_fps=30 # Frame rate +``` +More parameters: https://github.com/ERSTT/redroid + +### Network Configuration + +Each pod is auto-configured with: +- **EIP**: Public IP with 100Mbps bandwidth +- **ENI**: TKE route-eni mode +- **Route Guardian**: Sidecar maintains routing tables + +## Architecture + +``` +┌─────────────────────────────────────────┐ +│ Module Loader DaemonSet │ +│ • Loads binder_linux, ashmem_linux │ +│ • Loads iptables modules │ +└─────────────────────────────────────────┘ + +┌─────────────────────────────────────────┐ +│ Redroid StatefulSet │ +│ ┌─────────────────────────────────┐ │ +│ │ Pod │ │ +│ │ ├─ redroid (Android 11) │ │ +│ │ └─ route-setup (Network) │ │ +│ └─────────────────────────────────┘ │ +└─────────────────────────────────────────┘ +``` + +### Components + +- **redroid**: Android 11 with Houdini ARM translation +- **route-setup**: Maintains routing for TKE ENI +- **module-loader**: Loads kernel modules on each node + +## Troubleshooting + +### Pod Not Starting + +```bash +# Check module loader +kubectl logs -l app=redroid-ubuntu-module-loader + +# Verify kernel modules +kubectl exec -it -- lsmod | grep -E "binder|ashmem" +``` + +### ADB Connection Failed + +```bash +# Check Redroid status +kubectl logs -c redroid + +# Verify ADB port +kubectl exec -it -c redroid -- netstat -tlnp | grep 5555 +``` + +### Network Issues + +```bash +# Check route configuration +kubectl logs -c route-setup + +# Test connectivity +kubectl exec -it -c redroid -- ping -c 3 8.8.8.8 +``` + +## Uninstall + +```bash +helm uninstall mobile-sandbox +``` + +**Note**: EIPs are retained by default. Delete manually from TKE console if needed. + +## References + +- [Redroid Documentation](https://github.com/ERSTT/redroid) +- [TKE Documentation](https://cloud.tencent.com/document/product/457) +- [Create TKE Cluster](https://cloud.tencent.com/document/product/457/103981) diff --git a/helm-charts/mobile-sandbox/README_zh.md b/helm-charts/mobile-sandbox/README_zh.md new file mode 100644 index 0000000..527b7be --- /dev/null +++ b/helm-charts/mobile-sandbox/README_zh.md @@ -0,0 +1,208 @@ +# mobile-sandbox + +在腾讯云容器服务(TKE)上部署 Android 容器,用于移动应用测试和自动化。 + +## 快速开始 + +### 前置要求 + +- TKE 集群 v1.28+ +- 节点池配置: + - 机型:**S6 系列** + - 操作系统:**Ubuntu Server 24.04 LTS 64位** (`img-mmytdhbn`) + - 内核版本:**6.8.0-87**(通过初始化脚本升级) + +### 安装步骤 + +**1. 创建节点池** + +选择 S6 系列机型和 Ubuntu 24.04: + +![节点池配置](docs/images/image1.png) + +在"节点初始化后"中添加内核升级脚本: + +![初始化脚本](docs/images/image2.png) + +脚本 `upgrade_kernel_to_6.8.0-87.sh` 已包含在本项目中。 + +**2. 部署 Chart** + +修改 values 指定节点池 id。 +```bash +helm install mobile-sandbox ./helm-charts/mobile-sandbox +``` + +**3. 验证部署** + +```bash +kubectl get pods -l k8s-app=redroid +kubectl get daemonset -l app=redroid-ubuntu-module-loader +``` + +## 使用方法 + +### 通过 ADB 连接 + +**直连方式(通过 EIP):** +```bash +# 获取 EIP +kubectl get pod -o jsonpath='{.metadata.annotations.tke\.cloud\.tencent\.com/eip-public-ip}' + +# 连接设备 +adb connect :5555 +``` + +### 示例:微信测试 + +**1. 连接设备** + +```bash +adb connect :5555 +``` + +![ADB 连接](docs/images/image3.png) + +**2. 安装 APK** + +```bash +adb push weixin.apk /sdcard/ +adb install /sdcard/weixin.apk +``` + +![传输 APK](docs/images/image4.png) + +**3. 打开界面** + +```bash +scrcpy -s +``` + +![Android 界面](docs/images/image5.png) + +![微信主屏](docs/images/image6.png) + +**4. 启动微信** + +![启动微信](docs/images/image7.png) + +**5. 功能测试** + +登入: + +![微信登录](docs/images/image8.png) + +发送消息: + +![发送消息](docs/images/image9.png) + +小程序: + +![小程序](docs/images/image10.png) + +截图: + +![截图](docs/images/image11.png) + +## 配置说明 + +### 主要参数 + +| 参数 | 说明 | 默认值 | +|------|------|--------| +| `redroid.replicas` | Android 实例数量 | `1` | +| `redroid.nodeAffinity.nodepoolIds` | 调度到的节点池 ID | `[np-xxxxxxxx]` | +| `redroid.container.resources.requests.cpu` | 单实例 CPU 请求 | `6500m` | +| `redroid.container.resources.requests.memory` | 单实例内存请求 | `10Gi` | +| `moduleLoader.nodeAffinity.nodepoolIds` | 模块加载器运行的节点池 | `[np-xxxxxxxx]` | + +### Android 配置 + +```yaml +redroid: + container: + args: + - androidboot.redroid_width=1080 # 屏幕宽度 + - androidboot.redroid_height=1920 # 屏幕高度 + - androidboot.redroid_dpi=480 # 屏幕 DPI + - androidboot.redroid_fps=30 # 帧率 +``` +更多参数参考: https://github.com/ERSTT/redroid + +### 网络配置 + +每个 Pod 自动配置: +- **EIP**:公网 IP,100Mbps 带宽 +- **ENI**:TKE 路由 ENI 模式 +- **路由守护**:Sidecar 容器维护路由表 + +## 架构说明 + +``` +┌─────────────────────────────────────────┐ +│ 模块加载器 DaemonSet │ +│ • 加载 binder_linux、ashmem_linux │ +│ • 加载 iptables 模块 │ +└─────────────────────────────────────────┘ + +┌─────────────────────────────────────────┐ +│ Redroid StatefulSet │ +│ ┌─────────────────────────────────┐ │ +│ │ Pod │ │ +│ │ ├─ redroid (Android 11) │ │ +│ │ └─ route-setup (网络) │ │ +│ └─────────────────────────────────┘ │ +└─────────────────────────────────────────┘ +``` + +### 组件说明 + +- **redroid**:Android 11 系统,支持 Houdini ARM 转译 +- **route-setup**:维护 TKE ENI 网络路由 +- **module-loader**:在每个节点上加载内核模块 + +## 故障排查 + +### Pod 无法启动 + +```bash +# 检查模块加载器 +kubectl logs -l app=redroid-ubuntu-module-loader + +# 验证内核模块 +kubectl exec -it -- lsmod | grep -E "binder|ashmem" +``` + +### ADB 连接失败 + +```bash +# 检查 Redroid 运行状态 +kubectl logs -c redroid + +# 验证 ADB 端口 +kubectl exec -it -c redroid -- netstat -tlnp | grep 5555 +``` + +### 网络问题 + +```bash +# 检查路由配置 +kubectl logs -c route-setup + +# 测试网络连通性 +kubectl exec -it -c redroid -- ping -c 3 8.8.8.8 +``` + +## 卸载 + +```bash +helm uninstall mobile-sandbox +``` + +**注意**:EIP 默认保留,如需删除请在 TKE 控制台手动操作。 + +## 参考文档 + +- [Redroid 文档](https://github.com/ERSTT/redroid) +- [TKE 文档](https://cloud.tencent.com/document/product/457) +- [创建 TKE 集群](https://cloud.tencent.com/document/product/457/103981) diff --git a/helm-charts/mobile-sandbox/docs/images/image1.png b/helm-charts/mobile-sandbox/docs/images/image1.png new file mode 100644 index 0000000..11fca0a Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image1.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image10.png b/helm-charts/mobile-sandbox/docs/images/image10.png new file mode 100644 index 0000000..14ebc73 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image10.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image11.png b/helm-charts/mobile-sandbox/docs/images/image11.png new file mode 100644 index 0000000..0873ee3 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image11.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image2.png b/helm-charts/mobile-sandbox/docs/images/image2.png new file mode 100644 index 0000000..05f1e70 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image2.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image3.png b/helm-charts/mobile-sandbox/docs/images/image3.png new file mode 100644 index 0000000..5eee618 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image3.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image4.png b/helm-charts/mobile-sandbox/docs/images/image4.png new file mode 100644 index 0000000..7164ab0 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image4.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image5.png b/helm-charts/mobile-sandbox/docs/images/image5.png new file mode 100644 index 0000000..4b6bc76 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image5.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image6.png b/helm-charts/mobile-sandbox/docs/images/image6.png new file mode 100644 index 0000000..c6b47ca Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image6.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image7.png b/helm-charts/mobile-sandbox/docs/images/image7.png new file mode 100644 index 0000000..494e10e Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image7.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image8.png b/helm-charts/mobile-sandbox/docs/images/image8.png new file mode 100644 index 0000000..b0d000f Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image8.png differ diff --git a/helm-charts/mobile-sandbox/docs/images/image9.png b/helm-charts/mobile-sandbox/docs/images/image9.png new file mode 100644 index 0000000..da73521 Binary files /dev/null and b/helm-charts/mobile-sandbox/docs/images/image9.png differ diff --git a/helm-charts/mobile-sandbox/templates/NOTES.txt b/helm-charts/mobile-sandbox/templates/NOTES.txt new file mode 100644 index 0000000..a915660 --- /dev/null +++ b/helm-charts/mobile-sandbox/templates/NOTES.txt @@ -0,0 +1,43 @@ +mobile-sandbox has been deployed successfully! + +{{- if .Values.moduleLoader.enabled }} +1. Module Loader DaemonSet: + - Loads Android kernel modules (binder_linux, ashmem_linux) + - Loads iptables modules (IPv4 and IPv6) + - Running on nodes: {{ if .Values.moduleLoader.nodeAffinity.nodepoolIds }}{{ .Values.moduleLoader.nodeAffinity.nodepoolIds | join ", " }}{{ end }} + + Check module loader status: + kubectl get daemonset {{ include "mobile-sandbox.fullname" . }}-module-loader + kubectl logs -l app=redroid-ubuntu-module-loader --tail=50 +{{- end }} + +{{- if .Values.redroid.enabled }} +2. Redroid StatefulSet: + - Running {{ .Values.redroid.replicas }} Android container(s) + - Image: {{ .Values.redroid.container.image.repository }}:{{ .Values.redroid.container.image.tag }} + - ADB port: 5555 + + Check redroid status: + kubectl get statefulset {{ include "mobile-sandbox.fullname" . }}-redroid + kubectl get pods -l k8s-app=redroid + + View logs: + kubectl logs {{ include "mobile-sandbox.fullname" . }}-redroid-0 -c redroid + kubectl logs {{ include "mobile-sandbox.fullname" . }}-redroid-0 -c logcat-sidecar + kubectl logs {{ include "mobile-sandbox.fullname" . }}-redroid-0 -c route-setup + + Connect to ADB (from pod with direct network access): + adb connect :5555 + + Or port-forward: + kubectl port-forward {{ include "mobile-sandbox.fullname" . }}-redroid-0 5555:5555 + adb connect localhost:5555 +{{- end }} + +{{- if .Values.service.enabled }} +3. Service: + export SERVICE_IP=$(kubectl get svc {{ include "mobile-sandbox.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') + adb connect $SERVICE_IP:{{ .Values.service.port }} +{{- end }} + +For more information, visit: https://github.com/remote-android/redroid-doc diff --git a/helm-charts/mobile-sandbox/templates/_helpers.tpl b/helm-charts/mobile-sandbox/templates/_helpers.tpl new file mode 100644 index 0000000..0db2ae2 --- /dev/null +++ b/helm-charts/mobile-sandbox/templates/_helpers.tpl @@ -0,0 +1,62 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "mobile-sandbox.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "mobile-sandbox.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "mobile-sandbox.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "mobile-sandbox.labels" -}} +helm.sh/chart: {{ include "mobile-sandbox.chart" . }} +{{ include "mobile-sandbox.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "mobile-sandbox.selectorLabels" -}} +app.kubernetes.io/name: {{ include "mobile-sandbox.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "mobile-sandbox.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "mobile-sandbox.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} diff --git a/helm-charts/mobile-sandbox/templates/daemonset.yaml b/helm-charts/mobile-sandbox/templates/daemonset.yaml new file mode 100644 index 0000000..b29d011 --- /dev/null +++ b/helm-charts/mobile-sandbox/templates/daemonset.yaml @@ -0,0 +1,234 @@ +{{- if .Values.moduleLoader.enabled }} +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ include "mobile-sandbox.fullname" . }}-module-loader + labels: + {{- include "mobile-sandbox.labels" . | nindent 4 }} + {{- with .Values.moduleLoader.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + revisionHistoryLimit: 10 + selector: + matchLabels: + {{- with .Values.moduleLoader.labels }} + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.daemonSetUpdateStrategy }} + updateStrategy: + {{- toYaml . | nindent 4 }} + {{- end }} + template: + metadata: + labels: + {{- with .Values.moduleLoader.labels }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- if .Values.moduleLoader.nodeAffinity.enabled }} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + {{- range .Values.moduleLoader.nodeAffinity.nodepoolIds }} + - matchExpressions: + - key: tke.cloud.tencent.com/nodepool-id + operator: In + values: + - {{ . }} + {{- end }} + {{- end }} + {{- with .Values.moduleLoader.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + hostNetwork: true + hostPID: true + dnsPolicy: {{ .Values.dnsPolicy }} + restartPolicy: {{ .Values.restartPolicy }} + terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }} + securityContext: + {{- toYaml .Values.securityContext | nindent 8 }} + + # Init container - installs kernel modules on host + initContainers: + - name: install-modules + image: "{{ .Values.moduleLoader.initContainer.image.repository }}:{{ .Values.moduleLoader.initContainer.image.tag }}" + imagePullPolicy: {{ .Values.moduleLoader.initContainer.image.pullPolicy }} + command: + - /bin/bash + - -c + - | + #!/bin/bash + set -e + + echo "Checking if kernel modules need to be installed..." + + # Use nsenter to run commands in host namespace + KERNEL_VERSION=$(nsenter --mount=/proc/1/ns/mnt -- uname -r) + echo "Host kernel version: $KERNEL_VERSION" + + # Check if modules already exist + if nsenter --mount=/proc/1/ns/mnt -- test -f /lib/modules/$KERNEL_VERSION/kernel/drivers/android/binder_linux.ko; then + echo "✓ binder_linux module already exists" + else + echo "Installing linux-modules-extra-$KERNEL_VERSION..." + nsenter --mount=/proc/1/ns/mnt -- apt-get update + nsenter --mount=/proc/1/ns/mnt -- apt-get install -y linux-modules-extra-$KERNEL_VERSION || echo "Failed to install, continuing..." + fi + + # Ensure /etc/modules-load.d directory exists on host + nsenter --mount=/proc/1/ns/mnt -- mkdir -p /etc/modules-load.d + + # Configure modules to load at boot (IPv4 and IPv6) + echo "Configuring modules to auto-load at boot..." + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "iptable_filter" > /etc/modules-load.d/iptables.conf' + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "iptable_mangle" >> /etc/modules-load.d/iptables.conf' + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "iptable_raw" >> /etc/modules-load.d/iptables.conf' + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "ip6table_filter" >> /etc/modules-load.d/iptables.conf' + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "ip6table_mangle" >> /etc/modules-load.d/iptables.conf' + nsenter --mount=/proc/1/ns/mnt -- sh -c 'echo "ip6table_raw" >> /etc/modules-load.d/iptables.conf' + echo "✓ Configured IPv4 and IPv6 iptables modules for auto-load" + + echo "Module installation check complete" + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - name: host-root + mountPath: /host + + # Main container - loads kernel modules + containers: + - name: module-loader + image: "{{ .Values.moduleLoader.container.image.repository }}:{{ .Values.moduleLoader.container.image.tag }}" + imagePullPolicy: {{ .Values.moduleLoader.container.image.pullPolicy }} + command: + - /bin/sh + - -c + - | + #!/bin/sh + set -e + + echo "Starting kernel module loader..." + + # Use Tencent Cloud mirror + sed -i 's/dl-cdn.alpinelinux.org/{{ .Values.moduleLoader.container.apkMirror }}/g' /etc/apk/repositories + + # Install kmod package for modprobe and lsmod + apk add --no-cache kmod + + # Load Android kernel modules + echo "=== Loading Android kernel modules ===" + + # Check if binder_linux module is already loaded + if lsmod | grep -q "^binder_linux"; then + echo "✓ binder_linux module is already loaded" + else + echo "Loading binder_linux module..." + modprobe binder_linux devices="binder,hwbinder,vndbinder" || echo "✗ Failed to load binder_linux" + fi + + # Check if ashmem_linux module is already loaded + if lsmod | grep -q "^ashmem_linux"; then + echo "✓ ashmem_linux module is already loaded" + else + echo "Loading ashmem_linux module..." + modprobe ashmem_linux || echo "✗ Failed to load ashmem_linux" + fi + + # Load iptables kernel modules (Fix for redroid networking issues) + echo "=== Loading IPv4 iptables kernel modules ===" + + if lsmod | grep -q "^iptable_filter"; then + echo "✓ iptable_filter module is already loaded" + else + echo "Loading iptable_filter module..." + modprobe iptable_filter || echo "✗ Failed to load iptable_filter" + fi + + if lsmod | grep -q "^iptable_mangle"; then + echo "✓ iptable_mangle module is already loaded" + else + echo "Loading iptable_mangle module..." + modprobe iptable_mangle || echo "✗ Failed to load iptable_mangle" + fi + + if lsmod | grep -q "^iptable_raw"; then + echo "✓ iptable_raw module is already loaded" + else + echo "Loading iptable_raw module..." + modprobe iptable_raw || echo "✗ Failed to load iptable_raw" + fi + + # Load IPv6 iptables kernel modules + echo "=== Loading IPv6 iptables kernel modules ===" + + if lsmod | grep -q "^ip6table_filter"; then + echo "✓ ip6table_filter module is already loaded" + else + echo "Loading ip6table_filter module..." + modprobe ip6table_filter || echo "✗ Failed to load ip6table_filter" + fi + + if lsmod | grep -q "^ip6table_mangle"; then + echo "✓ ip6table_mangle module is already loaded" + else + echo "Loading ip6table_mangle module..." + modprobe ip6table_mangle || echo "✗ Failed to load ip6table_mangle" + fi + + if lsmod | grep -q "^ip6table_raw"; then + echo "✓ ip6table_raw module is already loaded" + else + echo "Loading ip6table_raw module..." + modprobe ip6table_raw || echo "✗ Failed to load ip6table_raw" + fi + + echo "" + echo "=== Module loading summary ===" + echo "Android modules:" + lsmod | grep -E "binder|ashmem" || echo "No binder/ashmem modules found" + echo "" + echo "IPv4 iptables modules:" + lsmod | grep -E "iptable_filter|iptable_mangle|iptable_raw" || echo "No IPv4 iptables modules found" + echo "" + echo "IPv6 iptables modules:" + lsmod | grep -E "ip6table_filter|ip6table_mangle|ip6table_raw" || echo "No IPv6 iptables modules found" + + # Keep the container running + echo "" + echo "Module loader ready. Entering sleep mode..." + while true; do sleep 3600; done + {{- with .Values.moduleLoader.container.resources }} + resources: + {{- toYaml . | nindent 10 }} + {{- end }} + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - name: lib-modules + mountPath: /lib/modules + readOnly: true + - name: sys + mountPath: /sys + + # Volumes + volumes: + - name: lib-modules + hostPath: + path: /lib/modules + type: Directory + - name: sys + hostPath: + path: /sys + type: Directory + - name: host-root + hostPath: + path: / + type: Directory +{{- end }} diff --git a/helm-charts/mobile-sandbox/templates/service.yaml b/helm-charts/mobile-sandbox/templates/service.yaml new file mode 100644 index 0000000..7b5f666 --- /dev/null +++ b/helm-charts/mobile-sandbox/templates/service.yaml @@ -0,0 +1,20 @@ +{{- if .Values.service.enabled }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "mobile-sandbox.fullname" . }} + labels: + {{- include "mobile-sandbox.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: {{ .Values.service.targetPort }} + protocol: TCP + name: adb + selector: + {{- include "mobile-sandbox.selectorLabels" . | nindent 4 }} + {{- with .Values.redroid.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/helm-charts/mobile-sandbox/templates/statefulset.yaml b/helm-charts/mobile-sandbox/templates/statefulset.yaml new file mode 100644 index 0000000..033383c --- /dev/null +++ b/helm-charts/mobile-sandbox/templates/statefulset.yaml @@ -0,0 +1,162 @@ +{{- if .Values.redroid.enabled }} +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ include "mobile-sandbox.fullname" . }}-redroid + labels: + {{- include "mobile-sandbox.labels" . | nindent 4 }} + {{- with .Values.redroid.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + serviceName: "" + replicas: {{ .Values.redroid.replicas }} + podManagementPolicy: {{ .Values.redroid.podManagementPolicy }} + revisionHistoryLimit: 10 + {{- with .Values.redroid.persistentVolumeClaimRetentionPolicy }} + persistentVolumeClaimRetentionPolicy: + {{- toYaml . | nindent 4 }} + {{- end }} + selector: + matchLabels: + {{- include "mobile-sandbox.selectorLabels" . | nindent 6 }} + {{- with .Values.redroid.labels }} + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.updateStrategy }} + updateStrategy: + {{- toYaml . | nindent 4 }} + {{- end }} + template: + metadata: + annotations: + {{- with .Values.redroid.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: + {{- include "mobile-sandbox.labels" . | nindent 8 }} + {{- with .Values.redroid.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if .Values.redroid.nodeAffinity.enabled }} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + {{- range .Values.redroid.nodeAffinity.nodepoolIds }} + - matchExpressions: + - key: tke.cloud.tencent.com/nodepool-id + operator: In + values: + - {{ . }} + {{- end }} + {{- end }} + dnsPolicy: {{ .Values.dnsPolicy }} + restartPolicy: {{ .Values.restartPolicy }} + terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }} + securityContext: + {{- toYaml .Values.securityContext | nindent 8 }} + containers: + # Main redroid container + - name: redroid + image: "{{ .Values.redroid.container.image.repository }}:{{ .Values.redroid.container.image.tag }}" + imagePullPolicy: {{ .Values.redroid.container.image.pullPolicy }} + {{- with .Values.redroid.container.args }} + args: + {{- toYaml . | nindent 10 }} + {{- end }} + {{- with .Values.redroid.container.env }} + env: + {{- toYaml . | nindent 10 }} + {{- end }} + {{- with .Values.redroid.container.ports }} + ports: + {{- toYaml . | nindent 10 }} + {{- end }} + {{- with .Values.redroid.container.resources }} + resources: + {{- toYaml . | nindent 10 }} + {{- end }} + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + + # Route setup sidecar + - name: route-setup + image: "{{ .Values.redroid.routeSetup.image.repository }}:{{ .Values.redroid.routeSetup.image.tag }}" + imagePullPolicy: {{ .Values.redroid.routeSetup.image.pullPolicy }} + command: + - sh + - -c + - | + set -e + echo "=== Network Route Guardian ===" + + for i in $(seq 1 60); do + POD_IP=$(ip -4 addr show eth0 2>/dev/null | grep inet | awk '{print $2}' | cut -d/ -f1) + [ -n "$POD_IP" ] && break + sleep 1 + done + + echo "Pod IP: $POD_IP" + + for i in $(seq 1 30); do + GW_MAC=$(ip neigh show | grep -E "REACHABLE|STALE|DELAY|PERMANENT" | grep -oE "([0-9a-f]{2}:){5}[0-9a-f]{2}" | head -1) + if [ -z "$GW_MAC" ]; then + ip route add 169.254.1.1 dev eth0 scope link 2>/dev/null || true + ping -c 1 -W 1 169.254.1.1 >/dev/null 2>&1 || true + sleep 1 + else + break + fi + done + + configure_routes() { + ip route replace 169.254.1.1 dev eth0 scope link 2>/dev/null || true + [ -n "$1" ] && arp -s 169.254.1.1 $1 2>/dev/null || true + ip route replace default via 169.254.1.1 dev eth0 src $POD_IP 2>/dev/null || true + + for tid in 97 98 99; do + ip route replace 169.254.1.1 dev eth0 scope link table $tid 2>/dev/null || true + ip route replace default via 169.254.1.1 dev eth0 src $POD_IP table $tid 2>/dev/null || true + done + + ip rule show | grep -q "32000.*unreachable" && ip rule del pref 32000 2>/dev/null || true + } + + configure_routes "$GW_MAC" + ping -c 2 8.8.8.8 >/dev/null 2>&1 && echo "✓ Network OK" + + CHECK=0 + FIX=0 + + while true; do + [ $CHECK -lt 60 ] && INT=2 || INT=10 + + NEED=0 + ! ip route show table 97 | grep -q "^default" && NEED=1 + ip rule show | grep -q "32000.*unreachable" && NEED=1 + + [ $NEED -eq 1 ] && { FIX=$((FIX+1)); configure_routes "$GW_MAC"; } + + CHECK=$((CHECK+1)) + sleep $INT + done + {{- with .Values.redroid.routeSetup.resources }} + resources: + {{- toYaml . | nindent 10 }} + {{- end }} + securityContext: + privileged: true + capabilities: + add: + - NET_ADMIN + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File +{{- end }} diff --git a/helm-charts/mobile-sandbox/upgrade_kernel_to_6.8.0-87.sh b/helm-charts/mobile-sandbox/upgrade_kernel_to_6.8.0-87.sh new file mode 100644 index 0000000..4db587c --- /dev/null +++ b/helm-charts/mobile-sandbox/upgrade_kernel_to_6.8.0-87.sh @@ -0,0 +1,192 @@ +#!/bin/bash + +################################################################################ +# Ubuntu 24.04 Kernel Upgrade Script +# Target: linux-image-6.8.0-87-generic +################################################################################ + +set -euo pipefail + +# Configuration +readonly TARGET_KERNEL="6.8.0-87-generic" +readonly LOG_FILE="/var/log/kernel_upgrade_$(date +%Y%m%d_%H%M%S).log" +readonly MIN_BOOT_SPACE_MB=512 + +# Colors +readonly RED='\033[0;31m' +readonly GREEN='\033[0;32m' +readonly YELLOW='\033[1;33m' +readonly NC='\033[0m' + +################################################################################ +# Functions +################################################################################ + +log() { + local level=$1 + shift + local color=${NC} + case $level in + INFO) color=${GREEN} ;; + WARN) color=${YELLOW} ;; + ERROR) color=${RED} ;; + esac + echo -e "${color}[$level]${NC} $(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$LOG_FILE" +} + +check_root() { + [[ $EUID -eq 0 ]] || { log ERROR "Must run as root"; exit 1; } +} + +check_os() { + grep -q "Ubuntu 24.04" /etc/os-release || { + log WARN "Designed for Ubuntu 24.04, current OS may differ" + read -p "Continue? (yes/no): " -r + [[ $REPLY =~ ^[Yy]es$ ]] || { log INFO "Cancelled"; exit 0; } + } +} + +check_disk_space() { + # Note: No separate /boot partition, using root partition + local available_mb=$(df / | tail -1 | awk '{print int($4/1024)}') + + if [[ $available_mb -lt $MIN_BOOT_SPACE_MB ]]; then + log ERROR "Insufficient disk space. Available: ${available_mb}MB, Required: ${MIN_BOOT_SPACE_MB}MB" + exit 1 + fi + log INFO "Disk space check passed: ${available_mb}MB available" +} + +get_current_kernel() { + CURRENT_KERNEL=$(uname -r) + log INFO "Current kernel: $CURRENT_KERNEL" +} + +install_kernel() { + log INFO "Installing kernel $TARGET_KERNEL..." + + apt update >> "$LOG_FILE" 2>&1 || { log ERROR "apt update failed"; exit 1; } + + # Check availability + apt-cache search "linux-image-$TARGET_KERNEL" | grep -q "linux-image-$TARGET_KERNEL" || { + log ERROR "Kernel $TARGET_KERNEL not found in repositories" + exit 1 + } + + # Install packages + local packages=( + "linux-image-$TARGET_KERNEL" + "linux-headers-$TARGET_KERNEL" + "linux-modules-$TARGET_KERNEL" + "linux-modules-extra-$TARGET_KERNEL" + ) + + DEBIAN_FRONTEND=noninteractive apt install -y "${packages[@]}" >> "$LOG_FILE" 2>&1 || { + log ERROR "Kernel installation failed" + exit 1 + } + + log INFO "Kernel installed successfully" +} + +update_grub() { + log INFO "Updating GRUB configuration..." + update-grub >> "$LOG_FILE" 2>&1 || log WARN "GRUB update had warnings" + + grep -q "vmlinuz-$TARGET_KERNEL" /boot/grub/grub.cfg && \ + log INFO "GRUB configuration verified" || \ + log WARN "Target kernel not found in GRUB config" +} + +cleanup_old_kernels() { + log INFO "Checking old kernels..." + + local old_kernels=$(dpkg -l | \ + awk '/^ii.*linux-image-[0-9]/ && !/'"$TARGET_KERNEL"'/ && !/'"$CURRENT_KERNEL"'/ {print $2}') + + if [[ -z "$old_kernels" ]]; then + log INFO "No old kernels to remove" + return + fi + + log INFO "Found old kernels:" + echo "$old_kernels" + read -p "Remove old kernels? (yes/no): " -r + + if [[ $REPLY =~ ^[Yy]es$ ]]; then + echo "$old_kernels" | xargs apt remove -y >> "$LOG_FILE" 2>&1 + apt autoremove -y >> "$LOG_FILE" 2>&1 + log INFO "Old kernels removed" + fi +} + +create_verification_script() { + cat > /root/verify_kernel.sh << EOF +#!/bin/bash +EXPECTED="$TARGET_KERNEL" +CURRENT=\$(uname -r) + +if [[ "\$CURRENT" == "\$EXPECTED" ]]; then + echo "[SUCCESS] Kernel upgraded to \$CURRENT" + rm -f /root/verify_kernel.sh +else + echo "[FAILED] Current: \$CURRENT, Expected: \$EXPECTED" + exit 1 +fi +EOF + chmod +x /root/verify_kernel.sh + log INFO "Verification script created: /root/verify_kernel.sh" +} + +prompt_reboot() { + cat << EOF + +======================================== +Kernel Upgrade Summary +======================================== +Previous: $CURRENT_KERNEL +Target: $TARGET_KERNEL +Log: $LOG_FILE +======================================== + +EOF + + read -p "Reboot now? (yes/no): " -r + if [[ $REPLY =~ ^[Yy]es$ ]]; then + log INFO "Rebooting..." + sleep 3 + reboot + else + log WARN "Reboot required. Run manually: sudo reboot" + echo "After reboot, verify with: sudo /root/verify_kernel.sh" + fi +} + +################################################################################ +# Main +################################################################################ + +main() { + log INFO "Starting kernel upgrade to $TARGET_KERNEL" + + check_root + check_os + check_disk_space + get_current_kernel + + # Check if already on target kernel + if [[ "$CURRENT_KERNEL" == "$TARGET_KERNEL" ]]; then + log INFO "Already running $TARGET_KERNEL" + exit 0 + fi + + install_kernel + update_grub + create_verification_script + cleanup_old_kernels + prompt_reboot + + log INFO "Upgrade completed" +} + +main "$@" diff --git a/helm-charts/mobile-sandbox/values.yaml b/helm-charts/mobile-sandbox/values.yaml new file mode 100644 index 0000000..40c349b --- /dev/null +++ b/helm-charts/mobile-sandbox/values.yaml @@ -0,0 +1,203 @@ +# Default values for mobile-sandbox +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +# Override chart name +nameOverride: "" +fullnameOverride: "" + +# Redroid StatefulSet Configuration +redroid: + # Enable/disable the redroid StatefulSet + enabled: true + + # Number of replicas + replicas: 1 + + # Pod Management Policy + podManagementPolicy: OrderedReady + + # PVC Retention Policy + persistentVolumeClaimRetentionPolicy: + whenDeleted: Retain + whenScaled: Retain + + # Labels for the StatefulSet + labels: + k8s-app: redroid + qcloud-app: redroid + + # Pod annotations + podAnnotations: + tke.cloud.tencent.com/eip-attributes: '{"Bandwidth":"100","ISP":"BGP"}' + tke.cloud.tencent.com/eip-claim-delete-policy: Never + tke.cloud.tencent.com/networks: tke-route-eni + + # Pod labels + podLabels: + k8s-app: redroid + qcloud-app: redroid + + # Node affinity configuration + nodeAffinity: + enabled: true + nodepoolIds: + - np-xxxxxxxx + + # Redroid container configuration + container: + image: + repository: erstt/redroid + tag: 11.0.0_houdini_ChromeOS + pullPolicy: IfNotPresent + + # Android boot arguments + args: + - ro.enable.native.bridge.exec64=1 + - ro.dalvik.vm.native.bridge=libhoudini.so + - androidboot.redroid_gpu_mode=guest + - androidboot.redroid_width=720 + - androidboot.redroid_height=1280 + - androidboot.redroid_dpi=320 + - androidboot.redroid_fps=15 + - androidboot.use_memfd=true + - androidboot.redroid_net_ndns=2 + - androidboot.redroid_net_dns1=8.8.8.8 + - androidboot.redroid_net_dns2=8.8.4.4 + + # Environment variables + env: + - name: ANDROID_SKIP_BORINGSSL_SELF_TEST + value: "1" + + # Container ports + ports: + - containerPort: 5555 + name: adb + protocol: TCP + + # Resource requests and limits + resources: + requests: + cpu: 6500m + memory: 10Gi + tke.cloud.tencent.com/eip: "1" + tke.cloud.tencent.com/eni-ip: "1" + limits: + cpu: 6500m + memory: 10Gi + tke.cloud.tencent.com/eip: "1" + tke.cloud.tencent.com/eni-ip: "1" + + # Route setup sidecar configuration + routeSetup: + image: + repository: alpine + tag: latest + pullPolicy: IfNotPresent + + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 50m + memory: 1Gi + + # Logcat sidecar configuration + logcatSidecar: + image: + repository: sorccu/adb + tag: latest + pullPolicy: IfNotPresent + + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi + +# Module Loader DaemonSet Configuration +moduleLoader: + # Enable/disable the module loader DaemonSet + enabled: true + + # Labels for the DaemonSet + labels: + app: redroid-ubuntu-module-loader + + # Node affinity configuration + nodeAffinity: + enabled: true + nodepoolIds: + - np-xxxxxxxx + - np-yyyyyyyy + + # Tolerations for scheduling on all nodes + tolerations: + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists + + # Init container configuration (installs kernel modules) + initContainer: + image: + repository: ubuntu + tag: "22.04" + pullPolicy: IfNotPresent + + # Main container configuration (loads kernel modules) + container: + image: + repository: alpine + tag: latest + pullPolicy: IfNotPresent + + # APK mirror for faster package installation + apkMirror: mirrors.cloud.tencent.com + + resources: + requests: + cpu: 50m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi + +# Image pull secrets +imagePullSecrets: [] + +# Service configuration +service: + enabled: false + type: ClusterIP + port: 5555 + targetPort: adb + +# DNS Policy +dnsPolicy: ClusterFirst + +# Restart Policy +restartPolicy: Always + +# Termination Grace Period +terminationGracePeriodSeconds: 30 + +# Security Context +securityContext: {} + +# Update Strategy for StatefulSet +updateStrategy: + type: RollingUpdate + rollingUpdate: + partition: 0 + +# Update Strategy for DaemonSet +daemonSetUpdateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 0