diff --git a/Projects/Projects/Edge-AI-On-Mobile.md b/Projects/Projects/Edge-AI-On-Mobile.md index 56bede25..b1dc74db 100644 --- a/Projects/Projects/Edge-AI-On-Mobile.md +++ b/Projects/Projects/Edge-AI-On-Mobile.md @@ -43,17 +43,20 @@ Other devices also support SME2, including both Apple and Android - [see the ful ### Project Summary -Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing enabled by SME2. Build and optimize a proof-of-concept application on a vivo X300 phone or other device supporting SME2. +Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing accelerated by SME2, with real-time speech-to-speech translation, profanity filtering or filler-word removal, and on-device Small Language Models (SLMs) as key example use cases. Example project areas: - - Real-time video semantic segmentation (e.g., background removal + AR compositing) - - Live object detection + natural-language description (text summary of what the camera sees) - - Multi-sensor fusion (camera + IMU + microphone) for gesture + voice recognition - - On-device lightweight LLM or encoder-only transformer processing for mobile assistants +- Real-time speech-to-speech translation +- On-device SLM for contextual understanding, rewriting, or assistant tasks +- Profanity detection and filtering (masking, rewriting, audio bleeping) +- Filler-word removal / speech clean-up (e.g., removing “um”, “uh”, false starts) +- Real-time video semantic segmentation (e.g., AR captions + background removal) +- Live object detection with natural-language description +- Multi-sensor fusion (camera + IMU + microphone) for gesture + voice interaction Identify a model architecture that maps to wide matrix operations (e.g., ViT, MLP-Mixer, multi-branch CNN with large FC layers). Utilise a mobile-friendly framework (e.g., ExecuTorch, LiteRT, ONNX Runtime, MediaPipe) to leverage SME2 optimizations. Optimize quantization, memory layout, and verify that the large matrix multiplications get scheduled efficiently on the SME2-enabled CPU. Build a mobile app (Android or iOS) that executes the model and utilises it for a compelling use-case. -Utilise the resources and learning paths below and create an exciting and challenging application. Optionally, you could also compare performance vs a reference phone without SME2. +Utilise the resources and learning paths below and create an exciting and challenging application. As a bonus, you could compare performance vs a reference phone without SME2. ## Resources from Arm and our partners diff --git a/docs/_data/navigation.yml b/docs/_data/navigation.yml index 8184e5f4..037a258d 100644 --- a/docs/_data/navigation.yml +++ b/docs/_data/navigation.yml @@ -5,19 +5,19 @@ header: projects: - title: Projects children: - - title: Machine-Learning-on-AWS-Graviton - description: "This self-service project ports and tunes OpenSora text-to-video\ - \ transformers on AWS Graviton CPUs\u2014showcasing cost-efficient, quantized,\ - \ CPU-only inference pipelines and guiding best-practice optimization for Arm-based\ - \ cloud AI workloads." - url: /2025/05/30/Machine-Learning-on-AWS-Graviton.html + - title: Academic-Trends-Dashboard + description: "This self-service project creates a web-scraping, database-driven\ + \ dashboard that visualizes how computer-science research topics shift over\ + \ time\u2014helping Arm partners and chip architects align future hardware designs\ + \ with emerging algorithmic trends." + url: /2025/05/30/Academic-Trends-Dashboard.html subjects: - - ML - - Migration to Arm - - Performance and Architecture - - Cloud AI + - Web + - Databases platform: - Servers and Cloud Computing + - Laptops and Desktops + - Mobile, Graphics, and Gaming - AI sw-hw: - Software @@ -26,39 +26,43 @@ projects: - Arm Ambassador Support status: - Published - - title: R-Arm-Community-Support - description: "This self-service project boosts the R ecosystem on Windows on Arm\ - \ by identifying unsupported packages, upstreaming fixes, and automating builds\u2014\ - so data scientists can run their workflows natively on fast, efficient Arm64\ - \ laptops and desktops." - url: /2025/05/30/R-Arm-Community-Support.html + - title: Always-On-AI-with-Ethos-U85-NPU + description: The vision of Edge AI compute is to embed low-power intelligent sensing, + perception, and decision systems everywhere. A low-power always-on-AI island + continuously monitors sensory inputs to detect triggers. When a trigger is detected, + it wakes up a more capable processor to carry out high-value inference, interaction, + or control tasks. + url: /2025/11/27/Always-On-AI-with-Ethos-U85-NPU.html subjects: + - ML - Performance and Architecture - - Migration to Arm - - Libraries + - Embedded Linux + - RTOS Fundamentals + - Edge AI + - Physical AI platform: - - Laptops and Desktops + - IoT + - Embedded and Microcontrollers + - AI sw-hw: - Software + - Hardware support-level: - Self-Service - Arm Ambassador Support status: - Published - - title: Video-&-Audio-Provenance-In-The-Age-of-AI - description: Integrating transparent provenance - disclosing whether media is - AI-generated or AI-edited, and what other AI processing has occurred on any - media - is fundamental for accountability in domains like journalism, security, - and regulated industries. This project uses C2PA specification (www.c2pa.org) - revision 2.3 to record such actions as signed, machine-verifiable assertions - attached to the asset. - url: /2026/02/12/Video-&-Audio-Provenance-In-The-Age-of-AI.html + - title: Python-Porting-Challenge + description: "This challenge focuses on enabling Python support for Windows on\ + \ Arm (WoA) to improve developer experience. While Python is widely used in\ + \ research and industry, many popular packages\u2014such as Pandas\u2014still\ + \ lack pre-built WoA binaries (win_arm64 wheels). The goal is to validate and\ + \ optimise third-party packages, fix compatibility issues, and collaborate with\ + \ maintainers to upstream WoA support." + url: /2025/11/03/Python-Porting-Challenge.html subjects: - - ML - - Security - - Edge AI + - Libraries platform: - - AI - Laptops and Desktops sw-hw: - Software @@ -67,19 +71,19 @@ projects: - Arm Ambassador Support status: - Published - - title: AI-Agents - description: "This self-service project builds a sandboxed AI agent on Arm hardware\ - \ that harnesses appropriately sized LLMs to safely automate complex workflows\u2014\ - from DevOps pipelines to e-commerce tasks\u2014demonstrating secure, efficient\ - \ automation on accessible Arm platforms." - url: /2025/05/30/AI-Agents.html + - title: Machine-Learning-on-AWS-Graviton + description: "This self-service project ports and tunes OpenSora text-to-video\ + \ transformers on AWS Graviton CPUs\u2014showcasing cost-efficient, quantized,\ + \ CPU-only inference pipelines and guiding best-practice optimization for Arm-based\ + \ cloud AI workloads." + url: /2025/05/30/Machine-Learning-on-AWS-Graviton.html subjects: - ML - - Edge AI + - Migration to Arm + - Performance and Architecture - Cloud AI platform: - Servers and Cloud Computing - - Laptops and Desktops - AI sw-hw: - Software @@ -88,18 +92,19 @@ projects: - Arm Ambassador Support status: - Published - - title: Haskell-Compiler-Windows-on-Arm - description: "This self-service project brings native Glasgow Haskell Compiler\ - \ support to Windows on Arm\u2014unlocking efficient Arm-laptop builds, extending\ - \ Haskell\u2019s reach, and giving contributors hands-on experience with Arm64\ - \ code generation and runtime integration." - url: /2025/05/30/Haskell-Compiler-Windows-on-Arm.html + - title: SpecINT2017-benchmarking-on-Arm64 + description: "This self-service project profiles SPEC CPU2017 on Arm64 servers\u2014\ + using GCC, Clang, and Arm Compiler with top-down analysis\u2014to reveal how\ + \ compiler choices and Arm micro-architectural features impact execution time,\ + \ energy efficiency, and performance bottlenecks." + url: /2025/05/30/SpecINT2017-benchmarking-on-Arm64.html subjects: - - Migration to Arm - Performance and Architecture + - Migration to Arm platform: - Servers and Cloud Computing - Laptops and Desktops + - AI sw-hw: - Software - Hardware @@ -108,16 +113,18 @@ projects: - Arm Ambassador Support status: - Published - - title: C-Based-Application-from-Scratch - description: This self-service project goes back to the fundamentals. The challenge - is to develop an application of your choice but your are only permitted to use - the C language with as few dependencies as possible. - url: /2025/07/11/C-Based-Application-from-Scratch.html + - title: R-Arm-Community-Support + description: "This self-service project boosts the R ecosystem on Windows on Arm\ + \ by identifying unsupported packages, upstreaming fixes, and automating builds\u2014\ + so data scientists can run their workflows natively on fast, efficient Arm64\ + \ laptops and desktops." + url: /2025/05/30/R-Arm-Community-Support.html subjects: - Performance and Architecture + - Migration to Arm - Libraries platform: - - IoT + - Laptops and Desktops sw-hw: - Software support-level: @@ -125,19 +132,18 @@ projects: - Arm Ambassador Support status: - Published - - title: NPC-LLM-Runtime - description: This self-service project explores novel ways of integrating Large - Language Models (LLMs) into real-time gameplay to drive dynamic Non-Playable - Character (NPC) interactions. - url: /2025/08/28/NPC-LLM-Runtime.html + - title: Bioinformatic-Pipeline-Analysis + description: "This self-service project benchmarks Arm64 Bioconda packages in\ + \ real nf-core workflows\u2014measuring performance, diagnosing build failures,\ + \ and proposing fixes that accelerate truly native bioinformatics on the expanding\ + \ fleet of Arm-powered machines." + url: /2025/05/30/Bioinformatic-Pipeline-Analysis.html subjects: - - ML - - Gaming - - Graphics - - Edge AI + - Performance and Architecture + - Databases platform: - - AI - - Mobile, Graphics, and Gaming + - Servers and Cloud Computing + - Laptops and Desktops sw-hw: - Software support-level: @@ -145,41 +151,39 @@ projects: - Arm Ambassador Support status: - Published - - title: Python-Porting-Challenge - description: "This challenge focuses on enabling Python support for Windows on\ - \ Arm (WoA) to improve developer experience. While Python is widely used in\ - \ research and industry, many popular packages\u2014such as Pandas\u2014still\ - \ lack pre-built WoA binaries (win_arm64 wheels). The goal is to validate and\ - \ optimise third-party packages, fix compatibility issues, and collaborate with\ - \ maintainers to upstream WoA support." - url: /2025/11/03/Python-Porting-Challenge.html + - title: Processor-in-the-Loop-Automotive + description: Verify a Simulink automotive controller by running processor-in-the-loop + (PIL) tests on a virtual Arm Cortex M7 processor. + url: /2025/05/30/Processor-in-the-Loop-Automotive.html subjects: - - Libraries + - Embedded Linux + - RTOS Fundamentals + - Virtual Hardware platform: - Laptops and Desktops + - Automotive + - Embedded and Microcontrollers sw-hw: - Software + - Hardware support-level: - Self-Service - Arm Ambassador Support status: - Published - - title: AI-Powered-Porting-Tool - description: "This self-service project creates an AI-driven porting engine that\ - \ analyzes package dependencies, auto-generates fixes, and submits pull requests\u2014\ - accelerating native macOS and Windows-on-Arm support for bioinformatics and\ - \ R software so researchers can run demanding workflows directly on modern Arm\ - \ devices." - url: /2025/05/30/AI-Powered-Porting-Tool.html + - title: AI-Agents + description: "This self-service project builds a sandboxed AI agent on Arm hardware\ + \ that harnesses appropriately sized LLMs to safely automate complex workflows\u2014\ + from DevOps pipelines to e-commerce tasks\u2014demonstrating secure, efficient\ + \ automation on accessible Arm platforms." + url: /2025/05/30/AI-Agents.html subjects: - - CI-CD - ML - - Migration to Arm - Edge AI + - Cloud AI platform: - Servers and Cloud Computing - Laptops and Desktops - - Mobile, Graphics, and Gaming - AI sw-hw: - Software @@ -209,18 +213,16 @@ projects: - Arm Ambassador Support status: - Published - - title: Responsible-AI-and-Yellow-Teaming - description: "This self-service project equips teams with a YellowTeamGPT workflow\ - \ that probes Arm-based AI products for unintended impacts\u2014turning responsible-AI\ - \ stress-testing into a core step of the development cycle." - url: /2025/05/30/Responsible-AI-and-Yellow-Teaming.html + - title: C-Based-Application-from-Scratch + description: This self-service project goes back to the fundamentals. The challenge + is to develop an application of your choice but your are only permitted to use + the C language with as few dependencies as possible. + url: /2025/07/11/C-Based-Application-from-Scratch.html subjects: - - ML - - Cloud AI + - Performance and Architecture + - Libraries platform: - - Servers and Cloud Computing - - Laptops and Desktops - - AI + - IoT sw-hw: - Software support-level: @@ -228,39 +230,43 @@ projects: - Arm Ambassador Support status: - Published - - title: SpecINT2017-benchmarking-on-Arm64 - description: "This self-service project profiles SPEC CPU2017 on Arm64 servers\u2014\ - using GCC, Clang, and Arm Compiler with top-down analysis\u2014to reveal how\ - \ compiler choices and Arm micro-architectural features impact execution time,\ - \ energy efficiency, and performance bottlenecks." - url: /2025/05/30/SpecINT2017-benchmarking-on-Arm64.html + - title: Game-Dev-Using-Neural-Graphics-&-Unreal-Engine + description: "Build a playable Unreal Engine 5 game demo that utilises Arm\u2019\ + s Neural Graphics SDK UE plugin for features such as Neural Super Sampling (NSS).\ + \ Showcase near-identical image quality at lower resolution by driving neural\ + \ rendering directly in the graphics pipeline." + url: /2025/11/27/Game-Dev-Using-Neural-Graphics-&-Unreal-Engine.html subjects: - - Performance and Architecture - - Migration to Arm + - ML + - Gaming + - Libraries + - Graphics + - Edge AI platform: - - Servers and Cloud Computing + - Mobile, Graphics, and Gaming - Laptops and Desktops - AI sw-hw: - Software - - Hardware support-level: - Self-Service - Arm Ambassador Support status: - Published - - title: Processor-in-the-Loop-Automotive - description: Verify a Simulink automotive controller by running processor-in-the-loop - (PIL) tests on a virtual Arm Cortex M7 processor. - url: /2025/05/30/Processor-in-the-Loop-Automotive.html + - title: Ethos-U85-NPU-Applications + description: Push the limits of Edge AI by deploying the heaviest inference applications + possible on Ethos-U85. Students will explore transformer-based and TOSA-optimized + workloads that demonstrate performance levels on the next-gen of Ethos NPUs. + url: /2025/11/27/Ethos-U85-NPU-Applications.html subjects: - - Embedded Linux - - RTOS Fundamentals - - Virtual Hardware + - ML + - Performance and Architecture + - Edge AI + - Physical AI platform: - - Laptops and Desktops - - Automotive + - IoT - Embedded and Microcontrollers + - AI sw-hw: - Software - Hardware @@ -293,42 +299,65 @@ projects: - Arm Ambassador Support status: - Published - - title: Bioinformatic-Pipeline-Analysis - description: "This self-service project benchmarks Arm64 Bioconda packages in\ - \ real nf-core workflows\u2014measuring performance, diagnosing build failures,\ - \ and proposing fixes that accelerate truly native bioinformatics on the expanding\ - \ fleet of Arm-powered machines." - url: /2025/05/30/Bioinformatic-Pipeline-Analysis.html + - title: Computational-Photography + description: This project creates and implements a novel computational photography + pipeline that is optimized for Arm-based mobile devices using SME2 and neural + technology. This project comes with the possibility of a hardware donation to + support your efforts + url: /2026/01/07/Computational-Photography.html subjects: - - Performance and Architecture - - Databases + - ML + - Graphics + - Edge AI platform: - - Servers and Cloud Computing - Laptops and Desktops + - AI sw-hw: - Software support-level: - Self-Service - Arm Ambassador Support + - Direct Support from Arm status: - Published - - title: Ethos-U85-NPU-Applications - description: Push the limits of Edge AI by deploying the heaviest inference applications - possible on Ethos-U85. Students will explore transformer-based and TOSA-optimized - workloads that demonstrate performance levels on the next-gen of Ethos NPUs. - url: /2025/11/27/Ethos-U85-NPU-Applications.html + - title: Edge-AI-On-Mobile + description: Leverage the latest SME2 (Scalable Matrix Extension 2) available + on the newest vivo X300 smartphones (built on Arm Lumex CSS) or other SME2-enabled + devices for advanced image/video, audio and text processing edge AI. Explore + how SME2, via KleidiAI, enables larger matrix workloads, higher throughput, + and novel applications on device without cloud connectivity required. + url: /2025/11/27/Edge-AI-On-Mobile.html subjects: - ML - Performance and Architecture + - Libraries - Edge AI - - Physical AI platform: + - Mobile, Graphics, and Gaming + - AI - IoT - - Embedded and Microcontrollers + sw-hw: + - Software + support-level: + - Self-Service + - Arm Ambassador Support + status: + - Published + - title: HPC-Algorithm + description: "This self-service project is around finding a HPC algorithm and\ + \ accelerating it with Arm\u2019s SVE/SVE2 vectorization\u2014demonstrating\ + \ how next-generation Arm hardware can deliver significant, scalable performance\ + \ gains." + url: /2025/05/30/HPC-Algorithm.html + subjects: + - Performance and Architecture + - Cloud AI + platform: + - Servers and Cloud Computing + - Laptops and Desktops - AI sw-hw: - Software - - Hardware support-level: - Self-Service - Arm Ambassador Support @@ -354,36 +383,38 @@ projects: - Arm Ambassador Support status: - Published - - title: Computational-Photography - description: This project creates and implements a novel computational photography - pipeline that is optimized for Arm-based mobile devices using SME2 and neural - technology. This project comes with the possibility of a hardware donation to - support your efforts - url: /2026/01/07/Computational-Photography.html + - title: Haskell-Compiler-Windows-on-Arm + description: "This self-service project brings native Glasgow Haskell Compiler\ + \ support to Windows on Arm\u2014unlocking efficient Arm-laptop builds, extending\ + \ Haskell\u2019s reach, and giving contributors hands-on experience with Arm64\ + \ code generation and runtime integration." + url: /2025/05/30/Haskell-Compiler-Windows-on-Arm.html subjects: - - ML - - Graphics - - Edge AI + - Migration to Arm + - Performance and Architecture platform: + - Servers and Cloud Computing - Laptops and Desktops - - AI sw-hw: - Software + - Hardware support-level: - Self-Service - Arm Ambassador Support - - Direct Support from Arm status: - Published - - title: Academic-Trends-Dashboard - description: "This self-service project creates a web-scraping, database-driven\ - \ dashboard that visualizes how computer-science research topics shift over\ - \ time\u2014helping Arm partners and chip architects align future hardware designs\ - \ with emerging algorithmic trends." - url: /2025/05/30/Academic-Trends-Dashboard.html + - title: AI-Powered-Porting-Tool + description: "This self-service project creates an AI-driven porting engine that\ + \ analyzes package dependencies, auto-generates fixes, and submits pull requests\u2014\ + accelerating native macOS and Windows-on-Arm support for bioinformatics and\ + \ R software so researchers can run demanding workflows directly on modern Arm\ + \ devices." + url: /2025/05/30/AI-Powered-Porting-Tool.html subjects: - - Web - - Databases + - CI-CD + - ML + - Migration to Arm + - Edge AI platform: - Servers and Cloud Computing - Laptops and Desktops @@ -396,42 +427,19 @@ projects: - Arm Ambassador Support status: - Published - - title: Edge-AI-On-Mobile - description: Leverage the latest SME2 (Scalable Matrix Extension 2) available - on the newest vivo X300 smartphones (built on Arm Lumex CSS) or other SME2-enabled - devices for advanced image/video, audio and text processing edge AI. Explore - how SME2, via KleidiAI, enables larger matrix workloads, higher throughput, - and novel applications on device without cloud connectivity required. - url: /2025/11/27/Edge-AI-On-Mobile.html + - title: NPC-LLM-Runtime + description: This self-service project explores novel ways of integrating Large + Language Models (LLMs) into real-time gameplay to drive dynamic Non-Playable + Character (NPC) interactions. + url: /2025/08/28/NPC-LLM-Runtime.html subjects: - ML - - Performance and Architecture - - Libraries + - Gaming + - Graphics - Edge AI platform: - - Mobile, Graphics, and Gaming - - AI - - IoT - sw-hw: - - Software - support-level: - - Self-Service - - Arm Ambassador Support - status: - - Published - - title: HPC-Algorithm - description: "This self-service project is around finding a HPC algorithm and\ - \ accelerating it with Arm\u2019s SVE/SVE2 vectorization\u2014demonstrating\ - \ how next-generation Arm hardware can deliver significant, scalable performance\ - \ gains." - url: /2025/05/30/HPC-Algorithm.html - subjects: - - Performance and Architecture - - Cloud AI - platform: - - Servers and Cloud Computing - - Laptops and Desktops - AI + - Mobile, Graphics, and Gaming sw-hw: - Software support-level: @@ -461,48 +469,40 @@ projects: - Arm Ambassador Support status: - Published - - title: Always-On-AI-with-Ethos-U85-NPU - description: The vision of Edge AI compute is to embed low-power intelligent sensing, - perception, and decision systems everywhere. A low-power always-on-AI island - continuously monitors sensory inputs to detect triggers. When a trigger is detected, - it wakes up a more capable processor to carry out high-value inference, interaction, - or control tasks. - url: /2025/11/27/Always-On-AI-with-Ethos-U85-NPU.html + - title: Responsible-AI-and-Yellow-Teaming + description: "This self-service project equips teams with a YellowTeamGPT workflow\ + \ that probes Arm-based AI products for unintended impacts\u2014turning responsible-AI\ + \ stress-testing into a core step of the development cycle." + url: /2025/05/30/Responsible-AI-and-Yellow-Teaming.html subjects: - ML - - Performance and Architecture - - Embedded Linux - - RTOS Fundamentals - - Edge AI - - Physical AI + - Cloud AI platform: - - IoT - - Embedded and Microcontrollers + - Servers and Cloud Computing + - Laptops and Desktops - AI sw-hw: - Software - - Hardware support-level: - Self-Service - Arm Ambassador Support status: - Published - - title: Game-Dev-Using-Neural-Graphics-&-Unreal-Engine - description: "Build a playable Unreal Engine 5 game demo that utilises Arm\u2019\ - s Neural Graphics SDK UE plugin for features such as Neural Super Sampling (NSS).\ - \ Showcase near-identical image quality at lower resolution by driving neural\ - \ rendering directly in the graphics pipeline." - url: /2025/11/27/Game-Dev-Using-Neural-Graphics-&-Unreal-Engine.html + - title: Video-&-Audio-Provenance-In-The-Age-of-AI + description: Integrating transparent provenance - disclosing whether media is + AI-generated or AI-edited, and what other AI processing has occurred on any + media - is fundamental for accountability in domains like journalism, security, + and regulated industries. This project uses C2PA specification (www.c2pa.org) + revision 2.3 to record such actions as signed, machine-verifiable assertions + attached to the asset. + url: /2026/02/12/Video-&-Audio-Provenance-In-The-Age-of-AI.html subjects: - ML - - Gaming - - Libraries - - Graphics + - Security - Edge AI platform: - - Mobile, Graphics, and Gaming - - Laptops and Desktops - AI + - Laptops and Desktops sw-hw: - Software support-level: diff --git a/docs/_posts/2025-11-27-Edge-AI-On-Mobile.md b/docs/_posts/2025-11-27-Edge-AI-On-Mobile.md index c4a6d441..a33fcd67 100644 --- a/docs/_posts/2025-11-27-Edge-AI-On-Mobile.md +++ b/docs/_posts/2025-11-27-Edge-AI-On-Mobile.md @@ -45,17 +45,20 @@ full_description: |- ### Project Summary - Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing enabled by SME2. Build and optimize a proof-of-concept application on a vivo X300 phone or other device supporting SME2. + Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing accelerated by SME2, with real-time speech-to-speech translation, profanity filtering or filler-word removal, and on-device Small Language Models (SLMs) as key example use cases. Example project areas: - - Real-time video semantic segmentation (e.g., background removal + AR compositing) - - Live object detection + natural-language description (text summary of what the camera sees) - - Multi-sensor fusion (camera + IMU + microphone) for gesture + voice recognition - - On-device lightweight LLM or encoder-only transformer processing for mobile assistants + - Real-time speech-to-speech translation + - On-device SLM for contextual understanding, rewriting, or assistant tasks + - Profanity detection and filtering (masking, rewriting, audio bleeping) + - Filler-word removal / speech clean-up (e.g., removing “um”, “uh”, false starts) + - Real-time video semantic segmentation (e.g., AR captions + background removal) + - Live object detection with natural-language description + - Multi-sensor fusion (camera + IMU + microphone) for gesture + voice interaction Identify a model architecture that maps to wide matrix operations (e.g., ViT, MLP-Mixer, multi-branch CNN with large FC layers). Utilise a mobile-friendly framework (e.g., ExecuTorch, LiteRT, ONNX Runtime, MediaPipe) to leverage SME2 optimizations. Optimize quantization, memory layout, and verify that the large matrix multiplications get scheduled efficiently on the SME2-enabled CPU. Build a mobile app (Android or iOS) that executes the model and utilises it for a compelling use-case. - Utilise the resources and learning paths below and create an exciting and challenging application. Optionally, you could also compare performance vs a reference phone without SME2. + Utilise the resources and learning paths below and create an exciting and challenging application. As a bonus, you could compare performance vs a reference phone without SME2. ## Resources from Arm and our partners @@ -100,17 +103,20 @@ Other devices also support SME2, including both Apple and Android - [see the ful ### Project Summary -Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing enabled by SME2. Build and optimize a proof-of-concept application on a vivo X300 phone or other device supporting SME2. +Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing accelerated by SME2, with real-time speech-to-speech translation, profanity filtering or filler-word removal, and on-device Small Language Models (SLMs) as key example use cases. Example project areas: - - Real-time video semantic segmentation (e.g., background removal + AR compositing) - - Live object detection + natural-language description (text summary of what the camera sees) - - Multi-sensor fusion (camera + IMU + microphone) for gesture + voice recognition - - On-device lightweight LLM or encoder-only transformer processing for mobile assistants +- Real-time speech-to-speech translation +- On-device SLM for contextual understanding, rewriting, or assistant tasks +- Profanity detection and filtering (masking, rewriting, audio bleeping) +- Filler-word removal / speech clean-up (e.g., removing “um”, “uh”, false starts) +- Real-time video semantic segmentation (e.g., AR captions + background removal) +- Live object detection with natural-language description +- Multi-sensor fusion (camera + IMU + microphone) for gesture + voice interaction Identify a model architecture that maps to wide matrix operations (e.g., ViT, MLP-Mixer, multi-branch CNN with large FC layers). Utilise a mobile-friendly framework (e.g., ExecuTorch, LiteRT, ONNX Runtime, MediaPipe) to leverage SME2 optimizations. Optimize quantization, memory layout, and verify that the large matrix multiplications get scheduled efficiently on the SME2-enabled CPU. Build a mobile app (Android or iOS) that executes the model and utilises it for a compelling use-case. -Utilise the resources and learning paths below and create an exciting and challenging application. Optionally, you could also compare performance vs a reference phone without SME2. +Utilise the resources and learning paths below and create an exciting and challenging application. As a bonus, you could compare performance vs a reference phone without SME2. ## Resources from Arm and our partners diff --git a/docs/assets/badges/Trending.svg b/docs/assets/badges/Trending.svg index 0b45f07d..7563c89e 100644 --- a/docs/assets/badges/Trending.svg +++ b/docs/assets/badges/Trending.svg @@ -1 +1,4 @@ -TrendingTrending \ No newline at end of file + + + Spotlight + \ No newline at end of file