Open Issues Need Help
View All on GitHubAI Summary: This issue proposes implementing an AI-powered control mode for iOS app interactions, leveraging Vision Language Models (VLMs) to understand screen content and execute actions. It outlines two main approaches: integrating Gemini Live for real-time streaming or a more generic VLM post-request architecture, alongside core components like an efficient screen capture system, a VLM integration layer, and an action execution framework.
🏋️♂️ OpenAI Gym-like environment for iOS app testing and automation through iPhone Mirroring on macOS
AI Summary: This issue proposes implementing a comprehensive data persistence layer and a machine learning pipeline for the iOS Testing Gym app. The goal is to collect, store, and analyze user interaction recordings to train models that improve automated testing capabilities, encompassing local and cloud storage, data modeling, analytics, model training, and privacy compliance.
🏋️♂️ OpenAI Gym-like environment for iOS app testing and automation through iPhone Mirroring on macOS