How Apple Suggests Using Artificial Intelligence
Haebom
I touched on this topic in a previous post as well, but I believe the current way we use artificial intelligence is in a transitional phase. I'm studying and listening to many people's opinions, thinking there must be a better way. Recently, Apple published an interesting paper, so I brought it here.
Ferret-UI- Grounded Mobile UI Understanding with Multimodal LLMs.pdf1.08MB
This time, Ferret-UI is a new approach based on a multimodal large language model (MLLM) that can better understand public mobile UI screens and, following natural language instructions, reference or pinpoint specific UI elements. Simply put, it helps understand what the user is seeing right now and predicts their behavior to help them make better choices.
•
Overcoming UI complexity: Modern mobile apps are built with diverse and complex UIs, and users need to obtain information or execute commands through them. Ferret-UI understands such complicated UI structures and helps accurately identify the right UI elements based on user instructions.
•
Enhancing accessibility: Ferret-UI can greatly improve UI accessibility by leveraging visual understanding. This especially makes it easier for users with visual impairments to use apps comfortably.
•
Streamlining multi-step UI navigation: When users perform complex tasks within an app, Ferret-UI supports them by accurately identifying and pointing out the necessary UI elements. This lets users achieve their goals more efficiently.
Apple's Ferret-UI technology takes an innovative approach by leveraging multimodal large-scale language models to maximize mobile UI understanding and, in turn, enhance user experience. I think this is a solid approach in several respects—tackling app complexity, improving accessibility, and making the app development process more efficient.
Still, there are ongoing concerns about privacy and security risks, and whether it might end up like those incomplete auto-complete features. In Apple's case, rather than disclosing the model itself or making preemptive moves in AI, it seems they're aiming for a practical approach—smoothly integrating it into iOS and MacOS, which they have always excelled at.
Subscribe to 'haebom'
📚 Welcome to Haebom's archives.
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
haebom
해봄의 아카이브
AI is a hot topic, so why is Apple staying quiet?
Haebom
AI and generative models from companies like Google, Meta, and MS are receiving intense attention. In fact, established Big Tech firms are actively collaborating with companies that have LLMs, such as OpenAI and Anthropic. Yet, the one name we don't hear is Apple. Apple is probably the only brand that drives everyone around the world absolutely crazy, but this time they're keeping quiet.
Of course, there have been a handful of rumor-filled articles claiming that Apple GPT would be released. Stories have circulated suggesting Apple isn't interested in AI and has put all its focus into MR. In reality, Apple has been running its own AI/ML team for years with in-house language models like Ajax. According to a recent Tim Cook interview, they've already developed products internally using generative AI technology and are now planning the timeline for public release.
Recently, Apple unveiled their Matryoshka Diffusion Models, which propose a way to overcome the limitations of traditional image generation methods using diffusion models—making efficient image and video generation possible even at high resolutions. This approach has proven effective and delivers fast, high-quality results, especially for 1024-square images. Based on these developments, it seems Apple is achieving meaningful results with its own image and language models. As with the M1 chip, I expect they'll go public once they've finished optimizing. https://arxiv.org/abs/2310.15111
I remember seeing a comment on Hacker News a while ago about generative AI that stuck with me: "If you don't like the result, it's hallucination. If you do, it's creativity. (It's hallucination if you hate it, and creativity if you like it.)" Someone might look at what Apple's doing and say they don't know what they're doing with AI—but as always, it could just be that they're moving carefully. You can see this even in their PR—they never go overboard.
As a long-time Apple fan and user, I believe they will deliver real value and meaningful results, as always.
Subscribe to 'haebom'
📚 Welcome to Haebom's archives.
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
1
Haebom
https://www.bloomberg.com/news/newsletters/2023-10-22/what-is-apple-doing-in-ai-revamping-siri-search-apple-music-and-other-apps-lo1ffr7p
See latest comments