How Apple Suggests Using Artificial Intelligence
Haebom
I touched on this topic in a previous post as well, but I believe the current way we use artificial intelligence is in a transitional phase. I'm studying and listening to many people's opinions, thinking there must be a better way. Recently, Apple published an interesting paper, so I brought it here.
Ferret-UI- Grounded Mobile UI Understanding with Multimodal LLMs.pdf1.08MB
This time, Ferret-UI is a new approach based on a multimodal large language model (MLLM) that can better understand public mobile UI screens and, following natural language instructions, reference or pinpoint specific UI elements. Simply put, it helps understand what the user is seeing right now and predicts their behavior to help them make better choices.
•
Overcoming UI complexity: Modern mobile apps are built with diverse and complex UIs, and users need to obtain information or execute commands through them. Ferret-UI understands such complicated UI structures and helps accurately identify the right UI elements based on user instructions.
•
Enhancing accessibility: Ferret-UI can greatly improve UI accessibility by leveraging visual understanding. This especially makes it easier for users with visual impairments to use apps comfortably.
•
Streamlining multi-step UI navigation: When users perform complex tasks within an app, Ferret-UI supports them by accurately identifying and pointing out the necessary UI elements. This lets users achieve their goals more efficiently.
Apple's Ferret-UI technology takes an innovative approach by leveraging multimodal large-scale language models to maximize mobile UI understanding and, in turn, enhance user experience. I think this is a solid approach in several respects—tackling app complexity, improving accessibility, and making the app development process more efficient.
Still, there are ongoing concerns about privacy and security risks, and whether it might end up like those incomplete auto-complete features. In Apple's case, rather than disclosing the model itself or making preemptive moves in AI, it seems they're aiming for a practical approach—smoothly integrating it into iOS and MacOS, which they have always excelled at.
Subscribe to 'haebom'
📚 Welcome to Haebom's archives.
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
haebom
해봄의 아카이브
Do you find the current way of using artificial intelligence convenient?
Haebom
With the emergence of artificial intelligence, especially large language models (LLMs), chat- and thread-based user interfaces (UIs) and user experiences (UXs) are drawing a lot of attention. Users input questions or requests in chat windows, and the AI responds to them with text. These text-based interfaces are reminiscent of the days of DOS, virtual terminals, and computer bulletin board systems (BBS).
나우누리, 하이텔 등 Virtual Terminal 등의 시대와 무척 유사합니다.
But from past experience, we know that these text-centered interfaces eventually evolved into graphical user interfaces (GUIs). GUIs were developed to help ordinary users interact with computers in a more intuitive and convenient way. So, today's chat- and thread-based UIs are also transitional, and it's expected that they will evolve into even more user-friendly forms in the future.
Looking at the direction of these changes, Apple's recent moves are worth noting. Apple is showing attempts to fundamentally reimagine interaction between people and devices using technologies like voice recognition, augmented reality (AR), and motion recognition. For instance, voice assistants like Siri, AR applications built using ARKit, and the hand gesture recognition in Apple Vision Pro all offer a glimpse of what's possible for interfaces of the future.
We also need to pay close attention to this wave of technological change and practice thinking ahead to future UI/UX trends. Just as we once saw interactions shift from PC communication to GUIs, we can expect that interfaces will get more natural and intuitive by combining modalities like voice, vision, and movement, rather than sticking to text alone. I believe it's more important than ever to recognize where we are in this transition and to develop the insight to anticipate what's coming next.
Privacy and ethics are often brought up here—people want services that help them contextually, whenever and wherever, but they don't want tech companies watching their every move in order to make that possible.
Recently, I got together with friends and colleagues to study how services might change in the next few years, and I was surprised by how enthusiastic everyone was. The more we discussed, the more it felt like an AI-powered version of HCI—coming back, in the end, to voice, movement, and eye tracking. I think simply taking time to reflect on these possibilities right now is very valuable.
On the other hand, seeing recent successful Neuralink implants and realizing it's now possible to post to Twitter just by thinking, it makes me feel that change in the world may happen much faster than I'd modestly predicted.
*If I were to open up this kind of study to the public, would many people be interested? I’m also curious if there are folks out there who’d like to join in for some laid-back study and discussion.
Subscribe to 'haebom'
📚 Welcome to Haebom's archives.
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com
2
이
이승현
해봄님, AI 시대 서비스들에 UX/UI에 관심이 많습니다. 스터디가 열리면 너무 좋을 것 같아요!
김
김민주
저도 스터디가 열리면 같이 공부하고 싶어요!
See latest comments