Apple AI: Not many surprises, but it set an example for Android phones

Although Apple does not have many technological innovations, it has provided a lot of engineering practices for AI implementation in mobile phones and other terminals.

In the nearly two-hour main forum speech, Apple devoted half of the time to artificial intelligence.

At 1 a.m. on June 11th, Beijing time, Apple's 2024 Developer Conference opened in Cupertino. If the limelight of last year's developer conference was still the MR device Vision Pro, this year Apple, which was criticized for lagging behind its peers in the artificial intelligence track, finally released a large number of artificial intelligence-related products and features.

For example, the new Siri can understand the context and conduct multiple rounds of conversations, can transfer information between multiple apps, and has access to ChatGPT capabilities; for example, it can automatically classify emails and generate replies, voice memos can be transcribed into text and summaries, it supports image search and image removal, and supports automatic video arrangement... Apple is connecting these AI capabilities with various apps and scenarios in the phone.

Although there are not many eye-catching things, a big advantage of terminal manufacturers such as Apple is that they do not need to create demand and scenarios. They only need to use AI to solve the many pain points that already exist on mobile phones to bring users a different experience.

Apple uses a very tricky word to summarize these AI capabilities - Apple Intelligence, trying to cultivate users' minds into thinking that Apple Intelligence is equal to Artificial Intelligence.

01 How will AI change the iPhone experience?

An hour into the press conference, Apple finally began to release Apple Intelligence, with the application scenarios being introduced first.

The first is to generate emoticons and pictures. When users can't find the right emoticons to express what they want to express, they can freely create their own emoticons through natural language. Moreover, when sending messages to friends, you can use friends' photos to generate pictures of different styles to make the chat more interesting. Currently, it supports the generation of pictures in three styles: sketch, illustration and animation.

The second is call transcription and summary. The iPhone has not supported the call recording function before due to privacy concerns, but it will support this function in the future and provide smart summaries, but when the user starts recording, the other party will receive a notification, which will be an embarrassing scene.

The third is information extraction and generation. This includes the extraction of web page information from the Safari browser, as well as email classification and summaries. It can also generate email reply content or perform text polishing. In addition, Apple AI will also summarize the key information pushed by mobile apps and give priority notifications.

Fourth, image removal and vlog generation. Users can simply enter a description, and the Apple system will use the existing image library to create a "movie with a unique narrative arc." Or you can directly tell the phone what kind of pictures you need to find, and then the system will find the corresponding pictures from the massive image library. Users can also remove unnecessary elements in the picture, achieving the effect of one-click removal.

Fifth, Siri has become smarter, supporting multiple rounds of natural language conversations, understanding context, and will also support text input. Siri has been launched as a voice assistant for more than a decade, but it has never been smart enough and can only handle some simple tasks, such as "setting a timer" and "creating a reminder". In addition, for many questions, it will only give you a bunch of search links, which is not smart at all.

After the big model was added, Siri has a new logo, and when users call Siri, there is no longer a circle as before, but a light surround effect appears on the entire screen.

Moreover, Apple has officially announced its cooperation with Open AI. Siri can directly call on the capabilities of ChatGPT. For example, if you want to know which plants are suitable for planting in the yard, you can take a photo and Siri can use ChatGPT to provide answers. However, Apple also stated that before using this service, users will be asked whether they want to use ChatGPT. In addition, Apple users can access GPT-4o for free without creating an account, and ChatGPT subscribers can connect their accounts and access paid features directly in the Apple system.

However, since ChatGPT services cannot be directly used in China, Siri's capabilities in China may be limited. Prior to this, Apple was reported to be in talks with Baidu to integrate Wenxin Yiyan's services into the iPhone, but this news has not been confirmed by both parties.

Another not-so-good news is that because large models require stronger computing power, many iPhone models cannot experience these AI capabilities. According to current information, only iPhone 15 Pro and iPad and Mac equipped with M1 or newer chips can use them.

This means that as the replacement cycle of mobile phones becomes longer and longer, Apple may want to use AI to boost sales of new iPhones.

In addition to the AI ​​highlight, Apple also made regular updates to its operating systems for mobile phones, computers, tablets and watches. For example, iOS added app locks, satellite communications, and support for cross-device operations between mobile phones and computers. Another surprising feature is that the calculator has finally been introduced to the iPad, supporting the use of a stylus to take various mathematical notes. The calculator can recognize various expressions, directly display the answer, or create a parabola graph, etc., and can even imitate your handwriting.

02‍ Emphasize system-level usage across apps

From the AI ​​features launched by Apple, none of them are very eye-catching, including call summary, picture editing and elimination, which have already been implemented on Android phones such as Samsung and OPPO. But compared with the fragmented updates of other manufacturers, Apple has shown its thinking on generative AI from a system level.

Apple clearly introduced its five core AI concepts at the conference: powerful, intuitive, deeply integrated, personal, and private. This also means that Apple's AI will not provide many cool features like many large-scale products, but will improve the complete experience of the product in the system.

"It has to understand you and be rooted in your personal context, like your daily life, your relationships, your communications, and so on. It goes beyond artificial intelligence. It's personal intelligence," Cook said.

Apple has been emphasizing cross-app collaboration, and behind cross-app is that Apple will call on your personal information and context. It can not only analyze the most relevant content from your various searches, but also refer to the content on your screen, such as the email or calendar you are viewing.

For example, when you ask Siri when your loved one's flight is landing, it will cross-reference flight data that may have been shared with you via text message or email. Siri can also integrate other information, such as upcoming lunch reservations, so you can ask it how long it will take to get from the airport to the restaurant without switching between apps.

For example, when someone sends you a text message with an address, you can ask Siri to add the address to the corresponding contact. For example, through screen perception, you can ask for photos of a friend at a certain location, and then ask Siri to add a specific photo to a note, which means Siri browses the photos and memos at the same time.

For example, if a meeting is rescheduled to the evening, and you want to know whether you can catch your daughter's performance after the meeting, Apple Intelligence will process relevant personal data to provide you with assistance. First, it knows who your daughter is and the performance she sent a few days ago, and also knows the time and place of your meeting, and estimates your transportation conditions from the company to the theater.

Currently, the new version of Siri can perform more than 100 operations. These functions run through emails, notes, Safari browser, Pages, Keynote and even third-party apps.

Of course, domestic users may have very limited perception of Apple's AI capabilities. In addition to the inability to access ChatGPT, it also has a lot to do with the usage habits of domestic users. First, the usage habits of many domestic users are different from those of overseas users. For example, the press conference repeatedly mentioned the content of processing emails, but in the work habits of domestic users, emails are not a rigid demand. In addition, functions like iMessage are not popular in China, and everyone still likes to communicate on WeChat, which will also make it difficult for the system to obtain personal information.

03‍ End-to-cloud deployment tests user privacy and security

Apple defines its AI capabilities as a personalized intelligent system, and the prerequisite for achieving intelligence is to obtain a large amount of personal data and environmental information, including pictures, memos, messages, emails, etc.

At present, Apple tries its best to run AI calculations locally on the terminal side, so that data can be processed on mobile phones and computers without uploading it to the server. However, the computing performance on the terminal side is limited after all. Only mobile phones and computers with A16Pro and M1 or higher processors can support Apple's AI functions. And even these models may still need to call on cloud capabilities when facing larger tasks.

In fact, mobile phone and PC manufacturers are currently emphasizing the ability of end-cloud collaboration. However, Apple did not disclose which specific tasks need to be completed in the cloud. But at least the part that calls ChatGPT must be connected to ChatGPT's cloud service.

Once the data is uploaded to the cloud, it will always bring more worries to users, and it will not be as reassuring as storing it on their own mobile phones.

In order to solve the problem of limited computing power on the client side and data security on the cloud side, Apple has specially launched Private Cloud Compute, a private cloud computing technology. It will use a special server of Apple Silicon, which uses Apple's chips. If the request needs to be sent to the cloud server, Apple said it will only send task-related data in an "encrypted" and secure way.

Apple claims that the data will never be stored, not even by Apple, and will only be used to execute requests. Apple's efforts in data security and user privacy are worth learning from for domestic mobile phone manufacturers. Perhaps in the future, both mobile phone manufacturers and PC manufacturers will need to build dedicated data centers to handle AI requests instead of relying on public cloud services.

Of course, this does not completely dispel people's concerns about data security.

OpenAI's arch-enemy Musk left a message directly under Cook's tweet, "If Apple integrates OpenAI at the operating system level, then my company will be banned from using Apple devices. This is an unacceptable security violation."