Taylor Swift speaks Chinese, will this lead to a major change in marketing?

This article starts with the recently popular "Taylor Swift Speaks Mandarin" video, and introduces the current voice cloning, AI face-changing, digital human software, and their advantages and possible problems. It is recommended for those who are interested in AI digital human technology.

Recently, a video about "translation without dubbed film accent" has gone viral. In the video, Taylor Swift (American singer: Taylor Swift) speaks fluent Mandarin to the host. Not only does her voice not feel out of place, but even her lip movements are perfectly matched. It makes people sigh that translation has become so complicated.

The original author of the video, "johnhuu teaches English", often translates, but AI mouth-changing and voice-simulating translations are rare. Many netizens in the comment area said, "I was wondering when Taylor Swift's Mandarin became so standard." Not only that, Musk, Emma Watson and others have also embarked on the path of "localization" with the help of AI translation.

01 The man who made Taylor Swift speak Mandarin is from China

In fact, this video looks amazing at first glance, but it is nothing new. It can be broken down into three aspects: translation + voice cloning + mouth changing. In terms of translation, most translation software on the market can be accurate, but in terms of beauty, manual processing is still required. There are also quite a lot of tools for voice cloning and AI mouth changing. For example, the current mainstream voice cloning software includes Audio Clone, Voice Cloaker, Voice Changer, and face changing apps include Faceapp, Openfaceswap, Face Changer, etc.

After the video went viral, some netizens found out that the blogger was most likely using HeyGen, a software that can directly combine translation, voice cloning and mouth changing into one.

If you don’t dig into it, you won’t know how fast HeyGen has risen. According to statistics from Twitter analyst Will, the number of visits to the website increased by 92% from August to September, and it now ranks first among all independent AI websites. The developer behind this popular software is a real Chinese team - Shiyun Technology. "Shiyun" is taken from Liu Cixin's novel, and its core concept "technology cannot replace art" coincides with the founder Joshua Xu's philosophy.

In December 2020, Shiyun Technology was established in Shenzhen, and in just three months, it received a seed fund of US$2 million to US$3 million from Sequoia China and Zhen Fund. For a company that is only three months old, what they really need to do at this time is to find product-market fit.

In fact, as early as when Xu Zhuo worked at Snapchat, some cross-border e-commerce companies came to consult about "AI face-changing" technology, especially those clothing brands. Clothing brands have a large SKU and many new products are launched every quarter. Finding models of different skin colors to shoot a large number of new products is a very time-consuming and economically costly task. It is understood that the cost of European and American models is 2-3 times that of Asian models. After using AI face-changing technology, merchants only need to hire domestic models and then replace them to obtain model display pictures of different races, which greatly reduces costs and improves efficiency, and effectively solves the pain points of internationalization of clothing brands.

I think this is why HeyGen is currently mainly aimed at the B-side - click on the HeyGen page, the slogan reads "providing support for the world's best teams from startups to mature companies." Compared with traditional video production, HeyGen does not require equipment, actors, or a third-party team. It only takes a few minutes to generate a video that takes a crew many days to produce. It can indeed greatly reduce the costs of a large number of companies with video needs.

In addition, we can also see its founder Joshua Xu (Xu Zhuo) advertising online, "What you see here is my video 100% generated by AI." His expression, words, and pictures are all very natural, and you can even see the leaves blowing outside the window.

Some netizens who have experienced it say that HeyGen is the best tool for video generation. You only need to upload a video of more than ten seconds to get a video of you speaking a foreign language with the same voice and lip shape. In addition, you can choose a voice that matches the character's image from more than 300 voice packages, covering more than 40 languages. Currently, the most basic monthly payment package is 15 points (1 point is consumed for 1 minute), and the price is US$29 (about RMB 200). You can generate a 15-minute video for RMB 200 per month, and each video is no longer than 5 minutes. This is very time-saving and labor-saving for small and medium-sized enterprises that have foreign language broadcasting needs.

Relying on this demand, Shiyun Technology has opened up the cross-border e-commerce market. Of course, this may just be a microcosm of the application of AIGC in the marketing industry, or AIGC is essentially becoming a new paradigm in the marketing industry.

02 AI digital human, the first space for reducing costs and increasing efficiency

So the essence of "Taylor Swift speaking Mandarin" is nothing more than the latest face-changing technology with a little creativity, and what is really applied to commercial marketing is a more mature system, such as replicating or creating a digital person. Many domestic celebrities have also used digital avatars in live broadcasts and videos. From IP type to functional type, digital people are moving towards a broader market.

In this regard, Lehua Entertainment has made the biggest splash. On May 22, "Du Hua" wearing classic pink skin appeared in the live broadcast room. As soon as she opened her mouth, fans were stunned. How could she speak Mandarin so well? A little later, another Du Hua who looked exactly the same also appeared on the screen, using plastic to introduce her AI avatar "Hua Huazi" next to her. Fans said that this was just right. In just ten minutes, the number of viewers in the live broadcast room exceeded 320,000. Therefore, Lehua launched its artist Chen Xinwei's AI avatar and digital human "Le Le" to explain and promote Ms. Du Hua's skin care brand Dr. JE in the live broadcast room.

Last year, Liu Run also revealed that the short videos of his oral content have become digital clones. He only needs to let AI learn from previous videos to generate a "self". Then he can use audio to drive this digital person to produce videos. The feeling is very efficient. Faced with a large number of repetitive jobs that require "self" to appear on camera, they all chose to replicate a "silicon-based workforce".

Zhu Zixiao, Marco and others even used their digital avatars to sell goods in the live broadcast room. Calvin Chen's AI image continuously ate for 15 hours. Although there were doubts and the audience said they did not buy it, this wave did attract a lot of traffic.

As we said, the essence of marketing is content creation. When AIGC appears, changes in the marketing industry will naturally follow. At present, there are already thousands of "basic" digital human anchors active in the live broadcast room. Although they are far less attractive than the "celebrity version" in attracting traffic, it is still a good deal for merchants.

According to iMedia Consulting data, the scale of China's live streaming e-commerce industry is expected to reach 2.1373 trillion yuan by 2025, and the transaction volume of brand self-broadcasting is expected to reach 50.0% of the overall live streaming e-commerce transaction volume this year.

How to find growth in brand self-broadcasting, no longer rely on top anchors to bring goods, and aim at digital people. This year's 618, JD.com's virtual anchor Yanxi appeared in multiple brand live broadcast rooms, and the number of merchants broadcasting increased nearly 5 times compared with last year's Double Eleven, and the actual transaction volume increased by 246% month-on-month.

Compared with real anchors, virtual anchors have the following advantages:

• Low cost: As mentioned above, the popularity of HeyGen hits the pain point of high production costs of traditional content. From scene construction to post-production, the cost of a live broadcast room is 60,000 to 100,000 yuan a year, while virtual anchors are only one-tenth of that. With extremely low costs, as long as there is a long period of time, there will be transaction flow and revenue.

• All-time: In this era where traffic is king, whoever occupies more user attention and time will be able to grasp wealth. No matter how popular a real anchor is, he or she cannot be more popular than a digital person who works 24 hours a day and is in good condition. At present, many businesses choose to let real anchors broadcast during peak traffic periods, and use virtual anchors to fill the blank time during low traffic periods.

• Stability: High-quality anchors are always the object of competition among major brands, especially for highly professional industries. It is common for an anchor with a good image, temperament and strong professional knowledge to be poached by other companies. Virtual anchors rely on big data and powerful computing power. As long as they are equipped with a knowledge base and AI generalization capabilities, they can directly take up their posts and will never "rebel".

• More flexible: Digital anchors can choose their appearance, voice and language according to the needs of merchants when they are generated, and can even be modified when they are used. According to iMedia Research, the scale of China's cross-border live e-commerce market is expected to reach 284.58 billion yuan in 2023, a year-on-year growth rate of 155%, and the market scale will reach 828.7 billion yuan in 2025. Faced with strong export demand, digital anchors with changeable images and languages ​​have become a hot commodity.

AIGC is not just a gimmick of digital human anchors, but it permeates the entire process of live broadcast interaction - reaching, attracting, recognizing, and guiding purchases. In the entire marketing process, digital humans can not only provide users with product information, but also display the image as a brand IP, thereby attracting some users. This is also the direction that merchants should focus on when digital human anchors are widely popularized.

03 Dangers and opportunities coexist, All In Digital People, you have to face...

Today, it has become an industry consensus to go all-in on AI marketing, and digital humans with generative AI entry attributes have become a battleground. However, dangers and opportunities coexist, and problems are inevitable in an industry in its growth stage. At present, there are obviously four key issues that need to be solved for digital humans:

• Innovation issues: At present, the use of digital people is still very limited. Most digital people cannot achieve the process from zero to one. In other words, cost reduction and efficiency improvement can be achieved, but most of them are only used as a supplement to real anchors. It is obviously a false demand to let digital people achieve the effect that real anchors of the same level cannot achieve. In addition, the construction of digital scenes in the live broadcast room all comes from the real world and relies on human imagination. There is still a long way to go to build a virtual world with digital people.

• Platform issues: Douyin, as the largest live broadcast platform, has very strict control over content. On May 17, the Douyin platform changed its blocking policy to "require real people to participate in digital live broadcasts." Sima Huapeng, CEO of Silicon Intelligence, commented that "this is like the driver of an unmanned vehicle cannot leave the steering wheel." Data shows that the temporary blocking rate of 100 accounts on the Douyin platform (blocked for 1-3 days, no permanent blocking) is around 30%. Only 15 out of 30 accounts can be pushed by the platform and generate daily trading volume.

• Emotional issues: Platform strategies do not support it, and platform users are not very interested. Live streaming is an interest-based e-commerce, and it is full of randomness. Most of the goods are not necessities, and the transaction of non-standard goods needs to stimulate consumers' emotional needs. Compared with real people, digital people currently lack the ductility and empathy to communicate with people. Therefore, digital people seem to be very struggling to promote innovative products. When the real anchor gives timely care and converts it into a purchase rate, the digital people are still dryly introducing the product features. Products with a strong repurchase rate and a purchase habit are the main battlefield for digital people.

• Infringement issues: Since digital humans are called "humans", it not only involves technical issues, but also legal issues are the most concerned points for the public and businesses. 2D real-life digital humans generate videos by ingesting original video information, which obviously infringes on the portrait rights of real people and may even become a tool for crime. Infringements of 3D virtual humans mostly occur in copyright infringement. In April this year, the first case involving "virtual digital humans" infringement was decided in Hangzhou. A video released by a Hangzhou Internet company used the virtual digital human image and related content released by Mofa Company, and was eventually found to constitute copyright infringement.

Currently, advertisers and service providers are testing the waters with digital humans. The AIGC track is in the stage of enclosure and horse racing. The market direction has gradually become clear, but whether a healthy development model can be formed after high-profile efforts depends on the players' minds and awareness of norms.