The Internet is full of "eye-catching" "AI photos"

From revealing AI beauties to suggestive AI handsome men, these "eye-catching" contents not only violate the values ​​that social platforms should have, but also arouse public doubts about platform supervision. This article will explore the reasons behind this phenomenon for your reference.

“AI is good and powerful, but if it changes during use, it is worth thinking about.”

Recently, Lin, a post-90s girl who is addicted to AI photoshoots, lamented that the various AI photoshoots on social platforms are "eye-catching". "Now, whenever I open those social platforms, I feel uncomfortable."

"This kind of discomfort is a direct visual impact." Sister Lin said bluntly, "What popped out directly are all kinds of AI beauties, or AI handsome guys walking on the 'edge'. This is far from the AI ​​photos we imagine and see in daily life. It's not normal at all, and it's 'eye-catching'."

What is more common is that when you open social platforms such as Xiaohongshu and Douyin, you will see bloggers who operate such accounts posting "Professional Attire Ladies Series" and "Swimsuit Series", etc. Some of them even label the content as "no harmful content".

Many users think this is simply "calling a horse a zebra". One netizen even said, "At a time when social platforms are calling for strict regulation and actively building a high-quality content ecosystem, I feel like I've returned to the days of browsing adult websites."

Some netizens also expressed strong doubts, "Has the platform ever thought about the harm to minors when it allows adult content?"

As for more netizens, they have been disappointed with the platform more than once.

Xiao Ran, a heavy user of Xiaohongshu, said, "Every time the content is expanded, Xiaohongshu seems to be associated with 'borderline'. From the travel hookup incident to the socialite training class to AI applications, all kinds of bad content have never been clean."

Obviously, there is a paradox between corporate goals and content operations, which is difficult for users to understand.

Regardless of whether the platform's review standards are detailed, whether creators take advantage of the gray areas of the platform's mechanism, or whether there are loopholes in the entire supervision, in the era of "AI for Everything", all users need is a clean and high-quality content ecosystem.

In fact, the regulatory authorities attach great importance to this. On March 5, 2025, at the first "Minister's Channel" of the Third Session of the 14th National People's Congress, the State Administration for Market Regulation stated that this year's focus will be on rectifying "big data killing old customers" and "anchor malicious hype" and other content.

Perhaps it is time for platforms to really pay serious attention to bad content that is maliciously created using AI.

01 “Eye-catching” AI content

Sister Lin is a post-90s generation. Her daily job is new media operation. She also operates related content on social platforms. After seeing many people on the platform use AI to take photos to "revive old photos", pretending to travel in the Hawaiian Islands, embedding multiple sets of styles and hairstyles, and realizing "taking photos" with a variety of styles, Sister Lin also wanted to play with AI.

According to experience, Sister Lin opened the social platform and searched for tutorials on AI photo shooting. Unexpectedly, after entering the keyword "AI photo", most of the pages that popped up were "AI beauties" and "AI handsome guys".

But what shocked Lin was that the photos marked "Generated by AI" in the lower left corner were not portraits in the traditional sense. Although these contents were synthesized by AI, both men and women were wearing revealing clothes, showing off their bodies, or doing indecent actions, which gave people an overall feeling of "spicy eyes".

What Sister Lin couldn’t understand in particular was that this was her first time searching for “AI photo” on a social platform and she was not interested in this type of photo at all. So why was she pushed to this content by big data?

As a cyber native, Lin experienced the early days of the Internet. When she first went online, she saw adult content everywhere, but she didn't expect that "it's 2025, and these borderline content is still active in front of the public. Watching short videos is like watching those adult web pages."

Xiaoran, who is used to visiting Xiaohongshu, also hates this.

Xiaoran is a post-00s generation who likes beauty and traveling, and thus has become a frequent visitor to Xiaohongshu. Xiaoran is also a fan of trendy new technology applications. After learning about "AI photo", she was the first to browse bloggers' experience of using and shooting "AI photo" on Xiaohongshu.

Figure: Content about “AI photo” on social platforms

Source: Xiaohongshu (left), Douyin (right) Screenshot of "HandsetTech"

“You don’t know until you see it, and you’ll be shocked when you do.” Searching for “AI photo”, Xiaoran’s Xiaohongshu page was instantly dominated by “beauties” and “handsome guys” showing off their bodies.

"They are atypical photo albums. Some pictures show 'suspected to contain AI-created information', including 'bathrobe', 'fitness OOTD', 'man running a horse' and other series, but the content is directly 'borderline'." Xiaoran sighed, "Is this Xiaohongshu or Xiaohuangshu?"

At one time, when "travel partners" were popular, Xiaohongshu broke out the news of "bad guidance". Xiaoran also found a similar case when looking for travel partners on Xiaohongshu.

What Xiaoran didn’t expect was that Xiaohongshu, which has always used high-quality content as its platform label, still maintained its original appearance. In particular, after Tiktok was blocked, Xiaohongshu once topped the download list in the United States, and a large number of overseas celebrities also joined Xiaohongshu. Even Xiaoran proudly said, “It has to be said that Chinese social applications have also made progress overseas.”

But now, a small experience has once again disappointed Xiaoran. "I feel a little speechless and a little disdainful. If the content ecosystem continues to maintain this level, I might slowly escape from this platform."

According to Xiaoran, in fact, there are a lot of similar content not only on Xiaohongshu, but also on other social platforms such as Douyin, Kuaishou, and WeChat Video Account.

Xiao Ran said that since he searched for "AI photo" on Douyin, his recommended content has "changed dramatically." Now, Xiao Ran said that his Douyin account is completely unbearable to look at, "I have turned Douyin into a brothel."

02 Quickly create an account and be eager to attract money

"AI is flexing its muscles like crazy" and "real people are becoming more and more fake, while AI is becoming more and more real" seem to be slowly becoming the most appropriate descriptions.

The bloggers who operate these "AI beauty" accounts are taking advantage of the marginal appeal to quickly create accounts and make money.

On Douyin, some "AI beauties" wear revealing clothes and constantly post dance videos on the platform, gaining more than 1 million followers in just half a year.

According to media reports, last year, there was a blogger on Douyin named "Xiao Yimei". Each video featured different products, and the prices were generally less than 10 yuan. However, the cover picture was an AI-generated beauty with a product introduction. She once gained more than 1.1 million fans.

There are even bloggers who see this as the next gold-digging track for "making money from side jobs". On Douyin, many bloggers seriously explain "how to become an AI blogger".

Figure: Content on social platforms about “How to become an AI blogger”

Source: Screenshot of Douyin "HandsetTech"

In fact, on Xiaohongshu, AI photo beauties with exquisite makeup and hot bodies have been quickly gaining popularity on the platform since 2024. These bloggers often post photos of beauties wearing swimsuits on vacation at the beach or taking photos on fashionable streets, with more than a few thousand likes.

Take a blogger in Zhejiang as an example. He is a certified "design service company" whose services include "digital video and digital photo". He started to publish his first note in March 2024. The content is mostly borderline, involving "pure desire style" and "royal sister style". So far, he has more than 36,000 fans and more than 200,000 likes and collections.

In fact, in the early days of the platform, many people could gain good traffic by simply "shooting and writing". However, as the platform's content ecology gradually matures, the content distribution mechanism relies more and more on algorithmic logic rather than pure luck.

A sports blogger on Xiaohongshu also revealed to us that if her content is "slightly off the mark", the traffic will be better than normal content.

In the opinion of the sports blogger, although the platform has not admitted that "borderline content can attract traffic", after her many attempts, at least in her opinion, "borderline content has much higher traffic than normal content."

The sports blogger said that similar problems exist not only on Xiaohongshu, but also on short video platforms such as Douyin, Kuaishou, and WeChat Video Account. "I have tried to post swimsuit AI short videos on these short video platforms, and the traffic is very good." But considering her sports IP image, she ultimately did not follow this style.

An insider who works in operations at a social platform also said that, generally speaking, the internal ecosystem will provide traffic support for specific content, allowing it to quickly gain popularity with the support of the platform.

The above-mentioned insider said, "The reference standards for traffic support include keyword matching, interest recommendations, and different reference dimensions such as interaction and dwell time. For example, users' likes, collections, comments, and dwell time on notes directly determine whether the content will be recommended to more people."

This has also led to a misunderstanding that many people think that popular content is "random", but in fact, there is a clear traffic logic behind the popular content. Of course, the above-mentioned insider also said that "brushing traffic" is also an important means of pushing traffic.

Regarding the question of "whether there is support in a certain sense for the wild growth of AI-based photo-shoots", the above-mentioned person said, "It's hard to say, but there are definitely algorithm problems on the platform."

03 It’s time to pay close attention

Whether it is the existence of a large amount of AI photo content that borders on the edge, or the sale of courses that peddle AI-bordering business, all of this points directly to one problem: there are loopholes in platform supervision.

And this is not the first time this has happened.

Take Xiaohongshu as an example. In the past few years, the platform’s involvement in pornography and illegal transactions have become the focus of public opinion.

In November 2020, Xiaohongshu was exposed for pushing large-scale nude photos and videos of beautiful women, with explicit and suggestive content. In September 2021, CCTV revealed that Xiaohongshu contained a large number of videos that leaked the physical privacy of minors.

What is more widely known is that in August 2023, an article titled "Xiaohongshu's "All-inclusive Travel Hitchhiking" is Pornographic" caused an uproar on the Internet, claiming that the travel hookup notes on Xiaohongshu, under the banner of "finding hookups", actually involved sexual transactions.

Although Xiaohongshu has stated every time that it will investigate and handle the matter, the actual results have not been ideal.

Even in 2025, when TikTok was facing a ban, a large number of American users turned to Xiaohongshu. Almost overnight, a large number of overseas bloggers' marginal information and photos appeared on the Xiaohongshu platform.

This once again exposed the weakness of Xiaohongshu's content ecological supervision. At that time, some industry insiders said that Xiaohongshu really needed to further refine its review standards to adapt to content supervision requirements under different legal environments and avoid legal risks caused by content violations.

Xiaohongshu is not without action. On February 13 this year, Xiaohongshu's official account "Shu Guanjia" issued an announcement, saying that since December last year, the governance team has punished 1,695 brands in 15 industries, including beauty and personal care, food, 3C digital, and life services, for violating marketing regulations.

But judging from the current situation, Xiaohongshu seems to have "good intentions but insufficient strength".

As for Douyin, although it has carried out supervision on "pornographic" and other related content many times, it has not been able to effectively improve the platform's ecology so far.

In fact, this recurring problem not only damages the image and reputation of the platform, but also greatly reduces users' trust in it. In particular, with increasingly stringent supervision today, how the platform can effectively curb the spread of bad content remains a problem that needs to be solved.

However, an algorithm engineer told us, "In fact, all major mainstream apps have also launched AI identification functions, and are also strengthening the review of AI pictures and videos. But from the current situation, the effect is average. Especially in the face of increasingly advanced AI tools, sometimes even the platform cannot really distinguish them."

Of course, the state has also taken some actions in this regard. In September 2024, the Cyberspace Administration of China issued a notice for soliciting opinions on the "Measures for Identifying Synthetic Content Generated by Artificial Intelligence (Draft for Comments)", requiring platforms and service providers to clearly identify AI-generated content, such as adding watermarks.

But this still requires strong support from the platform.

Earlier, Liu Xiaochun, associate professor at the School of Law of the University of Chinese Academy of Social Sciences and director of the Internet Rule of Law Research Center, said, "To prevent platforms from using AI to output violent and insulting content, different technical means can be adopted. At the output end, service providers should screen and review again."

However, facing various questions from users, the platform's supervision is still clearly insufficient. At least for now, this cannot change users' distrust of the platform and the entire ecosystem.

"You are as beautiful as AI" is an irresistible compliment for girls. The progress of AI technology is irreversible, and related applications have indeed made people's lives better.

"But just from the perspective of borderline photos, if we or our next generation are still fed with all kinds of pornographic content and low tastes that can be found everywhere, that is probably the painful price that mankind pays for technological progress and the profit-seeking behavior of platforms." Xiaoran said bluntly.

"We still hope that AI can be used for good and not be used maliciously." In the view of Xiaoran and Sister Lin, "We are not criticizing the platform. We hope that the platform can strengthen supervision and provide netizens with a truly pure space."

All of this is based on the fact that platforms, technology, regulatory authorities and society as a whole maintain basic respect and awe for content production.

Text | Xiao Ting Editor | Rao Yan