How to build a content evaluation system for entertainment platforms?

Editor's note: Nowadays, content is an important factor for brands or platforms to attract users. In order to ensure content quality and improve the content ecosystem, platforms need to establish a content evaluation system. Entertainment platforms are no exception. In this article, the author summarizes the strategies for entertainment platforms to build content evaluation systems. Let's take a look.

For entertainment platforms, continuous content iteration and innovation is an important survival rule for the platform, and the ability to screen high-quality content is the core capability of the platform. Only by establishing a scientific content evaluation system and continuously satisfying users' demand for "good" content can users continue to use and retain. Therefore, it is imperative for entertainment platforms to build a content evaluation system and create a robust content ecosystem.

1. Why build a content evaluation system?

There are three reasons why we conduct content evaluation to distinguish good and bad content on entertainment platforms:

First of all, users need to consume "good" content. In terms of attracting new users, "good" content has advantages. For example, interesting content can attract new users to download, and content going viral can bring about natural traffic growth. At the same time, only by continuously satisfying users' demand for "good" content can users continue to use and retain.

Secondly, "good" content should get the "right" traffic. The essence of the content industry is innovation. It is the nature of users to pursue fresh content. Continuous content iteration and innovation is the most important survival rule for content platforms. The ability to screen high-quality new content is the core capability of the platform. In addition, traffic resources are precious and scarce. Only by operating traffic in a refined manner can we give this traffic to the most suitable content as much as possible.

Finally, content has costs and benefits. On the one hand, the introduction of content is often accompanied by the investment of money, employees’ energy, and users’ traffic. Therefore, a screening mechanism is needed when creating content to identify high-quality content in the early stages of resource investment and to efficiently support content.

On the other hand, content can generate revenue, for example, users purchase membership value-added services to view content, attract new users, and users consume more traffic, which can indirectly bring economic value to the platform. With a content evaluation system, the ROI of content can be clarified to assist decision-making and achieve refined content operations.

2. How to build a content evaluation system?

Commonly used content evaluation methods include manual evaluation and data evaluation.

1. Human Evaluation

Manual evaluation requires qualitative analysis of the content. In scenarios where there is no data or where there is a strong reliance on manual aesthetics, we will give priority to manual evaluation. However, in the qualitative process, we should also try to quantify the degree to which we can do this. This way, when we apply the manual evaluation results later, the classification can be clearer and the application more accurate.

For example, if we manually evaluate the beauty of the sound of a certain content, it may be divided into three levels of beauty, whether it is the heavenly sound at the professional level, the beauty at the ordinary professional level, or the beautiful voice among ordinary people.

Manual evaluation involves manual aesthetics, and can solve many evaluation scenarios without data quantification. However, it also has the characteristic of large deviations in subjective judgment. Even if the standards are clear, people’s feelings will be different. Each reviewer’s understanding of the content is very prone to deviations. Generally, a sampling review method is used to ensure that the final result meets expectations. However, manual review as a whole must have a certain tolerance for deviations.

Manual evaluation usually includes three types: preliminary content evaluation, content classification evaluation and content perception evaluation.

2. Data evaluation

Data evaluation is a quantitative evaluation of content. It relies on massive user behavior. The various feedback from users on the content is the most objective and fair evaluation. It is also the actual content effect that each entertainment platform is most concerned about.

Data evaluation has the advantages of being more scientific, objective, and comparable, and timely feedback can help improve traffic distribution. The value of content can be broken down and evaluated specifically from the following dimensions:

1) Revenue value

Content can directly generate revenue. Users will pay for certain content, which is usually divided into single-article revenue and membership revenue. These revenues can be attributed to each individual content.

2) Growth value

The dissemination of high-quality content outside the platform can bring in new users and help lost users return to the platform. It is also costly to obtain these users in other ways, so the new returning users brought by the content are the value of the content. This can be evaluated through indicators such as the number of new users, the number of returning users, the value evaluation of new returning users, retention and duration.

3) Community Value

• Consumption value: High-quality content has consumption value. Users will like to watch it and even watch it repeatedly, so it can be evaluated by completion rate, deep consumption rate, consumption time, collection rate and like rate (different indicators for different products are different).
• Promoting production value: Content has the function of promoting production. When ordinary users see the magic emojis used by celebrities on Douyin and Kuaishou, they rush to try them out. When experts on Xiaohongshu and Weibo publish their dressing tips, users often imitate them to produce their own content. These contents are an incentive for users to produce, and the production value of content is also a representative dimension of "quality".
• Social value: Users’ social behaviors on the site can increase user stickiness. The social attributes attached to the content help users deepen their connection with the platform. The social value of the content also needs to be quantified.

It should be noted that the evaluation indicators of data evaluation are often complicated, and effective information needs to be extracted in combination with the evaluation objectives. Usually, the complete content evaluation system of entertainment platforms is formed by the combination and complementation of manual evaluation and data evaluation.

3. Introduction to content evaluation model

1. Quantitative Model

Whether the content is high-quality can be evaluated from multiple perspectives. For entertainment platforms, when faced with multi-dimensional data, they are often confused about whether the content is good or not, and which of these contents is the best. Therefore, a unified quantitative model is needed to evaluate the content. The unified quantitative model will eventually condense the evaluation indicators of multiple dimensions into a comprehensive evaluation score.

What is a quantitative model? It refers to the establishment of a model based on applied statistics and econometrics using mathematical statistical methods. The goal of establishing a model is to establish a functional relationship between indicators. The statistical model ultimately describes the relationship between variables and is an approximate estimate of the real data.

When building an entertainment content evaluation system, the most core goal is retention. So when we do content quality evaluation, we need to find content that can promote user retention. The consumption value, production value, and social value of the content are all factors that influence user retention on the site.

So in this fitting model, y is retention, and x is a specific indicator of the three major values, such as listening time, number of likes, number of productions, number of comments, number of followers, etc. The x and y of the model have been determined. In the application of mathematical statistics models, the most suitable regression equation for this model is logit regression. Our final fitting formula will get the following result:

Comprehensive score of content quality assessment = a*playing time + b*number of produced content + c*number of followers + ...

The higher the comprehensive score of content quality assessment, the higher the contribution of this content to user retention. Therefore, under this model system, we can assume that the content with a higher score is more "high-quality" content.

But at the same time, this model also has limitations in application, because the data is fitted with the user behavior data of the current site, which means that it only represents the current situation within the site, and this quantitative relationship cannot be applied to other apps outside the site; and if the product form of our site or the logic of distributing content to users changes, then this quantitative relationship will change, so under this limitation, the model needs to be updated regularly.

2. Unified measurement - monetization

Even with the model's comprehensive quality assessment score, it may still not be enough in some scenarios. For example, when we purchase content, we know that the comprehensive quality assessment score of the content is 100 points. However, if we need to spend 100,000 yuan to purchase the 100-point content, is it worth it?

For such problems, we need to further quantify the value of the content and convert the score into currency. For example, if the value created by this content on the site is 1 million yuan, then we can more easily perceive the value of the content.

From a monetary perspective, the value of content can be broken down into three core levels: revenue value, growth value, and community value. Each content can get a specific value amount in a fixed time period.

It should be noted that this value amount is not rigorous enough when calculating financial data, and it does not represent how much money each content really makes for the business. The purpose of calculating such a quantitative value is to help the business measure the relative value of each content, with the aim of being clearer and more quantifiable when iterating content, and assisting decision-making.

3. Model modification

The definition of high-quality content may be different at different stages of development. For example, in the early stage, more emphasis is placed on the value of the content itself, while in the middle and late stages, more emphasis is placed on community value. The coefficients of various indicators can be adjusted to a certain extent in specific scenarios.

The model is fitted based on the goal of user retention, but it may not be applicable to all scenarios. For example, when we adjust the community atmosphere, we tend to use social interaction data to evaluate content quality. At this time, we do not necessarily have to use this model, and we can also directly weight the relevant indicators.

The content value ranking summarized by the model may be inconsistent with the cognition of App operators, and needs to be analyzed one by one according to specific circumstances.

With the content evaluation model and system, whether it is content introduction evaluation, refined operation and distribution of different groups of people or content ROI evaluation, it has certain applications and assistance for entertainment platforms. In order to improve the supply of high-quality content, entertainment platforms need traffic incentives to support the conversion of traffic into more fans and higher interactions, and encourage creators to produce more high-quality content; at the same time, negative content should be suppressed to avoid bad money driving out good money, and feed back content introduction strategies, ultimately helping entertainment platforms to truly implement refined operations, bring better experience to users, and promote the healthy development of the platform.