[User Growth] Channel Growth SOP: Easily implement internal channel efficiency review and analysis

WeChat is an important internal channel distribution channel, so how to improve the distribution efficiency through mature game nodes? The data group has accumulated a set of analysis SOPs for this purpose. The author of this article shares the problems encountered in the construction of the internal channel efficiency growth SOP and the disassembly method.

WeChat is an important channel for internal distribution. The Growth Center has in-depth cooperation with WeChat to improve the distribution efficiency of IEG games on WeChat through various means such as mature game node activities. The Growth Data Group has developed a set of analysis SOPs around internal channel efficiency growth to facilitate the rapid location of problems encountered during delivery. Next, I will share with you the problems encountered in the construction of our internal channel efficiency growth SOP and our disassembly methods.

1. Business Background

First, for easier understanding, let’s briefly introduce the WeChat distribution scenario.

Its product form is shown in the figure below. The main distribution resources include official accounts, mini-programs, video account game centers, etc. Among them, the "WeChat Games" official account and the discovery page game red dot + center advertisements are internal channel advertising resources distributed through the virtual gold allocation mechanism.

The data analysis link is package release → push → game red dot/public account message exposure → click → download → register.

2. Overall Analysis Framework and Focus Indicators

The WeChat delivery process is that after the algorithm has produced the A game user package, WeChat will filter the users based on data such as user activity, and then bid with other game population packages. After a successful bid, game messages will be pushed to the WeChat public account and red dot location to attract users to click and download.

The North Star indicator of the entire delivery is the Exposure Download Registration Rate (EVR). Based on the North Star indicator, we can divide the delivery funnel into two parts: model effect and channel performance. The model effect mainly refers to the user identification process, and the channel performance is mainly the effect data on the channel side after the user identification is completed. Through process decomposition, each step of the delivery funnel has a clear inspection target and focus indicator, so that the changes in the North Star indicator can be accurately attributed.

1. First, we will use a series of strategies to screen out the candidate set for sorting. The candidate set is a large user package. The subsequent refined sorting algorithm will sort within this user range to find users who are more likely to register. Therefore, the candidate set coverage of the target user is the first core indicator. By evaluating the proportion of registered users in the candidate set to WeChat relationship chain registrations in the same period, we can know whether the candidate set is sufficient. For example, there are 100 registered users in a certain period, and there are only 50 registered users in the candidate set. No matter how the subsequent sorting is done, the number of registered users who will eventually be out of the package will not exceed 50.
2. During the user identification stage of the model, we mainly focus on the model effect, that is, whether the model can screen out high-potential registered users. Due to the limit on the number of packets, we need to maximize the accuracy of end-user identification, that is, the recall rate of the top 5 million users of the model.
3. After configuring the user package to WeChat, the distribution of the original model layer will change after WeChat filtering and bidding. Since this step is currently a black box, we can only evaluate the impact of WeChat filtering on registration by comparing the package registration rate and exposure registration rate.
4. After WeChat pushes advertisements to target users, the quality of the ad copy and materials will directly affect the exposure and click-through rate.
5. If the user is interested in the game and continues to click to download, this step will cause the channel package to be replaced due to interception by some manufacturers. We can evaluate the loss of this step through the channel share and relationship chain share.
6. Finally, when it comes to the registration stage, we can compare the user download situation and analyze the situation of users who registered but did not download, as well as users who clicked to download but did not ultimately convert, in addition to the users who came through the normal funnel. This will help us understand the diversity of user conversion links.

Through user segmentation, we can check whether the model accurately identifies top users. How can we further evaluate which segment of users’ changes caused the final click-through rate to drop? We define the segmentation contribution index to describe and help locate the impact of segmentation changes on the overall index.

• Different layers of users have different registration conversion potentials. Compared with the two campaigns, due to WeChat black box filtering, the actual exposure of the user layer distribution has changed from uniform distribution when the original user was circled to uneven distribution, and the actual exposure of different layers in the two campaigns has changed differently. The actual exposure and click-through rate of different layers are different. The change of layer exposure is affected by filtering, and the click-through rate is affected by the material. The two are independent of each other.
• The layered CTR contribution index can describe the impact of the material. Assuming that the exposure volume remains unchanged, it can evaluate the overall impact of changes in CTR at different layers. For each layer, the difference between the current CTR and the previous CTR is calculated. Assuming that the exposure volume remains unchanged, the CTR contribution is calculated using the ratio of the current exposure of the layer to the total exposure of the layer as the weight.
• The layered exposure contribution index is mainly used to describe the impact on the overall click-through rate caused by changes in exposure distribution due to black box filtering. For each layer, the ratio of the difference between the current exposure and the previous exposure to the total exposure is calculated as the ratio of layer traffic change. Assuming that the click-through rate remains unchanged, the difference between the current click-through rate and the total click-through rate of the layer is used as the weight to calculate the exposure contribution.
• By combining the two deliveries, we can calculate the total contribution of the layers brought about by the changes in click-through rate and traffic. When the click-through rate is lower than that of the previous delivery, the layer contributions of layers 1 and 17 are large, which are the main factors leading to the decrease in click-through rate. By observing the changes in traffic, we can find that the actual exposure of the head layer 1 (high-click-rate users who the model believes have a high conversion probability) has decreased significantly, while the actual exposure of the tail layer 17 (low-click-rate users who the model believes have a low conversion probability) has increased. When the overall exposure does not change much, the overall click-through rate decreases due to the change in layer exposure.

3. Case Analysis

Next, we use the above SOP process to analyze an actual delivery case. The background of this case is the fluctuation of indicators in the virtual gold delivery of a certain game. It is observed that the exposure registration rate of the delivery in March has a trend of a sharp decline, while the EVR in April has increased significantly. It is necessary to analyze the reasons for the EVR fluctuation.

The EVR change trend and exposure level of the game from March to April (the coordinate axis values ​​are hidden for data security reasons):

According to the above funnel, we analyze layer by layer and attribute the fluctuations of the North Star indicator.

First, before releasing the package, we will first check whether the coverage of the candidate set for the registered population has changed significantly. The indicator we mainly focus on here is the recall rate of the candidate set for registered users. This indicator represents the upper limit of the number of registered people that our model can select. Under normal circumstances, if there is no major adjustment to the candidate set, the proportion of registered people covered by the candidate set should be relatively stable.

If the recall rate of the candidate set drops significantly, then before the entire package delivery process begins, we can expect a certain degree of decline in the effectiveness of this delivery, because the upper limit of the number of registered people we can reach has become lower.

At this time, we need to find out what caused the change in the candidate set, for example, whether it is because the previous user activity filtering strategy filtered out too many users, or the scene click filtering strategy screened out too many users, or some other reason, and make corresponding adjustments.

The above figure shows the changing trend of the recall rate of the candidate set of the game. It can be seen that the recall rate of the candidate set and the EVR have basically the same changing trend, and the inflection points are basically the same. Between 3.7 and 4.4, the proportion of registered people covered by the candidate set increases significantly. From the perspective of the exposure and click active candidate set, it increases from 24.7% in 3.7 to 45.8% in 4.4, nearly doubling. The EVR in the same period also increases from 0.005% to 0.191%. In contrast, the recall rate of the candidate set in the 4.25 period drops significantly, and the EVR is correspondingly much lower than other periods in April.

This shows that the number of people covered in the candidate set of the model dropped sharply in early March, resulting in a reduction in the upper limit of the number of registered people that the model could select, thus affecting the exposure registration rate. After that, the number of registered people in the WeChat relationship chain covered by the candidate set gradually recovered, and by early April it had almost returned to the level of the end of February. The upper limit of the number of conversions that the model could select has increased. If the model capability is stable, more high-potential conversion users can be selected to improve the overall EVR.

After determining the impact of the candidate set on EVR, the next step is to evaluate whether the model's capabilities are stable. The indicators selected here are the candidate set accuracy and the top user registration rate, which represent the number of registered people covered by the candidate set and the number of registered people covered by the top 5 million users. When the total number of people in the candidate set does not change much, by comparing the changing trends of these two indicators, we can determine whether the model's performance meets expectations.

If the model capability is relatively stable, the number of registered people in the candidate set and the number of registered people selected by the model should increase or decrease proportionally. That is, the registration rate of the candidate set and the registration rate of the model's top users will have the same upward/downward trend. If the registration rate of the candidate set is increasing, while the registration rate of the model's top users is decreasing significantly, then the model is performing abnormally poorly and you need to see what went wrong with the model in this issue.

Below we compared the accuracy of the game's candidate set and the registration rate of the top 5 million users. We can see that in most periods, the two have basically the same rising and falling trend. When the registration rate of the top 5 million users is low, the accuracy of the candidate set in the same period also decreases by basically the same proportion, indicating that the decrease in the number of registered people in the top 5 million in March is due to the decrease in the total number of registered people in the candidate set.

In addition, it can be observed that from March 11 to April 7, the registration rate of the top 5 million users has increased significantly, which is much greater than the increase in the candidate set during this period. This shows that while the number of registered users in the candidate set has not increased much, the model has screened out more than twice as many registered users as in previous periods. It can be seen that the improvement in model capabilities in these periods is also one of the main reasons for the increase in EVR.

In addition to comparing the accuracy of the candidate set and the registration rate of the top users, we also evaluate the performance of the model through the recall rate of the top users . This indicator represents the proportion of the total number of registered users that the top users selected by the model can cover. If the model has strong predictive ability, then the selected high-quality users should be able to represent high-potential registered users, and the top users should be able to cover most of the registered users. If the users at the end of the list include more registered users than the top users, it means that the probability predicted by the model may be biased.

Below is the recall rate of the top 5 million users of the game in the past two months. It can be seen that from the release on March 7 to April 4, the recall rate of top users increased from 13.3% to 88.6%, an increase of 75.3%. This increase is far greater than the change in the candidate set, which also verifies the improvement in model capabilities, indicating that the high-quality users selected by the model since the end of March have covered a higher proportion of registered users.

From the first part of the candidate set and model effects, it can be concluded that the changes in the candidate set and model from March to April are the two main reasons for the large fluctuations in the EVR of the game's virtual gold delivery. Next, we will pay attention to whether there are other factors that lead to the EVR change trend in the channel performance of the subsequent links of the funnel during the actual delivery process.

After the package is released, in the process of pushing the ad creative to users, due to user bidding or filtering by the business side, there will be a certain percentage of loss, and not all of them can be exposed. If most users, especially the top high-quality users, are not fully exposed in a certain delivery, while the exposure ratio of users with poor conversion rate at the end is higher, then the effect of the delivery is expected to be worse.

However, this result is not absolute. If only a small number of top users are exposed, but this exposure has covered most users, then the conversion effect may be better. Therefore, before looking at the user's click-to-download registration behavior, we need to monitor the proportion of users who actually push the package to users.

The following figure compares the exposure rates of the two campaigns (3.31 and 4.4) with the same level of packet generation and relatively close candidate set recall. It can be seen that the exposure rates of the top packet generation in these two campaigns are not high, ranging from 10% to 30%. The exposure rate of the 3.31 campaign is actually higher. Therefore, insufficient exposure of top quality users is not the main reason for the lower EVR of the 3.31 campaign compared to the 4.4 campaign.

In order to better explain the difference in EVR caused by the difference in exposure of top users, the following is a comparison of the exposure rate of another game in two releases. The EVR of the game released on March 3rd is much higher than that on April 8th. From the following figure, it can be clearly seen that the exposure ratio of top users in the release of March 3rd is much higher than that on April 8th, and the exposed users are concentrated in the top. On the contrary, the exposed users in the release of April 8th are concentrated in the tail, which is significantly higher than the tail exposure of the release of March 3rd. In the two releases of this game, the difference in the exposure rate of the package is one of the main factors leading to the good or bad conversion effect.

After pushing the copywriting materials to users, we need to pay attention to the user's channel performance. The next step after pushing is clicks. We need to monitor whether there are major changes in the overall click-through rate of the delivery. If so, what causes the click-through rate to fluctuate? Is it because the copywriting and materials are of poor quality, resulting in users not being willing to click, or is it because the quality of the delivered users is not high and they are not interested in the game's ads? Or is it because the exposure level of the tail users is too large, resulting in an increase in the total exposure level when the number of people who click is similar, reducing the overall click-through rate? For the latter two points, we designed a layered click-through rate contribution indicator to evaluate the impact of the click-through rate and exposure level of each 1 million users on the overall click-through rate.

The above figure shows the click-through rate of the game. It can be observed that the loss in the click link in March is much greater than that in April. The click-through rate in April is significantly higher than that in March. The click-through rate of 4.4 is nearly 3 times that of 3.31.

Taking March 31 as the experimental group and April 4 as the control group, we calculated the click-through rate contribution. The following figure shows that the difference in click-through rate is mainly caused by two reasons:

First, the CTR of the top 1 million users in the two campaigns is quite different (as can be seen in the red box on the left, the difference in CTR of the top 1 million users contributes 20% to the result that the CTR of 3.31 is lower than that of 4.4). This is because the quality of the top users in the two campaigns is different. The number of registered users among the top 5 million users in 4.4 is more than twice that of 3.31.

Second, the number of users exposed at the end of the 3.31 period is much higher than that of the 4.4 period. This can be seen from the exposure rate chart above. The large number of exposed tail users lowered the overall click-through rate (as can be seen from the red box on the right, the contribution of the tail exposure level is relatively high, indicating that there are too many users with a click-through rate below the average at the end of the 3.31 release, which lowered the overall click-through rate).

After the user clicks, it is time to download and register. There are two aspects here: one is whether the user registers after downloading, and the other is whether the registered user downloads through our channel/scenario.

From the first perspective, it is difficult to analyze why users do not register after downloading, because we believe that most of the downloading users have a preference for the game, but do not register the game after downloading. Perhaps the registration process is cumbersome or there is insufficient memory, etc. However, it is difficult for us to intervene in these issues. Therefore, in this link, we mainly focus on analyzing whether registered users have downloaded the game in our channel scenarios.

These users will be converted among the users who have received package delivery, but they are not attributed to our channel in the download link. If a high proportion of registered users do not download from our channel, what is the reason?

Here, the indicator we focus on is the proportion of users who registered but did not download. As can be seen from the figure below, in the 4.4, 4.7, and 4.14 periods with good conversion, only about 10% of registered users did not download from our channel (which could not contribute to the increase in EVR), while in March and 4.25, the conversion of several launches was poor, the proportion of registered users who did not download was about 15%-20%, indicating that more high-potential registered users lost in the download link, which was also one of the reasons for the poor conversion of the March launch.

If the above situation occurs, and a high proportion of registered users do not download from our channel, we need to analyze the reasons for this situation.

There may be two situations at this time. One is that these users downloaded through non-WeChat channels or external channels (not through WeChat, QQ, App Store, etc.). The other is that they downloaded through WeChat channels but not in our scenarios (official accounts, red dots). We will monitor these two situations separately in the SOP process.

For the first situation, the indicator we monitor is the proportion of WeChat channels among registered users who have not downloaded the app. We look at how many of these registered users who have not downloaded the app downloaded the app through the WeChat channel, how many downloaded the app through other internal channels, or how many were preempted by external channels.

For the second situation, we will pay attention to the proportion of users who registered on the WeChat channel and have not downloaded. This indicator can show how many users who registered on the WeChat channel downloaded in our scenario and how many downloaded in other scenarios of the WeChat channel.

We take the 3.11 and 4.25 periods, which have the highest proportion of registered users without downloading, as examples, and we can clearly see the difference between the above two situations:

• The figure below shows the proportion of registered users without downloads who registered through the WeChat channel. It can be seen that the proportion of registered users without downloads on 3.11 was the lowest in two months, at only about 43%. This shows that the high proportion of registered users without downloads in the 3.11 period was due to more users being preempted by external channels in the download link. In the 4.25 period, which also had a high proportion of registered users without downloads, 80% of the registered users whose downloads were not attributed to our channels were still registered through WeChat channels. This shows that the high proportion of registered users without downloads was not caused by being preempted by external channels.

• The figure below shows the proportion of registered users on the WeChat channel who have not downloaded the app. It can be seen that Period 4.25 has the highest proportion of registered users on the WeChat channel who have not downloaded the app in the past two months. This shows that although downloads have not been preempted by external channels, a higher proportion of registered users on Period 4.25 have no preference for public accounts and red dots, and download in other scenarios on the WeChat channel.

By using the SOP framework to attribute the EVR changes in the game from March to April, we can draw several main conclusions:

At the end of March, the candidate set for out-of-band testing was adjusted, and the model also underwent major changes, resulting in:

1. The recall rate of candidate sets in April increased, and the number of registered people covered was significantly more than that in March;
2. The accuracy rate of top users has been greatly improved, the model capability has been enhanced, and the selected top users have brought more registrations;
3. The click-through rate of the March campaign was much lower than that of April. First, the quality of the top users of the two campaigns was different, which led to a large difference in the click-through rate of the top users. Second, the number of users exposed at the end of the 3.31 campaign was much higher than that of the 4.4 campaign. The large number of exposed users at the end lowered the overall click-through rate.
4. In March and April 25, a higher proportion of high-potential registered users were lost in the download link. The March 11 period was due to more users being preempted by external channels in the download link, while the April 25 period was due to a higher proportion of registered users having no preference for the official account and red dot scenarios.

4. Final Thoughts

The V1.0 growth SOP focuses on post-package delivery, with the goal of reviewing the effects of a single delivery and quickly locating abnormal problems. It is a drill-down analysis after a specific delivery with a problem is found. How to find the problematic game and the corresponding delivery from the fluctuation of the market data is the key issue we need to solve next.

Next, we will pay more attention to the game delivery link. After the algorithm is released but before delivery, we will insert a mechanism to check the health of the package and find some common problems. This will facilitate timely replacement of some tail users based on the inspection results and improve the conversion efficiency of the package, that is, the pre-delivery analysis SOP, and the post-delivery analysis together to continuously improve the delivery efficiency.

Author: harrylhcao

Source: WeChat official account: Tencent Lecture Hall