How to evaluate the value of paid membership programs? This method is given to you for free

Editor's note: In penetration projects, testing and comparison are often needed to support quick operational decision-making. So which kind of testing and comparison is scientific and effective? Starting from the limitations of full-scale AB testing and comparison, this article proposes a new paid member comparison and evaluation method. Let's take a look.

1. Current Problems

As the business develops, we find that the full-scale comparison of AB testing has great limitations. For example, in the scenario of penetration projects, due to the low penetration rate in the early stage of the project, the full-scale AB testing comparison often leads to low overall incremental value and cannot support the operation to make quick decisions on whether to continue to invest in the project.

What is a penetration project?

Penetration projects refer specifically to products that are foreseeably unable to quickly achieve high user penetration during the operation cycle, such as paid memberships, points, etc. They are often in contrast to conventional AB testing projects where elements such as copy adjustment and UI optimization can have a 100% impact on the experimental group of users.

2. Solution

After reading relevant information in the industry, the author was unable to find a feasible solution that has been implemented in the industry, so he began to think about this issue.

The key points to solve this problem are:

Find the control group of users who have penetrated the node and compare the user value data of the two types of users.

What is a penetration node?

It can be understood that the core user behavior nodes of the penetration project are penetration nodes. For example, the "Purchase Paid Membership" behavior node of the paid membership project and the "Receive Points" behavior node of the points project.

Evaluation steps:

After thinking about it, I found a method. Since no one else in the industry has published this method, I will temporarily name this comparison method "Biluo Comparison Method". The specific steps are as follows:

1. Data testing

Test the penetration project according to the conventional AB testing method (the knowledge of AB testing is not introduced in detail here, and students who do not understand can search and learn on their own).

Assume that the project is divided into two groups, experimental group A and control group B.

2. Data Recovery

1) Collect the data on the number of users entering the experiment and user value of Group A and Group B respectively.

2) Collect the number of users and user value data of the infiltration nodes in the experimental group.

3. Data Analysis

1) User incremental value assessment of AB group penetration nodes

Calculation of the number and value of the unpenetrated population in experimental group A: Assuming that the number of unpenetrated users in experimental group A and the corresponding user value are a3 and A3, then: a3=a1-a2, A3=A1-A2.

Calculation of the theoretical number of penetrated/non-penetrated users in the control group B: Assuming that a2 and a3 correspond to two groups of users in the control group B, theoretically the proportion of the number of users should be the same. We assume that they correspond to b2 and b3, then: b2=a2*(a1/b1), b3=a3*(a1/b1)=b1-b2.

• Calculation of the value of the theoretically unpenetrated population in the control group B: In theory, the per capita value of the b3 population is consistent with that of the a3 population. We define the total user value of the b3 population as B3, then: B3=b3*(A3/a3) .
• Calculation of the theoretical penetration population value of the control group B: Based on the above data, it is easy to derive the theoretical penetration population value of the control group B, which we identify as B2, that is, B2=B1-B3 .

Calculation of incremental user value of penetration nodes in experimental group A:

△ The incremental value increase ratio of users in the penetration node of experimental group A = (b2*A2/a2-B2)/B2

2) Comparison of user value data of the A and B groups

Calculation of incremental value of users in experimental group A: △Increase ratio of incremental value of users in experimental group A = (b1*A1/a1-B1)/B1

3) Estimate the upper limit of the penetration rate of the penetration project, and use this to estimate the theoretical incremental value of the project at a high penetration rate

Assuming that the estimated upper limit of the penetration rate of the project is X, it is obvious that: 1≥X≥a2/a1. Assuming that under this penetration rate, the per capita value of the a2 and a3 groups remains unchanged (in fact, as the penetration rate gradually increases, the per capita value of both types of users may decline), then, under this penetration rate of the project, the incremental user value of the market will be improved overall under the new state, and the calculation logic is consistent with the calculation under the current state.

4. Give an example

1) Data testing

Taking a certain APP paid membership project as an example, we conducted AB testing on the paid membership project with a ratio of 5:5.

2) Data recovery

a. Collect the number of users entering the large-scale experiment and cross-user data of paid members in groups A and B respectively.

Cross-definition: Users who complete two or more business transactions on a certain platform.

b. Collect the number of users who purchased paid membership in the experimental group and the corresponding cross-user data. "The cross-user rate of purchasing paid membership is 24% = 240/1000"

3) Data analysis

a. Evaluation of the incremental value of users who purchase paid memberships

The number of users in experimental group A who did not purchase paid membership and the corresponding number of cross-users:

• The number of users in experimental group A who did not purchase paid membership = 10,000-1,000 = 9,000.
• The number of crossover users in experimental group A who did not purchase paid membership = 1050-240 = 810. "The crossover rate of non-paid members is 9% = 810/9000"

The theoretical number of users who purchased/did not purchase paid membership in control group B is calculated as follows: theoretically, the number of users who purchased paid membership and the number of users who did not purchase paid membership are 1,000 and 9,000 respectively.

• Theoretically, the cross-over efficiency of users in the control group B who did not purchase paid membership should be consistent with that of users in the experimental group A. Therefore, the number of cross-over users in the control group B who did not purchase paid membership = 9000* (810/9000) = 810.
• Theoretically, the number of cross-users who purchase paid membership in the control group B is calculated as follows: Based on the above data, it is easy to conclude that the theoretical number of cross-users who purchase paid membership in the control group B is 1000-810=190.

The proportion of incremental cross-users who purchased paid memberships in experimental group A:

△The incremental cross-user increase ratio of experimental group A who purchased paid membership = (240-190)/240 = 20.83%.

b. Comparison of user value data of Group A and Group B

Calculation of the incremental cross-user growth of experimental group A: △The incremental cross-user growth ratio of experimental group A who purchase paid memberships = (1050-1000)/1000=5%.

c. Estimate the upper limit of the penetration rate of the penetration project, and use this to estimate the theoretical incremental value of the project under high penetration rate. Assume that the estimated upper limit of the penetration rate of the project is 50%.

Assuming that at this penetration rate, the cross-efficiency between users who purchase paid memberships and those who do not purchase paid memberships remains unchanged (in fact, as the penetration rate gradually increases, the per capita value of both types of users is likely to decline), then at a penetration rate of 50%, the corresponding increase in the market's incremental cross-user ratio = (2600-1000)/1000 = 130%

Conclusion

To summarize:

The core of this solution is to find the value data of the same users in the control group of the penetration node users through simulation calculation, so as to obtain a relatively reasonable comparison.

This method has certain advantages:

When the project penetration rate is low and the incremental value comparison of the market is not obvious, this solution can find a control group of theoretical penetration node users, which can be applied to operations to quickly decide whether the project can be further promoted.

But it also has its limitations:

Just like AB testing, all incremental data can only reflect the current data increment and cannot be directly regarded as long-term data increment. As the penetration rate increases, the theoretical incremental value estimated by the project under high penetration rate is actually not accurate and may be blindly overestimated.

Author: Biluo, WeChat public account: Operational Meditations