How to build an activity effectiveness evaluation system?

Students who do data segmentation often encounter the problem of evaluation activities. This article introduces common mistakes in evaluation activities, key questions, how to answer them, etc., and I hope it will be helpful to you.

"If you were asked to evaluate this event, how would you analyze it?" Whether it is an interview or a job, students who do data analysis often encounter this question. Today we will explain it systematically.

Scene restoration:

A music APP offers a 7-day membership to new users upon registration. After registration, users can decide whether to click to receive the membership. The membership period lasts for one month. Question: How would you evaluate this activity?

1. Common mistakes in activity evaluation

First of all, remember that for all questions that use evaluate/evaluate/judge as verbs, there is only one answer: "good or bad".

For example, how would you evaluate this activity? You can answer:

• This activity is very good and should be continued;
• This activity is not good and cannot be done
• This activity is neither good nor bad, it's useless
• This activity has not changed anything, so it is useless to do it

This is the core conclusion of the evaluation analysis. Without these four sentences, everything else is nonsense. For example:

• 40,000 new users registered during the event
• The number of registered people during the event was 10,000 more than before the event
• The click-through rate of new users during the event was 80%
• During the event, new users will enjoy a 30% interest rate

These are not conclusions at all, but just analysis processes. If you don’t have a conclusion and just throw out these process indicators, it is easy to be asked back by the business side: “So what? So what? What did you analyze? Where is the conclusion!” In the end, you will be embarrassed.

II. Key issues in activity evaluation

When evaluating an activity, we first need to get a good/bad evaluation. If the evaluation is good, we then see whether we can continue to do it and how many times we can do it. If the evaluation is bad, we then see where the problem is, whether it is hopelessly bad or can be saved and continued.

Data + labeling = judgment. Therefore, to make a good/bad judgment, you need two things:

• Clear assessment indicators
• Clear target value

If it meets the standard, it is good; if it does not meet the standard, it is not good. It is that simple (as shown in the figure below).

It seems simple, but in fact, operations often do:

• Confused: I just do it because my boss tells me to do it, but why? I don't know, and I don't dare to ask.
• Dai Toudainao: I just want to recruit new people, and that’s all, OMG!
• Taking shortcuts: It has been done before/others are doing it, so just do it.
• Fishing in troubled waters: This is about changing users’ mental resources, which cannot be measured by data!

In short, only two out of ten operations can accurately describe the current situation and goals. At this time, data analysts need to have independent judgment ability. Only by analyzing business logic and sorting out business processes can objective conclusions be drawn. Here we take a completely confusing scenario as an example to see how to help operations clarify their goals.

3. How to build an evaluation model from scratch

1. Sort out the activity process

Operational activities will change user behavior, which will then be reflected in changes in data indicators. To build an evaluation model from scratch, the first step is to understand the specific process of the activity and the changes in user behavior that may be caused by the activity. For example, the new user benefits in the question can be sorted out as follows (as shown in the figure below):

After understanding the changes in behavior, we can further see what data can be used to record these behaviors and what indicators can be reflected. After sorting out, we can clearly see the indicators for measuring the results of the activities. These tasks should be done by operations in the planning stage. If they are not done well beforehand, they will have to be made up afterwards.

2. Screening main indicators

An activity may affect many aspects. For example, in the question above, there is a gift giving event. You said:

1. Can increase new user registrations - no problem;
2. Can increase the chances of membership purchases - seems to have a chance
3. Can increase loyalty and reduce churn - seems to make sense
4. Can increase DAU! ——Well, in theory, if there are more new registrations, DAU will also increase.

If you don't look at the data and just listen to what people say, of course all of the above makes sense. But if you really want to calculate the percentage of users who have not lost, the percentage of free memberships, the percentage of the product itself, and the percentage of the number of songs, it would be a miracle if you can calculate it clearly. Therefore, the evaluation indicators should be prioritized to make it easier to explain the problem.

If the goal is set in advance, the main indicator of the activity should be closely linked to the goal, and the indicator that is directly affected should be preferred. For example, if the activity is to attract new users, the main indicator is the number of new registered users; if the activity is to increase the retention rate of new users, the main consideration is the retention within 1-7 days.

This may seem simple, but it is easy for operations to fish in muddy waters. Operations often like to talk about a bunch of impact indicators, and even say things like "my activities have deeply changed the user's mental cognition, thus achieving a far-reaching impact that cannot be measured by data." In short, they bring in a bunch of indicators, and talk about which ones look good, and don't talk about the ones that don't look good.

When doing data evaluation, the worst thing to do is to use hundreds of indicators and then create a very complicated evaluation formula. The more things that are mixed in, the more room there is for word games, and the easier it is to whitewash the situation. The simpler and clearer the evaluation, the easier it is to spot problems.

3. Set judgment criteria

With clear main indicators, we can find the judgment criteria. There are four basic ideas for finding the criteria:

1. From the overall results, look at the total amount. For example, if you need 100,000 new users this month, you must achieve 100,000.
2. Compare with no activity and look at the increment. For example, if there is no activity for 50,000 a month, the activity must be 5+50,000, and the extra 50,000 is used as the standard.
3. Compare with past activities to see the efficiency. For example, the new customer acquisition activity usually costs 100 yuan per new customer, so this time it cannot exceed that.
4. Compare with non-participating users to see the difference. For example, divide into non-participating group/participating group, and compare the number of new registrations/retention rate of the participating group.

From the company's perspective, the first method is definitely the most realistic. But from the perspective of organizing activities, everyone likes to highlight their own contributions, so they tend to use methods 2, 3, and 4. Operations like to talk about: natural growth rate (what is the natural growth without activities). And they always tend to calculate the natural growth rate too low, or even make it a negative number, so that the activities appear to be extremely awesome.

It should be noted that the so-called natural growth rate can only be calculated when activities are not frequent. In many businesses (such as e-commerce and O2O), there are constant activities, and large activities are embedded in small activities, which cannot be distinguished at all. Therefore, it is not applicable in this case.

Similarly, the premise of setting up a reference group is that the reference group does not have the activity reminder and activity participation functions, and the quality of users in the reference group and the activity group is similar. If this restriction is not met, it is difficult to directly conclude that the activity is effective. These methods are all seemingly scientific, but in fact they are full of room for tricks. If you want to discuss the problem clearly, just set the standard directly.

4. Find the process that affects the results

Steps 1, 2, and 3 above are all for the purpose of making a judgment. With a "good/bad" judgment, we can further analyze why it is good or bad. At this time, we need to refine and sort out the business process to find the key points that can affect the results. For example, new customer acquisition activities, advertising channels, advertising copy, registration process, the way to prompt benefits after entering, and the process of obtaining membership benefits will all have an impact (as shown in the figure below).

When analyzing these indicators, you should pay attention to the order of priority. For example, regarding new user registration, you should first look at the strength of the campaigns from each user source channel, whether the campaign is launched in a timely manner, and when it is combined with the campaign. Only then can you conduct an in-depth analysis of the copywriting, event gifts, post-receipt behavior, etc.

IV. Summary

If you use this question in an interview, you can tell the difference between rookies, novices, and veterans at a glance:

Rookie: I think we should consider four aspects: user growth, product experience, brand influence, and commercial revenue, etc. Then you can ask him how to measure brand influence, and you can basically ask him to death. The biggest difference between rookies who have not yet started and novices is that they don’t know that data needs to be collected. Rookies like to make up some indicators that are difficult to collect but sound awesome.

Newbie: I think we should consider the number of new user registrations, the retention of new users on the next day, 3 days, and 7 days, the membership claim rate of new users, the retention market of new users, the number of new user clicks, the number of new user membership purchases, the frequency of new user rights use... (more than 100 indicators are omitted here). Establish a neural network model for comprehensive evaluation.

Yes, the first thing newcomers like is to pile up indicators, as if the more indicators, the better. The second thing they like is to talk about models, regardless of whether the model can be used or not, and whether they have used it themselves or not. They think they are awesome if they talk about it. The only thing they will ignore is the standard. If you want to kill newcomers, just ask "Why is a growth of 50,000 good, but 49,127 is not good!" "You build a neural network evaluation model, who will label good/bad activities? Do you label the entire activity or certain indicators? Why are different types of activities labeled together?" These questions basically confuse newcomers.

In the final analysis, activity evaluation is a matter where clear thinking is more important than advanced technology, and maintaining integrity is more important than clear thinking. As long as you have a basic understanding of common operations activities: attracting new customers, promoting activation, retention, conversion, fission, etc., and have an understanding of regular operating data and understanding of past activities, you can basically answer the correct answer. The only thing to worry about is that you know too little about the business process and don't even understand what others are doing.

Author: Down-to-earth Teacher Chen, from the official account: "Down-to-earth Teacher Chen"