The entire process of implementing data growth experiments

Data growth is the key to data analysis for data analysts, and data growth experiments can identify problems in advance and ensure growth. In this article, the author introduces the entire process of implementing data growth experiments, using false data growth and basic growth models, and deeply considers the dimensions of growth foundation, cycle, and implementation. I believe this will bring a lot of experience to new data analysts~

I have shared before: Growth is the best way for data analysts to make contributions. Today, let’s take a look at an example of how to design growth experiments through data.

Problem scenario:

A fast-moving consumer goods company with multiple product lines hopes to launch a new beverage (2 SKUs) to boost overall sales. This product is new and lacks experience, so it plans to conduct a trial this year and then promote it on a large scale after observing the effect.

Q: How should growth experiments be designed to detect problems in advance and ensure growth?

1. False data growth

Many new students raised their hands and said that they would:

1. Get all the information from Toutiao, Tencent and Alibaba big data
2. Establish a conversion funnel from user to store - shelf - selection - adding to shopping cart - checkout
3. Conduct AB test, and automatically code and divert users entering the store for AAAB comparison
4. Build user portraits to accurately identify target users’ gender, age, income, and hobbies
5. Build an artificial intelligence big data model to accurately predict organic sales

The reality is: there is no data. Because we don’t have our own channels, we can only get the purchase data. Forget about other data, they don’t exist. We don’t have any data. We can rely on store supervisors to regularly check whether the stores have stocked the products.

So, what to do?

2. The most basic growth model

The simplest idea is that the purpose of launching new products is to boost sales, so after launching new products, the channel orders more than before launching new products. So the simplest model comes out (as shown below):

Then, the experimental design seems to be very simple:

1. Find a few stores
2. Distribution
3. Observe sales after distribution
4. Done

Is it really done?

3. Consider the foundation for growth

The first question: Do you look for stores randomly or with a specific goal in mind?

It is very likely that some stores are born to sell well, while others are born to sell poorly. If you do not analyze the store's past order situation in advance, you are likely to overestimate/underestimate the growth capacity. Pay special attention to whether there are specialty stores. If there are too many of these stores, it may affect the overall judgment (as shown below):

When selecting pilot sample stores in the early stage, do a good job of screening in advance and consider:

• Store location: Community store/CBD store/Pedestrian street store
• Store performance: overall performance is good/average/poor
• Category performance: Beverages good/average/poor
• Store hours: New store/Old store

These data are available. Data 1 is recorded in the supervisor's store inspection table, and data 2, 3, and 4 are recorded in the order form, so they are completely available. What needs to be done is to analyze the data in advance and do a good job of stratification and labeling.

The intersection of so many dimensions raises a new question: how many stores should be selected for the pilot? Statistics will tell you that the minimum sample size for a single group is 30, and the best is 384, so that the sampling error is 5% at a 95% confidence level - but these have little to do with the problem at hand.

Because the question at hand is:

• The draw needs to be done on a store-by-store basis, and it is possible that not all the stores combined will have enough draws.
• The products being tested are new products, and the testing cycle may be long, which means that the supply may not be sufficient.
• The products being tested are newly listed products, which require the business side to distribute to each store and point, so the workload has to be considered.

Therefore, when designing the number of samples, we first estimate the sales volume of a single store during the test period to ensure that there is stock. This is the only way to truly test whether it meets expectations. After determining the total number of stores, we can then add samples according to the above dimensions. The final result is to ensure that there are as many samples as possible in each category.

If you have classified the stores into first-level, second-level, and third-level stores in advance, it will be much easier. Because the first-level, second-level, and third-level classifications are likely to have comprehensively considered factors such as sales capacity and store size. However, you should pay attention to a few issues before using them:

• Are the previous 1, 2, and 3 level classifications still accurate? Don't let the situation of 3rd level>2nd level>1st level occur, as that would be very embarrassing for the subsequent analysis.
• Consider the types of first, second and third level stores. Avoid having all first-level stores be of the same type (for example, all CBD stores). This will lead to a serious lack of samples of other stores in the post-evaluation.
• Are levels 1, 2, and 3 related to beverage sales? Note that the company in question is a full-line company, and it is very likely that levels 1, 2, and 3 are classified according to overall performance. Corresponding to the beverage category, there will be an extreme situation of level 3>level 2>level 1.

As long as the above problems do not exist, level 1, 2, and 3 classifications can be used directly.

Considering the growth basis not only makes the design more comprehensive, but also greatly facilitates the subsequent evaluation. Avoid embarrassing problems such as:

• Why the test results are not good? Because the stores I found are all very bad.
• Why can’t I analyze the promotion potential? Because I’m looking for stores of the same type.
• Why do level 1 stores sell poorly? Because they are not naturally good at selling.

And in the post-analysis, we can conduct an in-depth analysis of the situation under each type of store label, specifically the effect of each type of store label. This way, when iterating experiments, we will have a clearer direction and more ideas when implementing them (as shown in the figure below).

So, is this enough to take this step into consideration?

4. Consider the growth cycle

Step 2: Consider when to test and how long to test.

Generally, all commodities have their own sales cycles, and beverages have a more special cycle, which may be concentrated in the summer, or may be affected by the climate of each region, or may be affected by the weather in the short term. Therefore, when designing the test cycle, it is necessary to first sort out the trends of beverages with similar prices, similar types, and similar target groups, so that you can have a global judgment (as shown in the figure below).

After making a global judgment, you can set a relatively long observation period to cover as many scenarios as possible. This way, you can also analyze various situations during the post-evaluation analysis (as shown in the figure below).

5. Consider growth implementation

Step 3: Consider business implementation actions.

When a new product is launched, publicity, distribution and promotion are often carried out together. These implementation actions are the factors that ultimately determine the test results. These actions all rely on the execution of branches/offices in various places, and execution is crucial.

Here is a very profound question: once the test results are not good...

• Is it that there is no demand for the product itself?
• Or is it because I didn’t do my job well?
• Or did the data analyst make a mistake?

Only if you monitor the business execution process, you are qualified to say: the business is done well or not. If you do not monitor the business execution process, people can always say: the data analysis is not calculated. "Now it's all artificial intelligence and big data, it must be that our data analysts are too stupid, and we can hire a data analyst who can definitely calculate it clearly" - this pot has been prepared for you, so you must understand it clearly.

The information to be obtained includes:

• Distribution start time
• Delivery completion time
• Replenishment order time

With this information, we can combine the order data to do more analysis:

• Is there any delay in starting for a long time?
• Have you started the slow progress?
• Is there anyone who paved the road with his eyes closed regardless of scale?
• Is there any shortage of stock?

Of course, the subsequent verification must keep up, and during the verification, several key dimensions can be checked, such as:

• In the hot summer, put the goods on the shelves instead of putting them in the freezer.
• Hypermarkets don't make shelves, they only make shelves
• There are promotional materials that are not brought into the store.

These verification data must also be collected from the supervisor and analyzed together with the data to make it easier to see the results.

In this way, when explaining the results, you will naturally be more confident: if there is any failure in execution, you are not allowed to blame the product/data, and you should reflect on your own failure. This will also help to find the answer to the real problem.

V. Summary

The biggest problem in the field of data analysis is that the learning process is book-based and out of touch with reality. In order to make the algorithms, statistical principles, user portraits, funnel models, and AB tests work, books, teachers, and courses that teach data analysis specifically pick out some data sets with rich fields and cleanliness to run the algorithms. Newcomers treat their work like studying, and they are happy and self-satisfied when they run a few data sets.

The result of these two is that when newcomers encounter practical problems, they either imagine that Touteng has a panacea, or they rush to look for answers in books, or they go to various groups to ask: "Is there any big boss in the Internet beverage industry? It's urgent. I'm waiting online. I can pay!" The only thing they have lost is the ability to analyze specific problems.

The solution to the problem is to get rid of superstition, be down-to-earth, study the business process carefully, and design reasonable methods. There are simple ways to deal with simple data, and there are rich ways to deal with rich data. Simple data can be enriched through business process improvements. The combination of these three is the ability that a qualified data analyst should have.

Author: Down-to-earth Teacher Chen

Source: WeChat public account "Down-to-earth Teacher Chen (ID: gh_abf29df6ada8)"