The ultimate question of data analysis: How big is the fluctuation of the indicator considered large?

In the world of data analysis, understanding the importance and impact of indicator fluctuations is a skill that every analyst must master. This article explores in depth how to define and deal with indicator fluctuations, from hard indicators to soft indicators to marginal indicators, revealing the meaning behind fluctuations and their impact on business decisions.

"Analyze today's fluctuations" is the most common task for data analysts. It is also the most troublesome, the most tangled, the most difficult to figure out, and the task that needs to be done every day.

A 1% drop is not considered a big fluctuation.

A 5% drop is not considered a big fluctuation.

Is a 10% drop considered a big fluctuation?

Is a 50% drop considered a big fluctuation?

Why is it that sometimes the business does not respond when the number drops by 50%, but the business is very anxious when the number drops by 1%!!! Today we will take a look at the system.

1. The nature of indicator fluctuations

For example, a temperature of 37.4 degrees VS a temperature of 36 degrees, there is only a 3.9% fluctuation. But if you are found with a temperature of 37.4 degrees at an epidemic temperature measurement point, you will probably be asked to leave by the security guard immediately. Why? Because people are not afraid of the 5.5% fluctuation, but the virus! A temperature of 37.4 degrees indicates: there may be a virus! This is what people are really afraid of.

Therefore: indicator fluctuations are not scary, but the business implications of indicator fluctuations are scary! Talking about indicator fluctuations without considering their business implications is just a hooligan. Only by understanding this can we continue the discussion.

2. Meaning of Index Fluctuation

▌ Category 1: Fluctuations in hard indicators.

Some indicators are rigid assessment indicators for business departments.

Sales Assessment: Performance and Payment Collection

Assessment of goods: inventory, gross profit

Assessment of customer service: answering calls and complaints

These indicators are rigid assessment results, which means that the specified number must be achieved, otherwise even a 1% difference will be a problem. Therefore, they are often called hard indicators. This is most obvious in sales. If the performance target is set, even if it is only 0.5%, it is not achieved, and there will be no bonus!

Departments that are responsible for rigid indicators are the most sensitive to fluctuations and are the most meticulous. Failure to meet rigid indicators may directly lead to being scolded or having their salary deducted. Therefore, fluctuations in rigid indicators are of particular concern.

▌ The second category: soft indicator fluctuations.

Such as indicators such as the number of registered users, user click-through rate, and conversion rate.

These indicators are often the process leading to business results, just like a registered user first, then browsing, adding to the shopping cart, and consumption. The rise and fall of soft indicators is not necessarily a problem, it may be a new business form (as shown in the figure below), or it may be an accidental change.

Therefore, changes in soft indicators will not directly trigger business actions. People are more concerned about whether the change is good or bad, and whether it will have a potential impact on hard indicators. This kind of tangled emotions will make analysis particularly troublesome.

Note: The distinction between hard and soft indicators is not static. For example, many Internet companies will examine "user growth", and the number of registered users is a hard indicator that forces the promotion department to complete. Therefore, the distinction between hard and soft indicators depends on the specific KPI requirements of the department.

▌ The third category: marginal indicator fluctuations.

Such as satisfaction, popularity and other indicators. These indicators have common characteristics:

1. The data is obtained through sampling survey, not full statistics. This means that non-business actions such as sampling method, questionnaire method, and survey time may also affect the results. It cannot directly reflect business issues.

2. It has little to do with hard indicators and process indicators, or the results are difficult to verify directly. For example, high satisfaction does not necessarily mean 100% purchase; low satisfaction does not necessarily mean no purchase.

3. Human manipulation has a great impact. For example, changing the sampling method will immediately change the results. For example, throwing in a wave of advertisements/discounts will immediately increase the value.

This kind of inaccurate, useless, and easy-to-manipulate indicator will also fluctuate and attract people's attention. But if you understand the logic of these indicators, you will find that it is too easy to control the fluctuations, just play with numbers.

After understanding the three major types, you will have a sense of direction when dealing with indicator fluctuations: hard indicators > soft indicators > marginal indicators. Focus on the key points in this order. Don't be anxious and scratch your head when faced with a screen full of high or low indicators.

With the distinction between primary and secondary, we can further consider the standards for judging size.

3. Criteria for judging the size of fluctuations

▌The first step: eliminate false fluctuations.

A lot of fluctuations are natural fluctuations.

For example, the difference in transaction volume between weekends, holidays, and weekdays.

For example, the changes in the number of users when a product is launched, becomes a hot seller, or is delisted.

For example, the reading rate of a public account article decreases within 7 days after it is published. These indicators are naturally variable.

If you summarize the empirical indicator patterns more often, you can find the rules (as shown below):

After discovering the law, any fluctuation that conforms to the law is a pseudo-fluctuation! Don't panic if the pseudo-fluctuation is large, it is a common occurrence. But if it goes against the law, then there must be something wrong! Regardless of the size of the fluctuation, it is a major change and must be carefully observed.

▌Step 2: Quantify proactive behavior.

Many fluctuations are actively triggered by the business.

For example, promotion, boost sales, training, and enhance work capabilities

For example, clearing inventory and getting rid of inventory as quickly as possible are changes in these indicators that are caused by the business itself.

In this situation, we must first gather information about what the business is doing. Otherwise, after analyzing for a long time, people will say, "I knew it a long time ago" or "I did it." This will make people laugh.

Secondly, we need to clearly collect the goals and results of each business action, so that it is easy to evaluate whether the indicator fluctuation meets the business expectations. This is an important evaluation standard, so be sure to mark it in red and bold. When the behavior is proactive and the indicator fluctuation meets the expectations, the business will not be entangled. When it fails to meet expectations, they will want to know: "Where is the difference?" At this time, it is very important to find the gap with the business expectation value (as shown below)

For those who have achieved business expectations, no matter how large the fluctuation range is, it is acceptable. Since it is an active increase/decrease, the larger the indicator change, the better. For those who have not achieved expectations, we need to look at the gap between expectations, and the gap is the fluctuation value to be analyzed.

▌Step 3: Quantify external impacts.

There are many fluctuations caused by collectible external behaviors, such as policy restrictions, weather, competitors, etc. Note: There are many external factors that cannot collect data and verify the impact. There are also many that, even if the impact is known, there is nothing you can do - always saying that rain affects performance, but you can't burn incense and pray to the Dragon King.

Therefore, when evaluating the fluctuation of external influences, we should not look at the absolute number of one day, but rather calculate the expected duration of the impact and estimate the total impact value within this time. This value is the standard for measuring fluctuations.

▌Step 4: Other unexpected fluctuations. Are there any fluctuations that are not in line with the rules, have no business initiative, no external factors, and no problems with the data itself?

Yes! At this time, we should first locate the point where the fluctuation occurs:

Global fluctuations or local fluctuations

Continuous or sudden

Fluctuation value, large or small

Criteria for judging the size of a problem:

Global > Local Issues

Persistent > Short-term issues

The bigger the number, the bigger the problem

After locking in the problem point, you can consider the countermeasures based on the attributes of the indicator (as shown below):

Regarding fluctuations in hard indicators: As long as hard indicators are not met, it is a major problem. Consider taking measures to maintain the indicators.

Regarding the fluctuation of soft indicators: as long as the related hard indicators do not collapse, it is not a major problem. Don’t dwell on the fluctuations of one day or one night, and focus on discovering the underlying reasons.

Regarding marginal indicator fluctuations: Don’t be afraid! It only takes a few minutes to turn it around.

After making this distinction, you will have a clear direction for handling. If you need to use drastic measures, you should act decisively, and if you need to take your time, you should take your time. Otherwise, if you do not distinguish between the important and the urgent, and just slowly "disassemble the data-disassemble the data-disassemble the data", you will either be disdained for "making a mountain out of a molehill" and "I knew it a long time ago", or the business has already dealt with the problem, but the data analysis report here is not yet ready.

4. Why do people always worry about fluctuations?

To summarize: If you want to calmly deal with indicator fluctuations, you need the following two points:

▌Business departments know what they need to do:

1. Know which are hard indicators, soft indicators, and marginal indicators

2. Know the extent to which your behavior can affect the indicators

3. Know what you can do in the short, medium and long term

4. Be clear about whether your short-term actions have achieved results

▌Data departments, know what happened:

1. What are the business initiatives and how much do they want to achieve?

2. What are the regular changes and what is the range?

3. What quantifiable external factors can bring about changes?

4. What are the abnormal changes and where are they located?

Unfortunately, the reality is that the business department only works hard. They never quantify how much they want to do, how much they can do, and how much they have done. They get scared when they see a little fluctuation in the indicators (as shown below).

The data department does not understand the business meaning of indicators, does not know what the business is doing, and does not know how to quantify the rules. They only take an indicator and cross it with gender, age, region, channel, etc., and put a bunch of pillars together. If one of them is short, they will shout: "The fluctuation is because this one is short!" They also call it "multi-dimensional decomposition method" and write articles on the Internet to poison newcomers...

This is what it means to be a blind man riding a blind horse. As for hoping that "a senior data scientist from Tencent or Alibaba can build an artificial intelligence big data model and know everything", it is like hoping for a life-saving elixir.

1. Quantify business goals and business behaviors

2. Sort out business logic and summarize it into reports

3. Summarize historical experience and development trends

4. Assessing the current situation and measuring the gap are the basic, simple and detailed tasks that are the best way to deal with "indicator fluctuation anxiety".