In addition to ABtest, there are better testing methods!

Do you ever feel stressed when facing the business department's requirements for accurate predictions? Don't panic! Today, we will show you how to safeguard business growth through scientific testing methods without a data foundation. From deployment to review, the four-stage detailed explanation of the entire testing process will help you avoid blind predictions, speak with data, and make business growth based on evidence.

The business side wants you to make accurate predictions with artificial intelligence and big data. If the prediction is off by a certain amount, your salary will be deducted. Are you afraid?! There are many similar scenarios. Without further ado, here are the practical tips:

Problem scenario: An Internet financial company is planning to acquire investment clients through advertising. Recently, it has heard that private domain traffic is very popular, but it has never been tested on the official account channel. Now it plans to advertise on the official account channel. Please ask the data analysis department to use big data and artificial intelligence to accurately predict the advertising effect and how much money can be made from one advertising. Suppose you are the data analyst of this company...

Think for a minute

1. Don't forget you still have this weapon

Correct answer: Predict shit! The business department doesn’t understand the principles of data, and if they can’t figure out something, they can just bring out “big data artificial intelligence”. But people who work with data need to be extra clear-headed. According to the scenario in the question, the business has never done it once, and there is no data, so how can you predict it? At this time, you should do a test, collect some data first, and then talk about prediction after you have accumulated enough data.

Testing has always been a means of solving problems with data analysis. As the saying goes: whether it is a mule or a horse, take it out for a walk. Testing is the process of walking a horse, and the results are clear at a glance. Especially for innovations such as new channels, new products, and new teams. If old data cannot fully deduce innovation, testing is even more necessary.

However, in recent years, AB tests based on self-owned apps have been popular, so many newcomers have ignored the more common test design methods. Today we will explain them systematically.

2. Basic requirements for design testing

Many newcomers will take it for granted that testing is just about letting the business do it a few times, so let them do it and we can just wait to collect the data. This lazy thinking will add endless troubles to yourself afterwards.

First, testing has a business cost.

For example, in this scenario, the purpose of the investment is to attract new users and new investments. You have to see results after spending the money, otherwise you will definitely be dissed by your boss.

Then the related questions arise:

• How much does it cost?
• How many effects do you want?
• How many rounds of testing are needed to see the effect?
• Can you afford the losses from several rounds of testing?

These must be clearly defined in advance to avoid confusion.

Second, the test is designed with content.

For example, in this scenario, whether users will be attracted or not is related to the type of number, timing, copywriting, conversion path, product selection, and CTA action. If you don’t do a careful design at the beginning and just simply throw one, many other ones may not be able to be compared and tested, and no effective conclusion can be drawn.

Again, testing is impacted by investment.

For example, in this scenario, it is possible that high-quality channels need to spend a lot of money, and it is possible that the user subsidy is a little larger than other channels. The result is that the first round may not be effective, but the second round of increased efforts will be effective! Therefore, whether to make additional investments or not must be considered in advance.

Due to the above three points, the test needs to be divided into four stages, and full preparation must be made before going on the road (as shown in the figure below).

3. Deployment Phase

The deployment phase addresses strategic issues:

• How long should the test last?
• How much resources can be put into testing
• What test results are satisfactory?

In this scenario, as a test of a new delivery channel, you must first clarify the positioning of the channel. Common ones are:

• Main channel: responsible for more than 50% of traffic sources and the main investment direction
• Assisting channels: responsible for more than 20% of traffic sources, secondary investment direction
• Marginal channels: single channel traffic does not exceed 5%, selective delivery
• Scattered channels: It doesn’t matter if you have it or not, but it’s better than nothing

You can reverse the required traffic based on the current overall channel delivery target; then, according to the business strategy (determined to establish a new channel or just follow the trend), clearly allocate tasks and define the positioning of the new channel for this test. With a clear positioning, it is easy to figure out how much money to invest and how many times to do it. With the definition of financial resources, manpower, and time, the subsequent design plan is simple.

4 Preparation

The preparation phase is about solving tactical problems:

• Which channels to test?
• Which versions are tested?
• Which products are tested?

In this scenario, since there is absolutely no experience, we need third-party/peer cases and data to support us. Although we cannot get 100% accurate data, we can at least copy it, for example:

• Which accounts are invested by peers?
• What is the observable article exposure?
• What is the conversion rate from hearsay?
• What form is used in the industry

After sorting out, at least there is a general direction, which is much better than blindly doing it with your eyes closed. Note that from the user's perspective, the factors that affect user behavior are comprehensive. For example, the public account channel release, title, length, release time, content writing, CTA, conversion path, product price, product attributes, etc. will all have an impact.

When testing with data, it is difficult to break down all the above factors in one test. Therefore, it is necessary to prepare multiple test versions in advance. The differences between the test versions should not be too large and there should be a certain continuity, so as to prepare for later analysis.

All of the above are highlighted in red because in actual work, business people often like to get hung up on details, which results in different versions that are completely incomparable. Apart from looking at the overall conversion results, the details are completely unmatched, so it is difficult to do in-depth analysis.

It can be said that 80% of the difficulties in post-analysis are caused by the lack of prior planning . Remember this.

5. Testing and review phase

After preparation, you can launch the test and review. This scenario is channel delivery, and the goal is to acquire new investment users, so the evaluation result indicators are relatively simple and clear, and you can look at the number of converted users, user investment rate, and user investment amount. As long as the test results can achieve the goals of the deployment phase, the channel is qualified and the task is completed. If not, you can perform iterative optimization according to the pre-made iteration plan to further observe the effect.

Here are some details to emphasize:

1. The result judgment and cause analysis should be separated. First determine whether the result is acceptable, and then analyze which link has problems.
2. Iterations should be done in order, with products, prices, and content separated. It is best to change one at a time, and at worst, do not change all three at once.
3. Prioritize changing products and prices, not changing the content one by one. The content involves too many details, and it is too costly to measure them all, so give priority to products and prices.

When looking at the data in this way, you can expand it in the following order:

VI. Summary

Why do we ask at the beginning how to make accurate predictions with artificial intelligence and big data? Because many students really think that predictions can be made! Not only do business departments believe in predictions, but even many data analysts believe in them. They really think that just a few numbers are big data, and that just a model and a parameter adjustment is artificial intelligence. They really think that artificial intelligence is an omniscient and omnipotent God, and that a golden light will fall from the sky and that the code will turn into money, and that it will burst out of the screen as the keyboard is tapped...

• Data analysis, only with data can we analyze
• No data, collect data first
• I have never done business before, so I will try it first.
• Don't try blindly, make a plan carefully and try step by step

These traditional processes that seem to have no technical content are the secrets to using data to ensure business growth. Since this scenario involves external channels, the idea of ​​using AB test to divide traffic for comparison is not applicable.