Data science also uses a statistical method called "multivariate analysis" that analyzes and interprets multiple variables simultaneously. For example, in a survey on people's health, various items such as "age, sex, height, weight, eyesight..." are collected in addition to physical condition. Each of these items is treated as a variable, but if you analyze only "height" as data without looking at other variables, it will be difficult to obtain the intended analysis results on people's health. Multivariate analysis analyzes variables in a complex manner, and can derive analysis results such as what tendencies can be found in certain types of people.
However, while it would be ideal if all the data could be collected smoothly, some data will be lost, for example, if someone forgets to answer a question. This is called "missing values." Roderick JA Little and Donald B. Rubin have proposed the following three main mechanisms for how missing values occur:
The first, "MCAR," is a case where, for example, in a questionnaire survey, a person does not answer a question due to an oversight. In this case, the probability of missing values occurring is completely random, not dependent on the variable in question or other variables. The second, "MAR," is a case where missing data does not occur completely randomly, but is dependent on other variables and can be explained by those variables. For example, in data on income, the probability of not answering questions about income increases as people get older, so the data contains a variable related to age. In this case, missing data on income can be explained as occurring randomly with age as a condition. And the third, "MNAR," is a case where, in the previous example of income, income and age are not dependent, and people with higher incomes are more likely to not answer questions about income. In other words, the data does not contain a variable that can explain missing income.
When such missing values exist, even if they are "MNAR", data analysis cannot be performed unless the missing values are dealt with somehow, and the value of the data cannot be found. Of course, there is the option of discarding the entire part containing the missing values, but this would be a waste of time and money spent on aggregating the data, and it is not possible to re-collect the data multiple times, so how to fill in the missing values or how to handle data containing missing values is a major issue in modern data science. The graph below proposes that statistical methods can be used for data containing missing values, and the approximation accuracy can also be improved by adding a mathematical approach to achieve higher approximation accuracy.
In clinical trials of pharmaceuticals, the occurrence of missing values is inevitable, such as when a participating patient is unable to continue the trial due to the effects of their disease or for some other reason, so the handling of missing data is also described in the guidelines. In the future, anyone who handles data, not just in the pharmaceutical field, will need to have knowledge of how to handle missing values.
Simulation results for data with missing values
If the sample size is large enough, the curve approaches the red curve. By adding a mathematical approach modification (pink) to the proposed method (blue), the accuracy of the approximation can be improved even if the sample size is not large enough.
