Open Access Mini Review

A Note on Treatment Effect Estimation in Randomized and Observational Clinical Trails

Ying Wang2, Ao Yuan1* and Ming T Tan3

Department of Biostatistics, Bioinformatics and Biomathematics, Georgetown University, USA

Corresponding Author

Received Date: July 30, 2025;  Published Date: August 19, 2026

Abstract

The randomized clinical trial is used as golden standard, and various classical methods can be used to estimate the treatment effect. When the clinical trial is not randomized it is known that the naive estimate (group sample mean difference) of treatment effect is biased, and causal inference methods is needed to get unbiased estimate. However, in this case it is still not clear for other non-causal classical estimates, such as least squares estimate, regression model (linear or non-linear) estimate, can they implement the treatment effect and give unbiased estimates? Here we look into this question by reasoning and simulation studies. Our conclusions are provided at the end in terms of linear and nonlinear models.

Keywords:Randomized clinical trial (RCT); Inverse propensity scores weighted estimate (IPSWE); Doubly robust estimator (DRE); Naive estimate (NE); Regression coefficient estimate (RCE)

Introduction

The Randomized Clinical Trial (RCT) is used as golden standard, and various classical methods can be used to estimate the treatment effect in this case [1-4]. However, in practice, RCT has its drawbacks, such as time consuming and high costs [5], conflict of interest dangers [6], and ethics and feasibility [7]. Also, sometimes randomization is not possible or difficult to implement strictly. In these cases of observational studies, it is known that the naive estimate (group sample mean difference) of treatment effect is biased [8], and causal inference methods is needed to get unbiased estimate, such as the recently commonly used doubly robust estimator [9-13]. However, in this case it is still not clear for other non-causal classical estimates, such as least squares estimate, regression model (linear or non-linear) estimate, can they implement the treatment effect and give unbiased estimates? Here we look into this question by reasoning and simulation studies.

The Question and Method

Consider the observed data irispublishers-openaccess-biostatistics-biometric-applications iid (Y, X , A), i y is response, xi is covariate, and ai is binary treatment assignment, ai =1/ 0 for treatment/control. The interest is to estimate

irispublishers-openaccess-biostatistics-biometric-applications

where Y(1) is the potential outcome under treatment, and Y(0) is that under control? It is known that when the treatment assignment is not random or the covariates are not balanced among the treatment groups, or the , ai, s depend on the , xi, s, then

irispublishers-openaccess-biostatistics-biometric-applications
irispublishers-openaccess-biostatistics-biometric-applications

are biased estimates of μ1 and μ0 [8], so is the estimate of mean treatment effect difference

irispublishers-openaccess-biostatistics-biometric-applications

based on the above two estimates, and so causal inference methods are needed to get unbiased estimates.

Now consider the following regression model, for some given link function g (⋅) ,

irispublishers-openaccess-biostatistics-biometric-applications

The question of interest is: in randomized or observational clinical trial, can we get unbiased estimate of Δ using the above model? Our understanding is that for randomized clinical trial model (1) can be the correct specification and it can be used to estimate Δ unbiasedly. For observational study, model (1) is incorrect, and the estimate of Δ based on it is biased. In this case, they xi, s are not iid over n, so the consistency of irispublishers-openaccess-biostatistics-biometric-applications is questionable. In fact, this conclusion applies also to any other data in which some components of the covariates are not iid.

Commonly Used Estimation Method

Below we list some commonly used estimation method of Δ for randomized and observational clinical trials. The naive and regression coefficient method are valid only for randomized trial. The causal inference methods will provide unbiased estimates of Δ in either randomized or observational study. Here we consider five of them.

Naive Estimator (NE). It just uses the sample mean difference

irispublishers-openaccess-biostatistics-biometric-applications

Regression Coefficient Estimator (RCE): It just uses the estimated regression coefficient irispublishers-openaccess-biostatistics-biometric-applications of δ in model (1), i.e., set irispublishers-openaccess-biostatistics-biometric-applications . Inverse Propensity Scores Weighted Estimate (IPSWE) proposed by Horvitz and Thompson (1952) [8]. In this method, it requires a propensity score model

irispublishers-openaccess-biostatistics-biometric-applications

After obtaining the estimate irispublishers-openaccess-biostatistics-biometric-applications , the estimate irispublishers-openaccess-biostatistics-biometric-applications is constructed as

irispublishers-openaccess-biostatistics-biometric-applications

Outcome Model Estimator (OME). This estimator is also called g-estimator in Robins (2009) [9]. It requires the outcome model

irispublishers-openaccess-biostatistics-biometric-applications

After obtaining estimate irispublishers-openaccess-biostatistics-biometric-applications the estimator irispublishers-openaccess-biostatistics-biometric-applications is

irispublishers-openaccess-biostatistics-biometric-applications

Doubly Robust Estimator (DRE), The DRE is a recent development in causal inference. It requires a propensity score model (2) and outcome model (4). After obtaining estimate irispublishers-openaccess-biostatistics-biometric-applications , the DRE irispublishers-openaccess-biostatistics-biometric-applications is

irispublishers-openaccess-biostatistics-biometric-applications

The estimator is unbiased if either the propensity score model or the outcome model is correctly specified.

Simulation Studies

Here we simulate data from models (1) and (4), and compute the Naive Estimate (NE), and the Regression Coefficient Estimate (RCE), and the causal estimates IPSWE (3), OME (5) and the DRE (6), and compare the results in corresponding Tables. We consider two cases: case 1), g (⋅) , g1 (⋅ ) and g0(⋅ ) all be identity function, so models (1) and (4) are linear models; case 2), g (⋅) , g1(⋅ ) and g0( ⋅ ) are non-linear link functions. The simulation distinguishes for randomized observational studies.

Simulation Results for Randomized Study

In the randomized study, we considered three different covariate dimensions: p = 3 , p = 5 , and p = 8 . For all three settings, the total sample size was fixed at n = 300 , with n1 =10 0treated subjects and n0 = 200 control subjects. We consider two cases, linear model, g (⋅) be identity, and non-linear model, g (⋅) is non-identity. The treatment assignment was completely randomized, so the average proportion of treated subjects was approximately 0.333 in every setting.

Linear Model: The response variable was generated from the linear model

irispublishers-openaccess-biostatistics-biometric-applications

with Δ0 = 2.0 and error standard deviation 0.5. The true regression coefficients were taken as β0 =( 1.0, − 1.5,0.8)T for the 3-dimensional setting; β0 =(1.0,− 1.5,0.8,1.2, − 0.7 )T for the 5-dimensional setting; and β0 =(1.0,− 1.5,0.8,1.2, − 0.7 )T for the 8-dimensional setting.

For each setting, the simulation was repeated 400 times. Table 1 reports the average estimate and the standard deviation for the five estimators. From Table 1, all five estimators produce estimates very close to the treatment effect Δ0 = 2.0 in all three dimensions. This is expected because the treatment assignment is randomized, so there is no confounding bias. However, there is a clear difference in variability. The Naive estimator has notice-ably larger standard deviations, and its variability increases as the dimension grows. In contrast, RCE, IPSWE, OME, and DRE remain more stable because they ad-just for covariates. Among these adjusted methods, OME, DRE, and RCE give the smallest standard deviations across all three settings, while IPSWE is also close to unbiased but slightly less stable. Overall, the randomized study shows that all five methods are approximately unbiased, but the adjusted estimators, especially OME and DRE, are more efficient than the Naive estimator.

Table 1:Estimation of Δ0 with five methods in the randomized study.

irispublishers-openaccess-biostatistics-biometric-applications

Non-linear Model. In this part, we continue the randomized simulation study by taking the link function as a nonlinear distribution function g(.). The response variable was generated from model (1):

irispublishers-openaccess-biostatistics-biometric-applications

where irispublishers-openaccess-biostatistics-biometric-applications. The treatment assignment was randomized, with n = 300 , n 1 =100 , and n0 = 200 . Thus, the treatment indicator Ai was independent of the covariates Xi .

The index-scale treatment coefficient was set to be δ =1.000 . Since the link function is nonlinear, the treatment effect on the response mean scale is not equal to δ. For each simulated dataset, the response-scale treatment effect was calculated as

irispublishers-openaccess-biostatistics-biometric-applications

The true regression coefficient vectors were taken as β0 =(0.2,1.0,− 1.5,0.8 )T for the 3-dimensional setting; β0 =(1.0,− 1.5,0.8,1.2, − 0.7 )T for the 5-dimensional setting; and

β0 =(0.2,1.0, 1.5,0.8,1.2,− 0.7,0.9,− 1.1 for the 8-dimensional setting.

For each setting, the simulation was repeated 400 times. Table 2 reports the simulation results. In this table, the Naive, IPSWE, OME, and DRE estimators estimate the response-scale treatment effect Δ0 . However, the RCE estimates the index-scale coefficient η directly. Therefore, the RCE column should be compared with η =1.000 , while the other methods should be compared with the response-scale Δ0 . From Table 2, the Naive estimator is close to the response-scale treatment effect Δ0 in all three dimensions. This is reasonable because the data come from a randomized clinical trial, so the treatment assignment is independent of the covariates. The IPSWE estimator gives almost the same results as the Naive estimator because the propensity score is known and constant in this randomized setting.

Table 2:Estimation results with nonlinear g (u) in the randomized study.

irispublishers-openaccess-biostatistics-biometric-applications

The RCE results need to be interpreted differently. In the nonlinear model

irispublishers-openaccess-biostatistics-biometric-applications

the regression coefficient δ ≠ Δ . Therefore, RCE is estimating η, not the response-scale treatment effect Δ0 . The simulation results show that the RCE estimates are very close to η =1.000 in the 3-, 5-, and 8-dimensional settings.

The OME and DRE estimators also estimate the response-scale treatment effect Δ0 , and their estimates are close to the corresponding values. They have smaller standard deviations than the Naive and IPSWE estimators because they use the correct outcome model with the known link function g (⋅) . Overall, in this randomized setting, the Naive estimator works well for the response-scale treatment effect, while RCE should be reported and interpreted as an index-scale estimator.

Simulation Results for Observational Study

In the observational study, we again considered three different covariate dimensions: p = 3 , p = 5 , and p = 8 , with total sample size n = 300 . In this case, the treatment assignment was no longer randomized. Instead, the treatment indicator i A was generated from a propensity score model

irispublishers-openaccess-biostatistics-biometric-applications

where irispublishers-openaccess-biostatistics-biometric-applications. The true γ was chosen so that the treated proportion was not close to one half. The average proportions of treated subjects were approximately 0.296, 0.284, and 0.270 for the 3-, 5-, and 8-dimensional settings, respectively

Linear Model: The outcome model followed the separate regression structure

irispublishers-openaccess-biostatistics-biometric-applications

The error standard deviation was 0.5, and the true intercepts were 0 α0 = 0.5 and α1 = 2.0 , so that the treatment effect was Δ0 =α1 −α0 =1.5.

For the 3-, 5-, and 8-dimensional settings, the true parameters were chosen as follows:

irispublishers-openaccess-biostatistics-biometric-applications

For each setting, the simulation was repeated 400 times. Table 3 reports the average estimate and the standard deviation for the five estimators. From Table 3, the NE is strongly biased in all three observational settings. The RCE is also biased upward, although the bias is smaller than that of the NE because the regression model adjusts for covariates. This shows that when treatment assignment depends on covariates, methods that do not properly account for confounding can give misleading results. By contrast, IPSWE, OME, and DRE all produce estimates much closer to the target value. Among these three methods, OME and DRE are more stable, with substantially smaller standard deviations than IPSWE. Although IPSWE is approximately unbiased, its variability is considerably larger, especially in the 3- and 8-dimensional settings. This is consistent with the general feature of inverse weighting methods, which often suffer from high variance. Overall, the observational study supports the main conclusion of this project. In the presence of covariate-dependent treatment assignment, the NE and RCE are not reliable, whereas IPSWE, OME, and DRE perform much better. Among them, OME and DRE provide the best balance between accuracy and stability.

Table 3:Estimation of Δ0 with five methods in the observational study.

irispublishers-openaccess-biostatistics-biometric-applications

Nonlinear Model

In this part, we consider the observational simulation study with nonlinear out-come models. Different from the previous nonlinear randomized setting, the data in this part were generated from model (4), not from model (1). That is, the treatment and control groups were generated from two separate outcome models,

irispublishers-openaccess-biostatistics-biometric-applications

where irispublishers-openaccess-biostatistics-biometric-applications . The treatment assignment was no longer randomized. Instead, i A was generated from the propensity score model

irispublishers-openaccess-biostatistics-biometric-applications

Therefore, the treatment assignment depended on the covariates, and the treated and control groups were not balanced.

For the nonlinear outcome models, g0(⋅ ) and g1( ⋅)were chosen as two different distribution functions. The parameters were chosen so that μ1 > μ0 . The true treatment effect was calculated by formula (5),

irispublishers-openaccess-biostatistics-biometric-applications

Here

irispublishers-openaccess-biostatistics-biometric-applications

For the 3-, 5-, and 8-dimensional settings, the true parameters were chosen as follows:

irispublishers-openaccess-biostatistics-biometric-applications

Since this setting is based on the separate nonlinear outcome models in formula (4), there is no index-scale treatment coefficient δ . Therefore, the RCE is not reported in this table. We compare four estimators: NE, IPSWE, OME, and DRE. For each setting, the simulation was repeated 400 times. Table 4 reports the average estimate and the standard deviation for each estimator. From Table 4, the NE is clearly biased in the observational nonlinear setting. For example, when the dimension is 3, the true treatment effect is Δ0 = 0.260 , but the NE estimate is 0.520. This happens because the treatment assignment depends on the covariates, so the treated and control groups have different covariate distributions. As a result, the simple sample mean difference does not estimate the causal treatment effect. By contrast, IPSWE, OME, and DRE are much closer to the true treatment effect Δ0 . The OME performs well because the outcome model is correctly specified. The DRE also performs well because it uses both the propensity score model and the outcome models. The IPSWE is also close to the target value, although it has larger standard deviations than OME and DRE. Overall, this observational nonlinear study shows that causal adjustment is necessary when the treatment assignment depends on covariates, while the NE is not reliable.

Table 4:Estimation of Δ0 with nonlinear outcome models in the observational study

irispublishers-openaccess-biostatistics-biometric-applications

Concluding Remarks

We examined commonly used regression models and estimates for treatment effect under the randomized clinical trial and observational study, and used simulation studies to confirm our reasonings. Our conclusions are: regression models are generally different for randomized clinical trial and observational study; for the five estimators we examined, some of them can be used for both trials, some are not, which are summarized below.

• For randomized trial with linear model irispublishers-openaccess-biostatistics-biometric-applications , all the five estimators give consistent estimate of Δ .

• For randomized trial with non-linear model irispublishers-openaccess-biostatistics-biometric-applications, all the estimators except RCE give consistent estimate of Δ . The RCE estimator irispublishers-openaccess-biostatistics-biometric-applications is not an estimator of Δ in this case since typically δ ≠ Δ in non-linear model.

• For observational trial with linear model irispublishers-openaccess-biostatistics-biometric-applications and

irispublishers-openaccess-biostatistics-biometric-applications, the NE and RCE are biased estimates of Δ , and the IPSWE, OME and DRE are unbiased estimators of Δ.

• For observational trial with non-linear model irispublishers-openaccess-biostatistics-biometric-applications and

irispublishers-openaccess-biostatistics-biometric-applicationsthe NE is a biased estimate of Δ , and the IPSWE, OME and DRE are unbiased estimators of Δ . In this setting, the RCE is not reported because model (4) does not contain a single index-scale treatment coefficient δ .

References

  1. Coover JE, Angell F (1907) General practice effect of special exercise. The American Journal of Psychology 18(3): 328-340.
  2. Neyman J (1923) [1990]. On the application of probability theory to agricultural experiments. Essay on Principles. Section 9. Statistical Science 5(4): 465-472.
  3. Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, et al. (1981) A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2(1): 31-49.
  4. Lachin JM (1988) Statistical properties of randomization in clinical trials. Controlled Clinical Trials, 9(4): 289-311.
  5. Sanson-Fisher RW, Bonevski B, Green LW, D’Este C (2007) Limitations of the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventive Medicine 33(2): 155-161.
  6. Bekelman JE, Li Y, Gross CP (2003) Scope and impact of financial conflicts of interest in biomedical research: a systematic review. Journal of American Medical Association 289(4): 454-465.
  7. Black N (1996) Why we need observational studies to evaluate the effectiveness of health care. British Medical Journal 312(7040): 1215-1218.
  8. Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association 47: 663-685.
  9. Robins J, Rotnitzky A, Zhao L (1994) Estimation of regression coefficients when some of the regressors are not always observed. Journal of the American Statistical Association 89(427): 846-866.
  10. Rotnitzky A, Lei QH, Sued M, Robins JM (2012) Improved double-robust estimation in missing data and causal inference models. Biometrika 99(2): 439-456.
  11. Yuan A, Yin A, Tan MT (2021) Enhanced doubly robust procedure for causal inference. Statistics in Biosciences 13(3): 454-478.
  12. Yin A, Yuan A, Tan MT (2024a) Doubly robust semiparametric estimation for multi-group causal comparisons. Statistics in Biosciences 16: 45-68.
  13. Yin A, Yuan A, Tan MT (2024b) Highly robust causal semiparametric U-statistic with applications in biomedical studies. International Journal of Bio-statistics 20(1): 69-91.
Citation
Keywords
Signup for Newsletter
Scroll to Top