Abstract

The survey experiment is widely used in economics and social sciences to evaluate

the effects of treatments or programs. In a standard population-based survey experiment,

the experimenter randomly draws experimental units from a target population of interest

and then randomly assigns the sampled units to treatment or control conditions to explore

the treatment effect of an intervention. Simple random sampling and treatment assignment

can balance covariates on average. However, covariate imbalance often exists in finite samples. To address the imbalance issue, we study a stratified approach to balance covariates in

a survey experiment. A stratified rejective sampling and rerandomization design is further

proposed to enhance the covariate balance. We develop a design-based asymptotic theory

for the widely used stratified difference-in-means estimator of the average treatment effect

under the proposed design. In particular, we show that it is consistent and asymptotically

a convolution of a normal distribution and two truncated normal distributions. This limiting distribution is more concentrated at the true average treatment effect than that under

the existing experimental designs. Moreover, we propose a covariate adjustment method in

the analysis stage, which can further improve the estimation efficiency. Numerical studies

demonstrate the validity and improved efficiency of the proposed method.

Key words and phrases: Blocking, covariate adjustment, design-based inference, stratification, rerandomization

Information

Preprint No.SS-2025-0315
Manuscript IDSS-2025-0315
Complete AuthorsPengfei Tian, Jiyang Ren, Yingying Ma
Corresponding AuthorsYingying Ma
Emailsmayingying_11@163.com

References

  1. Bai, Y., J. P. Romano, and A. M. Shaikh (2022). Inference in experiments with matched pairs. Journal of the American Statistical Association 117(540), 1726–1737.
  2. Banerjee, A. V., S. Chassang, and E. Snowberg (2017). Decision theoretic approaches to experiment design and external validity. In Handbook of Economic Field Experiments, Volume 1, pp. 141–174. Elsevier.
  3. Bickel, P. J. and D. A. Freedman (1984). Asymptotic normality and the bootstrap in stratified sampling. The Annals of Statistics 12, 470–482.
  4. Bilias, Y. (2000). Sequential testing of duration data: The case of the Pennsylvania ‘reemployment bonus’ experiment. Journal of Applied Econometrics 15(6), 575–594.
  5. Bloniarz, A., H. Liu, C. H. Zhang, J. Sekhon, and B. Yu (2016). Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences of the United States of America 113, 7383–7390.
  6. Branson, Z. and S. Shao (2021). Ridge rerandomization: An experimental design strategy in the presence of covariate collinearity. Journal of Statistical Planning and Inference 211, 287–314.
  7. Christenson, D. P. and D. L. Kriner (2017). Constitutional qualms or politics as usual? The factors shaping public support for unilateral action. American Journal of Political Science 61(2), 335–349.
  8. Cochran, W. G. (1977). Sampling Techniques (3rd ed.). New York: Wiley.
  9. Cytrynbaum, M. (2023). Optimal stratification of survey experiments. arXiv preprint arXiv:2111.08157.
  10. Dancey, L. and G. Sheagley (2018). Partisanship and perceptions of party-line voting in congress. Political Research Quarterly 71(1), 32–45.
  11. Fisher, R. A. (1926). The arrangement of field experiments. Journal of Ministry of Agriculture of Great Britain 33, 503–513.
  12. Fisher, R. A. (1935). The Design of Experiments (1st ed.). Oliver and Boyd, Edinburgh.
  13. Fuller, W. A. (2009). Some design properties of a rejective sampling procedure. Biometrika 96(4), 933–944.
  14. Gaines, B. J., J. H. Kuklinski, and P. J. Quirk (2007). The logic of the survey experiment reexamined. Political Analysis 15(1), 1–20.
  15. Imbens, G. and K. Menzel (2021). A causal bootstrap. The Annals of Statistics 49(3), 1460–1488.
  16. Imbens, G. W. and T. Lancaster (1996). Efficient estimation and stratified sampling. Journal of Econometrics 74(2), 289–318.
  17. Imbens, G. W. and D. B. Rubin (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press.
  18. Kriner, D. (2017). CCES 2014, Team Module of Boston University (BOS).
  19. Kuriwaki, S. (2022). Cumulative CCES Common Content.
  20. Li, X. and P. Ding (2017). General forms of finite population central limit theorems with applications to causal inference. Journal of the American Statistical Association 112, 1759–1769.
  21. Li, X. and P. Ding (2020). Rerandomization and regression adjustment. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82(1), 241–268.
  22. Li, X., P. Ding, and D. B. Rubin (2018). Asymptotic theory of rerandomization in treatment-control experiments. Proceedings of the National Academy of Sciences of the United States of America 115(37), 9157–9162.
  23. Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics 7, 295–318.
  24. Lin, Y., M. Zhu, and Z. Su (2015). The pursuit of balance: an overview of covariate-adaptive randomization techniques in clinical trials. Contemporary Clinical Trials 45, 21–25.
  25. Liu, H., J. Ren, and Y. Yang (2024). Randomization-based joint central limit theorem and efficient covariate adjustment in randomized block 2K factorial experiments. Journal of the American Statistical Association 119(545), 136–150.
  26. Liu, H. and Y. Yang (2020). Regression-adjusted average treatment effect estimators in stratified randomized experiments. Biometrika 107, 935–948.
  27. Lu, X., T. Liu, H. Liu, and P. Ding (2023). Design-based theory for cluster rerandomization. Biometrika 110(2), 431–447.
  28. Morgan, K. L. and D. B. Rubin (2012). Rerandomization to improve covariate balance in experiments. The Annals of Statistics 40, 1263–1282.
  29. Morgan, K. L. and D. B. Rubin (2015). Rerandomization to balance tiers of covariates. Journal of the American Statistical Association 110(512), 1412–1421.
  30. Mutz, D. C. (2011). Population-based Survey Experiments. Princeton, NJ, USA: Princeton University Press.
  31. Pashley, N. E. and L. W. Miratrix (2021). Insights on variance estimation for blocked and matched pairs designs. Journal of Educational and Behavioral Statistics 46(3), 271–296.
  32. Schindl, K. and Z. Branson (2024). A unified framework for rerandomization using quadratic forms. arXiv preprint arXiv:2403.12815.
  33. Schultzberg, M. and P. Johansson (2022). Re-randomization: A complement or substitute for stratification in randomized experiments? Journal of Statistical Planning and Inference 218, 43–58.
  34. Splawa-Neyman, J. (1923). On the application of probability theory to agricultural experiments. Essay on principles (with discussion). Section 9. Reprinted. Statistical Science 5, 465–472.
  35. Tabord-Meehan, M. (2023). Stratification trees for adaptive randomisation in randomised controlled trials. Review of Economic Studies 90(5), 2646–2673.
  36. Wang, X., T. Wang, and H. Liu (2023). Rerandomization in stratified randomized experiments. Journal of the American Statistical Association 118(542), 1295–1304.
  37. Wang, Y. and X. Li (2022). Rerandomization with diminishing covariate imbalance and diverging number of covariates. The Annals of Statistics 50(6), 3439–3465.
  38. Yang, Z., T. Qu, and X. Li (2023). Rejective sampling, rerandomization, and regression adjustment in survey experiments. Journal of the American Statistical Association 118(542), 1207–1221.
  39. Yu, H., K. Zhu, and H. Liu (2025). Sharp variance estimator and causal bootstrap in stratified randomized experiments. Statistics in Medicine 44(13-14), e70139.
  40. Zhang, H., G. Yin, and D. B. Rubin (2024). PCA rerandomization. Canadian Journal of Statistics 52(1), 5–25.
  41. Zhao, A. and P. Ding (2022). Reconciling design-based and model-based causal inferences for split-plot experiments. The Annals of Statistics 50, 1170–1192.
  42. Zhao, A. and P. Ding (2024). No star is good news: A unified look at rerandomization based on p-values from covariate balance tests. Journal of Econometrics 241(1), 105724.
  43. Zhu, K., H. Liu, and Y. Yang (2025). Design-based theory for lasso adjustment in randomized block experiments and rerandomized experiments. Journal of Business & Economic Statistics 43(3), 544– 555. Pengfei Tian Qiuzhen College, Tsinghua University, Beijing, 100084, China

Supplementary Materials

includes additional theoretical results, additional simulation

results, and proofs of all the theoretical results.

Supplementary materials are available for download.