Abstract

While classic off-policy evaluation literature typically assumes evenly

spaced decision times for analytical convenience, this assumption is often violated

in real-world applications, such as those involving user-initiated visits. To accommodate irregularly-spaced and potentially outcome-dependent decision times in

off-policy evaluation, we introduce a policy evaluation criterion defined through

integrated rewards rather than the traditional cumulative-reward formulation. To

ensure the estimability of the value function of a policy, we further develop a new

reinforcement learning framework that incorporates not only the state-action process but also an observation process, where the gap times between observations

can interact with states, actions, and rewards in a more flexible way. Statistical

inference for the proposed policy value is established by positing a modulated

renewal process model for the observation times. The validity of the proposed

approach is further supported by theoretical analysis, simulation studies, and a

real-world application that uses electronic health records to evaluate periodontal

Corresponding

author:

Wenbin

Lu.

ORCID:

disease treatments.

Key words and phrases: Modulated renewal process, Policy evaluation, Rein- forcement learning, Statistical inference

Information

Preprint No.SS-2026-0176
Manuscript IDSS-2026-0176
Complete AuthorsXin Chen, Wenbin Lu, Shu Yang, Dipankar Bandyopadhyay
Corresponding AuthorsWenbin Lu
Emailswlu4@ncsu.edu

References

  1. Barto, A. G. and S. Mahadevan (2003, Oct). Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 13(4), 341–379.
  2. Bradtke, S. and M. Duff (1994). Reinforcement learning methods for continuous-time markov decision problems. Advances in Neural Information Processing Systems 7, 393–400.
  3. Chen, X., J. Ding, and L. Sun (2018). A semiparametric additive rate model for a modulated renewal process. Lifetime Data Analysis 24, 675–698.
  4. Clarkson, J. E., N. B. Pitts, D. Bonetti, D. Boyers, H. Braid, R. Elford, P. A. Fee, R. Floate,
  5. B. Goul˜ao, G. Humphris, et al. (2018). INTERVAL dental recalls trial: a multicentre randomised controlled trial investigating the best dental recall interval for optimum, costeffective maintenance of oral health in dentate adults attending dental primary care. BMC Oral Health 18, 1–10.
  6. Clarkson, J. E., N. B. Pitts, B. Goulao, D. Boyers, C. R. Ramsay, R. Floate, H. J. Braid,
  7. P. A. Fee, F. S. Ord, H. V. Worthington, et al. (2020). Risk-based, 6-monthly and 24monthly dental check-ups for adults: the INTERVAL three-arm RCT. Health Technology Assessment (Winchester, England) 24(60), 1–138.
  8. Cox, D. R. (1972). The statistical analysis of dependencies in point processes. In Stochastic Point Processes., pp. 55–66. New York: Wiley.
  9. Crites, R. H. and A. G. Barto (1998, Nov). Elevator group control using multiple reinforcement learning agents. Machine Learning 33(2), 235–262.
  10. Das, T. K., A. Gosavi, S. Mahadevan, and N. Marchalleck (1999). Solving semi-markov decision problems using average reward reinforcement learning. Management Science 45(4), 560– 574.
  11. Du, J., J. Futoma, and F. Doshi-Velez (2020). Model-based reinforcement learning for semimarkov decision processes with neural odes. Advances in Neural Information Processing Systems 33, 19805–19816.
  12. Fatemi, M., M. Wu, J. Petch, W. Nelson, S. J. Connolly, A. Benz, A. Carnicelli, and M. Ghassemi (2022). Semi-markov offline reinforcement learning for healthcare. In Proceedings of the Conference on Health, Inference, and Learning, Volume 174, pp. 119–137.
  13. Guan, Q., B. J. Reich, E. B. Laber, and D. Bandyopadhyay (2020). Bayesian nonparametric policy search with application to periodontal recall intervals. Journal of the American Statistical Association 115, 1066–1078.
  14. Holtfreter, B., K. Kuhr, K. Borof, M. S. Tonetti, M. Sanz, K. Kornman, S. Jepsen, G. Aarabi,
  15. H. V¨olzke, T. Kocher, et al. (2024). ACES: A new framework for the application of the 2018 periodontal status classification scheme to epidemiological survey data. Journal of Clinical Periodontology 51, 512–521.
  16. Ireland, R. (2010). A Dictionary of Dentistry (1 ed.). Oxford, U.K.: Oxford University Press.
  17. Jiang, N. and L. Li (2016). Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, Volume 48, New
  18. York, USA, pp. 652–661. PMLR.
  19. Kallus, N. and M. Uehara (2020). Double reinforcement learning for efficient off-policy evaluation in markov decision processes. Journal of Machine Learning Research 21, 1–63.
  20. Kallus, N. and M. Uehara (2022). Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research 70(6), 3282–3302.
  21. Lin, F.-C., Y. K. Truong, and J. P. Fine (2013). Robust analysis of semiparametric renewal process models. Biometrika 100(3), 709–726.
  22. Lin, H., D. O. Scharfstein, and R. A. Rosenheck (2004). Analysis of longitudinal data with irregular, outcome-dependent follow-up. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 66(3), 791–813.
  23. Liu, Q., L. Li, Z. Tang, and D. Zhou (2018). Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in Neural Information Processing Systems 31, 5356–5366.
  24. Luckett, D. J., E. B. Laber, A. R. Kahkoska, D. M. Maahs, E. Mayer-Davis, and M. R. Kosorok
  25. (2019). Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association 115, 692–706.
  26. Maei, H. R. and R. S. Sutton (2010). Gq(lambda): A general gradient algorithm for temporaldifference prediction learning with eligibility traces. In Proceedings of the 3d Conference
  27. on Artificial General Intelligence (2010), pp. 100–105. Atlantis Press.
  28. Murphy, S. A. (2003). Optimal Dynamic Treatment Regimes. Journal of the Royal Statistical Society Series B: Statistical Methodology 65, 331–355.
  29. Oakes, D. and L. Cui (1994). On semiparametric inference for modulated renewal processes. Biometrika 81(1), 83–90.
  30. Parr, R. E. (1998). Hierarchical Control and Learning for Markov Decision Processes. Ph. D. thesis, University of California,Berkeley.
  31. Pons, O. and E. de Turckheim (1988). Cox’s periodic regression model. The Annals of Statistics 16, 678–693.
  32. Riley, P., H. V. Worthington, J. E. Clarkson, and P. V. Beirne (2013). Recall intervals for oral health in primary care patients. Cochrane Database of Systematic Reviews 12, 1–54.
  33. Robins, J. M. (2004). Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seattle Symposium in Biostatistics: Analysis of Correlated Data, New York, NY, pp. 189–326. Springer.
  34. Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American statistical Association 100(469), 322–331.
  35. Shi, C., S. Luo, Y. Le, H. Zhu, and R. Song (2022). Statistically efficient advantage learning for offline reinforcement learning in infinite horizons. Journal of the American Statistical Association 119(545), 232–245.
  36. Shi, C., M. Uehara, J. Huang, and N. Jiang (2022). A minimax learning approach to off-policy evaluation in confounded partially observable Markov decision processes. In Proceedings of the 39th International Conference on Machine Learning, Volume 162, pp. 20057–20094. PMLR.
  37. Shi, C., S. Zhang, W. Lu, and R. Song (2022). Statistical Inference of the Value Function for Reinforcement Learning in Infinite-Horizon Settings. Journal of the Royal Statistical Society Series B: Statistical Methodology 84, 765–793.
  38. Sutton, R. S. and A. G. Barto (2018). Reinforcement learning: An introduction. Cambridge: MIT press.
  39. Tang, X., Z. T. Qin, F. Zhang, Z. Wang, Z. Xu, Y. Ma, H. Zhu, and J. Ye (2019). A deep value-network based approach for multi-driver order dispatching. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM.
  40. Uehara, M., C. Shi, and N. Kallus (2022). A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355, 1–27.
  41. Yang, S. (2021). Semiparametric estimation of structural nested mean models with irregularly spaced longitudinal observations. Biometrics 78, 937–949.
  42. Yang, S., K. Pieper, and F. Cools (2019). Semiparametric Estimation of Structural Failure Time Models in Continuous-Time Processes. Biometrika 107, 123–136.

Acknowledgments

The authors gratefully acknowledge funding support from the National Natural Science Foundation of China, Grant Number 12001371; National Sci-

ence Foundation, Grant Number DMS-2113637 and SES 2242776; United

States National Institutes of Health, Grant Number R01DE031134.

Supplementary Materials

The supplementary material includes the proofs of the results in Section

3.1, Lemmas 1-2 and Theorem 1, as well as additional numerical results

from the simulation and the application, respectively.

Supplementary materials are available for download.