Perhaps the most important unresolved question in HIV care is when to initiate highly active antiretroviral therapy for a HIV patient. We present a Markov decision process (MDP) model that considers the optimal time to start a patient on therapy as a function of the CD4 count.

Methods:

MDPs are mathematical techniques for solving stochastic, dynamic decision problems. They are applied to industrial engineering problems such as optimal machine replacement, in which periodic observations are made of a stochastic system and certain actions may be taken each period. This framework is well-suited to HIV therapy where physicians makes periodic observations of an asymptomatic patient and decide whether or not to start therapy. We model monthly observations of a patient's (categorized) CD4 count. The monthly probabilistic transitions from one CD4 category to other CD4 categories or death are based on a Markov model of the natural history of a patient's CD4 count. We estimated survival after starting therapy using a Weibull regression model with CD4 count as a covariate. With these two components, the MDP provides an analytic solution for the optimal pre-therapy treatment policy (i.e., start therapy or continue to wait) as a function of the CD4 category each month. MDPs have “structural properties”: provable statements indicating how the model will behave under specific conditions. These structural properties may provide insights concerning the actual process being modeled.

Results:

We created 5 CD4 categories: cat1: [0,50); cat2: [50,200); cat3: [200,350); cat4: [350, 500); cat5: [500, infinity). The Weibull-estimated post-therapy survival for each category was: 8.6, 10.0, 12.6, 15.9, and 20.1 years. We used a pre-therapy Markov transition probability matrix that implied the expected remaining life years of patients who never take therapy are: 3.9, 7.7, 10.9, 12.7, and 13.9 across categories. The solution of the MDP indicated that the optimal policy starts a patient on therapy when in cat1, waits when in cats2-4, and starts when in cat5. This resulted in expected remaining life years of: 8.6, 11.2, 15.2, 17.8, and 20.1.

Conclusion:

The data satisfied sufficient conditions to prove the optimal value vector was monotonically increasing in the CD4 category: higher CD4 categories resulted in higher expected remaining life years. However, the form of the optimal policy was not expected and requires further investigation.