Curiosity motivates exploration and is beneficial for learning, but curiosity is not always experienced when facing the unknown. In the present research, we address this selectivity: what causes curiosity to be experienced under some circumstances but not others? Using a Bayesian reinforcement learning model, we disentangle four possible influences on curiosity that have typically been confounded in previous research: surprise, local uncertainty/expected information gain, global uncertainty, and global expected information gain. In two experiments, we find that backward-looking influences (concerning beliefs based on prior experience) and forward-looking influences (concerning expectations about future learning) independently predict reported curiosity, and that forward-looking influences explain the most variance. These findings begin to disentangle the complex environmental features that drive curiosity.