We are interested in how the nervous system learns from experience in dynamic environments. We trained mice on a dynamic foraging task, in which they freely chose between two alternatives that delivered reward with changing probabilities. We found that serotonin neurons in the dorsal raphe represented a quantity related to reward uncertainty over long timescales (tens of seconds), consistent with a modulatory signal used to adjust learning rates of ongoing decision variables in frontal cortex. In locus coeruleus, we found two types of norepinephrine neuron, including one that provided a reward prediction error for cortex. Our results provide quantitative links between activity of two key neuromodulators--serotonin and norepinephrine--and dynamic behavior.