I know that both my Xrite 361 and my Speedmaster tr-90 recommend a warmup period, though I don't remember exactly how long offhand.
It's not surprising to see a small drift from a warmed-up unit to one that has cooled back to ambient temperature. Electronic components warm up, and their values can change slightly with temperature. How sensitive a device may be to those component changes is a function of the circuit design.
If you only need to make a single reading or two, then there's no harm in turning on, calibrating and taking your readings in a fairly short period of time, the thermal state of the unit won't change much in those couple of minutes. If you're going to be taking a number of readings over a period of time, especially those which need to be compared to others, it's probably best to allow a warm up period prior to calibrating and to leave the unit on until you're done with all your readings. For critical work, you should check against the calibration plaque prior to making each set of readings, re-calibrating as necessary.