You won't often see a balloon airborne at lunch time. Why? Because the heat of the sun will have warmed the ground, the ground in turn will have warmed the adjacent air, and nature will have produced its own massive and invisible hot-air balloon (invisible until these thermals, as they are known, rise above condensation level and turn into puffy cotton wool cumulus clouds).
It's very uncomfortable to be caught in a thermal when flying in a balloon because you can't stop your ascent and may reach an altitude of thousands of feet.
Nature also abhors a vacuum, so where the thermal breaks away from the ground local breezes spring-up as surrounding air moves in to equalise the pressure. The result is unpredictable takeoff and landing conditions. Occasionally weak winter sun conditions do allow all-day flying but as a rule balloons prefer the gentler airs of early morning and evening, before the thermals start or after the have finished for the day.
If you think about it a balloon is very similar to a yacht, the envelope acting in the manner of a sail without a mast. With a breeze behind it the envelope can exert a force of several tons and, while sophisticated devices are available to tether the craft to a heavy vehicle or handy tree, it is obviously more sensible not to try and fly when the wind is too strong.
A belt of trees can often be found that will shelter the balloon from the wind during the inflation and takeoff phase, but good pilots remember what goes up must come down - and there may not be any shelter to drop behind when the balloon lands.
Rain is also best avoided. Moisture adds significant weight to the envelope in flight while a balloon packed away wet may suffer from mould and mildew.