Because you're shockingly ignorant.

You can't farm in the winter. That doesn't make the ground infertile, you absolute parsnip. If anything, it still deposits water into the soil, which is why spring exists.

The soil conditions in a lot of Africa are generally poor. In Sub-Saharan Africa, there are many forms of clay which are not filled with nutrients, and are the result of little water, and little vegetation. Even that can be stripped by the enormous fauna that eats all the damn vegetation. And again, there are inconsistent rains and rivers that don't regularly flow and deposit water to other areas.

Then there are places which get too much water, like tropical jungles. Most of the foods you eat can't tolerate being water-logged.