Before agriculture you weren't likely to find enough plant based calories to fuel the next day's foraging.

We've always eaten lots of animal products. Some survived on animal sources alone, like the Inuit. The difference is that now we are capable of much larger human population sizes than in the past, when we relied on what we could individually hunt or fish for. A properly fed cow eats grasses, not corn. So modern agriculture makes it possible to mass produce cheap, fatty meats, but not only is it not necessary, it actually harms the nutrition of the resulting product. And fish obviously don't require agriculture to produce. If you did not have access to seafood or domesticated animals like cows, sheep, lamb, etc., then you relied on hunting wild animals such as moose, elk, deer, or bison. (which made you nomadic since you followed the herd).

While those people certainly supplemented meat with berries or other vegetation, they would not be able to survive on that alone. The Indians of America were the ones who helped settlers understand the importance of Vitamin C to cure scurvy (not that they understood it in a scientific way, it was just intuition). Their primary source of that was not potatoes or pine needles (though the latter is what they encouraged them to use as medicine), it was actually the organs of the animals they hunted. They knew which ones would provide them what they needed to sustain themselves, at least before they lost that ancestral knowledge.

There are books discussing remote populations which did not take advantage of modern technology, and the healthy ones all consumed the following sources of nutrition: butter, milk, cheese, animal organs, fish, and fish eggs. Certain foods like fish eggs were rare and highly prized for fertility benefits, so you are correct about that, but most of these things were staples. The main difference in modern times is that we no longer congregate in groups where the food is plentiful, but have the food delivered to us instead.

I've read stories that the strongest African tribes preferred cattle, and not only would they drink the milk and eat their flesh, they also had a technique where they would puncture a vein to draw blood from the animal, and drink the blood to sustain themselves. Of course they would not eat meat every day the way we do now, but they didn't need to do that. If you live on milk, meat, and fat, then you don't need to consume three meals a day. I can drink three cups of raw milk a day and not have to eat anything because there are so many calories, vitamins, and minerals in that alone.

Just consider, we cannot sustain ourselves on plant sources alone. We require animals to digest and extract the nutrients from the plant material for us, since their bodies can manufacture certain things that we require but can't make on our own, and they do a lot better at digesting grasses than we do. The plants pull the minerals from the earth for us, and the animals add some vitamins that we can't do without. Additionally, the best sources of nutrients that we need are animals, which makes sense because they are basically collecting the same minerals for their own body building and life sustaining activities.