being heavy means that you are carrying along all that mass that does not contribute anything to the effort. You will fatigue faster as you are doing more work.
By eating a diet that is completely and totally without sugar or carbs including fruit, juice, chocolate, alcohol, even salad dressings with sugar in the ingredient list, after about 7-10 days, the body will switch over to burning that stored fat as energy. I felt it happen. If you stay with that diet, and ride at a moderate pace for extended periods of time, you can lose weight fast. After desired weight loss goals have been met, you don't even want simple carbs any more. You can reintroduce fruit and complex carbs and maintain the fat burning process.
When you have all that quick sugar available, your body won't ever dig deep to get to the stored energy that it has put away. So your tank empties, and you bonk. That is why afternoon rides are harder for you because you have used up the glycogen (sugars) in your blood during your daily activities compared to morning when they are replenished after rest.