While I'm not road riding, I have noticed the same thing just by working out during the week and changing up my diet. I used to get by eating horribly and just doing my weekly (weakly) ride on the dirt...on occasion I would maybe do 20-30 min of riding during the week and some push-ups and situps, but nothing consistent, nothing that pushed myself in the least. Like EC said, I could get by for a while on my dirt rides, but I would fade quickly and just go into survival mode.
Since changing my diet, I've dropped 10lbs and since starting a workout program, another 5 and all this added exercise during the week is making my weekend rides so much more enjoyable. I can now take my big bike out, keep up with everyone, and have fun instead of just trying to get to the end of the trail so I can rest.