I would not just take it in stride. Smaller poops can be a sign of intestinal problems, including (but not limited to) hairballs. Has his eating or drinking lessened any, or is he still eating the same amount? If he is eating and drinking normally, there is less to worry about (gastrointestinal stasis would be less likely then), but he could still have worms. Pinworms are the most common type of worms in rabbits, though they do serve as a secondary host for tapeworms as well.
Have you changed the diet at all or is his water different (different source, etc.) ? If so, that could be why. Diet changes can make for different size poops.
If you have not changed his diet and it continues, or if he is eating or drinking less, definitely have a rabbit knowledgeable vet take a look at him.
The huge one sounds like he might have started to get soft poops, which can also be a sign of digestive problems. The fact that it was in the middle of extra small pellets is a bit worrisome.
If he is not used to a lot of veggies, do NOT switch him over to mostly veggies, please, or you can end up with a dead rabbit very quickly. Veggies have to be introduced slowly. They are good for rabbits, but some rabbits cannot handle certain veggies and some cannot handle many veggies. Also, sudden diet changes are always very stressful on a rabbit's digestive system, and a sudden overload of veggies can result in severe diarrhea as well as gastrointestinal stasis (a very deadly problem).