He shouldn't be living with the girls as he could get them all pregnant. In fact, male degus will sometimes mate with females right after they give birth which is very unhealthy. I would either neuter him or separate him and pair him with one of his sons if a boy is born.
Anyways, did you weigh the degus? And has he gone to a vet? It's definitely possible that he's simply smaller than the others but I would ask for a vet's opinion

. Without knowing his weight or even seeing pictures, it's going to be hard for anyone to tell you if he's underweight over the internet.
I would avoid feeding raisins, by the way, as they're quite high in sugar. Stick to the nuts and seeds

.
Oh, and what brand of food are you feeding? Maybe the food is causing the weight problems.
Good luck and welcome to Paw-Talk!