Small Tanks and Genetics
Yes, you're overcrowded. I don't know how much you can or are willing to spend but you should have a minimum of a 29 gallon for the main fish, then you can use the 10 gallon as a fry tank. If you're planning on keeping the offspring then you should think about a 55 or so, and even at that continual breeding will still lead to overstocking. If you're hoping to breed to get to a good level of fish, like I am, then after you reach that number stop seperating the fry. Inevitably, a few fry will make it and through time a few of you adults will die out so you should end up with a good balance. If you're planning on selling then keep the fry tanks going and continue to seperate, but definitely move up to atleast the 29 gallon.
I'm no geneticist or so I can't help much on your other questions. I've breed mollies plenty of times and the results were almost always random since I didn't seperate the parents to breed, I only seperated the mother to give birth. I don't know what traits are past, say males control fin type and size and female controls the colors. It has to be even more difficult since you won't know the true history of your parents. Say an orange could have some dalmation, albeit a minute amount, in it's genetic coding so if you breed it with another orange hoping to get a vibrant orange group of fry, you will most likely end up with some vibrant, a few light colored, and a couple of dalmation.
Say you have BB = Black Molly and BW = Dalmation:
X | B | B
B | BB | BB
W| BW | BW
You would get, theoretically, half black, half dalmation. However, if the black had some dalmation in it's genetics making BB = Bw then:
X | B | w
B | BB | Bw
W| BW | Ww
There would be a good chance that you would end up with some 25% dominate white, 25% black, 25% dominate dalmation, and 25% dominate black with some dalmation characteristics.
X | B | W
B | BB | BW
W| BW | WW
50% Dalmations, 25% Black, and 25% White.
As you can see, it can get very confusing, when you add more strands and more characteristics like fins it gets even worse. To make a long post short, Good Luck, let me know if you figure it out.
Enjoy the experience and the fish,