Babies aren't compatible.

Actually I've noticed I'll have sims standing around more when there are infants and toddlers in the house than not. Even if someone else is doing something with the little ones, teens and older must get the thought "I can't do anything with that kid right now, but as soon as that other sim stops I'll own that kid." And this is without anything in the action queue, kinda like there's an unseen queue that's blocking anything else from happening.