My feeling is that you generally only want to align with like items. That is, align boxes with boxes and cards with cards, but not cards with boxes.
So you could choose a convention. I think if there are multiple boxes selected then align those boxes (bringing their contents along). If there is only one box selected, then align the cards, assuming the user selected the box by accident.
To cover the smaller case of mixed alignment, that could be accomplished by another feature where you snap items when they are close to other objects (perhaps with a keyboard modifier)
