...
Set the number of items that are cross sold to appear in the cross sold lightbox or mini cart to be 4 or less
Set the number of times the lightbox appears to a LOW number (less than 4)
Keep the number of associated cross sell products low (again, 4 or fewer)
The combination of those three settings, when they have high values is very disruptive, and dilutes the impact that you are looking for when you employ cross sell. Change the settings until you feel you’ve found the sweet spot.
...