During the collision, the bouncy ball’s momentum changed from mv to –mv, corresponding to a momentum change of 2mv (rightward).
The lazy bag’s momentum changed from mv to 0, corresponding to a momentum change of only mv (rightward).
Assuming that the duration of collision was about the same, this implies that the tower pushed the bouncy ball (rightward) harder than it pushed the lazy ball. Because F=Δp/Δt.
By N3L, the tower was pushed harder (leftward) by the bouncy ball than by the lazy ball.