Principle of conservation of momentum is the reason behind recoils. During the firing of the bullet, we have the rifle and the bullet pushing each other. As far as the bullet-rifle system is concerned, this is a pair of internal forces. Without any external force acting on the bullet-rifle system, its total momentum remain unchanged. Since we started with the rifle and bullet both at rest, the total momentum must remain at zero.
The bullet, though small in mass, does carry a large forward momentum thanks to its speed. This forward momentum must be matched by a backward momentum of the same magnitude.