Shorter distance between the sights makes pistols much more difficult.

Rifles are easier to hold steady, particularly as the weapon cycles. E.g. if you limp wrist your rifle, it still cycles. Limp wrist a handgun and it'll jam. Additionally, rifles can maintain a sight picture between shots. Handguns cannot. Rifles also have a longer barrel to stabilize the bullet than handguns.

Completely different skills.

Maybe you just got lucky on the rifle range...