Sharp affine stability estimates for Hammer's problem