Weak$^*$ fixed point property and the space of affine functions