Complex dynamics and multistability with increasing rationality in market games