very rough shift from high to low

So not always but most of the time when I shift from high to low gear it doesn’t engage right away now before anyone says this, I am holding both hand brakes and foot brake as well, after shifting into low I let go all the brakes and lightly push on the throttle and nothing then touch the brakes again and wam into low it goes. Anyone else with this problem if so any solution or any idea why? Thanks