Rear prop shaft removal

I’m trying to remove my Rear prop shaft on my 99 Magnum 500 u-joint needs replaced i remove the rear diff and axle and remove the roll pin but can’t get enough clearance to get the shaft out the yoke is hitting the swing arm looks like i have to remove the swing arm but my problem is can’t find anything on you-tube to remove it plus i have no play at all in the swing arm bearings if i remove them could it damage to bearings thanks in advance…