CV Joint Refuses to Come Off Axle Shaft

Have bad CV boots. Trying to remove the outer CV joint, caliper end, and it refuses to come off. The other side was no problem. Purchased the special punch still will not come off. Are there any secrets others have discovered to get the CV joint off? At this point a little concerned and thinking a new axle maybe the way to go, but all this just to replace a torn boot? Any help out in forum land?