Front Breaks Only Work when I pump Them

I have a 2003 Foreman 450 ES, and the front breaks are driving me nuts. If you pull the break lever one time it goes straight back to handle with no pressure, not working at all. If you pump the lever quickly 3 or 4 time pressure builds up and they are rock solid and work great until you release the handle, so I have to pump the handle a few times each time I break. I replaced the pads and got a vacuum bleeder and bled them, but no change, still doing same thing. any ideas?

Front breaks are stock drums.