Pedestrians often approach intersections and see "push to walk" buttons and actually push them thinking this will make a difference in stopping traffic just for them. Silly pedestrians. That's not how they work.
I would like to see traffic come crashing to a halt on my whim - who wouldn't? - but it would be dangerous for vehicles and can you imagine the mischief kids could get up if they did work that way?
During less busy times, some traffic cycles don't include a walk phase at all. The button is for those times. Pushing the button forces a 'walk' phase into a traffic cycle that does not include one, or ironically, during a time when nobody will be requiring a walk phase because nobody is out walking at those times. It will not speed up an existing traffic cycle so a pedestrian can get across the street sooner.
So yeah, go ahead and keep pushing that button. It will give you something to do with your time while you wait for the walk button which will appear exactly when it was programmed to appear and no sooner.