You should take a look at the type of polymerase that you are using. Some polymerases extend more readily from nonspecifically bound primers while others only extend from a "better" priming event. Also--you can add DMSO to the reaction mixture. This prevents mispriming from occurring. Probably 1-5% overall in your reaction should help.