(CNN) - Anti-tax activist Grover Norquist said Monday that his group, Americans for Tax Reform, would work to unseat Republicans who break their pledge to never vote for higher taxes.

His vow came after prominent GOP lawmakers said over the weekend they would consider breaking the Taxpayer Protection Pledge in order to reach a deal with Democrats and President Barack Obama to avoid tumbling over the fiscal cliff – the combination of sweeping spending cuts and tax increases that would go into effect at the end of the year if negotiators can't reach a deal on reducing the federal debt.

Norquist said his group would "certainly highlight who has kept their commitment and who hasn't" when it comes time for lawmakers like Sen. Lindsey Graham and Rep. Peter King to run for re-election, though Norquist claimed voters generally decide on their own to oust elected officials who vote to raise taxes.

"Historically the people who lose do so because the people in their state have figured that out," Norquist said on CNN's "Starting Point with Soledad O'Brien."

On Sunday, Graham and King joined other Republican lawmakers, including Georgia Republican Sen. Saxby Chambliss, in publicly breaking from Norquist, whose pledge stipulates signers will "oppose any and all efforts to increase the marginal income tax rates for individuals and/or businesses" and will "oppose any net reduction or elimination of deductions and credits, unless matched dollar for dollar by further reducing tax rates."

Read More.