Angry residents confronted Lord Smith during a visit to the flood-stricken Somerset levels today as the beleaguered peer refused to quit as chairman of the Environment Agency.
The former Labour minister, who was making his first visit to the area since the flooding began at Christmas, said he remained "very proud" of the Environment Agency (EA) staff.
Residents spoke of their anger at Lord Smith's refusal to apologise for the EA's handling of the crisis and said they were not satisfied at the answers he gave to their questions.
Lord Smith's visit coincided with Prime Minister David Cameron, who was also seeing for himself first hand the damage caused by the floods on the Levels.
Lord Smith insisted the top priority for authorities was "protecting lives", followed by protecting homes and businesses.
The under-fire peer said: "I have no intention of resigning because I'm very proud of the work the Environment Agency and its staff have been doing right round the country in the face of the most extreme weather."
Local resident Jim Winkworth, a farmer and pub landlord, said he was "bloody mad" at Lord Smith's refusal to apologise.
"We thought that's the least he could do today and he's not apologising or admitting any liability," Mr Winkworth.
"He hasn't come down here to apologise, which is what he should be here for.
"If you apologise it means you're admitting you got it wrong, I made a mistake, I'm sorry, I messed up but he's not fit to do that."
Lord Smith's visit - a week after Environment Secretary Owen Paterson was heckled by local residents - comes as Royal Marines were helping evacuate some 140 properties in the village of Moorland.
Another night of heavy rain overwhelmed local flood defences and, despite advice from police, a handful of people have chosen to remain in their homes.
Around 5,000 properties have been affected by flooding across the country, including 40 in Somerset, and more flood misery is expected in the coming days, with severe weather alerts in place for south east England, the South West and Wales as more wet weather is forecast.
Lord Smith's visit to the Somerset Levels was his first since it was hit by floods, and he met residents who have been worst affected.
The peer repeated his refusal to resign as he was confronted by an angry land-owner live on television.
The man told him: "We just need something done, it's as simple as that."
He said farmers were desperately trying to move livestock and save what they could.
"Our house is going under, it's as simple as that. I am one of the last ones."
Asked what he wanted the chairman of the EA to do, he said: "Sort the rivers out."
The angry land-owner, who did not give his name, said he agreed with calls for Lord Smith's resignation, saying: "I am not the only one round here who would like him to resign. We have had this for too long, we've had this for five weeks."
But the agency boss refused to quit, and said: "As I have said before, I have no intention of doing so because there is important work to be done, getting the dredging done as soon as possible, and getting the long-term solutions for the Somerset Levels sorted out."
Speaking earlier from the Willows and Wetlands Visitor Centre in Stoke St Gregory, Lord Smith said: "I am very proud of the work that the Environment Agency staff have been doing up and down the country over the course of the last two months.
"We have been faced with the most extreme weather that we have seen for years, we have had the wettest January since records began, this has been a major, major challenge for everyone up and down the country.
"The Environment Agency staff have been working their socks off to try and sort this out for everyone."
Asked why he had not apologised to residents, he said: "I have said to the people here what we did last year, what we've been preparing now, the work we're currently doing, and I think the important thing now is to work out what we can do for the future of Somerset, what can now happen, and that's what I'm primarily talking about with the local people here."
Lord Smith denied making the controversial comment that Britain may have to choose whether it wants to save "town or country" from future flooding because it is too costly to defend both.
He said: "I have never said it is a choice between saving the town and saving the country.
"What I have said is that the clear priorities that have been set for us by successive governments is: our top priority is protecting lives; our second priority is protecting people's homes and people's businesses; our third priority is protecting as much agricultural land as we can.
"That's the order of priority, that happens in both the town and the country."
Asked if he could promise there would not be a repeat of the disaster next year, the peer said: "I don't think anyone can promise that we won't see the sort of rainfall that we have been having over the last few months.
"What we are seeing is increasing levels of erratic weather, very extreme amounts of rainfall. I suspect we are going to see more of that over the next few years.
"What we need to do is find the best possible ways - dredging is probably one of the ways but it's not the only way - and there will be lots of other things we need to see if we can do in order to try and protect Somerset for the future."
The EA chairman, who said he could "absolutely understand" residents' frustration, said he would be visiting other parts of the Somerset Levels to see the impact of the flooding.
Local councillor Julian Taylor, who was evacuated on Wednesday and has moved into a holiday cottage, said earlier: "I think it's all very well for politicians in Westminster to send condolences and say that they're going to do things, but the issue is now that we're reaping failure of something like about 50 years of bad maintenance and short-term policies.
"And it's individual suffering of people having to cope... I've seen farming friends pushed to the brink of disaster on their farms."
Somerset County Council leader John Osman, who has also visited flood-hit residents today, said: "To see families forced to leave their homes because of floodwater is just devastating and I can only imagine what they are going through.
"The message from everyone I met came across loud and clear - more needs to be done, and I am determined that the views of those worst affected will be heard further up the line."
Lord Smith's visit to Somerset came as Transport Secretary Patrick McLoughlin travelled to Devon to see the Dawlish damage, which is going to take around six weeks to repair.
Deputy Prime Minister Nick Clegg said he wanted to "hear answers" from the EA, but dismissed criticism of Lord Smith for failing to go to Somerset earlier, saying it was not right to "get under the feet" of the emergency services,
Speaking on his regular LBC radio phone-in, the Liberal Democrat leader said: "The Environment Agency have a very difficult job but also has to learn lessons from what has happened.
"I actually don't think the measure of the response to this should be whether X individual or Y individual has been there. The measure should be have we got the right flood defences in place? Clearly we don't in Somerset."