Chamberlain was certainly wrong in his appeasement policies. And no, he wasn't buying time with it. It is clear from his statements from the period that he believed that it was possible to stop Hitler with diplomatic concessions. In this, he was certainly in accordance to the general public sentiment, which was, as other answers and comments point, against war. Certainly, Britain was rearming during the period, but the point of Chamberlain's politics was to avoid war, not to postpone it to a time when Britain was better equipped.
It is important to remember the course of German demands and acquisitions in the 1930's, to understand that those demands were not unreasonable or demented. Germany reannexed the Saar in 1935, through a plebiscite that was provisioned by the Versailles Treaty. It was a region with a majority of German population. Then Germany remilitarised the Rhineland, which was, and never ceased to be, metropolitan German territory. That put an end to a buffer zone between Germany and France, but it can hardly be called an absurd.
Then Germany demanded four other things: 1. the Anschluss of Austria (which had an overwhelmingly German population, that gave strong popular support for the idea), and the annexation of 2. the Sudettenland, 3. Memel, and 4. Dantzig/Gdansk. All those regions were German-majority regions; so, as an abstraction, such demands did not offend the sence of justice of most people. The fact that these apparently reasonable demands were part of a German strategy for war and domination of Europe was not immediately apparent - and was in fact the kernel of the debate about appeasement.
Those who thought that the demands were reasonable and fair, and that Hitler, albeit being a tyrant, was a tyrant of the old kind, with limited strategic goals that could be negotiated, favoured appeasement: to give Germany its reasonable demands, with reasonable assurances for reasonable protections of the rights of the ethnic minorities in those regions. The fate of political oppositionists in Memel, Gdansk, the Suddeten, or Austria, was of minimal concern - after all, those people were in great part already subjected to brutal dictatorial regimes, like those of Dolfuss/Schuschnnig, Bock/Pilsudsky, or Smetona, and that was considered pretty normal.
Those who thought that the German demands, reasonable as they were, were just a part of an expansionist strategy, which could be much more ambitious than the mere political reunification of German ethnicity, opposed appeasement, and proposed stronger diplomacy - of which, of course, threats of war were an integral part. Those people were not visionaries - Hitler himself had extensively written about his strategy, and made no secret of the fate he intended to impose into the Slavic or otherwise non-German populations of Eastern Europe.
And so, the Sudetten crisis was to be the watershed moment that finally cleared which of those political currents was correct. Austrians were for the most part happy to be anschlussed - and those who weren't, either were, like Communists and Socialdemocrats, already being repressed under Austria's own national government, or, like Schuschnigg's loyalists, didn't attract much sympathy, as they were the ones doing such repression. Czechoslovakia was different - it was a democracy, and its inhabitants weren't German or happy with German domination. Hitler promised to annex the Sudetten but to otherwise respect Czechoslovakia's independence. He broke that promise and invaded and subjected the Czech part of Czechoslovakia, showing in practice that he wouldn't be stopped by diplomacy and that his politics wasn't merely a politics of unifying ethnic Germans.
Now, Czechoslovakia had a quite formidable defensive line in the Suddeten. While they obviously could not counter-attack and invade Germany, they could have set up fierce resistance, much more than Poland did. The terrain is difficult, not the plains and prairies of Poland, and Czechoslovakian fortifications were strong and modern. This point shows, I think, the extent of Chamberlain's mistakes. If he was intent to rearm Britain for a future war, then he shouldn't allow the Germans to remove the Czechoslovakian defensive line, leaving the poor republic defenceless when the following, predictable, onslaught came. He was really deluded about Hitler's intentions and strategies.
Only then public opinion turned against appeasement. That would be the precise moment when Chamberlain could have been ousted. But Chamberlain himself recognised that we was wrong, and changed his mind on the possibility of containing Hitler through diplomacy. He was weakened by the blunder, and Churchill, who had opposed the policy, was strengthened. But it didn't cost Chamberlain his leadership within the Tories, and he remained prime minister. The moment of his possible fall had passed. When the invasion of Poland came, Chamberlain was no longer defending a policy of appeasement - and indeed his government immeadiately declared war on Germany, as it had promised before. And so, there was no particular reason why Chamberlain would be ousted in September 1939. He survived the crisis of March, when Hitler occupied "Bohemia and Moravia"; he was not to survive the crisis of the failure to defend Norway. But there was no particular British internal crisis due to the invasion of Poland.
What ensued was the drôle de guerre - the inaction of the allies in the Western Front, while the Wehrmacht slaughtered Poland. This was another wrong policy, but it cannot be blamed upon Chamberlain alone; any action would have to be started from French territory, and the French government, not Chamberlain, was the main culprit of the drôle de guerre.
His fall came with the invasion of Denmark and Norway, not because the British public opinion realised that war was unavoidable, but because it realised the war was iminent, and that the invasion of France, and probably Belgium and the Netherlands, was a matter of days. Thence Chamberlain lost his position, not directly as a punishment for appeasement or drôle de guerre, but because it was consensual that all main parties should be included in government, and Churchill was by far more acceptable to Labour (and Liberals, though that probably didn't matter as much).
Luckacs' two books on the subject (The Duel: 10 May–31 July 1940: the Eighty-Day Struggle between Churchill and Hitler and Five Days in London, May 1940) are a good read, mapping quite well the positions of Churchil and Chamberlain (and Labour's. And Halifax's - whose dellusions seem to have been more persistent than Chamberlain's) during the crisis that lead to the fall of France.