Got the same problem using hardware mode, Nexus-s on a E6700, 2gb ram, 8800gtx, msi "power up"-edition.. Using Windows XP SP2.

Seems it only freezes when I either use fullscreen or Tv Window if I use Tv-out and then make the normal window fit as if it was fullscreen then I can record without any ...