After much testing, and with help from several other forum members, I believe we've come up with a HandBrake preset for DVDs that works properly on the Touchpad. Files encoded with this preset should show in the photos and videos app, allow seeking throughout the file, and play at a smooth frame rate.
Once the preset is imported, please check the "Keep Aspect Ratio" box under the Picture tab, as the preset export seems to lose this setting.
This may work with sources other than DVDs, but that's all I've tested it with so far.
I have not verified that files larger than 1.6GB will play, but I did have one 2.1GB file that would not - so the limit is likely somewhere in between.
This preset was created with HandBrake 0.9.5 in Windows.