Well, fool around enough with something and you can make it work. Turns out the line has to read:
ErrorDocument 404 /404.php
For those who are interested, I implemented this hack in Movable Type:
Which lets me capture incoming referral links to help people find what they want on my blog; now the 404 page includes the same code. (I'm going to have to modify the 404 one a bit to include both blogs hosted at my domain, but you get the idea.)
Now, when they get referred to a non-existent page from a search engine, the first thing the 404 page does is execute a query against my blogs - if there are posts that contain the search strings used at the search engine, it will present links to those posts before just presenting the visitor with a standard search interface.