ssh hangs in forked children

I have attempted to simultaneously ssh to a number of boxes (with
openssh 3.5p1 - 3.9p1) via forking subprocesses. However, there always
seem to be a few ssh sessions got stuck and thus, my control (parent)
process cannot exit. I have searched for similar problems regarding
ssh hanging with fork, but most results indicated that the issue was
on the remote machine with an inappropriately demonized process (ie.
not redirect 3 input streams). In my scenerio, the forking doesn't
happen on remote site and the remote command is just a simple one
(uname) as demonstrated in the following scripts.
In this setting, I have an 'expect' script (spawn.exp) used to cache
the password and feed it to spawned processes (ssh.pl).

% ./spawn.exp ./ssh.pl

ssh.pl:

#!/usr/local/bin/perl -w
use strict;

my %pids;

for my $id ('01'..'05') {
my $pid = fork();
die "fork $id: $!\n" unless defined $pid;
if ($pid == 0) {
#open STDIN, '</dev/null';
#open STDOUT, '>/dev/null';
#open STDERR, '>/dev/null';
exec 'ssh', "build-$id", 'uname', '-a';
}
else {
$pids{$pid} = "build-$id";
}
}
while (%pids) {
my $pid = wait();
last if $pid == -1;
delete $pids{$pid} if exists $pids{$pid};
}

__END__


spawn.exp:

#!/usr/bin/expect --

set passwd ""
set timeout -1

eval spawn -noecho $argv
expect {
"Please try again." {
set passwd
exp_continue
}
"password:" {
if {$passwd == ""} {
stty -echo
send_user " (script) "
expect_user -re "(.*)\n"
stty echo
set passwd $expect_out(1,string)
} else {
send_user " (supplied by script) "
}
send -- "$passwd\r"
exp_continue
}
}