読者です 読者をやめる 読者になる 読者になる

[マルチスレッド] [分散] xargs,gnu parallel比較 追記

xargsの方が早い?他の検証データ見ないと。

1)GNU parallelのダウンロード

macmini$ wget http://ftp.gnu.org/gnu/parallel/parallel-20110722.tar.bz2
2011-08-13 16:49:47 http://ftp.gnu.org/gnu/parallel/parallel-20110722.tar.bz2
ftp.gnu.org (ftp.gnu.org) をDNSに問いあわせています... 140.186.70.20
ftp.gnu.org (ftp.gnu.org)|140.186.70.20|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 179658 (175K) [application/x-bzip2]
`parallel-20110722.tar.bz2' に保存中

100%[======================================>] 179,658      112K/s 時間 1.6s    

2011-08-13 16:49:49 (112 KB/s) - `parallel-20110722.tar.bz2' へ保存完了 [179658/179658]

2)展開

macmini$ tar xvjf parallel-20110722.tar.bz2 
x parallel-20110722/
x parallel-20110722/COPYING
x parallel-20110722/README
x parallel-20110722/configure.ac
x parallel-20110722/src/
x parallel-20110722/src/parallel.pod
x parallel-20110722/src/sem.html
x parallel-20110722/src/sql
x parallel-20110722/src/niceload.html
x parallel-20110722/src/sem.1
x parallel-20110722/src/parallel.1
x parallel-20110722/src/sql.1
x parallel-20110722/src/parallel
x parallel-20110722/src/niceload.pod
x parallel-20110722/src/Makefile.in
x parallel-20110722/src/sql.html
x parallel-20110722/src/sem
x parallel-20110722/src/sem.pod
x parallel-20110722/src/niceload
x parallel-20110722/src/Makefile.am
x parallel-20110722/src/parallel.html
x parallel-20110722/src/niceload.1
x parallel-20110722/NEWS
x parallel-20110722/Makefile.in
x parallel-20110722/configure
x parallel-20110722/missing
x parallel-20110722/install-sh
x parallel-20110722/aclocal.m4
x parallel-20110722/Makefile.am
x parallel-20110722/config.h.in

3)インストール

macmini$ cd ./parallel-20110722
macmini$ ls
COPYING		NEWS		config.h.in	install-sh
Makefile.am	README		configure	missing
Makefile.in	aclocal.m4	configure.ac	src
macmini$ 
macmini$ ./configure 
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating config.h
macmini$ make
make  all-recursive
Making all in src
pod2man --release='20110722' --center='parallel' \
		--section=1 ./parallel.pod > ./parallel.1
pod2man --release='20110722' --center='parallel' \
		--section=1 ./sem.pod > ./sem.1
pod2man --release='20110722' --center='parallel' \
		--section=1 ./sql > ./sql.1
pod2man --release='20110722' --center='parallel' \
		--section=1 ./niceload.pod > ./niceload.1
./sem --fg --id pod2html pod2html ./parallel.pod > ./parallel.html
./sem --fg --id pod2html rm -f ./pod2htm*
./sem --fg --id pod2html pod2html ./sem.pod > ./sem.html
./sem --fg --id pod2html rm -f ./pod2htm*
./sem --fg --id pod2html pod2html ./sql > ./sql.html
./sem --fg --id pod2html rm -f ./pod2htm*
./sem --fg --id pod2html pod2html ./niceload.pod > ./niceload.html
./sem --fg --id pod2html rm -f ./pod2htm*
make[2]: Nothing to be done for `all-am'.
macmini$ 
macmini$ sudo make install
Making install in src
test -z "/usr/local/bin" || .././install-sh -c -d "/usr/local/bin"
 /usr/bin/install -c parallel sem sql niceload '/usr/local/bin'
test -z "/usr/local/share/doc/parallel" || .././install-sh -c -d "/usr/local/share/doc/parallel"
 /usr/bin/install -c -m 644 parallel.html sem.html sql.html niceload.html '/usr/local/share/doc/parallel'
test -z "/usr/local/share/man/man1" || .././install-sh -c -d "/usr/local/share/man/man1"
 /usr/bin/install -c -m 644 parallel.1 sem.1 sql.1 niceload.1 '/usr/local/share/man/man1'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.

4) 通常、xargs、paralleの処理時間比較

macmini$ time md5 To* > /dev/null

real	0m22.126s
user	0m1.467s
sys	0m1.251s

macmini$ time ls To* | xargs -L 1 -P 2 md5 > /dev/null

real	0m1.671s
user	0m1.379s
sys	0m0.895s

macmini$ time parallel -j200% md5 ::: To* > /dev/null

real	0m1.935s
user	0m1.655s
sys	0m1.102s