case-kの備忘録

日々の備忘録です。データ分析とか基盤系に興味あります。

fluentd でjRubyを使う

rubyでマルチスレッド処理を使うとGIPの制約CPUのコアをうまく使えなかったのでjRubyで実行する方法を試してみました。

Parallel.map(list, in_threads: @parallel_in_processes) do |rows| puts rows end

https://stackoverflow.com/questions/29215989/collecting-results-for-ruby-parallel-gem

# use jruby to use multi core to avoid gastric inhibitory polypeptide (GIP) for `router.emit_stream` method
# https://github.com/puckel/docker-airflow/issues/182#issuecomment-444683455
# https://www.jruby.org/download
mkdir -p /usr/share/man/man1
apt-get install openjdk-8-jre -y
wget https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.2.17.0/jruby-dist-9.2.17.0-bin.tar.gz
tar zxf jruby-dist-9.2.17.0-bin.tar.gz
mv jruby-9.2.17.0 /usr/local/jruby
/usr/local/jruby/bin/jruby -v

# ref
# jruby: https://stackoverflow.com/questions/29215989/collecting-results-for-ruby-parallel-gem
# je: https://github.com/kzk/jemalloc-rb#usage
/usr/local/jruby/bin/jruby -S je fluentd -c /usr/src/app/config/fluent-local.conf -p /usr/src/app/fluent-plugin-sql-server/lib/fluent/plugin