■debian strechにElasticsearchを導入してみる。
Install Elasticsearch with Debian Package | Elasticsearch Reference [5.6] | Elastic
https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
■「SSH サーバ」のみでクリーンインストールしたdebian stretchに、
プロキシ経由でsudo、vimをインストール
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.1 (stretch)
Release: 9.1
Codename: stretch
$ tasksel --list-tasks | awk '/^i/'
i ssh-server SSH サーバ
$ tr '[:digit:]' 'X' < /etc/apt/apt.conf
Acquire::http::Proxy "http://XXX.XX.XX.XX:XXXX/";
$ su
usermod -a -G sudo labunix; \
id labunix | sed -e 's/ /\n/g'; \
exit
uid=1000(labunix)
gid=1000(labunix)
groups=1000(labunix),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(netdev)
exit
■前提条件のjava、wgetと動作確認のためのcurlの導入とプロキシ設定
$ java -version || sudo apt-get install -y openjdk-8-jre-headless; \
dpkg -l | grep "^ii.* wget" >/dev/null || sudo apt-get install -y wget; \
dpkg -l | grep "^ii.* curl" >/dev/null || sudo apt-get install -y curl
$ dpkg -l | awk '/^ii/&&/ wget| curl|openjdk/{print $2,$3}'
curl 7.52.1-5
openjdk-8-jre-headless:amd64 8u141-b15-1~deb9u1
wget 1.18-5
$ java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15)
OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode)
$ MYPROXY="http://172.31.31.60:8080/"; \
echo -e "http_proxy=$MYPROXY\nhttps_proxy=$MYPROXY\nftp_proxy=$MYPROXY" | \
awk '{print "sed -i -e \"/^#https_proxy/i "$1"\" /etc/wgetrc"}' | sudo sh
$ tr '[:digit:]' 'X' < /etc/wgetrc | grep ^[hf].*proxy
http_proxy=http://XXX.XX.XX.XX:XXXX/
https_proxy=http://XXX.XX.XX.XX:XXXX/
ftp_proxy=http://XXX.XX.XX.XX:XXXX/
$ MYPROXY="http://172.31.31.60:8080/"; \
echo '#!/bin/bash' | sudo tee /etc/profile.d/proxy.sh; \
echo -e "http_proxy=$MYPROXY\nhttp_proxys=$MYPROXY\nftp_proxy=$MYPROXY" | \
awk '{print "export "$1}' | sudo tee -a /etc/profile.d/proxy.sh
echo -e "HTTP_PROXY=$MYPROXY\nHTTPS_PROXY=$MYPROXY\nFTP_PROXY=$MYPROXY\nno_proxy=\"\"" | \
awk '{print "export "$1}' | sudo tee -a /etc/profile.d/proxy.sh
$ tr '[:digit:]' 'X' < /etc/profile.d/proxy.sh
export http_proxy=http://XXX.XX.XX.XX:XXXX/
export http_proxys=http://XXX.XX.XX.XX:XXXX/
export ftp_proxy=http://XXX.XX.XX.XX:XXXX/
export HTTP_PROXY=http://XXX.XX.XX.XX:XXXX/
export HTTPS_PROXY=http://XXX.XX.XX.XX:XXXX/
export FTP_PROXY=http://XXX.XX.XX.XX:XXXX/
export no_proxy=""
■apt-getの最新化
$ cat myscripts/update.sh && chmod +x myscripts/update.sh
echo -e "update\nupgrade\nautoremove\nautoclean" | \
awk '{print "apt-get -y "$1};END{print "apt-get clean all"}' | sh
$ sudo ./myscripts/update.sh
■ワンライナーでelasticsearchをインストール
Install Elasticsearch with Debian Package | Elasticsearch Reference [5.6] | Elastic
https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
sudo apt-key add - && \
sudo apt-get install apt-transport-https; \
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | \
sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list; \
sudo apt-get update && sudo apt-get install elasticsearch; \
ps -p 1 | grep system[d] && \
$(sudo /bin/systemctl daemon-reload; \
sudo /bin/systemctl enable elasticsearch.service; \
sudo systemctl start elasticsearch.service)
$ apt-key finger grep -A 1 -B 3 "4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4"
pub rsa2048 2013-09-16 [SC]
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
uid [ unknown] Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>
sub rsa2048 2013-09-16 [E]
$ dpkg -l | awk '/^ii/&&/elasticsearch/{print $2,$3}'
elasticsearch 5.6.0
$ sudo journalctl --unit elasticsearch
-- Logs begin at Mon 2017-09-18 23:35:27 JST, end at Tue 2017-09-19 00:28:06 JST. --
9月 19 00:27:25 vm-datasearch systemd[1]: Starting Elasticsearch...
9月 19 00:27:25 vm-datasearch systemd[1]: Started Elasticsearch.
$ curl -XGET 'localhost:9200/?pretty'
{
"name" : "o0yAXcv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "wWtUcCgASmGptaY4I9B43Q",
"version" : {
"number" : "5.6.0",
"build_hash" : "781a835",
"build_date" : "2017-09-07T03:09:58.087Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}
■起動してからのログに限定するには。
$ sudo systemctl status elasticsearch.service | \
awk '/since/{print "sudo journalctl --unit elasticsearch --since \""$6,$7"\""}' | sh
-- Logs begin at Mon 2017-09-18 23:35:27 JST, end at Tue 2017-09-19 00:42:16 JST. --
9月 19 00:27:25 vm-datasearch systemd[1]: Starting Elasticsearch...
9月 19 00:27:25 vm-datasearch systemd[1]: Started Elasticsearch.
■動作確認
$ top -b -n 1 | awk '/elastic/'
5307 elastic+ 20 0 4200480 1.621g 9324 S 0.0 82.8 0:25.88 java
$ sudo ss -tnp | awk '/java/'
ESTAB 0 0 ::ffff:127.0.0.1:9200 ::ffff:127.0.0.1:55988 users:(("java",pid=5307,fd=126))
ESTAB 0 0 ::ffff:127.0.0.1:9200 ::ffff:127.0.0.1:55990 users:(("java",pid=5307,fd=129))
■クラスタの状態とインデックスの状態を確認
greenが正常。現在はyellow。
$ echo "cluster status";curl 127.0.0.1:9200/_cat/health?vcluster status
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1505750046 00:54:06 elasticsearch yellow 1 1 1 1 0 0 1 0 - 50.0%
$ echo "index status";curl 127.0.0.1:9200/_cat/indices?v
index status
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana cL0AZAUmQPeQeablxsc6Sw 1 1 1 0 3.2kb 3.2kb
■clusterに名前を付ける。
$ curl -X PUT http://127.0.0.1:9200/myelastic;echo
{"acknowledged":true,"shards_acknowledged":true,"index":"myelastic"}
$ curl "http://127.0.0.1:9200/_cat/indices?v&index=myelastic"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open myelastic CjvlAhrPQV61KD-PacZ71w 5 1 0 0 810b 810b
■Shards の状態を確認、Replica Shards の数を0にする。
nodeが複数の場合に必要となるReplica Shardsだが、nodeが1つなので0にする。
$ curl "http://127.0.0.1:9200/_cat/shards?v&index=myelastic"
index shard prirep state docs store ip node
myelastic 2 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 2 r UNASSIGNED
myelastic 1 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 1 r UNASSIGNED
myelastic 3 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 3 r UNASSIGNED
myelastic 4 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 4 r UNASSIGNED
myelastic 0 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 0 r UNASSIGNED
$ curl -X PUT -d '{"index" : { "number_of_replicas": 0}}' http://127.0.0.1:9200/myelastic/_settings;echo
{"acknowledged":true}
$ curl "http://127.0.0.1:9200/_cat/indices?v&index=myelastic"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open myelastic CjvlAhrPQV61KD-PacZ71w 5 0 0 0 810b 810b
$ curl "http://127.0.0.1:9200/_cat/shards?v&index=myelastic"
index shard prirep state docs store ip node
myelastic 2 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 1 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 3 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 4 p STARTED 0 162b 127.0.0.1 o0yAXcv
myelastic 0 p STARTED 0 162b 127.0.0.1 o0yAXcv
$ curl -XGET '127.0.0.1:9200/_cat'
=^.^=
/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates
■X-Packと日本語のためのkuromojiプラグインをプロキシ経由でインストール。
$ dpkg -L elasticsearch | grep plugin\$
/usr/share/elasticsearch/bin/elasticsearch-plugin
$ sudo ES_JAVA_OPTS="-DproxyHost=172.31.31.60 -DproxyPort=8080" \
/usr/share/elasticsearch/bin/elasticsearch-plugin install --batch x-pack
-> Downloading x-pack from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission \\.\pipe\* read,write
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission sun.nio.ch.bugLevel write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.
-> Installed x-pack
$ dpkg -L kibana | grep plugin\$
/usr/share/kibana/node_modules/extract-text-webpack-plugin
/usr/share/kibana/bin/kibana-plugin
/usr/share/kibana/src/cli_plugin
$ sudo ES_JAVA_OPTS="-DproxyHost=172.31.31.60 -DproxyPort=8080" \
/usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-kuromoji
-> Downloading analysis-kuromoji from elastic
[=================================================] 100%
-> Installed analysis-kuromoji
$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list
analysis-kuromoji
x-pack
$ sudo systemctl stop elasticsearch.service
$ sudo systemctl start elasticsearch.service
$ sudo systemctl status elasticsearch.service | \
awk '/since/{print "sudo journalctl --unit elasticsearch --since \""$6,$7"\""}' | sh
-- Logs begin at Mon 2017-09-18 23:35:27 JST, end at Tue 2017-09-19 01:55:24 JST. --
9月 19 01:54:28 vm-datasearch systemd[1]: Starting Elasticsearch...
9月 19 01:54:28 vm-datasearch systemd[1]: Started Elasticsearch.
■入門ドキュメント
知識ゼロからElasticsearchを実践で使えるようになろう！ - $shibayu36->blog;
http://blog.shibayu36.org/entry/2016/09/05/110000
Elasticsearchチュートリアル
http://code46.hatenablog.com/entry/2014/01/21/115620
Elasticsearch入門 – シリーズ –
http://dev.classmethod.jp/series/elasticsearch%E5%85%A5%E9%96%80/
「Elasticsearchと対話Botによる対話型の検索システム」連載一覧
https://codezine.jp/article/corner/673
全文検索エンジン「Elasticsearch」を調べて使ってみた色々まとめ
https://vitalify.jp/blog/2017/02/elasticsearch_matome.html
■公式の販路を探す
Elasticsearch
https://www.elastic.co/jp/products/elasticsearch
ニーズに応じたサブスクリプション
https://www.elastic.co/jp/subscriptions
日本国内の契約会社
https://www.elastic.co/jp/about/partners/reseller
■Beatsを探す
「Filebeat」「Packetbeat」「Winlogbeat」「Metricbeat」「Heartbeat」「Auditbeat」「Topbeat」
Pick a Beat. Any Beat.
https://www.elastic.co/downloads/beats
■その他の情報源
Elastic Stack and Product Documentation
https://www.elastic.co/guide/index.html
ビデオ
https://www.elastic.co/jp/videos
エキスパートに問い合わせ
https://www.elastic.co/jp/contact
API Documentation
http://elasticsearch-py.readthedocs.io/en/master/api.html
