Project

General

Profile

Feature #6767

[cmi] Update Kibana to 6.5, filebeat and proxypass improvements

Added by Albert Sole Domingo 3 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
Start date:
02/18/2019
Due date:
02/28/2019
% Done:

30%

Estimated time:
Module:
CMI
web_user_name:
web_email:
qa test:

Files

kibana_viapps_mapping.json (3.34 KB) kibana_viapps_mapping.json Felipe Tavares, 02/26/2019 05:51 PM
kibana_viapps_data.json (110 KB) kibana_viapps_data.json Felipe Tavares, 02/26/2019 05:51 PM

History

#1

Updated by Albert Sole Domingo 3 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30
#2

Updated by Albert Sole Domingo 3 months ago

  • Assignee changed from Albert Sole Domingo to Daniel Lopez
#3

Updated by Felipe Tavares 3 months ago

First step is to remove the old ELK stack.
Important: At this point, the data in Elastic will be lost.

rpm -evh elasticsearch logstash kibana elasticdump elasticsearch-curator --nodeps
rm -rfv /etc/logstash /etc/kibana /etc/elasticsearch /var/lib/logstash /var/lib/elasticsearch /opt/kibana /opt/elasticsearch-curator /opt/logstash;

Now we install the software packages. For now, we used an openNAC repository, which had them all:

yum install kibana-6.5.0-1.x86_64 elasticsearch-curator-5.4.1-1.x86_64 elasticsearch-6.5.0-1.noarch logstash-6.5.0-1.noarch
npm install elasticdump -g

systemctl daemon-reload

We also needed to change the way that we would change Elasticsearch's configuration path.data.
This happens, because now elasticsearch already come with it uncommented:

conf_file=$(ls /etc/elasticsearch/elasticsearch.yml*)
sed -i -e 's_^path.data.*_path.data: /var/log/syslog/elasticsearch/data_' $conf_file

Also, we now need to edit the elasticsearch's jvm.options, to configure minimun and maximum JVM Heap Size. Elastic recommends to have 50% of the systems RAM as JVM Heap Size and also to let mimimun and maximum to the same number, so the JVM won't need to resize inflight:

vim /etc/elasticsearch/jvm.options
-Xms4g
-Xmx4g

So, as we changed the kibana.yml file location, we need to change the KIBANA_CONF:

KIBANA_CONF=/etc/kibana/kibana.yml

Here is the snippet that we use in openNAC to configure the needed parameters in the kibana.yml file:

if ! (grep -E -q '^server.basePath' /etc/kibana/kibana.yml 2>/dev/null)
then
    sed -i '/#server.basePath:.*/a server.basePath: "\/cmi-portal-html5\/rest\/elasticsearch"' /etc/kibana/kibana.yml
else
    echo "Kibana 'server.basePath' property is ok." >&2
fi

if ! (grep -E -q '^server.rewriteBasePath' /etc/kibana/kibana.yml 2>/dev/null)
then
        sed -i '/^server.basePath:.*/a server.rewriteBasePath: false' /etc/kibana/kibana.yml
fi

if ! (grep -E -q '^server.host' /etc/kibana/kibana.yml 2>/dev/null)
then
    sed -i '/#server.host:.*/a server.host: "0.0.0.0"' /etc/kibana/kibana.yml
else
    echo "Kibana 'server.host' property is ok." >&2
fi

After this, we restart kibana and remove it's menus:

systemctl restart kibana

#remove kibana menus:
echo "Updating Kibana information ..." 
if [ "$(grep ".global-nav {" /usr/share/kibana/optimize/bundles/commons.style.css -A1 | grep -c "\!important")" -eq 0 ]
then

    echo "Hidding Kibana's menus..." 

    ##/usr/share/kibana/optimize/bundles/commons.style.css
    sed -i -e '/.global-nav {/a\' -e "  display: none \!important;" /usr/share/kibana/optimize/bundles/commons.style.css
    sed -i -e '339s/left: 53px/left: 0px \!important/' /usr/share/kibana/optimize/bundles/commons.style.css
    cat >> /usr/share/kibana/optimize/bundles/commons.style.css << 'EOF'
.euiPage--restrictWidth-default {
    max-width: none !important;
}
EOF

    ##/usr/share/kibana/optimize/bundles/vendors.style.css
    sed -i -e '2593s/display: flex;/display: none \!important;/' /usr/share/kibana/optimize/bundles/vendors.style.css
    cat >> /usr/share/kibana/optimize/bundles/vendors.style.css << 'EOF'
.kuiLocalMenuItem[data-test-subj="dashboardFullScreenMode"] {
  display: none !important;
}

.kuiLocalMenuItem[data-test-subj="shareTopNavButton"] {
  display: none !important;
}

.kuiLocalMenuItem[data-test-subj="dashboardClone"] {
  display: none !important;
}

.kuiLocalMenuItem[data-test-subj="dashboardEditMode"] {
  display: none !important;
}
EOF

## /usr/share/kibana/node_modules/@kbn/ui-framework/dist/ui_framework.css
sed -i -e '2210s/display: flex;/display: none \!important;/' /usr/share/kibana/node_modules/\@kbn/ui-framework/dist/ui_framework.css

    if [ "$(grep ".global-nav {" /usr/share/kibana/optimize/bundles/commons.style.css -A1 | grep -c "\!important")" -eq 0 ]
    then
        echo "Kibana's menus failed to hide." 
    fi
fi

Now, we need to change 2 of the logstash's config files:
  • 50-filter-postfix.conf:
    sed "trim" for "trim_value" 
    
  • 99-end.conf:
    message_format won't work anymore.
    Changed the codec to the new format:
    if [program] =~ /^postfix.*$/ {
      file {
            path => "/var/log/logstash/hosts/%{logsource}/%{+YYYY}/%{+MM}/%{+dd}/maillog.log" 
            codec => line { format => "%{timestamp} %{logsource} %{program}[%{pid}]: %{[message]}" }
      }
    }
    

This line should be commented, as there is no health index at the elasticsearch (at least, not while I was working on it):

#curl -X DELETE 'localhost:9200/health'

All the other steps of our postinstall should be the same.

#4

Updated by Albert Sole Domingo 3 months ago

  • Due date set to 02/28/2019
  • Status changed from In Progress to Resolved
  • Assignee changed from Daniel Lopez to Albert Sole Domingo
  • % Done changed from 30 to 80
#5

Updated by Daniel Lopez 3 months ago

  • Status changed from Resolved to In Progress
  • Assignee changed from Albert Sole Domingo to Daniel Lopez
  • % Done changed from 80 to 30

packaging and postinstall

#6

Updated by Daniel Lopez about 2 months ago

fixing update

Also available in: Atom PDF