Eugeny Shtoltc - IT Cloud

In Grafana, the initial login is admin and this password. First, we are prompted to select a source – select Prometheus, enter localhost: 9090, select the connection not as to the server, but as to the browser (that is, over the network) and select that we have basic authentication – that's all – click Save and Test and Prometheus is connected.

It is clear that it is not worth giving out a password and login from admin rights to everyone. To do this, you will need to create users or integrate them with an external user database such as Microsoft Active Directory.

I will select in the Dashboard tab and activate all three reconfigured dashboards. From the New Dashboard list in the top menu, select the Prometheus 2.0 Stats dashboard. But, there is no data:

I click on the "+" menu item and select "Dashboard", it is proposed to create a dashboard. A dashboard can contain several widgets, for example, charts that can be positioned and customized, so click on the add chart button and select its type. On the graph itself, we select edit by choosing a size, click edit, and the most important thing here is the choice of the displayed metric. Choosing Prometheus

Complete assembly available:

essh @ kubernetes-master: ~ / prometheus $ wget \

https://raw.githubusercontent.com/grafana/grafana/master/devenv/docker/ha_test/docker-compose.yaml

–-2019-10-30 07: 29: 52– https://raw.githubusercontent.com/grafana/grafana/master/devenv/docker/ha_test/docker-compose.yaml

Resolving raw.githubusercontent.com (raw.githubusercontent.com) … 151.101.112.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com) | 151.101.112.133 |: 443 … connected.

HTTP request sent, awaiting response … 200 OK

Length: 2996 (2.9K) [text / plain]

Saving to: 'docker-compose.yaml'

docker-compose.yaml 100% [=========>] 2.93K –.– KB / s in 0s

2019-10-30 07:29:52 (23.4 MB / s) – 'docker-compose.yaml' saved [2996/2996]

Obtaining application metrics

Up to this point, we have looked at the case where Prometheus polled the standard metric accumulator, getting the standard metrics. Now let's try to create an application and submit our metrics. First, let's take a NodeJS server and write an application for it. To do this, let's create a NodeJS project:

vagrant @ ubuntu: ~ $ mkdir nodejs && cd $ _

vagrant @ ubuntu: ~ / nodejs $ npm init

This utility will walk you through creating a package.json file.

It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields

and exactly what they do.

Use `npm install –save` afterwards to install a package and

save it as a dependency in the package.json file.

name: (nodejs)

version: (1.0.0)

description:

entry point: (index.js)

test command:

git repository:

keywords:

author: ESSch

license: (ISC)

About to write to /home/vagrant/nodejs/package.json:

{

"name": "nodejs",

"version": "1.0.0",

"description": "",

"main": "index.js",

"scripts": {

"test": "echo \" Error: no test specified \ "&& exit 1"

},

"author": "ESSch",

"license": "ISC"

}

Is this ok? (yes) yes

First, let's create a WEB server. I'll use the library to create it:

vagrant @ ubuntu: ~ / nodejs $ npm install Express –save

npm WARN deprecated Express@3.0.1: Package unsupported. Please use the express package (all lowercase) instead.

nodejs@1.0.0 / home / vagrant / nodejs

└── Express@3.0.1

npm WARN nodejs@1.0.0 No description

npm WARN nodejs@1.0.0 No repository field.

vagrant @ ubuntu: ~ / nodejs $ cat << EOF> index.js

const express = require ('express');

const app = express ();

app.get ('/ healt', function (req, res) {

res.send ({status: "Healt"});

});

app.listen (9999, () => {

console.log ({status: "start"});

});

EOF

vagrant @ ubuntu: ~ / nodejs $ node index.js &

[1] 18963

vagrant @ ubuntu: ~ / nodejs $ {status: 'start'}

vagrant @ ubuntu: ~ / nodejs $ curl localhost: 9999 / healt

{"status": "Healt"}

Our server is ready to work with Prometheus. We need to configure Prometheus for it.

The Prometheus scaling problem arises when the data does not fit on one server, more precisely, when one server does not have time to record data and when the processing of data by one server does not suit the performance. Thanos solves this problem by not requiring federation setup, by providing the user with an interface and API that it broadcasts to Prometheus instances. A web interface similar to Prometheus is available to the user. He himself interacts with agents that are installed on instances as a side-car, as Istio does. He and the agents are available as containers and as a Helm chart. For example, an agent can be brought up as a container configured on Prometheus, and Prometheus is configured with a config followed by a reboot.

docker run –rm quay.io/thanos/thanos:v0.7.0 –help

docker run -d –net = host –rm \

–v $ (pwd) /prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \

–-name prometheus-0-sidecar-eu1 \

–u root \

quay.io/thanos/thanos:v0.7.0 \

sidecar \

–-http-address 0.0.0.0:19090 \

–-grpc-address 0.0.0.0:19190 \

–-reloader.config-file /etc/prometheus/prometheus.yml \

–-prometheus.url http://127.0.0.1:9090

Notifications are an important part of monitoring. Notifications consist of firing triggers and a provider. A trigger is written in PromQL, as a rule, with a condition in Prometheus. When a trigger is triggered (metric condition), Prometheus signals the provider to send a notification. The standard provider is Alertmanager and is capable of sending messages to various receivers such as email and Slack.

For example, the metric "up", which takes the values 0 or 1, can be used to poison a message if the server is off for more than 1 minute. For this, a rule is written:

groups:

– name: example

rules:

– alert: Instance Down

expr: up == 0

for: 1m

When the metric is equal to 0 for more than 1 minute, then this trigger is triggered and Prometheus sends a request to the Alertmanager. Alertmanager specifies what to do with this event. We can prescribe that when the InstanceDown event is received, we need to send a message to the mail. To do this, configure Alertmanager to do this:

global:

smtp_smarthost: 'localhost: 25'

smtp_from: 'youraddress@example.org'

route:

receiver: example-email

receivers:

– name: example-email

email_configs:

– to: 'youraddress@example.org'

Alertmanager itself will use the installed protocol on this computer. In order for it to be able to do this, it must be installed. Take Simple Mail Transfer Protocol (SMTP), for example. To test it, let's install a console mail server in parallel with the Alert Manager – sendmail.

Fast and clear analysis of system logs

OpenSource full-text search engine Lucene is used for quick search in logs. On its basis, two low-level products were built: Sold and Elasticsearch, which are quite similar in capabilities, but differ in usability and license. Many popular assemblies are built on them, for example, just a delivery set with ElasticSearch: ELK (Elasticsearch (Apache Lucene), Logstash, Kibana), EFK (Elasticsearch, Fluentd, Kibana), and products, for example, GrayLog2. Both GrayLog2 and assemblies (ELK / EFK) are actively used due to the lesser need to configure non-test benches, for example, you can put EFK in a Kubernetes cluster with almost one command

helm install efk-stack stable / elastic-stack –set logstash.enabled = false –set fluentd.enabled = true –set fluentd-elastics