Decanter Elasticsearch ReST appender network traffic

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Decanter Elasticsearch ReST appender network traffic

dschulten
Hi,

I have combined

    decanter-collector-log
    decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Decanter Elasticsearch ReST appender network traffic

jbonofre
Hi,

That’s a good question ;)

Currently, Decanter Elasticsearch appender sends each event followed by a refresh.

It directly uses Elasticsearch REST native client via performRequest.

A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it’s stored in Elasticsearch. Using bulk would dramatically decrease network usage.
I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update.

Regards
JB

> Le 18 févr. 2020 à 08:01, dschulten <[hidden email]> a écrit :
>
> Hi,
>
> I have combined
>
>    decanter-collector-log
>    decanter-appender-elasticsearch-rest
>
> and decanter sends the log events to my elasticsearch nicely. However, I am
> a bit concerned about network traffic and blocking rest requests. I am not
> entirely sure, but the code just seems to make blocking http requests to ES.
>
> The ES client benchmark
> https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
> shows that up to 30.000 ingestions/s could be doable under optimal
> circumstances, that seems not all that much.
>
> How does the ES Rest Appender send the log events? What if ES ingestion
> slows down?
> Does the Rest Appender do anything to cope with heavy load on the network or
> reduce the call frequency?
> Are there any advisable strategies which have been proven to work?
>
> Best
> Dietrich
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

Reply | Threaded
Open this post in threaded view
|

Re: Decanter Elasticsearch ReST appender network traffic

fpapon
Hi,
There is some improvement on the ES7 client so may be we could update the ES appender.

Another solution is that we could detect slow down ingestion and add a temporization but I also like the bulk usage :)

regards,
François
[hidden email]
Le 18/02/2020 à 09:04, Jean-Baptiste Onofre a écrit :
Hi,

That’s a good question ;)

Currently, Decanter Elasticsearch appender sends each event followed by a refresh.

It directly uses Elasticsearch REST native client via performRequest.

A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it’s stored in Elasticsearch. Using bulk would dramatically decrease network usage.
I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update.

Regards
JB

Le 18 févr. 2020 à 08:01, dschulten [hidden email] a écrit :

Hi,

I have combined 

   decanter-collector-log
   decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

    
François Papon
fpapon@apache.org
Yupiik - https://www.yupiik.com
Reply | Threaded
Open this post in threaded view
|

Re: Decanter Elasticsearch ReST appender network traffic

jbonofre
I think slow down is not a big deal as eventadmin dispatcher act as a buffer (with timeout). 

So the current state is already good. The only drawback is about the number of append / refresh performed on ES. That?s why bulk could be interesting. 

Regards 
JB

Le mar. 18 f?vr. 2020 ? 15:31, Francois Papon <[hidden email]> a ?crit :
Hi,
There is some improvement on the ES7 client so may be we could update the ES appender.

Another solution is that we could detect slow down ingestion and add a temporization but I also like the bulk usage :)

regards,
Fran?ois
[hidden email]
Le 18/02/2020 ? 09:04, Jean-Baptiste Onofre a ?crit :
Hi,

That?s a good question ;)

Currently, Decanter Elasticsearch appender sends each event followed by a refresh.

It directly uses Elasticsearch REST native client via performRequest.

A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it?s stored in Elasticsearch. Using bulk would dramatically decrease network usage.
I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update.

Regards
JB

Le 18 f?vr. 2020 ? 08:01, dschulten [hidden email] a ?crit :

Hi,

I have combined 

   decanter-collector-log
   decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

    
Reply | Threaded
Open this post in threaded view
|

Re: Decanter Elasticsearch ReST appender network traffic

fpapon
Yes but I was thinking the case where we are sending a lot of request without pause during slow down ingestion, may be it can have an impact on the ES side to deal with the ingestion.
If we add a temporization, it can help ES server to deal more easily with the slow down and backup to normal state.

Sometimes it could end with a OOM because the ES slowdown never stop and EventAdmin become full.

regards, 
François
[hidden email]
Le 18/02/2020 à 20:05, Jean-Baptiste Onofré a écrit :
I think slow down is not a big deal as eventadmin dispatcher act as a buffer (with timeout). 

So the current state is already good. The only drawback is about the number of append / refresh performed on ES. That?s why bulk could be interesting. 

Regards 
JB

Le mar. 18 f?vr. 2020 ? 15:31, Francois Papon [hidden email] a ?crit :
Hi,
There is some improvement on the ES7 client so may be we could update the ES appender.

Another solution is that we could detect slow down ingestion and add a temporization but I also like the bulk usage :)

regards,
Fran?ois
[hidden email]
Le 18/02/2020 ? 09:04, Jean-Baptiste Onofre a ?crit :
Hi,

That?s a good question ;)

Currently, Decanter Elasticsearch appender sends each event followed by a refresh.

It directly uses Elasticsearch REST native client via performRequest.

A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it?s stored in Elasticsearch. Using bulk would dramatically decrease network usage.
I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update.

Regards
JB

Le 18 f?vr. 2020 ? 08:01, dschulten [hidden email] a ?crit :

Hi,

I have combined 

   decanter-collector-log
   decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
François Papon
fpapon@apache.org
Yupiik - https://www.yupiik.com
Reply | Threaded
Open this post in threaded view
|

Re: Decanter Elasticsearch ReST appender network traffic

jbonofre
Eventadmin topic has a max size to avoid OOM. So not a big deal. 

Regards
JB

Le mar. 18 f?vr. 2020 ? 21:33, Francois Papon <[hidden email]> a ?crit :
Yes but I was thinking the case where we are sending a lot of request without pause during slow down ingestion, may be it can have an impact on the ES side to deal with the ingestion.
If we add a temporization, it can help ES server to deal more easily with the slow down and backup to normal state.

Sometimes it could end with a OOM because the ES slowdown never stop and EventAdmin become full.

regards, 
Fran?ois
[hidden email]
Le 18/02/2020 ? 20:05, Jean-Baptiste Onofr? a ?crit :
I think slow down is not a big deal as eventadmin dispatcher act as a buffer (with timeout). 

So the current state is already good. The only drawback is about the number of append / refresh performed on ES. That?s why bulk could be interesting. 

Regards 
JB

Le mar. 18 f?vr. 2020 ? 15:31, Francois Papon [hidden email] a ?crit :
Hi,
There is some improvement on the ES7 client so may be we could update the ES appender.

Another solution is that we could detect slow down ingestion and add a temporization but I also like the bulk usage :)

regards,
Fran?ois
[hidden email]
Le 18/02/2020 ? 09:04, Jean-Baptiste Onofre a ?crit :
Hi,

That?s a good question ;)

Currently, Decanter Elasticsearch appender sends each event followed by a refresh.

It directly uses Elasticsearch REST native client via performRequest.

A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it?s stored in Elasticsearch. Using bulk would dramatically decrease network usage.
I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update.

Regards
JB

Le 18 f?vr. 2020 ? 08:01, dschulten [hidden email] a ?crit :

Hi,

I have combined 

   decanter-collector-log
   decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html