Problems with Jetty Session Managemenent?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with Jetty Session Managemenent?

lechlukasz

Hello,

 

Did anyone had problems with Jetty Session Management with Karaf 4.2.x ?

 

After updating my Karaf and Java I’ve executed load tests that simulated many concurrent users. The test went through, and I’ve waited until all remaining sessions get scavenged (I have Prometheus metrics registered in session listener), and after that I’ve repeated the test with smaller number of users and to my surprise, the container died immediately.

 

I have health checks that do sample memory-intensive operation, that should, in theory, kill container when Java reaches OutOfMemory, but none of them has triggered. I’ve tried to heal the container restarting DB connection pool, etc., then I’ve got an idea that the Jetty could be broken, and after restarting pax-web-jetty bundle, the container went alive. So it looks like, something went stuck inside the Jetty container.

 

I’ll try adding creating HttpSession to health checks and make a test that logs in/out a few users for a longer period of time to try to spot the problem more exacly…

 

Has someone notices similar unstable behavior with Jetty? Some resources get not released internally? Some undocumented session limit was reached?

 

Thanks for any input,

Lukasz Lech

 

Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
Hi Lukasz,

not I'm aware off. Do you have a scenario ?

Regards
JB

On 09/04/2019 10:12, Lukasz Lech wrote:

> Hello,
>
>  
>
> Did anyone had problems with Jetty Session Management with Karaf 4.2.x ?
>
>  
>
> After updating my Karaf and Java I’ve executed load tests that simulated
> many concurrent users. The test went through, and I’ve waited until all
> remaining sessions get scavenged (I have Prometheus metrics registered
> in session listener), and after that I’ve repeated the test with smaller
> number of users and to my surprise, the container died immediately.
>
>  
>
> I have health checks that do sample memory-intensive operation, that
> should, in theory, kill container when Java reaches OutOfMemory, but
> none of them has triggered. I’ve tried to heal the container restarting
> DB connection pool, etc., then I’ve got an idea that the Jetty could be
> broken, and after restarting pax-web-jetty bundle, the container went
> alive. So it looks like, something went stuck inside the Jetty container.
>
>  
>
> I’ll try adding creating HttpSession to health checks and make a test
> that logs in/out a few users for a longer period of time to try to spot
> the problem more exacly…
>
>  
>
> Has someone notices similar unstable behavior with Jetty? Some resources
> get not released internally? Some undocumented session limit was reached?
>
>  
>
> Thanks for any input,
>
> Lukasz Lech
>
>  
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
jbonofre wrote
> Hi Lukasz,
>
> not I'm aware off. Do you have a scenario ?

Not something application-agnostic. I have a memory intensive application,
that stores big objects in session. The session is invalidated on user
logout. The test brings the container near to JVM memory limit, but there is
no single OutOfMemory in logs.

But I need to analyste if it's not caused by cleanup operation in session
listener, for example they take too long and Jetty do not release resources
correctly because of that etc.




--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
In reply to this post by jbonofre
I've found something very disturbing, I need to rollback to Karaf 4.1.3 and
redo the tests.

I've seen that my SessionListener is registered *twice*. I've quickly
started 4.1.3 locally and confirmed, there the SessionListener is called
only once.

But to find out, if doubling of SessionListener is responsible for the
problem with Jetty I need to rollback the whole CI pipeline configuration
and produce Karaf 4.1.3-based docker image...

This is how I register the session listener:

        ((WebContainer) /*org.osgi.service.http.HttpService*/
httpService).registerEventListener((EventListener) resource,
/*org.osgi.service.http.HttpContext*/ httpContext);

I've added logging before that instruction, so I can confirm, that I call
registerEventListener only once.





--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
Yes, that could be an issue. It's maybe my fault when I updated to the
new Jetty version.

Let me check and add a test.

Regards
JB

On 09/04/2019 15:12, lechlukasz wrote:
--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
jbonofre wrote
> Yes, that could be an issue. It's maybe my fault when I updated to the
> new Jetty version.
>
> Let me check and add a test.

What is the potential solution? Waiting for Jetty fix or rollback to last
stable version?

Is it possible to rollback manually without big issues, or some other
bundles will not start with older jetty?




--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
Let me check first, I'm another fix, I will take a look just after.

Regards
JB

On 09/04/2019 17:17, lechlukasz wrote:

> jbonofre wrote
>> Yes, that could be an issue. It's maybe my fault when I updated to the
>> new Jetty version.
>>
>> Let me check and add a test.
>
> What is the potential solution? Waiting for Jetty fix or rollback to last
> stable version?
>
> Is it possible to rollback manually without big issues, or some other
> bundles will not start with older jetty?
>
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
Hello,

Have you found some solution or workaroud? Is it possible to downgrade
Jetty? I start having strange feeling I'd be stuck with Karaf version 4.1.3
until Oracle finally kills Java project... ;)



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
Hi,

The problem is certainly in Pax Web (not in Jetty).

I will fix that in Pax Web.

Regards
JB

On 29/04/2019 11:54, lechlukasz wrote:

> Hello,
>
> Have you found some solution or workaroud? Is it possible to downgrade
> Jetty? I start having strange feeling I'd be stuck with Karaf version 4.1.3
> until Oracle finally kills Java project... ;)
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
Hello,

Is it fixed in Karaf 4.2.6?
I can't find the issue in the release notes for pax web. What is the issue
number?

Best regards,
Lukasz Lech



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
Hi Lukasz,

I did a change but I don't think it's enough. I will create a Jira to
track this (at Karaf) and I will cut 4.2.7 soon.

Regards
JB

On 11/06/2019 10:16, lechlukasz wrote:

> Hello,
>
> Is it fixed in Karaf 4.2.6?
> I can't find the issue in the release notes for pax web. What is the issue
> number?
>
> Best regards,
> Lukasz Lech
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

jbonofre
In reply to this post by lechlukasz
Correction: it's fixed ;)

Here's the fix I did:

https://ops4j1.jira.com/browse/PAXWEB-1197

Regards
JB

On 11/06/2019 10:16, lechlukasz wrote:

> Hello,
>
> Is it fixed in Karaf 4.2.6?
> I can't find the issue in the release notes for pax web. What is the issue
> number?
>
> Best regards,
> Lukasz Lech
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Jetty Session Managemenent?

lechlukasz
OK great, I've updated Karaf and tested, ans everything is fine



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html