Slow shutdown of karaf

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow shutdown of karaf

P Magon
Hello,
I have a problem stopping karaf in a production environment: after using the system for days, the stop takes minutes (> 30 min). 

I am able to reproduce this behavior by continually running the integration tests during 1 hour...

I tried to monitor karaf and everything seems ok (CPU, memory, no garbage collection....).
I also tried to monitor all involved sub-systems (kafka, solr...) but everything looks ok.

There is no error during the shutdown but all steps execution are slower... 
By using a profiler, I found the following:
- Many "ClassNotFoundException" are thrown (but not logged) during the shutdown (millions of "ClassNotFoundException)
Most of them comes from "org.apache.felix.framework.ExtentionManager" (method getClassByDelegation)...

- I also found that lsof shows a lot of files (bundles jars) (that are duplicated by thread ??)
These "open files" increase continually...

I'm a little lost with these problems...
I am using karaf 4.1.2

Any suggestion or hints to investigate are welcome !

Thanks

Patrick
Reply | Threaded
Open this post in threaded view
|

Re: Slow shutdown of karaf

jbonofre
Hi Patrick,

which OSGi framework are you using ? Equinox or Felix ?

Regards
JB

On 13/06/2019 17:20, Patrick Magon wrote:

> Hello,
> I have a problem stopping karaf in a production environment: after using
> the system for days, the stop takes minutes (> 30 min). 
>
> I am able to reproduce this behavior by continually running the
> integration tests during 1 hour...
>
> I tried to monitor karaf and everything seems ok (CPU, memory, no
> garbage collection....).
> I also tried to monitor all involved sub-systems (kafka, solr...) but
> everything looks ok.
>
> There is no error during the shutdown but all steps execution are slower... 
> By using a profiler, I found the following:
> - Many "ClassNotFoundException" are thrown (but not logged) during the
> shutdown (millions of "ClassNotFoundException)
> Most of them comes from "org.apache.felix.framework.ExtentionManager"
> (method getClassByDelegation)...
>
> - I also found that lsof shows a lot of files (bundles jars) (that are
> duplicated by thread ??)
> These "open files" increase continually...
>
> I'm a little lost with these problems...
> I am using karaf 4.1.2
>
> Any suggestion or hints to investigate are welcome !
>
> Thanks
>
> Patrick

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Slow shutdown of karaf

P Magon
Hello JB,

I'm using Felix

On Thu, Jun 13, 2019 at 5:33 PM Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Patrick,

which OSGi framework are you using ? Equinox or Felix ?

Regards
JB

On 13/06/2019 17:20, Patrick Magon wrote:
> Hello,
> I have a problem stopping karaf in a production environment: after using
> the system for days, the stop takes minutes (> 30 min). 
>
> I am able to reproduce this behavior by continually running the
> integration tests during 1 hour...
>
> I tried to monitor karaf and everything seems ok (CPU, memory, no
> garbage collection....).
> I also tried to monitor all involved sub-systems (kafka, solr...) but
> everything looks ok.
>
> There is no error during the shutdown but all steps execution are slower... 
> By using a profiler, I found the following:
> - Many "ClassNotFoundException" are thrown (but not logged) during the
> shutdown (millions of "ClassNotFoundException)
> Most of them comes from "org.apache.felix.framework.ExtentionManager"
> (method getClassByDelegation)...
>
> - I also found that lsof shows a lot of files (bundles jars) (that are
> duplicated by thread ??)
> These "open files" increase continually...
>
> I'm a little lost with these problems...
> I am using karaf 4.1.2
>
> Any suggestion or hints to investigate are welcome !
>
> Thanks
>
> Patrick

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com