Karaf hangs on startup (4.0.7)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Karaf hangs on startup (4.0.7)

Andrei Shakirin
Hi,

I have the following issue with Karaf (4.0.7) startup:
The startup process completely hangs after starting Jetty Server instance:

...
2018-06-17 04:00:42,326INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplDone.
2018-06-17 04:00:42,360INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplAdding features: spring-dm/[1.2.1,1.2.1], cxf-abdera/[3.1.8,3.1.8], cxf/[3.1.8,3.1.8], cxf-commands/[3.1.8,3.1.8], activemq-client/[5.14.5,5.14.5], camel/[2.17.3,2.17.3], camel-cxf/[2.17.3,2.17.3], camel-jms/[2.17.3,2.17.3], basit-server-availability/[3.23.2,3.23.2], basit-address-service/[3.34.1,3.34.1], basit-administration-service/[1.25.0,1.25.0], basit-article-domain/[7.26.0,7.26.0], basit-article-service/[4.28.0,4.28.0], basit-connector-atp-engine/[2.29.2,2.29.2], basit-cart-service/[6.21.0,6.21.0], basit-checkout-service/[9.2.0,9.2.0], basit-communication-engine/[1.24.0,1.24.0], basit-core-domain/[10.19.1,10.19.1], basit-customer-domain/[13.3.0,13.3.0], basit-datasource-ecom-pg/[9.11.0,9.11.0], basit-connector-ecom/[9.11.0,9.11.0], basit-datasource-hybris/[2.39.0,2.39.0], basit-connector-hybris/[2.39.0,2.39.0], basit-datasource-talendmd-pg/[3.34.0,3.34.0], basit-connector-talendmd/[3.34.0,3.34.0], basit-ebay-gateway/[1.28.0,1.28.0], basit-exclusion-engine/[1.14.0,1.14.0], basit-giftcard-service/[0,0.0.0], basit-connector-jms/[1.37.0,1.37.0], basit-messaging-service/[1.27.0,1.27.0], basit-newsletter-service/[2.8.3,2.8.3], basit-offer-service/[3.27.1,3.27.1], basit-order-service/[0,0.0.0], basit-payment-service/[1.33.0,1.33.0], basit-connector-sap/[4.16.0,4.16.0], basit-user-service/[11.11.1,11.11.1]
2018-06-17 04:00:42,384INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplSPDY not available, creating standard ServerConnector for Http
2018-06-17 04:00:42,398INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerImplPax Web available at [0.0.0.0]:[8040]
2018-06-17 04:00:42,418INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplNo ALPN class available
2018-06-17 04:00:42,418INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplSPDY not available, creating standard ServerConnector for Https
2018-06-17 04:00:42,420INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerImplPax Web available at [0.0.0.0]:[8041]
2018-06-17 04:00:42,431INFO [{{bundle.name,org.ops4j.pax.web.pax-web-runtime}{bundle.version,4.3.0}{bundle.id,95}}][null]org.ops4j.pax.web.service.internal.HttpServiceFactoryImplBinding bundle: [org.apache.karaf.webconsole.console_4.0.7 [56]] to http service
2018-06-17 04:00:42,469INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerWrapperwill add org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer to ServletContainerInitializers
2018-06-17 04:00:42,747INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerWrapperadded ServletContainerInitializer: org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer
2018-06-17 04:00:42,781INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.HttpServiceContextregistering context org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36e9b33f, with context-name:
2018-06-17 04:00:42,827INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandlerStarted HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36e9b33f}
2018-06-17 04:00:42,830INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20160908
2018-06-17 04:00:42,928INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted default@3616b12b{HTTP/1.1}{0.0.0.0:8040}
2018-06-17 04:00:42,959INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted secureDefault@4b972cf2{SSL-http/1.1}{0.0.0.0:8041}
2018-06-17 04:00:42,960INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @5804ms
----------------------------------------------------------
After that just nothing happens.

In normal case Karaf starts FeaturesServiceImpl with "Changes to perform:" message after jetty:

....
2018-06-17 05:00:35,815INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.HttpServiceContextregistering context org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77ca971, with context-name:
2018-06-17 05:00:35,849INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandlerStarted HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77ca971}
2018-06-17 05:00:35,852INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20160908
2018-06-17 05:00:35,929INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted default@2e0e40ef{HTTP/1.1}{0.0.0.0:8040}
2018-06-17 05:00:35,957INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted secureDefault@645d3768{SSL-http/1.1}{0.0.0.0:8041}
2018-06-17 05:00:35,958INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @7463ms
2018-06-17 05:00:48,501INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null] 2018-06-17 05:00:48,501INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplChanges to perform:
2018-06-17 05:00:48,502INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl  Region: root
2018-06-17 05:00:48,502INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl    Bundles to uninstall:
...

The problem happens not always - only sometimes, but quite often.

The full stacktrace and jetty configuration are attached (in stacktrace you can see that nothing happens after "2018-06-17 04:00:42,960INFO").

Any clue why it could happens and how to repair the issue?

Regards,
Andrei.

As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our contacts privacy notice at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>



jetty.xml (6K) Download Attachment
talend-runtime.log.2018-06-17.gz (6M) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Karaf hangs on startup (4.0.7)

jbonofre
Hi Andrei,

can you provide some details about the environment (OS, JVM, ...) ?

I guess it's a custom Karaf distribution ? Are you able to reproduce
with Karaf vanilla ?

Regards
JB

On 18/06/2018 13:02, Andrei Shakirin wrote:

> Hi,
>
> I have the following issue with Karaf (4.0.7) startup:
> The startup process completely hangs after starting Jetty Server instance:
>
> ...
> 2018-06-17 04:00:42,326INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplDone.
> 2018-06-17 04:00:42,360INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplAdding features: spring-dm/[1.2.1,1.2.1], cxf-abdera/[3.1.8,3.1.8], cxf/[3.1.8,3.1.8], cxf-commands/[3.1.8,3.1.8], activemq-client/[5.14.5,5.14.5], camel/[2.17.3,2.17.3], camel-cxf/[2.17.3,2.17.3], camel-jms/[2.17.3,2.17.3], basit-server-availability/[3.23.2,3.23.2], basit-address-service/[3.34.1,3.34.1], basit-administration-service/[1.25.0,1.25.0], basit-article-domain/[7.26.0,7.26.0], basit-article-service/[4.28.0,4.28.0], basit-connector-atp-engine/[2.29.2,2.29.2], basit-cart-service/[6.21.0,6.21.0], basit-checkout-service/[9.2.0,9.2.0], basit-communication-engine/[1.24.0,1.24.0], basit-core-domain/[10.19.1,10.19.1], basit-customer-domain/[13.3.0,13.3.0], basit-datasource-ecom-pg/[9.11.0,9.11.0], basit-connector-ecom/[9.11.0,9.11.0], basit-datasource-hybris/[2.39.0,2.39.0], basit-connector-hybris/[2.39.0,2.39.0], basit-datasource-talendmd-pg/[3.34.0,3.34.0], basit-connector-talendmd/[3.34.0,3.34.0], basit-ebay-gateway/[1.28.0,1.28.0], basit-exclusion-engine/[1.14.0,1.14.0], basit-giftcard-service/[0,0.0.0], basit-connector-jms/[1.37.0,1.37.0], basit-messaging-service/[1.27.0,1.27.0], basit-newsletter-service/[2.8.3,2.8.3], basit-offer-service/[3.27.1,3.27.1], basit-order-service/[0,0.0.0], basit-payment-service/[1.33.0,1.33.0], basit-connector-sap/[4.16.0,4.16.0], basit-user-service/[11.11.1,11.11.1]
> 2018-06-17 04:00:42,384INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplSPDY not available, creating standard ServerConnector for Http
> 2018-06-17 04:00:42,398INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerImplPax Web available at [0.0.0.0]:[8040]
> 2018-06-17 04:00:42,418INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplNo ALPN class available
> 2018-06-17 04:00:42,418INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFactoryImplSPDY not available, creating standard ServerConnector for Https
> 2018-06-17 04:00:42,420INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerImplPax Web available at [0.0.0.0]:[8041]
> 2018-06-17 04:00:42,431INFO [{{bundle.name,org.ops4j.pax.web.pax-web-runtime}{bundle.version,4.3.0}{bundle.id,95}}][null]org.ops4j.pax.web.service.internal.HttpServiceFactoryImplBinding bundle: [org.apache.karaf.webconsole.console_4.0.7 [56]] to http service
> 2018-06-17 04:00:42,469INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerWrapperwill add org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer to ServletContainerInitializers
> 2018-06-17 04:00:42,747INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyServerWrapperadded ServletContainerInitializer: org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer
> 2018-06-17 04:00:42,781INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.HttpServiceContextregistering context org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36e9b33f, with context-name:
> 2018-06-17 04:00:42,827INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandlerStarted HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36e9b33f}
> 2018-06-17 04:00:42,830INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20160908
> 2018-06-17 04:00:42,928INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted default@3616b12b{HTTP/1.1}{0.0.0.0:8040}
> 2018-06-17 04:00:42,959INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted secureDefault@4b972cf2{SSL-http/1.1}{0.0.0.0:8041}
> 2018-06-17 04:00:42,960INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @5804ms
> ----------------------------------------------------------
> After that just nothing happens.
>
> In normal case Karaf starts FeaturesServiceImpl with "Changes to perform:" message after jetty:
>
> ....
> 2018-06-17 05:00:35,815INFO [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.HttpServiceContextregistering context org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77ca971, with context-name:
> 2018-06-17 05:00:35,849INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandlerStarted HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77ca971}
> 2018-06-17 05:00:35,852INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20160908
> 2018-06-17 05:00:35,929INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted default@2e0e40ef{HTTP/1.1}{0.0.0.0:8040}
> 2018-06-17 05:00:35,957INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted secureDefault@645d3768{SSL-http/1.1}{0.0.0.0:8041}
> 2018-06-17 05:00:35,958INFO [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @7463ms
> 2018-06-17 05:00:48,501INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null] 2018-06-17 05:00:48,501INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplChanges to perform:
> 2018-06-17 05:00:48,502INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl  Region: root
> 2018-06-17 05:00:48,502INFO [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl    Bundles to uninstall:
> ...
>
> The problem happens not always - only sometimes, but quite often.
>
> The full stacktrace and jetty configuration are attached (in stacktrace you can see that nothing happens after "2018-06-17 04:00:42,960INFO").
>
> Any clue why it could happens and how to repair the issue?
>
> Regards,
> Andrei.
>
> As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our contacts privacy notice at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

RE: Karaf hangs on startup (4.0.7)

Andrei Shakirin
Hi JB,

JVM: 1.8.0_151
OS: Linux #16-Ubuntu.

Yes, it is custom Karaf distribution. Will try to reproduce it on clean Karaf.

I took thread dumps from hanging Karaf (attached). Can you see anything suspicious there?

Regards,
Andrei.


> -----Original Message-----
> From: Jean-Baptiste Onofré [mailto:[hidden email]]
> Sent: Montag, 18. Juni 2018 13:32
> To: [hidden email]
> Subject: Re: Karaf hangs on startup (4.0.7)
>
> Hi Andrei,
>
> can you provide some details about the environment (OS, JVM, ...) ?
>
> I guess it's a custom Karaf distribution ? Are you able to reproduce with Karaf
> vanilla ?
>
> Regards
> JB
>
> On 18/06/2018 13:02, Andrei Shakirin wrote:
> > Hi,
> >
> > I have the following issue with Karaf (4.0.7) startup:
> > The startup process completely hangs after starting Jetty Server instance:
> >
> > ...
> > 2018-06-17 04:00:42,326INFO
> [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id
> ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplDone.
> > 2018-06-17 04:00:42,360INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id,9}}][null]org.apache.karaf.features.internal.service.Features
> > ServiceImplAdding features: spring-dm/[1.2.1,1.2.1],
> > cxf-abdera/[3.1.8,3.1.8], cxf/[3.1.8,3.1.8],
> > cxf-commands/[3.1.8,3.1.8], activemq-client/[5.14.5,5.14.5],
> > camel/[2.17.3,2.17.3], camel-cxf/[2.17.3,2.17.3],
> > camel-jms/[2.17.3,2.17.3], basit-server-availability/[3.23.2,3.23.2],
> > basit-address-service/[3.34.1,3.34.1],
> > basit-administration-service/[1.25.0,1.25.0],
> > basit-article-domain/[7.26.0,7.26.0],
> > basit-article-service/[4.28.0,4.28.0],
> > basit-connector-atp-engine/[2.29.2,2.29.2],
> > basit-cart-service/[6.21.0,6.21.0],
> > basit-checkout-service/[9.2.0,9.2.0],
> > basit-communication-engine/[1.24.0,1.24.0],
> > basit-core-domain/[10.19.1,10.19.1],
> > basit-customer-domain/[13.3.0,13.3.0],
> > basit-datasource-ecom-pg/[9.11.0,9.11.0],
> > basit-connector-ecom/[9.11.0,9.11.0],
> > basit-datasource-hybris/[2.39.0,2.39.0],
> > basit-connector-hybris/[2.39.0,2.39.0],
> > basit-datasource-talendmd-pg/[3.34.0,3.34.0],
> > basit-connector-talendmd/[3.34.0,3.34.0],
> > basit-ebay-gateway/[1.28.0,1.28.0],
> > basit-exclusion-engine/[1.14.0,1.14.0],
> > basit-giftcard-service/[0,0.0.0], basit-connector-jms/[1.37.0,1.37.0],
> > basit-messaging-service/[1.27.0,1.27.0],
> > basit-newsletter-service/[2.8.3,2.8.3],
> > basit-offer-service/[3.27.1,3.27.1], basit-order-service/[0,0.0.0],
> > basit-payment-service/[1.33.0,1.33.0],
> > basit-connector-sap/[4.16.0,4.16.0],
> > basit-user-service/[11.11.1,11.11.1]
> > 2018-06-17 04:00:42,384INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFac
> > toryImplSPDY not available, creating standard ServerConnector for Http
> > 2018-06-17 04:00:42,398INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettySer
> > verImplPax Web available at [0.0.0.0]:[8040]
> > 2018-06-17 04:00:42,418INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFac
> > toryImplNo ALPN class available
> > 2018-06-17 04:00:42,418INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyFac
> > toryImplSPDY not available, creating standard ServerConnector for
> > Https
> > 2018-06-17 04:00:42,420INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettySer
> > verImplPax Web available at [0.0.0.0]:[8041]
> > 2018-06-17 04:00:42,431INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-runtime}{bundle.version,4.3.0
> > }{bundle.id,95}}][null]org.ops4j.pax.web.service.internal.HttpServiceF
> > actoryImplBinding bundle: [org.apache.karaf.webconsole.console_4.0.7
> > [56]] to http service
> > 2018-06-17 04:00:42,469INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettySer
> > verWrapperwill add
> > org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContai
> > nerInitializer to ServletContainerInitializers
> > 2018-06-17 04:00:42,747INFO
> > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0}{
> > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettySer
> > verWrapperadded ServletContainerInitializer:
> > org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContai
> > nerInitializer
> > 2018-06-17 04:00:42,781INFO [{{bundle.name,org.ops4j.pax.web.pax-web-
> jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.i
> nternal.HttpServiceContextregistering context
> org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36e9b
> 33f, with context-name:
> > 2018-06-17 04:00:42,827INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandler
> > Started
> > HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.se
> > rvlet.OsgiManagerHttpContext@36e9b33f}
> > 2018-06-17 04:00:42,830INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20
> > 160908
> > 2018-06-17 04:00:42,928INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted
> > default@3616b12b{HTTP/1.1}{0.0.0.0:8040}
> > 2018-06-17 04:00:42,959INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted
> > secureDefault@4b972cf2{SSL-http/1.1}{0.0.0.0:8041}
> > 2018-06-17 04:00:42,960INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @5804ms
> > ----------------------------------------------------------
> > After that just nothing happens.
> >
> > In normal case Karaf starts FeaturesServiceImpl with "Changes to perform:"
> message after jetty:
> >
> > ....
> > 2018-06-17 05:00:35,815INFO [{{bundle.name,org.ops4j.pax.web.pax-web-
> jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.i
> nternal.HttpServiceContextregistering context
> org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77ca9
> 71, with context-name:
> > 2018-06-17 05:00:35,849INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandler
> > Started
> > HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.se
> > rvlet.OsgiManagerHttpContext@77ca971}
> > 2018-06-17 05:00:35,852INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v20
> > 160908
> > 2018-06-17 05:00:35,929INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted
> > default@2e0e40ef{HTTP/1.1}{0.0.0.0:8040}
> > 2018-06-17 05:00:35,957INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStarted
> > secureDefault@645d3768{SSL-http/1.1}{0.0.0.0:8041}
> > 2018-06-17 05:00:35,958INFO
> > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v20160908
> > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted @7463ms
> > 2018-06-17 05:00:48,501INFO
> [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id
> ,9}}][null] 2018-06-17 05:00:48,501INFO
> [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id
> ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplChang
> es to perform:
> > 2018-06-17 05:00:48,502INFO
> [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id
> ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl
> Region: root
> > 2018-06-17 05:00:48,502INFO
> [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{bundle.id
> ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImpl
> Bundles to uninstall:
> > ...
> >
> > The problem happens not always - only sometimes, but quite often.
> >
> > The full stacktrace and jetty configuration are attached (in stacktrace you can
> see that nothing happens after "2018-06-17 04:00:42,960INFO").
> >
> > Any clue why it could happens and how to repair the issue?
> >
> > Regards,
> > Andrei.
> >
> > As a recipient of an email from Talend, your contact personal data
> > will be on our systems. Please see our contacts privacy notice at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__blog.nanthrax.net&d=DwIG-
> g&c=2w5q_42kFG40MI2alLPgJw&r=bWOqkHjIZE0sZtdpFMIhm-
> lcbhtB3cv08OlIr0lkKR4&m=LRcCRwvQdL0VBDlLPAYjd2DNfbeMm3wRTun9cjUP
> p8c&s=Wxx3gb1-rmuYShLTll-DDQmjIOgIMGtYB7cpZLoSkcs&e=
> Talend - http://www.talend.com

jstack.out (125K) Download Attachment
stacktrace.txt (137K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Karaf hangs on startup (4.0.7)

Andrei Shakirin
Hi,

I have identified the problem more precisely.
The issue occurs if feature URLs point to two different versions of the same wrapped thirdparties dependency (like Hystrix).
After fix features in way that they points on the same version, issue disappears.
I will try to distil the issue in simple example based on vanilla Karaf, anyway it would be nice to investigate and fix it (at least to produce clear log error messages).
Otherwise the problem is really weird and hard to investigate.

Regards,
Andrei.


> -----Original Message-----
> From: Andrei Shakirin [mailto:[hidden email]]
> Sent: Dienstag, 19. Juni 2018 09:54
> To: [hidden email]
> Cc: [hidden email]
> Subject: RE: Karaf hangs on startup (4.0.7)
>
> Hi JB,
>
> JVM: 1.8.0_151
> OS: Linux #16-Ubuntu.
>
> Yes, it is custom Karaf distribution. Will try to reproduce it on clean Karaf.
>
> I took thread dumps from hanging Karaf (attached). Can you see anything
> suspicious there?
>
> Regards,
> Andrei.
>
>
> > -----Original Message-----
> > From: Jean-Baptiste Onofré [mailto:[hidden email]]
> > Sent: Montag, 18. Juni 2018 13:32
> > To: [hidden email]
> > Subject: Re: Karaf hangs on startup (4.0.7)
> >
> > Hi Andrei,
> >
> > can you provide some details about the environment (OS, JVM, ...) ?
> >
> > I guess it's a custom Karaf distribution ? Are you able to reproduce
> > with Karaf vanilla ?
> >
> > Regards
> > JB
> >
> > On 18/06/2018 13:02, Andrei Shakirin wrote:
> > > Hi,
> > >
> > > I have the following issue with Karaf (4.0.7) startup:
> > > The startup process completely hangs after starting Jetty Server instance:
> > >
> > > ...
> > > 2018-06-17 04:00:42,326INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id
> >
> ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplDone.
> > > 2018-06-17 04:00:42,360INFO
> > > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}
> > > {b
> > > undle.id,9}}][null]org.apache.karaf.features.internal.service.Featur
> > > es ServiceImplAdding features: spring-dm/[1.2.1,1.2.1],
> > > cxf-abdera/[3.1.8,3.1.8], cxf/[3.1.8,3.1.8],
> > > cxf-commands/[3.1.8,3.1.8], activemq-client/[5.14.5,5.14.5],
> > > camel/[2.17.3,2.17.3], camel-cxf/[2.17.3,2.17.3],
> > > camel-jms/[2.17.3,2.17.3],
> > > basit-server-availability/[3.23.2,3.23.2],
> > > basit-address-service/[3.34.1,3.34.1],
> > > basit-administration-service/[1.25.0,1.25.0],
> > > basit-article-domain/[7.26.0,7.26.0],
> > > basit-article-service/[4.28.0,4.28.0],
> > > basit-connector-atp-engine/[2.29.2,2.29.2],
> > > basit-cart-service/[6.21.0,6.21.0],
> > > basit-checkout-service/[9.2.0,9.2.0],
> > > basit-communication-engine/[1.24.0,1.24.0],
> > > basit-core-domain/[10.19.1,10.19.1],
> > > basit-customer-domain/[13.3.0,13.3.0],
> > > basit-datasource-ecom-pg/[9.11.0,9.11.0],
> > > basit-connector-ecom/[9.11.0,9.11.0],
> > > basit-datasource-hybris/[2.39.0,2.39.0],
> > > basit-connector-hybris/[2.39.0,2.39.0],
> > > basit-datasource-talendmd-pg/[3.34.0,3.34.0],
> > > basit-connector-talendmd/[3.34.0,3.34.0],
> > > basit-ebay-gateway/[1.28.0,1.28.0],
> > > basit-exclusion-engine/[1.14.0,1.14.0],
> > > basit-giftcard-service/[0,0.0.0],
> > > basit-connector-jms/[1.37.0,1.37.0],
> > > basit-messaging-service/[1.27.0,1.27.0],
> > > basit-newsletter-service/[2.8.3,2.8.3],
> > > basit-offer-service/[3.27.1,3.27.1], basit-order-service/[0,0.0.0],
> > > basit-payment-service/[1.33.0,1.33.0],
> > > basit-connector-sap/[4.16.0,4.16.0],
> > > basit-user-service/[11.11.1,11.11.1]
> > > 2018-06-17 04:00:42,384INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
> > > ac toryImplSPDY not available, creating standard ServerConnector for
> > > Http
> > > 2018-06-17 04:00:42,398INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
> > > er verImplPax Web available at [0.0.0.0]:[8040]
> > > 2018-06-17 04:00:42,418INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
> > > ac
> > > toryImplNo ALPN class available
> > > 2018-06-17 04:00:42,418INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
> > > ac toryImplSPDY not available, creating standard ServerConnector for
> > > Https
> > > 2018-06-17 04:00:42,420INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
> > > er verImplPax Web available at [0.0.0.0]:[8041]
> > > 2018-06-17 04:00:42,431INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-runtime}{bundle.version,4.3
> > > .0
> > > }{bundle.id,95}}][null]org.ops4j.pax.web.service.internal.HttpServic
> > > eF actoryImplBinding bundle:
> > > [org.apache.karaf.webconsole.console_4.0.7
> > > [56]] to http service
> > > 2018-06-17 04:00:42,469INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
> > > er
> > > verWrapperwill add
> > > org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerCont
> > > ai nerInitializer to ServletContainerInitializers
> > > 2018-06-17 04:00:42,747INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
> > > }{
> > > bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
> > > er verWrapperadded ServletContainerInitializer:
> > > org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerCont
> > > ai
> > > nerInitializer
> > > 2018-06-17 04:00:42,781INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-
> > jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.se
> > rvice.jetty.i nternal.HttpServiceContextregistering context
> > org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36
> > e9b
> > 33f, with context-name:
> > > 2018-06-17 04:00:42,827INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandl
> > > er
> > > Started
> > > HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.
> > > se rvlet.OsgiManagerHttpContext@36e9b33f}
> > > 2018-06-17 04:00:42,830INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v
> > > 20
> > > 160908
> > > 2018-06-17 04:00:42,928INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
> > > ed default@3616b12b{HTTP/1.1}{0.0.0.0:8040}
> > > 2018-06-17 04:00:42,959INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
> > > ed secureDefault@4b972cf2{SSL-http/1.1}{0.0.0.0:8041}
> > > 2018-06-17 04:00:42,960INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08 }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted
> > > @5804ms
> > > ----------------------------------------------------------
> > > After that just nothing happens.
> > >
> > > In normal case Karaf starts FeaturesServiceImpl with "Changes to perform:"
> > message after jetty:
> > >
> > > ....
> > > 2018-06-17 05:00:35,815INFO
> > > [{{bundle.name,org.ops4j.pax.web.pax-web-
> > jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.se
> > rvice.jetty.i nternal.HttpServiceContextregistering context
> > org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77
> > ca9
> > 71, with context-name:
> > > 2018-06-17 05:00:35,849INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandl
> > > er
> > > Started
> > > HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.
> > > se rvlet.OsgiManagerHttpContext@77ca971}
> > > 2018-06-17 05:00:35,852INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v
> > > 20
> > > 160908
> > > 2018-06-17 05:00:35,929INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
> > > ed default@2e0e40ef{HTTP/1.1}{0.0.0.0:8040}
> > > 2018-06-17 05:00:35,957INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08
> > > }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
> > > ed secureDefault@645d3768{SSL-http/1.1}{0.0.0.0:8041}
> > > 2018-06-17 05:00:35,958INFO
> > > [{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
> > > 08 }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted
> > > @7463ms
> > > 2018-06-17 05:00:48,501INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id ,9}}][null] 2018-06-17 05:00:48,501INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id
> > ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
> > mplChang
> > es to perform:
> > > 2018-06-17 05:00:48,502INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id
> > ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
> > mpl
> > Region: root
> > > 2018-06-17 05:00:48,502INFO
> > [{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
> > undle.id
> > ,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
> > mpl
> > Bundles to uninstall:
> > > ...
> > >
> > > The problem happens not always - only sometimes, but quite often.
> > >
> > > The full stacktrace and jetty configuration are attached (in
> > > stacktrace you can
> > see that nothing happens after "2018-06-17 04:00:42,960INFO").
> > >
> > > Any clue why it could happens and how to repair the issue?
> > >
> > > Regards,
> > > Andrei.
> > >
> > > As a recipient of an email from Talend, your contact personal data
> > > will be on our systems. Please see our contacts privacy notice at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > [hidden email]
> > https://urldefense.proofpoint.com/v2/url?u=http-
> > 3A__blog.nanthrax.net&d=DwIG-
> > g&c=2w5q_42kFG40MI2alLPgJw&r=bWOqkHjIZE0sZtdpFMIhm-
> >
> lcbhtB3cv08OlIr0lkKR4&m=LRcCRwvQdL0VBDlLPAYjd2DNfbeMm3wRTun9cjUP
> > p8c&s=Wxx3gb1-rmuYShLTll-DDQmjIOgIMGtYB7cpZLoSkcs&e=
> > Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

RE: Karaf hangs on startup (4.0.7)

jbonofre
Hi

That's probably around the features resolver. As we did large improvements in 4.2 compared to 4.0, I would start by checking if the issue happens in 4.2.x.

Regards
JB
Le 27 juin 2018, à 05:28, Andrei Shakirin <[hidden email]> a écrit:
Hi, 

I have identified the problem more precisely.
The issue occurs if feature URLs point to two different versions of the same wrapped thirdparties dependency (like Hystrix).
After fix features in way that they points on the same version, issue disappears.
I will try to distil the issue in simple example based on vanilla Karaf, anyway it would be nice to investigate and fix it (at least to produce clear log error messages).
Otherwise the problem is really weird and hard to investigate.

Regards,
Andrei.


-----Original Message-----
From: Andrei Shakirin [mailto:[hidden email]]
Sent: Dienstag, 19. Juni 2018 09:54
To: [hidden email]
Cc: [hidden email]
Subject: RE: Karaf hangs on startup (4.0.7)

Hi JB,

JVM: 1.8.0_151
OS: Linux #16-Ubuntu.

Yes, it is custom Karaf distribution. Will try to reproduce it on clean Karaf.

I took thread dumps from hanging Karaf (attached). Can you see anything
suspicious there?

Regards,
Andrei.


-----Original Message-----
From: Jean-Baptiste Onofré [mailto:[hidden email]]
Sent: Montag, 18. Juni 2018 13:32
To: [hidden email]
Subject: Re: Karaf hangs on startup (4.0.7)

Hi Andrei,

can you provide some details about the environment (OS, JVM, ...) ?

I guess it's a custom Karaf distribution ? Are you able to reproduce
with Karaf vanilla ?

Regards
JB

On 18/06/2018 13:02, Andrei Shakirin wrote:
Hi,

I have the following issue with Karaf (4.0.7) startup:
The startup process completely hangs after starting Jetty Server instance:

...
2018-06-17 04:00:42,326INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
undle.id

,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceImplDone.
2018-06-17 04:00:42,360INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}
{b
undle.id,9}}][null]org.apache.karaf.features.internal.service.Featur
es ServiceImplAdding features: spring-dm/[1.2.1,1.2.1],
cxf-abdera/[3.1.8,3.1.8], cxf/[3.1.8,3.1.8],
cxf-commands/[3.1.8,3.1.8], activemq-client/[5.14.5,5.14.5],
camel/[2.17.3,2.17.3], camel-cxf/[2.17.3,2.17.3],
camel-jms/[2.17.3,2.17.3],
basit-server-availability/[3.23.2,3.23.2],
basit-address-service/[3.34.1,3.34.1],
basit-administration-service/[1.25.0,1.25.0],
basit-article-domain/[7.26.0,7.26.0],
basit-article-service/[4.28.0,4.28.0],
basit-connector-atp-engine/[2.29.2,2.29.2],
basit-cart-service/[6.21.0,6.21.0],
basit-checkout-service/[9.2.0,9.2.0],
basit-communication-engine/[1.24.0,1.24.0],
basit-core-domain/[10.19.1,10.19.1],
basit-customer-domain/[13.3.0,13.3.0],
basit-datasource-ecom-pg/[9.11.0,9.11.0],
basit-connector-ecom/[9.11.0,9.11.0],
basit-datasource-hybris/[2.39.0,2.39.0],
basit-connector-hybris/[2.39.0,2.39.0],
basit-datasource-talendmd-pg/[3.34.0,3.34.0],
basit-connector-talendmd/[3.34.0,3.34.0],
basit-ebay-gateway/[1.28.0,1.28.0],
basit-exclusion-engine/[1.14.0,1.14.0],
basit-giftcard-service/[0,0.0.0],
basit-connector-jms/[1.37.0,1.37.0],
basit-messaging-service/[1.27.0,1.27.0],
basit-newsletter-service/[2.8.3,2.8.3],
basit-offer-service/[3.27.1,3.27.1], basit-order-service/[0,0.0.0],
basit-payment-service/[1.33.0,1.33.0],
basit-connector-sap/[4.16.0,4.16.0],
basit-user-service/[11.11.1,11.11.1]
2018-06-17 04:00:42,384INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
ac toryImplSPDY not available, creating standard ServerConnector for
Http
2018-06-17 04:00:42,398INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
er verImplPax Web available at [0.0.0.0]:[8040]
2018-06-17 04:00:42,418INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
ac
toryImplNo ALPN class available
2018-06-17 04:00:42,418INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyF
ac toryImplSPDY not available, creating standard ServerConnector for
Https
2018-06-17 04:00:42,420INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
er verImplPax Web available at [0.0.0.0]:[8041]
2018-06-17 04:00:42,431INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-runtime}{bundle.version,4.3
.0
}{bundle.id,95}}][null]org.ops4j.pax.web.service.internal.HttpServic
eF actoryImplBinding bundle:
[org.apache.karaf.webconsole.console_4.0.7
[56]] to http service
2018-06-17 04:00:42,469INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
er
verWrapperwill add
org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerCont
ai nerInitializer to ServletContainerInitializers
2018-06-17 04:00:42,747INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-jetty}{bundle.version,4.3.0
}{
bundle.id,94}}][null]org.ops4j.pax.web.service.jetty.internal.JettyS
er verWrapperadded ServletContainerInitializer:
org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerCont
ai
nerInitializer
2018-06-17 04:00:42,781INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-
jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.se
rvice.jetty.i nternal.HttpServiceContextregistering context
org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@36
e9b
33f, with context-name:
2018-06-17 04:00:42,827INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandl
er
Started
HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.
se rvlet.OsgiManagerHttpContext@36e9b33f}
2018-06-17 04:00:42,830INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v
20
160908
2018-06-17 04:00:42,928INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
ed default@3616b12b{HTTP/1.1}{0.0.0.0:8040}
2018-06-17 04:00:42,959INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
ed secureDefault@4b972cf2{SSL-http/1.1}{0.0.0.0:8041}
2018-06-17 04:00:42,960INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08 }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted
@5804ms


After that just nothing happens.

In normal case Karaf starts FeaturesServiceImpl with "Changes to perform:"
message after jetty:

....
2018-06-17 05:00:35,815INFO
[{{bundle.name,org.ops4j.pax.web.pax-web-
jetty}{bundle.version,4.3.0}{bundle.id,94}}][null]org.ops4j.pax.web.se
rvice.jetty.i nternal.HttpServiceContextregistering context
org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@77
ca9
71, with context-name:
2018-06-17 05:00:35,849INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.handler.ContextHandl
er
Started
HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.
se rvlet.OsgiManagerHttpContext@77ca971}
2018-06-17 05:00:35,852INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.Serverjetty-9.2.19.v
20
160908
2018-06-17 05:00:35,929INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
ed default@2e0e40ef{HTTP/1.1}{0.0.0.0:8040}
2018-06-17 05:00:35,957INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08
}{bundle.id,79}}][null]org.eclipse.jetty.server.ServerConnectorStart
ed secureDefault@645d3768{SSL-http/1.1}{0.0.0.0:8041}
2018-06-17 05:00:35,958INFO
[{{bundle.name,org.eclipse.jetty.util}{bundle.version,9.2.19.v201609
08 }{bundle.id,79}}][null]org.eclipse.jetty.server.ServerStarted
@7463ms
2018-06-17 05:00:48,501INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
undle.id ,9}}][null] 2018-06-17 05:00:48,501INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
undle.id
,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
mplChang
es to perform:
2018-06-17 05:00:48,502INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
undle.id
,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
mpl
Region: root
2018-06-17 05:00:48,502INFO
[{{bundle.name,org.apache.karaf.features.core}{bundle.version,4.0.7}{b
undle.id
,9}}][null]org.apache.karaf.features.internal.service.FeaturesServiceI
mpl
Bundles to uninstall:
...

The problem happens not always - only sometimes, but quite often.

The full stacktrace and jetty configuration are attached (in
stacktrace you can
see that nothing happens after "2018-06-17 04:00:42,960INFO").

Any clue why it could happens and how to repair the issue?

Regards,
Andrei.

As a recipient of an email from Talend, your contact personal data
will be on our systems. Please see our contacts privacy notice at
Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>



--
Jean-Baptiste Onofré
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-
3A__blog.nanthrax.net&d=DwIG-
g&c=2w5q_42kFG40MI2alLPgJw&r=bWOqkHjIZE0sZtdpFMIhm-

lcbhtB3cv08OlIr0lkKR4&m=LRcCRwvQdL0VBDlLPAYjd2DNfbeMm3wRTun9cjUP
p8c&s=Wxx3gb1-rmuYShLTll-DDQmjIOgIMGtYB7cpZLoSkcs&e=
Talend - http://www.talend.com