Shutdown Order on 2.3.3

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Shutdown Order on 2.3.3

Chris Geer
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now having some significant issues on shutdown. Previously, everything would shut down in a normal order but now for some reasons things with lower start levels are shutting down first. Our big problem is ActiveMQ. We have our activemq broker starting with a start level of 60 and our normal bundles (mostly camel) starting at 80. But on shutdown the first thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main                             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService                    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ 5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector               | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector               | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp stopped


As you can imagine that causes problems because now that the broker doesn't exist the bundles that use it start throwing errors and in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory          | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring connection ActiveMQConnection {id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false} on IOException: org.apache.activemq.transport.TransportDisposedIOException: peer (vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] | faultJmsMessageListenerContainer | 163 - org.springframework.jms - 3.2.4.RELEASE | Setup of JMS message listener invoker failed for destination 'permission-notifications' - trying to recover. Cause: peer (vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession                    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

jbonofre
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB

On 02/14/2014 06:54 PM, Chris Geer wrote:

> We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
> having some significant issues on shutdown. Previously, everything would
> shut down in a normal order but now for some reasons things with lower
> start levels are shutting down first. Our big problem is ActiveMQ. We
> have our activemq broker starting with a start level of 60 and our
> normal bundles (mostly camel) starting at 80. But on shutdown the first
> thing that shows up in the logs is this.
>
> 23:46:38,243 | INFO  | Thread-2         | Main
>    |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
> framework...
> 23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
>     | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
> BlueprintContainer for bundle activemq-broker.xml
> 23:46:38,290 | INFO  | FelixShutdown    | BrokerService
>     | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
> 5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
> 23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
>    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
> openwire stopped
> 23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
>    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
> stopped
>
>
> As you can imagine that causes problems because now that the broker
> doesn't exist the bundles that use it start throwing errors and
> in-flight transactions are failing.
>
>
> 23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
>     | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
> connection ActiveMQConnection
> {id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
> on IOException:
> org.apache.activemq.transport.TransportDisposedIOException: peer
> (vm://default#1) stopped.
> 23:46:38,444 | WARN  | n-notifications] |
> faultJmsMessageListenerContainer | 163 - org.springframework.jms -
> 3.2.4.RELEASE | Setup of JMS message listener invoker failed for
> destination 'permission-notifications' - trying to recover. Cause: peer
> (vm://default#3) stopped.
> 23:46:38,444 | WARN  | t-notifications] | PooledSession
>     | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
> trying rollback() when putting session back into the pool, will
> invalidate. javax.jms.IllegalStateException: The Session is closed
>
>
> Is there anything we can do to restore an orderly shutdown order?
>
>
> Thanks,
>
> Chris
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

Chris Geer
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

dantran


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com


Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

Chris Geer
Thanks Dan. Since 1.1.1 seems to have never been released I tried to upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any news on when 2.3.4 will be released?

Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this issue?

Chris


On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[hidden email]> wrote:


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com



Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

dantran
Karaf dev team is currently calling a vote to karaf-2.3.4, You may want to try it out  and provide feedback

-D


On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[hidden email]> wrote:
Thanks Dan. Since 1.1.1 seems to have never been released I tried to upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any news on when 2.3.4 will be released?

Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this issue?

Chris


On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[hidden email]> wrote:


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com




Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

Chris Geer
Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still destroying lower start level items before higher start level items. I'm surprised no one else is noticing this. Here is what I see in my logs (leaving out hundreds of error messages). The lines with the * are start level 60 and the ** is start level 80. The rest of the start level 80 ones are killed even later...after karaf bundles are gone.

10:46:58,361 | INFO  | Thread-2         | Main                             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
10:46:58,378 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle locationvalidation-google
**10:46:58,389 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle operations-service
10:46:58,416 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle utils
*10:46:58,418 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-broker.xml
10:47:00,987 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands
10:47:01,038 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-karaf
10:47:01,056 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands
10:47:01,067 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean
10:47:01,069 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.web
10:47:01,071 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev
10:47:01,073 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages
10:47:01,075 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.log
10:47:01,077 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.config
10:47:01,081 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.services
10:47:01,083 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.system
10:47:01,085 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.command
10:47:01,090 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.ssh
10:47:01,104 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.management
10:47:01,106 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.common
10:47:01,120 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.packages
10:47:01,127 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.core
10:47:01,165 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.command
10:47:01,167 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.dev
10:47:01,188 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.management
10:47:01,190 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.features.command
10:47:01,250 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.jaas.command
10:47:01,274 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.commands
10:47:01,333 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.console
10:47:01,365 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.log is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,372 | INFO  | rint Extender: 1 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.web is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,441 | INFO  | rint Extender: 2 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.osgi is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,478 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.config is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.1.0]
10:47:01,484 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.web
10:47:01,487 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.config
10:47:01,490 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.osgi
10:47:01,492 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.log
10:47:01,496 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.deployer.wrap
**10:47:01,505 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle application-bundle


On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[hidden email]> wrote:
Karaf dev team is currently calling a vote to karaf-2.3.4, You may want to try it out  and provide feedback

-D


On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[hidden email]> wrote:
Thanks Dan. Since 1.1.1 seems to have never been released I tried to upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any news on when 2.3.4 will be released?

Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this issue?

Chris


On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[hidden email]> wrote:


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com





Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

Chris Geer
As a follow-up, if I manually go into JMX and set the start level attribute on the osgi.core framework mbean to 1 it does shutdown the bundles in the correct order. Then I can stop the server and all is well. 

Chris


On Sat, Feb 15, 2014 at 10:56 AM, Chris Geer <[hidden email]> wrote:
Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still destroying lower start level items before higher start level items. I'm surprised no one else is noticing this. Here is what I see in my logs (leaving out hundreds of error messages). The lines with the * are start level 60 and the ** is start level 80. The rest of the start level 80 ones are killed even later...after karaf bundles are gone.

10:46:58,361 | INFO  | Thread-2         | Main                             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
10:46:58,378 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle locationvalidation-google
**10:46:58,389 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle operations-service
10:46:58,416 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle utils
*10:46:58,418 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-broker.xml
10:47:00,987 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands
10:47:01,038 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-karaf
10:47:01,056 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands
10:47:01,067 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean
10:47:01,069 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.web
10:47:01,071 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev
10:47:01,073 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages
10:47:01,075 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.log
10:47:01,077 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.config
10:47:01,081 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.services
10:47:01,083 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.system
10:47:01,085 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.command
10:47:01,090 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.ssh
10:47:01,104 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.management
10:47:01,106 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.common
10:47:01,120 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.packages
10:47:01,127 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.core
10:47:01,165 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.command
10:47:01,167 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.dev
10:47:01,188 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.management
10:47:01,190 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.features.command
10:47:01,250 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.jaas.command
10:47:01,274 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.commands
10:47:01,333 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.console
10:47:01,365 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.log is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,372 | INFO  | rint Extender: 1 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.web is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,441 | INFO  | rint Extender: 2 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.osgi is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,478 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.config is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.1.0]
10:47:01,484 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.web
10:47:01,487 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.config
10:47:01,490 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.osgi
10:47:01,492 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.log
10:47:01,496 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.deployer.wrap
**10:47:01,505 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle application-bundle


On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[hidden email]> wrote:
Karaf dev team is currently calling a vote to karaf-2.3.4, You may want to try it out  and provide feedback

-D


On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[hidden email]> wrote:
Thanks Dan. Since 1.1.1 seems to have never been released I tried to upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any news on when 2.3.4 will be released?

Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this issue?

Chris


On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[hidden email]> wrote:


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com






Reply | Threaded
Open this post in threaded view
|

Re: Shutdown Order on 2.3.3

Chris Geer
Just found this ticket, I think it's a good representation of what I'm seeing. 

KARAF-2075


On Sat, Feb 15, 2014 at 11:29 AM, Chris Geer <[hidden email]> wrote:
As a follow-up, if I manually go into JMX and set the start level attribute on the osgi.core framework mbean to 1 it does shutdown the bundles in the correct order. Then I can stop the server and all is well. 

Chris


On Sat, Feb 15, 2014 at 10:56 AM, Chris Geer <[hidden email]> wrote:
Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still destroying lower start level items before higher start level items. I'm surprised no one else is noticing this. Here is what I see in my logs (leaving out hundreds of error messages). The lines with the * are start level 60 and the ** is start level 80. The rest of the start level 80 ones are killed even later...after karaf bundles are gone.

10:46:58,361 | INFO  | Thread-2         | Main                             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
10:46:58,378 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle locationvalidation-google
**10:46:58,389 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle operations-service
10:46:58,416 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle utils
*10:46:58,418 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-broker.xml
10:47:00,987 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands
10:47:01,038 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-karaf
10:47:01,056 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands
10:47:01,067 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean
10:47:01,069 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.web
10:47:01,071 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev
10:47:01,073 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages
10:47:01,075 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.log
10:47:01,077 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.config
10:47:01,081 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.services
10:47:01,083 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.system
10:47:01,085 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.command
10:47:01,090 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.ssh
10:47:01,104 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.management
10:47:01,106 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.common
10:47:01,120 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.packages
10:47:01,127 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.core
10:47:01,165 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.command
10:47:01,167 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.dev
10:47:01,188 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.management
10:47:01,190 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.features.command
10:47:01,250 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.jaas.command
10:47:01,274 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.commands
10:47:01,333 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.console
10:47:01,365 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.log is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,372 | INFO  | rint Extender: 1 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.web is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,441 | INFO  | rint Extender: 2 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.osgi is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.0.0]
10:47:01,478 | INFO  | rint Extender: 3 | BlueprintContainerImpl           | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.config is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.1.0]
10:47:01,484 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.web
10:47:01,487 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.config
10:47:01,490 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.osgi
10:47:01,492 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.log
10:47:01,496 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.deployer.wrap
**10:47:01,505 | INFO  | FelixShutdown    | BlueprintExtender                | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle application-bundle


On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[hidden email]> wrote:
Karaf dev team is currently calling a vote to karaf-2.3.4, You may want to try it out  and provide feedback

-D


On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[hidden email]> wrote:
Thanks Dan. Since 1.1.1 seems to have never been released I tried to upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any news on when 2.3.4 will be released?

Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this issue?

Chris


On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[hidden email]> wrote:


On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[hidden email]> wrote:
JB,

It's the default config.properties files.

Chris


On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Chris,

did you change something in the etc config file or it's the default one ?

Regards
JB


On 02/14/2014 06:54 PM, Chris Geer wrote:
We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now
having some significant issues on shutdown. Previously, everything would
shut down in a normal order but now for some reasons things with lower
start levels are shutting down first. Our big problem is ActiveMQ. We
have our activemq broker starting with a start level of 60 and our
normal bundles (mostly camel) starting at 80. But on shutdown the first
thing that shows up in the logs is this.

23:46:38,243 | INFO  | Thread-2         | Main
   |  -  -  | Karaf shutdown socket: received shutdown command. Stopping
framework...
23:46:38,266 | INFO  | FelixShutdown    | BlueprintExtender
    | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying
BlueprintContainer for bundle activemq-broker.xml
23:46:38,290 | INFO  | FelixShutdown    | BrokerService
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache ActiveMQ
5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is shutting down
23:46:38,368 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector
openwire stopped
23:46:38,436 | INFO  | FelixShutdown    | TransportConnector
   | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector stomp
stopped


As you can imagine that causes problems because now that the broker
doesn't exist the bundles that use it start throwing errors and
in-flight transactions are failing.


23:46:38,440 | INFO  | : vm://default#0 | PooledConnectionFactory
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring
connection ActiveMQConnection
{id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID:CXTPRO-CG.local-49654-1392332684735-2:8,started=false}
on IOException:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://default#1) stopped.
23:46:38,444 | WARN  | n-notifications] |
faultJmsMessageListenerContainer | 163 - org.springframework.jms -
3.2.4.RELEASE | Setup of JMS message listener invoker failed for
destination 'permission-notifications' - trying to recover. Cause: peer
(vm://default#3) stopped.
23:46:38,444 | WARN  | t-notifications] | PooledSession
    | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught exception
trying rollback() when putting session back into the pool, will
invalidate. javax.jms.IllegalStateException: The Session is closed


Is there anything we can do to restore an orderly shutdown order?


Thanks,

Chris


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com