Karaf Failover deployments

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Karaf Failover deployments

Benjamin Debeerst-3
Hello Karaf Users,

I was having a look into redundant setups for a Karaf container. What I found is this documentation [1], which describes failover configurations with locks on a file or a database.

But doesn't a lock on a single resource introduce a new single point of failure? Or am I missing something here?

How can I setup a full-redundant setup without that single resource lock? Can I at all? Or will I have to roll this on my own completely?

The setup I imagine is a master-slave setup like the following:
- two instances run on (physically) separate machines
- one is dedicated (configured as) master, the other as slave
- similar to container level locking, the slave instance is on hot standby
- master and slave hold an active connection (heartbeat or ping pong)
- as soon as the slave doesn't see a heartbeat from the master, it assumes the master is dead and becomes (temporary) master
- when the (original) master shows up again, the slave goes back to standby
 
What do you think about such a setup? How would you realize it with Karaf and are there maybe already existing solutions for such a setup? Could I maybe realize this with some kind of child instances setup?

Thanks in advance for any comments,
Benjamin

[1] http://karaf.apache.org/manual/latest-2.3.x/users-guide/failover.html
Reply | Threaded
Open this post in threaded view
|

Re: Karaf Failover deployments

Achim Nierbeck
Hi, 

this exact scenario you've been describing is not available out-of-the-box. 
Though I think, regarding the jdbc-lock vs. an active ping-pong via network, there is not much of a difference, in the end you rely on the network interface for communicating either directly or via the data-base. I'd use a database cause this is something that could be "clustered" also and therefore be as good as a direct ping-pong. 
Another alternative would be to create this on top of cellar, where you'd use hazelcast as your shared medium containing the information of state. 
So in the end you have to "die a certain death" :)
In different environments databases are considered to be "fail-safe" in others not, so this solely depends on the environment you working in. 

Regarding the realization, if you really think the jdbc-lock with the default hot-stand-by mechanism isn't enough, I'd take the clustering of Karaf with Hazelcast (that would be Karaf-Cellar) into account. I would use the hazelcast map as a "shared" resource instead of a ping, this way all nodes know of each other and as soon as one is missing on adding it's timestamp the others know of the failed node. 

regards, Achim 



2014-05-04 21:35 GMT+02:00 Benjamin Debeerst <[hidden email]>:
Hello Karaf Users,

I was having a look into redundant setups for a Karaf container. What I found is this documentation [1], which describes failover configurations with locks on a file or a database.

But doesn't a lock on a single resource introduce a new single point of failure? Or am I missing something here?

How can I setup a full-redundant setup without that single resource lock? Can I at all? Or will I have to roll this on my own completely?

The setup I imagine is a master-slave setup like the following:
- two instances run on (physically) separate machines
- one is dedicated (configured as) master, the other as slave
- similar to container level locking, the slave instance is on hot standby
- master and slave hold an active connection (heartbeat or ping pong)
- as soon as the slave doesn't see a heartbeat from the master, it assumes the master is dead and becomes (temporary) master
- when the (original) master shows up again, the slave goes back to standby

What do you think about such a setup? How would you realize it with Karaf and are there maybe already existing solutions for such a setup? Could I maybe realize this with some kind of child instances setup?

Thanks in advance for any comments,
Benjamin

[1] http://karaf.apache.org/manual/latest-2.3.x/users-guide/failover.html



--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master

Reply | Threaded
Open this post in threaded view
|

Re: Karaf Failover deployments

Jeremy Jongsma
The easiest implementation (or the least custom code) would be to block on a distributed lock via Hazelcast or Zookeeper, both of which have high-availability designs that mitigate any single points of failure. When the primary dies/disappears, the lock is automatically released, the standby gets the lock, and it takes over as primary.

I'd also recommend not caring which is primary. Treat them as identical instances, and let the lock decide who is primary - whoever gets in first wins. That way you don't have to worry about maintaining multiple application configurations.


On Sun, May 4, 2014 at 3:32 PM, Achim Nierbeck <[hidden email]> wrote:
Hi, 

this exact scenario you've been describing is not available out-of-the-box. 
Though I think, regarding the jdbc-lock vs. an active ping-pong via network, there is not much of a difference, in the end you rely on the network interface for communicating either directly or via the data-base. I'd use a database cause this is something that could be "clustered" also and therefore be as good as a direct ping-pong. 
Another alternative would be to create this on top of cellar, where you'd use hazelcast as your shared medium containing the information of state. 
So in the end you have to "die a certain death" :)
In different environments databases are considered to be "fail-safe" in others not, so this solely depends on the environment you working in. 

Regarding the realization, if you really think the jdbc-lock with the default hot-stand-by mechanism isn't enough, I'd take the clustering of Karaf with Hazelcast (that would be Karaf-Cellar) into account. I would use the hazelcast map as a "shared" resource instead of a ping, this way all nodes know of each other and as soon as one is missing on adding it's timestamp the others know of the failed node. 

regards, Achim 



2014-05-04 21:35 GMT+02:00 Benjamin Debeerst <[hidden email]>:

Hello Karaf Users,

I was having a look into redundant setups for a Karaf container. What I found is this documentation [1], which describes failover configurations with locks on a file or a database.

But doesn't a lock on a single resource introduce a new single point of failure? Or am I missing something here?

How can I setup a full-redundant setup without that single resource lock? Can I at all? Or will I have to roll this on my own completely?

The setup I imagine is a master-slave setup like the following:
- two instances run on (physically) separate machines
- one is dedicated (configured as) master, the other as slave
- similar to container level locking, the slave instance is on hot standby
- master and slave hold an active connection (heartbeat or ping pong)
- as soon as the slave doesn't see a heartbeat from the master, it assumes the master is dead and becomes (temporary) master
- when the (original) master shows up again, the slave goes back to standby

What do you think about such a setup? How would you realize it with Karaf and are there maybe already existing solutions for such a setup? Could I maybe realize this with some kind of child instances setup?

Thanks in advance for any comments,
Benjamin

[1] http://karaf.apache.org/manual/latest-2.3.x/users-guide/failover.html



--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master


Reply | Threaded
Open this post in threaded view
|

Re: Karaf Failover deployments

jbonofre
Hi,

what's the advantage of Hazelcast or Zookeeper compared to a filesystem
on a NAS or SAN ?

To avoid single point of failure, you have to setup multiple Hazelcast
or Zookeeper instances. If you have only one instance, it's like having
a single filesystem.
Using a SAN or NAS (it could be simply a mdadm partition on Linux with
NFS for instance), you have the same thing as multiple
Hazelcast/Zookeeper but in a more straight forward/convenient way.

Regards
JB

On 05/05/2014 04:25 PM, Jeremy Jongsma wrote:

> The easiest implementation (or the least custom code) would be to block
> on a distributed lock via Hazelcast or Zookeeper, both of which have
> high-availability designs that mitigate any single points of failure.
> When the primary dies/disappears, the lock is automatically released,
> the standby gets the lock, and it takes over as primary.
>
> I'd also recommend not caring which is primary. Treat them as identical
> instances, and let the lock decide who is primary - whoever gets in
> first wins. That way you don't have to worry about maintaining multiple
> application configurations.
>
>
> On Sun, May 4, 2014 at 3:32 PM, Achim Nierbeck <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     this exact scenario you've been describing is not available
>     out-of-the-box.
>     Though I think, regarding the jdbc-lock vs. an active ping-pong via
>     network, there is not much of a difference, in the end you rely on
>     the network interface for communicating either directly or via the
>     data-base. I'd use a database cause this is something that could be
>     "clustered" also and therefore be as good as a direct ping-pong.
>     Another alternative would be to create this on top of cellar, where
>     you'd use hazelcast as your shared medium containing the information
>     of state.
>     So in the end you have to "die a certain death" :)
>     In different environments databases are considered to be "fail-safe"
>     in others not, so this solely depends on the environment you working
>     in.
>
>     Regarding the realization, if you really think the jdbc-lock with
>     the default hot-stand-by mechanism isn't enough, I'd take the
>     clustering of Karaf with Hazelcast (that would be Karaf-Cellar) into
>     account. I would use the hazelcast map as a "shared" resource
>     instead of a ping, this way all nodes know of each other and as soon
>     as one is missing on adding it's timestamp the others know of the
>     failed node.
>
>     regards, Achim
>
>
>
>     2014-05-04 21:35 GMT+02:00 Benjamin Debeerst <[hidden email]
>     <mailto:[hidden email]>>:
>
>         Hello Karaf Users,
>
>         I was having a look into redundant setups for a Karaf container.
>         What I found is this documentation [1], which describes failover
>         configurations with locks on a file or a database.
>
>         But doesn't a lock on a single resource introduce a new single
>         point of failure? Or am I missing something here?
>
>         How can I setup a full-redundant setup without that single
>         resource lock? Can I at all? Or will I have to roll this on my
>         own completely?
>
>         The setup I imagine is a master-slave setup like the following:
>         - two instances run on (physically) separate machines
>         - one is dedicated (configured as) master, the other as slave
>         - similar to container level locking, the slave instance is on
>         hot standby
>         - master and slave hold an active connection (heartbeat or ping
>         pong)
>         - as soon as the slave doesn't see a heartbeat from the master,
>         it assumes the master is dead and becomes (temporary) master
>         - when the (original) master shows up again, the slave goes back
>         to standby
>
>         What do you think about such a setup? How would you realize it
>         with Karaf and are there maybe already existing solutions for
>         such a setup? Could I maybe realize this with some kind of child
>         instances setup?
>
>         Thanks in advance for any comments,
>         Benjamin
>
>         [1]
>         http://karaf.apache.org/manual/latest-2.3.x/users-guide/failover.html
>
>
>
>
>     --
>
>     Apache Karaf <http://karaf.apache.org/> Committer & PMC
>     OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>     Committer & Project Lead
>     blog <http://notizblog.nierbeck.de/>
>
>     Software Architect / Project Manager / Scrum Master
>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Karaf Failover deployments

Achim Nierbeck
Exactly, 
that's why my still most preferred solution would be to use a Database. 
Cause to my knowledge database systems usually are regarded to be "fail-safe"
within the company setup. Besides that I fully agree with Jeremy that one shouldn't treat one node to be the primary, and go back to it when it is up and running again. I'd keep it as backup then. 

regards, Achim 


2014-05-05 17:09 GMT+02:00 Jean-Baptiste Onofré <[hidden email]>:
Hi,

what's the advantage of Hazelcast or Zookeeper compared to a filesystem on a NAS or SAN ?

To avoid single point of failure, you have to setup multiple Hazelcast or Zookeeper instances. If you have only one instance, it's like having a single filesystem.
Using a SAN or NAS (it could be simply a mdadm partition on Linux with NFS for instance), you have the same thing as multiple Hazelcast/Zookeeper but in a more straight forward/convenient way.

Regards
JB


On 05/05/2014 04:25 PM, Jeremy Jongsma wrote:
The easiest implementation (or the least custom code) would be to block
on a distributed lock via Hazelcast or Zookeeper, both of which have
high-availability designs that mitigate any single points of failure.
When the primary dies/disappears, the lock is automatically released,
the standby gets the lock, and it takes over as primary.

I'd also recommend not caring which is primary. Treat them as identical
instances, and let the lock decide who is primary - whoever gets in
first wins. That way you don't have to worry about maintaining multiple
application configurations.


On Sun, May 4, 2014 at 3:32 PM, Achim Nierbeck <[hidden email]
<mailto:[hidden email]>> wrote:

    Hi,

    this exact scenario you've been describing is not available
    out-of-the-box.
    Though I think, regarding the jdbc-lock vs. an active ping-pong via
    network, there is not much of a difference, in the end you rely on
    the network interface for communicating either directly or via the
    data-base. I'd use a database cause this is something that could be
    "clustered" also and therefore be as good as a direct ping-pong.
    Another alternative would be to create this on top of cellar, where
    you'd use hazelcast as your shared medium containing the information
    of state.
    So in the end you have to "die a certain death" :)
    In different environments databases are considered to be "fail-safe"
    in others not, so this solely depends on the environment you working
    in.

    Regarding the realization, if you really think the jdbc-lock with
    the default hot-stand-by mechanism isn't enough, I'd take the
    clustering of Karaf with Hazelcast (that would be Karaf-Cellar) into
    account. I would use the hazelcast map as a "shared" resource
    instead of a ping, this way all nodes know of each other and as soon
    as one is missing on adding it's timestamp the others know of the
    failed node.

    regards, Achim



    2014-05-04 21:35 GMT+02:00 Benjamin Debeerst <[hidden email]
    <mailto:[hidden email]>>:


        Hello Karaf Users,

        I was having a look into redundant setups for a Karaf container.
        What I found is this documentation [1], which describes failover
        configurations with locks on a file or a database.

        But doesn't a lock on a single resource introduce a new single
        point of failure? Or am I missing something here?

        How can I setup a full-redundant setup without that single
        resource lock? Can I at all? Or will I have to roll this on my
        own completely?

        The setup I imagine is a master-slave setup like the following:
        - two instances run on (physically) separate machines
        - one is dedicated (configured as) master, the other as slave
        - similar to container level locking, the slave instance is on
        hot standby
        - master and slave hold an active connection (heartbeat or ping
        pong)
        - as soon as the slave doesn't see a heartbeat from the master,
        it assumes the master is dead and becomes (temporary) master
        - when the (original) master shows up again, the slave goes back
        to standby

        What do you think about such a setup? How would you realize it
        with Karaf and are there maybe already existing solutions for
        such a setup? Could I maybe realize this with some kind of child
        instances setup?

        Thanks in advance for any comments,
        Benjamin

        [1]
        http://karaf.apache.org/manual/latest-2.3.x/users-guide/failover.html




    --

    Apache Karaf <http://karaf.apache.org/> Committer & PMC
    OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
    Committer & Project Lead
    blog <http://notizblog.nierbeck.de/>

    Software Architect / Project Manager / Scrum Master



--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com



--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master

Reply | Threaded
Open this post in threaded view
|

karaf using 100% cpu

lacigas
In reply to this post by jbonofre
Hi
I noticed that our karaf runtime is using 100% CPU.
We are running karaf version 2.2.6 with some camel routes. There is
hardly any load.

Looking at the log file with level DEBUG didn't reveal anything.


What can I do to find out what's causing the high cpu load?

Regards
Laci
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

jgoodyear
Connect to the JMX management console and see what's running up the CPU.

--Jamie

On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]> wrote:

> Hi
> I noticed that our karaf runtime is using 100% CPU.
> We are running karaf version 2.2.6 with some camel routes. There is hardly
> any load.
>
> Looking at the log file with level DEBUG didn't reveal anything.
>
>
> What can I do to find out what's causing the high cpu load?
>
> Regards
> Laci
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

Achim Nierbeck
Also, do you have some trace levels activated with your camel routes?
This might also produce a high load :)
But as Jamie already said, it's good to enable JMX Monitoring ... this way you also find out about the camel routes ;)

regards, Achim 


2014-05-06 14:07 GMT+02:00 Jamie G. <[hidden email]>:
Connect to the JMX management console and see what's running up the CPU.

--Jamie

On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]> wrote:
> Hi
> I noticed that our karaf runtime is using 100% CPU.
> We are running karaf version 2.2.6 with some camel routes. There is hardly
> any load.
>
> Looking at the log file with level DEBUG didn't reveal anything.
>
>
> What can I do to find out what's causing the high cpu load?
>
> Regards
> Laci



--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master

Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

lacigas
Ok, after a little struggle, I installed jmxterm because I have only a terminal access to the host.
Unfortunately I have no idea what to do now... no jmx experience.
When I list the mbeans, I get TONS of entries representing my ActiveMQ queues.  Is that correct?

Then, how can I find out which process uses most cpu?

Thanks for your help.
Laci


On 06.05.2014 14:44, Achim Nierbeck wrote:
Also, do you have some trace levels activated with your camel routes?
This might also produce a high load :)
But as Jamie already said, it's good to enable JMX Monitoring ... this way you also find out about the camel routes ;)

regards, Achim 


2014-05-06 14:07 GMT+02:00 Jamie G. <[hidden email]>:
Connect to the JMX management console and see what's running up the CPU.

--Jamie

On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]> wrote:
> Hi
> I noticed that our karaf runtime is using 100% CPU.
> We are running karaf version 2.2.6 with some camel routes. There is hardly
> any load.
>
> Looking at the log file with level DEBUG didn't reveal anything.
>
>
> What can I do to find out what's causing the high cpu load?
>
> Regards
> Laci



--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master


Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

jbonofre
Hi Laci,

What's the version of ActiveMQ that you use ? If you use ActiveMQ 5.7,
there's a bug in the ConnectionFactory (it doesn't release correctly the
session). Maybe it's your issue (but not fully sure).

Regards
JB

On 05/06/2014 04:37 PM, Laci Gaspar wrote:

> Ok, after a little struggle, I installed jmxterm because I have only a
> terminal access to the host.
> Unfortunately I have no idea what to do now... no jmx experience.
> When I list the mbeans, I get TONS of entries representing my ActiveMQ
> queues.  Is that correct?
>
> Then, how can I find out which process uses most cpu?
>
> Thanks for your help.
> Laci
>
>
> On 06.05.2014 14:44, Achim Nierbeck wrote:
>> Also, do you have some trace levels activated with your camel routes?
>> This might also produce a high load :)
>> But as Jamie already said, it's good to enable JMX Monitoring ... this
>> way you also find out about the camel routes ;)
>>
>> regards, Achim
>>
>>
>> 2014-05-06 14:07 GMT+02:00 Jamie G. <[hidden email]
>> <mailto:[hidden email]>>:
>>
>>     Connect to the JMX management console and see what's running up
>>     the CPU.
>>
>>     --Jamie
>>
>>     On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]
>>     <mailto:[hidden email]>> wrote:
>>     > Hi
>>     > I noticed that our karaf runtime is using 100% CPU.
>>     > We are running karaf version 2.2.6 with some camel routes. There
>>     is hardly
>>     > any load.
>>     >
>>     > Looking at the log file with level DEBUG didn't reveal anything.
>>     >
>>     >
>>     > What can I do to find out what's causing the high cpu load?
>>     >
>>     > Regards
>>     > Laci
>>
>>
>>
>>
>> --
>>
>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>> Committer & Project Lead
>> blog <http://notizblog.nierbeck.de/>
>>
>> Software Architect / Project Manager / Scrum Master
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

lacigas

Hi JB
We're using AMQ 5.9

Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]>:
Hi Laci,

What's the version of ActiveMQ that you use ? If you use ActiveMQ 5.7, there's a bug in the ConnectionFactory (it doesn't release correctly the session). Maybe it's your issue (but not fully sure).

Regards
JB

On 05/06/2014 04:37 PM, Laci Gaspar wrote:
Ok, after a little struggle, I installed jmxterm because I have only a
terminal access to the host.
Unfortunately I have no idea what to do now... no jmx experience.
When I list the mbeans, I get TONS of entries representing my ActiveMQ
queues.  Is that correct?

Then, how can I find out which process uses most cpu?

Thanks for your help.
Laci


On 06.05.2014 14:44, Achim Nierbeck wrote:
Also, do you have some trace levels activated with your camel routes?
This might also produce a high load :)
But as Jamie already said, it's good to enable JMX Monitoring ... this
way you also find out about the camel routes ;)

regards, Achim


2014-05-06 14:07 GMT+02:00 Jamie G. <[hidden email]
<mailto:[hidden email]>>:

    Connect to the JMX management console and see what's running up
    the CPU.

    --Jamie

    On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]
    <mailto:[hidden email]>> wrote:
    > Hi
    > I noticed that our karaf runtime is using 100% CPU.
    > We are running karaf version 2.2.6 with some camel routes. There
    is hardly
    > any load.
    >
    > Looking at the log file with level DEBUG didn't reveal anything.
    >
    >
    > What can I do to find out what's causing the high cpu load?
    >
    > Regards
    > Laci




--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master



--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

dantran
In reply to this post by jbonofre
this is going to be tough, but possible.  You can scan thru all the thread stacks and look for the one that spins ( ie not locking on a pause/sleep/pause method)

-D


On Tue, May 6, 2014 at 8:30 AM, Jean-Baptiste Onofré <[hidden email]> wrote:
Hi Laci,

What's the version of ActiveMQ that you use ? If you use ActiveMQ 5.7, there's a bug in the ConnectionFactory (it doesn't release correctly the session). Maybe it's your issue (but not fully sure).

Regards
JB


On 05/06/2014 04:37 PM, Laci Gaspar wrote:
Ok, after a little struggle, I installed jmxterm because I have only a
terminal access to the host.
Unfortunately I have no idea what to do now... no jmx experience.
When I list the mbeans, I get TONS of entries representing my ActiveMQ
queues.  Is that correct?

Then, how can I find out which process uses most cpu?

Thanks for your help.
Laci


On 06.05.2014 14:44, Achim Nierbeck wrote:
Also, do you have some trace levels activated with your camel routes?
This might also produce a high load :)
But as Jamie already said, it's good to enable JMX Monitoring ... this
way you also find out about the camel routes ;)

regards, Achim


2014-05-06 14:07 GMT+02:00 Jamie G. <[hidden email]
<mailto:[hidden email]>>:


    Connect to the JMX management console and see what's running up
    the CPU.

    --Jamie

    On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar <[hidden email]
    <mailto:[hidden email]>> wrote:
    > Hi
    > I noticed that our karaf runtime is using 100% CPU.
    > We are running karaf version 2.2.6 with some camel routes. There
    is hardly
    > any load.
    >
    > Looking at the log file with level DEBUG didn't reveal anything.
    >
    >
    > What can I do to find out what's causing the high cpu load?
    >
    > Regards
    > Laci




--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master



--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

jbonofre
In reply to this post by lacigas
OK ;)

I'm surprised that you have a lot of Object for queues. Especially do
you have a bunch of objects for the same queue name ?

Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?

Regards
JB

On 05/06/2014 06:09 PM, Laci Gaspar wrote:

> Hi JB
> We're using AMQ 5.9
>
> Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
> <mailto:[hidden email]>>:
>
>     Hi Laci,
>
>     What's the version of ActiveMQ that you use ? If you use ActiveMQ
>     5.7, there's a bug in the ConnectionFactory (it doesn't release
>     correctly the session). Maybe it's your issue (but not fully sure).
>
>     Regards
>     JB
>
>     On 05/06/2014 04:37 PM, Laci Gaspar wrote:
>
>         Ok, after a little struggle, I installed jmxterm because I have
>         only a
>         terminal access to the host.
>         Unfortunately I have no idea what to do now... no jmx experience.
>         When I list the mbeans, I get TONS of entries representing my
>         ActiveMQ
>         queues.  Is that correct?
>
>         Then, how can I find out which process uses most cpu?
>
>         Thanks for your help.
>         Laci
>
>
>         On 06.05.2014 14:44, Achim Nierbeck wrote:
>
>             Also, do you have some trace levels activated with your
>             camel routes?
>             This might also produce a high load :)
>             But as Jamie already said, it's good to enable JMX
>             Monitoring ... this
>             way you also find out about the camel routes ;)
>
>             regards, Achim
>
>
>             2014-05-06 14:07 GMT+02:00 Jamie G.
>             <[hidden email] <mailto:[hidden email]>
>             <mailto:jamie.goodyear@gmail.__com
>             <mailto:[hidden email]>>>:
>
>                  Connect to the JMX management console and see what's
>             running up
>                  the CPU.
>
>                  --Jamie
>
>                  On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
>             <[hidden email] <mailto:[hidden email]>
>                  <mailto:[hidden email] <mailto:[hidden email]>>>
>             wrote:
>                  > Hi
>                  > I noticed that our karaf runtime is using 100% CPU.
>                  > We are running karaf version 2.2.6 with some camel
>             routes. There
>                  is hardly
>                  > any load.
>                  >
>                  > Looking at the log file with level DEBUG didn't
>             reveal anything.
>                  >
>                  >
>                  > What can I do to find out what's causing the high cpu
>             load?
>                  >
>                  > Regards
>                  > Laci
>
>
>
>
>             --
>
>             Apache Karaf <http://karaf.apache.org/> Committer & PMC
>             OPS4J Pax Web
>             <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
>             <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
>             Committer & Project Lead
>             blog <http://notizblog.nierbeck.de/__>
>
>             Software Architect / Project Manager / Scrum Master
>
>
>
>     --
>     Jean-Baptiste Onofré
>     [hidden email] <mailto:[hidden email]>
>     http://blog.nanthrax.net
>     Talend - http://www.talend.com
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

lacigas

I'm off work now so I can tell you tomorrow if I have several objects per queue name.
But yes, AMQ is installed in karaf, because in the beginning we used a broker inside karaf. We don't do that anymore, but AMQ is still installed in karaf.
Regards,
Laci

Am 06.05.2014 19:58 schrieb "Jean-Baptiste Onofré" <[hidden email]>:
OK ;)

I'm surprised that you have a lot of Object for queues. Especially do you have a bunch of objects for the same queue name ?

Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?

Regards
JB

On 05/06/2014 06:09 PM, Laci Gaspar wrote:
Hi JB
We're using AMQ 5.9

Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
<mailto:[hidden email]>>:

    Hi Laci,

    What's the version of ActiveMQ that you use ? If you use ActiveMQ
    5.7, there's a bug in the ConnectionFactory (it doesn't release
    correctly the session). Maybe it's your issue (but not fully sure).

    Regards
    JB

    On 05/06/2014 04:37 PM, Laci Gaspar wrote:

        Ok, after a little struggle, I installed jmxterm because I have
        only a
        terminal access to the host.
        Unfortunately I have no idea what to do now... no jmx experience.
        When I list the mbeans, I get TONS of entries representing my
        ActiveMQ
        queues.  Is that correct?

        Then, how can I find out which process uses most cpu?

        Thanks for your help.
        Laci


        On 06.05.2014 14:44, Achim Nierbeck wrote:

            Also, do you have some trace levels activated with your
            camel routes?
            This might also produce a high load :)
            But as Jamie already said, it's good to enable JMX
            Monitoring ... this
            way you also find out about the camel routes ;)

            regards, Achim


            2014-05-06 14:07 GMT+02:00 Jamie G.
            <[hidden email] <mailto:[hidden email]>
            <mailto:[hidden email]__com
            <mailto:[hidden email]>>>:

                 Connect to the JMX management console and see what's
            running up
                 the CPU.

                 --Jamie

                 On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
            <[hidden email] <mailto:[hidden email]>
                 <mailto:[hidden email] <mailto:[hidden email]>>>
            wrote:
                 > Hi
                 > I noticed that our karaf runtime is using 100% CPU.
                 > We are running karaf version 2.2.6 with some camel
            routes. There
                 is hardly
                 > any load.
                 >
                 > Looking at the log file with level DEBUG didn't
            reveal anything.
                 >
                 >
                 > What can I do to find out what's causing the high cpu
            load?
                 >
                 > Regards
                 > Laci




            --

            Apache Karaf <http://karaf.apache.org/> Committer & PMC
            OPS4J Pax Web
            <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
            <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
            Committer & Project Lead
            blog <http://notizblog.nierbeck.de/__>

            Software Architect / Project Manager / Scrum Master



    --
    Jean-Baptiste Onofré
    [hidden email] <mailto:[hidden email]>
    http://blog.nanthrax.net
    Talend - http://www.talend.com


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

Freeman-2
Hi,

I remember that some old jetty version have bugs can cause 100% CPU usage when use SSL, are you use jetty with SSL(through Camel/CXF/Pax-WEB,whatnot)?

What's the jetty version in your container?
-------------
Freeman(Yue) Fang

Red Hat, Inc. 
FuseSource is now part of Red Hat



On 2014-5-7, at 上午2:06, Laci Gaspar wrote:

I'm off work now so I can tell you tomorrow if I have several objects per queue name.
But yes, AMQ is installed in karaf, because in the beginning we used a broker inside karaf. We don't do that anymore, but AMQ is still installed in karaf.
Regards,
Laci

Am 06.05.2014 19:58 schrieb "Jean-Baptiste Onofré" <[hidden email]>:
OK ;)

I'm surprised that you have a lot of Object for queues. Especially do you have a bunch of objects for the same queue name ?

Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?

Regards
JB

On 05/06/2014 06:09 PM, Laci Gaspar wrote:
Hi JB
We're using AMQ 5.9

Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
<mailto:[hidden email]>>:

    Hi Laci,

    What's the version of ActiveMQ that you use ? If you use ActiveMQ
    5.7, there's a bug in the ConnectionFactory (it doesn't release
    correctly the session). Maybe it's your issue (but not fully sure).

    Regards
    JB

    On 05/06/2014 04:37 PM, Laci Gaspar wrote:

        Ok, after a little struggle, I installed jmxterm because I have
        only a
        terminal access to the host.
        Unfortunately I have no idea what to do now... no jmx experience.
        When I list the mbeans, I get TONS of entries representing my
        ActiveMQ
        queues.  Is that correct?

        Then, how can I find out which process uses most cpu?

        Thanks for your help.
        Laci


        On 06.05.2014 14:44, Achim Nierbeck wrote:

            Also, do you have some trace levels activated with your
            camel routes?
            This might also produce a high load :)
            But as Jamie already said, it's good to enable JMX
            Monitoring ... this
            way you also find out about the camel routes ;)

            regards, Achim


            2014-05-06 14:07 GMT+02:00 Jamie G.
            <[hidden email] <mailto:[hidden email]>
            <mailto:[hidden email]__com
            <mailto:[hidden email]>>>:

                 Connect to the JMX management console and see what's
            running up
                 the CPU.

                 --Jamie

                 On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
            <[hidden email] <mailto:[hidden email]>
                 <mailto:[hidden email] <mailto:[hidden email]>>>
            wrote:
                 > Hi
                 > I noticed that our karaf runtime is using 100% CPU.
                 > We are running karaf version 2.2.6 with some camel
            routes. There
                 is hardly
                 > any load.
                 >
                 > Looking at the log file with level DEBUG didn't
            reveal anything.
                 >
                 >
                 > What can I do to find out what's causing the high cpu
            load?
                 >
                 > Regards
                 > Laci




            --

            Apache Karaf <http://karaf.apache.org/> Committer & PMC
            OPS4J Pax Web
            <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
            <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
            Committer & Project Lead
            blog <http://notizblog.nierbeck.de/__>

            Software Architect / Project Manager / Scrum Master



    --
    Jean-Baptiste Onofré
    [hidden email] <mailto:[hidden email]>
    http://blog.nanthrax.net
    Talend - http://www.talend.com


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

lacigas
In reply to this post by jbonofre
I had to restart karaf because it rendered the system unusable. Of
course the CPU usage is down to 5% now.
Do you think it's a good idea to uninstall the activemq features I don't
need?
I probably only need activemq-karaf. Is that correct?

Regards,
Laci

On 06.05.2014 19:03, Jean-Baptiste Onofré wrote:

> OK ;)
>
> I'm surprised that you have a lot of Object for queues. Especially do
> you have a bunch of objects for the same queue name ?
>
> Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?
>
> Regards
> JB
>
> On 05/06/2014 06:09 PM, Laci Gaspar wrote:
>> Hi JB
>> We're using AMQ 5.9
>>
>> Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
>> <mailto:[hidden email]>>:
>>
>>     Hi Laci,
>>
>>     What's the version of ActiveMQ that you use ? If you use ActiveMQ
>>     5.7, there's a bug in the ConnectionFactory (it doesn't release
>>     correctly the session). Maybe it's your issue (but not fully sure).
>>
>>     Regards
>>     JB
>>
>>     On 05/06/2014 04:37 PM, Laci Gaspar wrote:
>>
>>         Ok, after a little struggle, I installed jmxterm because I have
>>         only a
>>         terminal access to the host.
>>         Unfortunately I have no idea what to do now... no jmx
>> experience.
>>         When I list the mbeans, I get TONS of entries representing my
>>         ActiveMQ
>>         queues.  Is that correct?
>>
>>         Then, how can I find out which process uses most cpu?
>>
>>         Thanks for your help.
>>         Laci
>>
>>
>>         On 06.05.2014 14:44, Achim Nierbeck wrote:
>>
>>             Also, do you have some trace levels activated with your
>>             camel routes?
>>             This might also produce a high load :)
>>             But as Jamie already said, it's good to enable JMX
>>             Monitoring ... this
>>             way you also find out about the camel routes ;)
>>
>>             regards, Achim
>>
>>
>>             2014-05-06 14:07 GMT+02:00 Jamie G.
>>             <[hidden email] <mailto:[hidden email]>
>>             <mailto:jamie.goodyear@gmail.__com
>>             <mailto:[hidden email]>>>:
>>
>>                  Connect to the JMX management console and see what's
>>             running up
>>                  the CPU.
>>
>>                  --Jamie
>>
>>                  On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
>>             <[hidden email] <mailto:[hidden email]>
>>                  <mailto:[hidden email] <mailto:[hidden email]>>>
>>             wrote:
>>                  > Hi
>>                  > I noticed that our karaf runtime is using 100% CPU.
>>                  > We are running karaf version 2.2.6 with some camel
>>             routes. There
>>                  is hardly
>>                  > any load.
>>                  >
>>                  > Looking at the log file with level DEBUG didn't
>>             reveal anything.
>>                  >
>>                  >
>>                  > What can I do to find out what's causing the high cpu
>>             load?
>>                  >
>>                  > Regards
>>                  > Laci
>>
>>
>>
>>
>>             --
>>
>>             Apache Karaf <http://karaf.apache.org/> Committer & PMC
>>             OPS4J Pax Web
>>             <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
>> <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
>>             Committer & Project Lead
>>             blog <http://notizblog.nierbeck.de/__>
>>
>>             Software Architect / Project Manager / Scrum Master
>>
>>
>>
>>     --
>>     Jean-Baptiste Onofré
>>     [hidden email] <mailto:[hidden email]>
>>     http://blog.nanthrax.net
>>     Talend - http://www.talend.com
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

Achim Nierbeck
Next time before "rebooting" do a stack-dump that should help you. 

regards, Achim 


2014-05-07 8:18 GMT+02:00 Laci Gaspar <[hidden email]>:
I had to restart karaf because it rendered the system unusable. Of course the CPU usage is down to 5% now.
Do you think it's a good idea to uninstall the activemq features I don't need?
I probably only need activemq-karaf. Is that correct?

Regards,
Laci


On <a href="tel:06.05.2014%2019" value="+49605201419" target="_blank">06.05.2014 19:03, Jean-Baptiste Onofré wrote:
OK ;)

I'm surprised that you have a lot of Object for queues. Especially do you have a bunch of objects for the same queue name ?

Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?

Regards
JB

On 05/06/2014 06:09 PM, Laci Gaspar wrote:
Hi JB
We're using AMQ 5.9

Am <a href="tel:06.05.2014%2018" value="+49605201418" target="_blank">06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
<mailto:[hidden email]>>:

    Hi Laci,

    What's the version of ActiveMQ that you use ? If you use ActiveMQ
    5.7, there's a bug in the ConnectionFactory (it doesn't release
    correctly the session). Maybe it's your issue (but not fully sure).

    Regards
    JB

    On 05/06/2014 04:37 PM, Laci Gaspar wrote:

        Ok, after a little struggle, I installed jmxterm because I have
        only a
        terminal access to the host.
        Unfortunately I have no idea what to do now... no jmx experience.
        When I list the mbeans, I get TONS of entries representing my
        ActiveMQ
        queues.  Is that correct?

        Then, how can I find out which process uses most cpu?

        Thanks for your help.
        Laci


        On <a href="tel:06.05.2014%2014" value="+49605201414" target="_blank">06.05.2014 14:44, Achim Nierbeck wrote:

            Also, do you have some trace levels activated with your
            camel routes?
            This might also produce a high load :)
            But as Jamie already said, it's good to enable JMX
            Monitoring ... this
            way you also find out about the camel routes ;)

            regards, Achim


            2014-05-06 14:07 GMT+02:00 Jamie G.
            <[hidden email] <mailto:[hidden email]>
            <mailto:[hidden email]__com
            <mailto:[hidden email]>>>:

                 Connect to the JMX management console and see what's
            running up
                 the CPU.

                 --Jamie

                 On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
            <[hidden email] <mailto:[hidden email]>
                 <mailto:[hidden email] <mailto:[hidden email]>>>
            wrote:
                 > Hi
                 > I noticed that our karaf runtime is using 100% CPU.
                 > We are running karaf version 2.2.6 with some camel
            routes. There
                 is hardly
                 > any load.
                 >
                 > Looking at the log file with level DEBUG didn't
            reveal anything.
                 >
                 >
                 > What can I do to find out what's causing the high cpu
            load?
                 >
                 > Regards
                 > Laci




            --

            Apache Karaf <http://karaf.apache.org/> Committer & PMC
            OPS4J Pax Web
            <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
<http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
            Committer & Project Lead
            blog <http://notizblog.nierbeck.de/__>

            Software Architect / Project Manager / Scrum Master



    --
    Jean-Baptiste Onofré
    [hidden email] <mailto:[hidden email]>
    http://blog.nanthrax.net
    Talend - http://www.talend.com






--

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead
blog <http://notizblog.nierbeck.de/>

Software Architect / Project Manager / Scrum Master

Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

jbonofre
In reply to this post by lacigas
Stupid question: did you use log:tail and what's the Karaf version ?

Regards
JB

On 05/07/2014 08:18 AM, Laci Gaspar wrote:

> I had to restart karaf because it rendered the system unusable. Of
> course the CPU usage is down to 5% now.
> Do you think it's a good idea to uninstall the activemq features I don't
> need?
> I probably only need activemq-karaf. Is that correct?
>
> Regards,
> Laci
>
> On 06.05.2014 19:03, Jean-Baptiste Onofré wrote:
>> OK ;)
>>
>> I'm surprised that you have a lot of Object for queues. Especially do
>> you have a bunch of objects for the same queue name ?
>>
>> Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?
>>
>> Regards
>> JB
>>
>> On 05/06/2014 06:09 PM, Laci Gaspar wrote:
>>> Hi JB
>>> We're using AMQ 5.9
>>>
>>> Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
>>> <mailto:[hidden email]>>:
>>>
>>>     Hi Laci,
>>>
>>>     What's the version of ActiveMQ that you use ? If you use ActiveMQ
>>>     5.7, there's a bug in the ConnectionFactory (it doesn't release
>>>     correctly the session). Maybe it's your issue (but not fully sure).
>>>
>>>     Regards
>>>     JB
>>>
>>>     On 05/06/2014 04:37 PM, Laci Gaspar wrote:
>>>
>>>         Ok, after a little struggle, I installed jmxterm because I have
>>>         only a
>>>         terminal access to the host.
>>>         Unfortunately I have no idea what to do now... no jmx
>>> experience.
>>>         When I list the mbeans, I get TONS of entries representing my
>>>         ActiveMQ
>>>         queues.  Is that correct?
>>>
>>>         Then, how can I find out which process uses most cpu?
>>>
>>>         Thanks for your help.
>>>         Laci
>>>
>>>
>>>         On 06.05.2014 14:44, Achim Nierbeck wrote:
>>>
>>>             Also, do you have some trace levels activated with your
>>>             camel routes?
>>>             This might also produce a high load :)
>>>             But as Jamie already said, it's good to enable JMX
>>>             Monitoring ... this
>>>             way you also find out about the camel routes ;)
>>>
>>>             regards, Achim
>>>
>>>
>>>             2014-05-06 14:07 GMT+02:00 Jamie G.
>>>             <[hidden email] <mailto:[hidden email]>
>>>             <mailto:jamie.goodyear@gmail.__com
>>>             <mailto:[hidden email]>>>:
>>>
>>>                  Connect to the JMX management console and see what's
>>>             running up
>>>                  the CPU.
>>>
>>>                  --Jamie
>>>
>>>                  On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
>>>             <[hidden email] <mailto:[hidden email]>
>>>                  <mailto:[hidden email] <mailto:[hidden email]>>>
>>>             wrote:
>>>                  > Hi
>>>                  > I noticed that our karaf runtime is using 100% CPU.
>>>                  > We are running karaf version 2.2.6 with some camel
>>>             routes. There
>>>                  is hardly
>>>                  > any load.
>>>                  >
>>>                  > Looking at the log file with level DEBUG didn't
>>>             reveal anything.
>>>                  >
>>>                  >
>>>                  > What can I do to find out what's causing the high cpu
>>>             load?
>>>                  >
>>>                  > Regards
>>>                  > Laci
>>>
>>>
>>>
>>>
>>>             --
>>>
>>>             Apache Karaf <http://karaf.apache.org/> Committer & PMC
>>>             OPS4J Pax Web
>>>             <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
>>> <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
>>>             Committer & Project Lead
>>>             blog <http://notizblog.nierbeck.de/__>
>>>
>>>             Software Architect / Project Manager / Scrum Master
>>>
>>>
>>>
>>>     --
>>>     Jean-Baptiste Onofré
>>>     [hidden email] <mailto:[hidden email]>
>>>     http://blog.nanthrax.net
>>>     Talend - http://www.talend.com
>>>
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

jbonofre
In reply to this post by Freeman-2
Good point Freeman. The workaround is to override the connector in
etc/jetty.xml to use the NIO one.

Regards
JB

On 05/07/2014 07:49 AM, Freeman Fang wrote:

> Hi,
>
> I remember that some old jetty version have bugs can cause 100% CPU
> usage when use SSL, are you use jetty with SSL(through
> Camel/CXF/Pax-WEB,whatnot)?
>
> What's the jetty version in your container?
> -------------
> Freeman(Yue) Fang
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
>
>
>
> On 2014-5-7, at 上午2:06, Laci Gaspar wrote:
>
>> I'm off work now so I can tell you tomorrow if I have several objects
>> per queue name.
>> But yes, AMQ is installed in karaf, because in the beginning we used a
>> broker inside karaf. We don't do that anymore, but AMQ is still
>> installed in karaf.
>> Regards,
>> Laci
>>
>> Am 06.05.2014 19:58 schrieb "Jean-Baptiste Onofré" <[hidden email]
>> <mailto:[hidden email]>>:
>>
>>     OK ;)
>>
>>     I'm surprised that you have a lot of Object for queues. Especially
>>     do you have a bunch of objects for the same queue name ?
>>
>>     Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?
>>
>>     Regards
>>     JB
>>
>>     On 05/06/2014 06:09 PM, Laci Gaspar wrote:
>>
>>         Hi JB
>>         We're using AMQ 5.9
>>
>>         Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré"
>>         <[hidden email] <mailto:[hidden email]>
>>         <mailto:[hidden email] <mailto:[hidden email]>>>:
>>
>>             Hi Laci,
>>
>>             What's the version of ActiveMQ that you use ? If you use
>>         ActiveMQ
>>             5.7, there's a bug in the ConnectionFactory (it doesn't
>>         release
>>             correctly the session). Maybe it's your issue (but not
>>         fully sure).
>>
>>             Regards
>>             JB
>>
>>             On 05/06/2014 04:37 PM, Laci Gaspar wrote:
>>
>>                 Ok, after a little struggle, I installed jmxterm
>>         because I have
>>                 only a
>>                 terminal access to the host.
>>                 Unfortunately I have no idea what to do now... no jmx
>>         experience.
>>                 When I list the mbeans, I get TONS of entries
>>         representing my
>>                 ActiveMQ
>>                 queues.  Is that correct?
>>
>>                 Then, how can I find out which process uses most cpu?
>>
>>                 Thanks for your help.
>>                 Laci
>>
>>
>>                 On 06.05.2014 14:44, Achim Nierbeck wrote:
>>
>>                     Also, do you have some trace levels activated with
>>         your
>>                     camel routes?
>>                     This might also produce a high load :)
>>                     But as Jamie already said, it's good to enable JMX
>>                     Monitoring ... this
>>                     way you also find out about the camel routes ;)
>>
>>                     regards, Achim
>>
>>
>>                     2014-05-06 14:07 GMT+02:00 Jamie G.
>>                     <[hidden email]
>>         <mailto:[hidden email]>
>>         <mailto:jamie.goodyear@gmail.__com
>>         <mailto:[hidden email]>>
>>                     <mailto:jamie.goodyear@gmail.
>>         <mailto:jamie.goodyear@gmail.>____com
>>                     <mailto:jamie.goodyear@gmail.__com
>>         <mailto:[hidden email]>>>>:
>>
>>                          Connect to the JMX management console and see
>>         what's
>>                     running up
>>                          the CPU.
>>
>>                          --Jamie
>>
>>                          On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
>>                     <[hidden email] <mailto:[hidden email]>
>>         <mailto:[hidden email] <mailto:[hidden email]>>
>>                          <mailto:[hidden email]
>>         <mailto:[hidden email]> <mailto:[hidden email]
>>         <mailto:[hidden email]>>>>
>>                     wrote:
>>                          > Hi
>>                          > I noticed that our karaf runtime is using
>>         100% CPU.
>>                          > We are running karaf version 2.2.6 with
>>         some camel
>>                     routes. There
>>                          is hardly
>>                          > any load.
>>                          >
>>                          > Looking at the log file with level DEBUG didn't
>>                     reveal anything.
>>                          >
>>                          >
>>                          > What can I do to find out what's causing
>>         the high cpu
>>                     load?
>>                          >
>>                          > Regards
>>                          > Laci
>>
>>
>>
>>
>>                     --
>>
>>                     Apache Karaf <http://karaf.apache.org/> Committer
>>         & PMC
>>                     OPS4J Pax Web
>>                     <http://wiki.ops4j.org/____display/paxweb/Pax+Web/
>>         <http://wiki.ops4j.org/__display/paxweb/Pax+Web/>
>>                     <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
>>         <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>>
>>                     Committer & Project Lead
>>                     blog <http://notizblog.nierbeck.de/____
>>         <http://notizblog.nierbeck.de/__>>
>>
>>                     Software Architect / Project Manager / Scrum Master
>>
>>
>>
>>             --
>>             Jean-Baptiste Onofré
>>         [hidden email] <mailto:[hidden email]>
>>         <mailto:[hidden email] <mailto:[hidden email]>>
>>         http://blog.nanthrax.net <http://blog.nanthrax.net/>
>>             Talend - http://www.talend.com <http://www.talend.com/>
>>
>>
>>     --
>>     Jean-Baptiste Onofré
>>     [hidden email] <mailto:[hidden email]>
>>     http://blog.nanthrax.net <http://blog.nanthrax.net/>
>>     Talend - http://www.talend.com <http://www.talend.com/>
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: karaf using 100% cpu

lacigas
In reply to this post by Freeman-2
Hi,
we have camel-jetty 2.9.2 in karaf.

Regards
Laci

On 07.05.2014 07:49, Freeman Fang wrote:
Hi,

I remember that some old jetty version have bugs can cause 100% CPU usage when use SSL, are you use jetty with SSL(through Camel/CXF/Pax-WEB,whatnot)?

What's the jetty version in your container?
-------------
Freeman(Yue) Fang

Red Hat, Inc. 
FuseSource is now part of Red Hat



On 2014-5-7, at 上午2:06, Laci Gaspar wrote:

I'm off work now so I can tell you tomorrow if I have several objects per queue name.
But yes, AMQ is installed in karaf, because in the beginning we used a broker inside karaf. We don't do that anymore, but AMQ is still installed in karaf.
Regards,
Laci

Am 06.05.2014 19:58 schrieb "Jean-Baptiste Onofré" <[hidden email]>:
OK ;)

I'm surprised that you have a lot of Object for queues. Especially do you have a bunch of objects for the same queue name ?

Do you install ActiveMQ in Karaf (using the ActiveMQ features) ?

Regards
JB

On 05/06/2014 06:09 PM, Laci Gaspar wrote:
Hi JB
We're using AMQ 5.9

Am 06.05.2014 18:02 schrieb "Jean-Baptiste Onofré" <[hidden email]
<mailto:[hidden email]>>:

    Hi Laci,

    What's the version of ActiveMQ that you use ? If you use ActiveMQ
    5.7, there's a bug in the ConnectionFactory (it doesn't release
    correctly the session). Maybe it's your issue (but not fully sure).

    Regards
    JB

    On 05/06/2014 04:37 PM, Laci Gaspar wrote:

        Ok, after a little struggle, I installed jmxterm because I have
        only a
        terminal access to the host.
        Unfortunately I have no idea what to do now... no jmx experience.
        When I list the mbeans, I get TONS of entries representing my
        ActiveMQ
        queues.  Is that correct?

        Then, how can I find out which process uses most cpu?

        Thanks for your help.
        Laci


        On 06.05.2014 14:44, Achim Nierbeck wrote:

            Also, do you have some trace levels activated with your
            camel routes?
            This might also produce a high load :)
            But as Jamie already said, it's good to enable JMX
            Monitoring ... this
            way you also find out about the camel routes ;)

            regards, Achim


            2014-05-06 14:07 GMT+02:00 Jamie G.
            <[hidden email] <mailto:[hidden email]>
            <mailto:[hidden email]__com
            <mailto:[hidden email]>>>:

                 Connect to the JMX management console and see what's
            running up
                 the CPU.

                 --Jamie

                 On Tue, May 6, 2014 at 9:31 AM, Laci Gaspar
            <[hidden email] <mailto:[hidden email]>
                 <mailto:[hidden email] <mailto:[hidden email]>>>
            wrote:
                 > Hi
                 > I noticed that our karaf runtime is using 100% CPU.
                 > We are running karaf version 2.2.6 with some camel
            routes. There
                 is hardly
                 > any load.
                 >
                 > Looking at the log file with level DEBUG didn't
            reveal anything.
                 >
                 >
                 > What can I do to find out what's causing the high cpu
            load?
                 >
                 > Regards
                 > Laci




            --

            Apache Karaf <http://karaf.apache.org/> Committer & PMC
            OPS4J Pax Web
            <http://wiki.ops4j.org/__display/paxweb/Pax+Web/
            <http://wiki.ops4j.org/display/paxweb/Pax+Web/>>
            Committer & Project Lead
            blog <http://notizblog.nierbeck.de/__>

            Software Architect / Project Manager / Scrum Master



    --
    Jean-Baptiste Onofré
    [hidden email] <mailto:[hidden email]>
    http://blog.nanthrax.net
    Talend - http://www.talend.com


--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com


12