Error when connecting Client

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Error when connecting Client

Skynet5
With Karaf 4.1.3 and Karaf 4.1.4 when I connect as a client.sh (with the log
level set to debug) and perform a log:tail I get the following...

2018-01-05 13:38:00,348 | DEBUG | 0]-nio2-thread-1 | Nio2Session                    
| 48 - org.apache.sshd.core - 1.6.0 |
handleCompletedWriteCycle(Nio2Session[local=/127.0.0.1:8101,
remote=/127.0.0.1:50379]) finished writing len=68

screams out on to the console, and unless I control C the whole karaf
container hangs and in the server window I get the following message:

2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
PaxOsgi

This is with a clean unzip running oracle 1.8 on both windows and ubuntu.  

Any ideas?





--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

jbonofre
Hmmm, weird. I just tested without problem on 4.1.4 (Ubuntu with Oracle JDK).

Anything special for the terminal ?

If you try to execute a command in the client session, does it work ?
Same issue with openssh client ?

Regards
JB

On 01/05/2018 02:40 PM, Skynet5 wrote:

> With Karaf 4.1.3 and Karaf 4.1.4 when I connect as a client.sh (with the log
> level set to debug) and perform a log:tail I get the following...
>
> 2018-01-05 13:38:00,348 | DEBUG | 0]-nio2-thread-1 | Nio2Session
> | 48 - org.apache.sshd.core - 1.6.0 |
> handleCompletedWriteCycle(Nio2Session[local=/127.0.0.1:8101,
> remote=/127.0.0.1:50379]) finished writing len=68
>
> screams out on to the console, and unless I control C the whole karaf
> container hangs and in the server window I get the following message:
>
> 2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
> PaxOsgi
> 2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
> PaxOsgi
> 2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
> PaxOsgi
> 2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
> PaxOsgi
>
> This is with a clean unzip running oracle 1.8 on both windows and ubuntu.
>
> Any ideas?
>
>
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

Skynet5
So let me be a bit more specific...

C:\>java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

Fresh unzip of karaf

start the server using server.bat/sh

run set:log debug

then connect to the karaf instance with client.bat/sh and then:

log:tail

wait for perhaps 10 seconds and you see it stream those error messages.

> 2018-01-05 13:38:00,348 | DEBUG | 0]-nio2-thread-1 | Nio2Session
> | 48 - org.apache.sshd.core - 1.6.0 |
> handleCompletedWriteCycle(Nio2Session[local=/127.0.0.1:8101,
> remote=/127.0.0.1:50379]) finished writing len=68

leave it until it hangs..perhaps another 10 seconds? then on the server
window you see:

2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender


If it doesn't do it, connect via another client process and log:tail
there...


All connections/clients are initiated from the server box itself (laptops in
case of windows/Ubuntu VM on datacenter Hyperisor) is done all on the same
box.  I've even had a colleague do it on their machine and it does the same.

So that is two windows 10 machines and an Ubuntu server.








--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

Skynet5
In reply to this post by jbonofre
To reply to your questions.

Nothing special, this si a fresh new Ubuntu install and for Windows a basic
console terminal.

Yes, if you connect via the client I can run all commands, it is only once
Ive run
log:set DEBUG

then run log:tail from the client terminal does it through a hissy.

Admittedly its unlikely we would change the ROOT logger to debug...but it
seems to pull the container down pretty quick.




--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

jbonofre
In reply to this post by Skynet5
Ahhh, interesting: it seems related to the log:set right ?

Regards
JB

On 01/05/2018 04:53 PM, Skynet5 wrote:

> So let me be a bit more specific...
>
> C:\>java -version
> java version "1.8.0_152"
> Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
> Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
>
> Fresh unzip of karaf
>
> start the server using server.bat/sh
>
> run set:log debug
>
> then connect to the karaf instance with client.bat/sh and then:
>
> log:tail
>
> wait for perhaps 10 seconds and you see it stream those error messages.
>
>> 2018-01-05 13:38:00,348 | DEBUG | 0]-nio2-thread-1 | Nio2Session
>> | 48 - org.apache.sshd.core - 1.6.0 |
>> handleCompletedWriteCycle(Nio2Session[local=/127.0.0.1:8101,
>> remote=/127.0.0.1:50379]) finished writing len=68
>
> leave it until it hangs..perhaps another 10 seconds? then on the server
> window you see:
>
> 2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender
>
>
> If it doesn't do it, connect via another client process and log:tail
> there...
>
>
> All connections/clients are initiated from the server box itself (laptops in
> case of windows/Ubuntu VM on datacenter Hyperisor) is done all on the same
> box.  I've even had a colleague do it on their machine and it does the same.
>
> So that is two windows 10 machines and an Ubuntu server.
>
>
>
>
>
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

Skynet5
Yeah!

So have you managed to replicate?



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

jbonofre
Yup, and I think I found the problem: it's probably the input pump thread,
saturated in DEBUG.

I think we create a bunch of threads (bug) saturating the connection.

Let me take a look.

Thanks for the report.

Regards
JB

On 01/05/2018 05:46 PM, Skynet5 wrote:
> Yeah!
>
> So have you managed to replicate?
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

jbonofre
In reply to this post by Skynet5
I created the following Jira for the tracking:

https://issues.apache.org/jira/browse/KARAF-5559

Regards
JB

On 01/05/2018 05:46 PM, Skynet5 wrote:
> Yeah!
>
> So have you managed to replicate?
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

Skynet5
In reply to this post by Skynet5
Ok. Cool.

So we're likely to push this to prod soon, we don't currently use karaf.  So
to give us some confidence...from what you have seen so far, is this
restricted only to log:tail when we have the ROOT logger set to debug?

Perhaps we avoid log:tail from an ops perspective and just use Linux tail?



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

jbonofre
Hi,

lot:tail directly in the console not via ssh looks to work fine, I have a check
the number of threads created.

As workaround you can use tail on the log file instead of log:tail on the log
LRU list.

Regards
JB

On 01/05/2018 09:38 PM, Skynet5 wrote:

> Ok. Cool.
>
> So we're likely to push this to prod soon, we don't currently use karaf.  So
> to give us some confidence...from what you have seen so far, is this
> restricted only to log:tail when we have the ROOT logger set to debug?
>
> Perhaps we avoid log:tail from an ops perspective and just use Linux tail?
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

rkmoquin
Maybe this is why when I ssh and use log:tail sometimes it freezes and every great once in a while Karaf will hang.  It's not all that prevalent for me but think I've been sometimes encountering something similar in the newest milestone versions.

Around confidence-wise for pushing Karaf into production, I've pushed it successfully into production at a previous company it's worked great.  I love developing solutions with it.  The Karaf team is always very responsive and quick to fix critical issues that come up.

Now back to fighting with CNFE errors with Solr on Karaf (what a mess the classloading and class heirarchy for Solr is...sheesh)

On Sat, Jan 6, 2018 at 2:35 AM Jean-Baptiste Onofré <[hidden email]> wrote:
Hi,

lot:tail directly in the console not via ssh looks to work fine, I have a check
the number of threads created.

As workaround you can use tail on the log file instead of log:tail on the log
LRU list.

Regards
JB

On 01/05/2018 09:38 PM, Skynet5 wrote:
> Ok. Cool.
>
> So we're likely to push this to prod soon, we don't currently use karaf.  So
> to give us some confidence...from what you have seen so far, is this
> restricted only to log:tail when we have the ROOT logger set to debug?
>
> Perhaps we avoid log:tail from an ops perspective and just use Linux tail?
>
>
>
> --
> Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: Error when connecting Client

Skynet5
Thanks.

I've used Karaf plenty before within Servicemix, years a go, and Fuse in the
last few years.

This is my first time using Karaf raw, so to speak, so just wanted to sanity
check.

Server is about to enter 2 weeks of severely heavy soak testing...so we will
see :-)



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html