ServiceUnavailableException after bundle restart

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ServiceUnavailableException after bundle restart

Harrison Tarr
Hi all,

I'm running into a weird problem with my blueprint bundle in Karaf. I have bundle A that exports a service (it uses a factory-method vs a constructor, as it's a third party interface 
(    <bean id="dataSession" class="com.datastax.driver.core.Session" factory-method="dataSession" factory-ref="cassandraSession"/> )
When I restart bundle B that uses the service, some of the calls that use the session throw a ServiceUnavailableException (but not all?)
In any case, shouldn't bundle B remake all the necessary service references?

Any ideas what could be causing this? The service is definitely still available. I'm happy to provide more info if it's necessary, I'm hoping maybe this is an easy fix though!

Regards,

Harrison
Reply | Threaded
Open this post in threaded view
|

Re: ServiceUnavailableException after bundle restart

Harrison Tarr
Oh one more piece of info I just thought to add that I think is relevant: the serviceunavailableexception is caused by "the blueprint container is being or has been destroyed". 

On Wed, Mar 25, 2020 at 11:31 AM Harrison Tarr <[hidden email]> wrote:
Hi all,

I'm running into a weird problem with my blueprint bundle in Karaf. I have bundle A that exports a service (it uses a factory-method vs a constructor, as it's a third party interface 
(    <bean id="dataSession" class="com.datastax.driver.core.Session" factory-method="dataSession" factory-ref="cassandraSession"/> )
When I restart bundle B that uses the service, some of the calls that use the session throw a ServiceUnavailableException (but not all?)
In any case, shouldn't bundle B remake all the necessary service references?

Any ideas what could be causing this? The service is definitely still available. I'm happy to provide more info if it's necessary, I'm hoping maybe this is an easy fix though!

Regards,

Harrison
Reply | Threaded
Open this post in threaded view
|

RE: ServiceUnavailableException after bundle restart

Siano, Stephan

Hi,

 

I have seen a similar issue before. I actually don’t think that the bean you have referenced directly creates that, but I bet you have a service reference (e.g. for cassandraSession) somewhere in your blueprint descriptor. A service reference does not resolve into the original service object, but into a blueprint proxy object that manages the actual service reference. If you stop the blueprint bundle, the blueprint context will be destroyed (and the dependency graph is usually considered in the order of destruction), which also includes the proxy object for the service reference. If any object leaks out of your blueprint context (e.g. by passing it to some other service or by storing it in a thread context), you will not notice this for regular beans (because they are actually pojos and live as long as they are referenced), but service proxies become broken as soon as the blueprint container is destroyed and you will get this exception on any invocation of the proxy.

 

In my case we were referencing a DataSource as a service and passing it to a Spring transaction manager. That stored the transaction synchronization in a thread local hash map using the DataSource (which is then a blueprint proxy) as the key. If the blueprint bundle is stopped why the transacted thread is still running the transaction manager cannot remove the transaction entry from the has map anymore because the DataStore proxy object will get you "the blueprint container is being or has been destroyed" even when calling the hashCode() and equals() method which is required to remove something from the hash map.

 

In short: When you see that error you should probably look which blueprint service proxies are involved and how they leak out of the context of your blueprint container. The stack trace of the exception might help you with that (who is trying to call the method that is throwing this exception).

 

Best regards

Stephan

 

From: Harrison Tarr <[hidden email]>
Sent: Mittwoch, 25. März 2020 19:44
To: [hidden email]
Subject: Re: ServiceUnavailableException after bundle restart

 

Oh one more piece of info I just thought to add that I think is relevant: the serviceunavailableexception is caused by "the blueprint container is being or has been destroyed". 

 

On Wed, Mar 25, 2020 at 11:31 AM Harrison Tarr <[hidden email]> wrote:

Hi all,

 

I'm running into a weird problem with my blueprint bundle in Karaf. I have bundle A that exports a service (it uses a factory-method vs a constructor, as it's a third party interface 
(    <bean id="dataSession" class="com.datastax.driver.core.Session" factory-method="dataSession" factory-ref="cassandraSession"/> )

When I restart bundle B that uses the service, some of the calls that use the session throw a ServiceUnavailableException (but not all?)
In any case, shouldn't bundle B remake all the necessary service references?

Any ideas what could be causing this? The service is definitely still available. I'm happy to provide more info if it's necessary, I'm hoping maybe this is an easy fix though!

 

Regards,

 

Harrison

Reply | Threaded
Open this post in threaded view
|

Re: ServiceUnavailableException after bundle restart

Tim Ward-2
Hi,

It is indeed likely that you are running into one of the “features” of the blueprint specification. The proxying behaviour is required by the OSGi blueprint specification, and it means that any service object that you use is wrapped in a delegating proxy - this means that the blueprint container only injects you once, and then “transparently” changes the backing service instance if the framework changes (this is known as “damping” see https://osgi.org/specification/osgi.cmpn/7.0.0/service.blueprint.html#i2630744). Furthermore if no service is available at the time the proxy is called then the blueprint implementation must block in the proxy until a timeout is hit. By default this timeout is 5 minutes…

It seemed like a good idea at the time, but in general service damping causes a lot of headaches in a dynamic environment. As people have described, you can end up in the situation where damped objects are called in various states, and it can lock up your system and/or cause weird errors elsewhere. Because there is no way to know when a damped service will block it’s also really hard to prevent these scenarios.

This is one of the main reasons that I decided to stop using blueprint (the improvements in Declarative Services tooling helped too).

All the best,

Tim

On 26 Mar 2020, at 08:06, Siano, Stephan <[hidden email]> wrote:

Hi,
 
I have seen a similar issue before. I actually don’t think that the bean you have referenced directly creates that, but I bet you have a service reference (e.g. for cassandraSession) somewhere in your blueprint descriptor. A service reference does not resolve into the original service object, but into a blueprint proxy object that manages the actual service reference. If you stop the blueprint bundle, the blueprint context will be destroyed (and the dependency graph is usually considered in the order of destruction), which also includes the proxy object for the service reference. If any object leaks out of your blueprint context (e.g. by passing it to some other service or by storing it in a thread context), you will not notice this for regular beans (because they are actually pojos and live as long as they are referenced), but service proxies become broken as soon as the blueprint container is destroyed and you will get this exception on any invocation of the proxy.
 
In my case we were referencing a DataSource as a service and passing it to a Spring transaction manager. That stored the transaction synchronization in a thread local hash map using the DataSource (which is then a blueprint proxy) as the key. If the blueprint bundle is stopped why the transacted thread is still running the transaction manager cannot remove the transaction entry from the has map anymore because the DataStore proxy object will get you "the blueprint container is being or has been destroyed" even when calling the hashCode() and equals() method which is required to remove something from the hash map.
 
In short: When you see that error you should probably look which blueprint service proxies are involved and how they leak out of the context of your blueprint container. The stack trace of the exception might help you with that (who is trying to call the method that is throwing this exception).
 
Best regards
Stephan
 
From: Harrison Tarr <[hidden email]> 
Sent: Mittwoch, 25. März 2020 19:44
To: [hidden email]
Subject: Re: ServiceUnavailableException after bundle restart
 
Oh one more piece of info I just thought to add that I think is relevant: the serviceunavailableexception is caused by "the blueprint container is being or has been destroyed". 
 
On Wed, Mar 25, 2020 at 11:31 AM Harrison Tarr <[hidden email]> wrote:
Hi all,
 
I'm running into a weird problem with my blueprint bundle in Karaf. I have bundle A that exports a service (it uses a factory-method vs a constructor, as it's a third party interface 
(    <bean id="dataSession" class="com.datastax.driver.core.Session" factory-method="dataSession" factory-ref="cassandraSession"/> )

When I restart bundle B that uses the service, some of the calls that use the session throw a ServiceUnavailableException (but not all?)
In any case, shouldn't bundle B remake all the necessary service references?

Any ideas what could be causing this? The service is definitely still available. I'm happy to provide more info if it's necessary, I'm hoping maybe this is an easy fix though!
 
Regards,
 
Harrison