Jenkins builds are failing

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Jenkins builds are failing

jbonofre
Hi guys,

FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke the build
(on master). It seems that the itests fail to resolve the artifacts. For instance:

Error downloading mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT

org.apache.karaf.features.internal.util.MultiException: Error:
        Error downloading mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT
        at
org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
        at
org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)

I'm checking the other builds too.

My plan is to fix all building issues asap.

Regards
JB
--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Jenkins builds are failing

Grzegorz Grzybek
Hello,

By strange coincidence (2nd this day), I'm working locally on
https://issues.apache.org/jira/browse/KARAF-5164 (karaf-maven-plugin fails
to verify artifacts if only available within local reactor).

I have successfully injected "@Component(role = MavenWorkspaceReader.class,
hint = "reactor")" into both assembly and verify mojos and I see successful
mvn build without "install" phase.
Of course this is not related to the above problem. But I'll have a look
too.

regards
Grzegorz Grzybek

2017-07-12 13:53 GMT+02:00 Jean-Baptiste Onofré <[hidden email]>:

> Hi guys,
>
> FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke the
> build (on master). It seems that the itests fail to resolve the artifacts.
> For instance:
>
> Error downloading mvn:org.apache.karaf.jms/org.a
> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>
> org.apache.karaf.features.internal.util.MultiException: Error:
>         Error downloading mvn:org.apache.karaf.jms/org.a
> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>         at org.apache.karaf.features.internal.download.impl.MavenDownlo
> adManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
>         at org.apache.karaf.features.internal.download.impl.MavenDownlo
> adManager.createDownloader(MavenDownloadManager.java:72)
>
> I'm checking the other builds too.
>
> My plan is to fix all building issues asap.
>
> Regards
> JB
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Jenkins builds are failing

jbonofre
In reply to this post by jbonofre
Just a quick update: master is now ok. I'm fixing the other Jenkins jobs.

Regards
JB

On 07/12/2017 01:53 PM, Jean-Baptiste Onofré wrote:

> Hi guys,
>
> FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke the build
> (on master). It seems that the itests fail to resolve the artifacts. For instance:
>
> Error downloading mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT
>
> org.apache.karaf.features.internal.util.MultiException: Error:
>      Error downloading
> mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT
>      at
> org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
>
>      at
> org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)
>
>
> I'm checking the other builds too.
>
> My plan is to fix all building issues asap.
>
> Regards
> JB

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Jenkins builds are failing

Grzegorz Grzybek
Hello

I love detailed explanations, so here's one.

First, an observation. If you check
https://repository.apache.org/content/groups/snapshots-group/org/apache/karaf/webconsole/org.apache.karaf.webconsole.http/4.2.0-SNAPSHOT/maven-metadata.xml,
you'll see this metadata declares latest SNAPSHOT version to be
"4.2.0-20170713.142530-162". But if you try to actually fetch this version
(that's what Aether is doing), you'll get HTTP 404, because there's *no
such version* - there's "4.2.0-20170713.142529-162" - 1 *second*
difference, but very important - this SNAPSHOT is *not resolvable*.

I started wondering about Nexus problems, about maven-deploy-plugin bugs,
but then I run this under debugger. I didn't notice this problem when using
Maven 3.5.0, but it failed under 3.3.9 (that's the version used by Jenkins).

Another worrying sign was this log:

[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @
org.apache.karaf.features.core ---
...
[INFO] Downloading:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/maven-metadata.xml
[INFO] Downloaded:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/maven-metadata.xml
(315 B at 4.6 KB/sec)
[INFO] Downloading:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/maven-metadata.xml
[INFO] Downloaded:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/maven-metadata.xml
(315 B at 10.6 KB/sec)
[INFO] Uploading:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/4.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/4.2.0-SNAPSHOT/maven-metadata.xml
(805 B at 31.4 KB/sec)
[INFO] Uploading:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/4.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded:
http://localhost:8081/nexus/content/repositories/snapshots/org/apache/karaf/features/org.apache.karaf.features.core/4.2.0-SNAPSHOT/maven-metadata.xml
(805 B at 30.2 KB/sec)
...

It looked as duplicate metadata upload. I was fooled by reading
maven-deploy-plugin source code, because it actually may upload metadata
multiple times (in case of attached artifacts).

But then, under debugger I saw this:

generators = {java.util.ArrayList@9874}  size = 4
 0 =
{org.apache.maven.repository.internal.RemoteSnapshotMetadataGenerator@9881}
  snapshots: java.util.Map  = {java.util.LinkedHashMap@9890}  size = 0
  legacyFormat: boolean  = false
 1 =
{org.apache.maven.repository.internal.RemoteSnapshotMetadataGenerator@9882}
  snapshots: java.util.Map  = {java.util.LinkedHashMap@9889}  size = 0
  legacyFormat: boolean  = false
 2 = {org.apache.maven.repository.internal.VersionsMetadataGenerator@9883}
  versions: java.util.Map  = {java.util.LinkedHashMap@9887}  size = 0
  processedVersions: java.util.Map  = {java.util.LinkedHashMap@9888}  size
= 0
 3 = {org.apache.maven.repository.internal.VersionsMetadataGenerator@9884}
  versions: java.util.Map  = {java.util.LinkedHashMap@9885}  size = 0
  processedVersions: java.util.Map  = {java.util.LinkedHashMap@9886}  size
= 0

which means that
org.eclipse.aether.internal.impl.DefaultDeployer#getMetadataGenerators()
returned/used too many generators. And I found that *this is the root
problem* - we have two instances of RemoteSnapshotMetadataGenerator and
both may transform "SNAPSHOT" version to two different timestamped versions
- then artifact is uploaded *once* (with first timestamped version) and
then metadata is uploaded twice - first time with correct version and then
with 1-second later version...

Why? Why duplicate generators? The problem is _class realm_ of ...
org.apache.karaf.tooling:karaf-services-maven-plugin when running on Maven
3.3.9. maven-aether-provider-3.3.9.jar declares
(META-INF/plexus/components.xml):
    <component>
      <role>org.eclipse.aether.impl.MetadataGeneratorFactory</role>
      <role-hint>snapshot</role-hint>

<implementation>org.apache.maven.repository.internal.SnapshotMetadataGeneratorFactory</implementation>
      <description />
      <isolated-realm>false</isolated-realm>
    </component>

and maven-resolver-provider-3.5.0.jar (direct dependency of
karaf-services-maven-plugin) has (META-INF/plexus/components.xml):
    <component>
      <role>org.eclipse.aether.impl.MetadataGeneratorFactory</role>
      <role-hint>snapshot</role-hint>

<implementation>org.apache.maven.repository.internal.SnapshotMetadataGeneratorFactory</implementation>
      <description />
      <isolated-realm>false</isolated-realm>
    </component>

Even if these declaration from different deps lead to single Class of
org.apache.maven.repository.internal.SnapshotMetadataGeneratorFactory (from
maven-aether-provider-3.3.9.jar), we still have TWO instances of such class.

I won't provide fix yet, I'm just letting you know that this dependency
list is wrong:
https://github.com/apache/karaf/blob/master/tooling/karaf-services-maven-plugin/pom.xml#L41-L143
- we should not mix Eclipse and Sonatype versions of Aether and we should
not depend on maven-resolver-provider. We can't easily exclude deps for
Maven plugins (like we can for dependencies) though.

I'll try to look at this issue tomorrow.

best regards
Grzegorz Grzybek

2017-07-13 13:49 GMT+02:00 Jean-Baptiste Onofré <[hidden email]>:

> Just a quick update: master is now ok. I'm fixing the other Jenkins jobs.
>
> Regards
> JB
>
>
> On 07/12/2017 01:53 PM, Jean-Baptiste Onofré wrote:
>
>> Hi guys,
>>
>> FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke the
>> build (on master). It seems that the itests fail to resolve the artifacts.
>> For instance:
>>
>> Error downloading mvn:org.apache.karaf.jms/org.a
>> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>>
>> org.apache.karaf.features.internal.util.MultiException: Error:
>>      Error downloading mvn:org.apache.karaf.jms/org.a
>> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>>      at org.apache.karaf.features.internal.download.impl.MavenDownlo
>> adManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
>>      at org.apache.karaf.features.internal.download.impl.MavenDownlo
>> adManager.createDownloader(MavenDownloadManager.java:72)
>>
>> I'm checking the other builds too.
>>
>> My plan is to fix all building issues asap.
>>
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Jenkins builds are failing

Grzegorz Grzybek
Hello

Tracked further here: https://issues.apache.org/jira/browse/KARAF-5250

regards
Grzegorz Grzybek

2017-07-13 18:07 GMT+02:00 Grzegorz Grzybek <[hidden email]>:

> Hello
>
> I love detailed explanations, so here's one.
>
> First, an observation. If you check https://repository.apache.org/
> content/groups/snapshots-group/org/apache/karaf/
> webconsole/org.apache.karaf.webconsole.http/4.2.0-
> SNAPSHOT/maven-metadata.xml, you'll see this metadata declares latest
> SNAPSHOT version to be "4.2.0-20170713.142530-162". But if you try to
> actually fetch this version (that's what Aether is doing), you'll get HTTP
> 404, because there's *no such version* - there's
> "4.2.0-20170713.142529-162" - 1 *second* difference, but very important -
> this SNAPSHOT is *not resolvable*.
>
> I started wondering about Nexus problems, about maven-deploy-plugin bugs,
> but then I run this under debugger. I didn't notice this problem when using
> Maven 3.5.0, but it failed under 3.3.9 (that's the version used by Jenkins).
>
> Another worrying sign was this log:
>
> [INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @
> org.apache.karaf.features.core ---
> ...
> [INFO] Downloading: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/maven-metadata.xml
> [INFO] Downloaded: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/maven-metadata.xml (315 B at 4.6 KB/sec)
> [INFO] Downloading: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/maven-metadata.xml
> [INFO] Downloaded: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/maven-metadata.xml (315 B at 10.6 KB/sec)
> [INFO] Uploading: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/4.2.0-SNAPSHOT/maven-metadata.xml
> [INFO] Uploaded: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/4.2.0-SNAPSHOT/maven-metadata.xml (805 B at 31.4 KB/sec)
> [INFO] Uploading: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/4.2.0-SNAPSHOT/maven-metadata.xml
> [INFO] Uploaded: http://localhost:8081/nexus/content/repositories/
> snapshots/org/apache/karaf/features/org.apache.karaf.
> features.core/4.2.0-SNAPSHOT/maven-metadata.xml (805 B at 30.2 KB/sec)
> ...
>
> It looked as duplicate metadata upload. I was fooled by reading
> maven-deploy-plugin source code, because it actually may upload metadata
> multiple times (in case of attached artifacts).
>
> But then, under debugger I saw this:
>
> generators = {java.util.ArrayList@9874}  size = 4
>  0 = {org.apache.maven.repository.internal.RemoteSnapshotMetadataGenerato
> r@9881}
>   snapshots: java.util.Map  = {java.util.LinkedHashMap@9890}  size = 0
>   legacyFormat: boolean  = false
>  1 = {org.apache.maven.repository.internal.RemoteSnapshotMetadataGenerato
> r@9882}
>   snapshots: java.util.Map  = {java.util.LinkedHashMap@9889}  size = 0
>   legacyFormat: boolean  = false
>  2 = {org.apache.maven.repository.internal.VersionsMetadataGenerator@9883}
>
>   versions: java.util.Map  = {java.util.LinkedHashMap@9887}  size = 0
>   processedVersions: java.util.Map  = {java.util.LinkedHashMap@9888}
> size = 0
>  3 = {org.apache.maven.repository.internal.VersionsMetadataGenerator@9884}
>
>   versions: java.util.Map  = {java.util.LinkedHashMap@9885}  size = 0
>   processedVersions: java.util.Map  = {java.util.LinkedHashMap@9886}
> size = 0
>
> which means that org.eclipse.aether.internal.impl.DefaultDeployer#getMetadataGenerators()
> returned/used too many generators. And I found that *this is the root
> problem* - we have two instances of RemoteSnapshotMetadataGenerator and
> both may transform "SNAPSHOT" version to two different timestamped versions
> - then artifact is uploaded *once* (with first timestamped version) and
> then metadata is uploaded twice - first time with correct version and then
> with 1-second later version...
>
> Why? Why duplicate generators? The problem is _class realm_ of ...
> org.apache.karaf.tooling:karaf-services-maven-plugin when running on
> Maven 3.3.9. maven-aether-provider-3.3.9.jar declares
> (META-INF/plexus/components.xml):
>     <component>
>       <role>org.eclipse.aether.impl.MetadataGeneratorFactory</role>
>       <role-hint>snapshot</role-hint>
>       <implementation>org.apache.maven.repository.internal.
> SnapshotMetadataGeneratorFactory</implementation>
>       <description />
>       <isolated-realm>false</isolated-realm>
>     </component>
>
> and maven-resolver-provider-3.5.0.jar (direct dependency of
> karaf-services-maven-plugin) has (META-INF/plexus/components.xml):
>     <component>
>       <role>org.eclipse.aether.impl.MetadataGeneratorFactory</role>
>       <role-hint>snapshot</role-hint>
>       <implementation>org.apache.maven.repository.internal.
> SnapshotMetadataGeneratorFactory</implementation>
>       <description />
>       <isolated-realm>false</isolated-realm>
>     </component>
>
> Even if these declaration from different deps lead to single Class of
> org.apache.maven.repository.internal.SnapshotMetadataGeneratorFactory
> (from maven-aether-provider-3.3.9.jar), we still have TWO instances of
> such class.
>
> I won't provide fix yet, I'm just letting you know that this dependency
> list is wrong: https://github.com/apache/karaf/blob/master/tooling/
> karaf-services-maven-plugin/pom.xml#L41-L143 - we should not mix Eclipse
> and Sonatype versions of Aether and we should not depend on
> maven-resolver-provider. We can't easily exclude deps for Maven plugins
> (like we can for dependencies) though.
>
> I'll try to look at this issue tomorrow.
>
> best regards
> Grzegorz Grzybek
>
> 2017-07-13 13:49 GMT+02:00 Jean-Baptiste Onofré <[hidden email]>:
>
>> Just a quick update: master is now ok. I'm fixing the other Jenkins jobs.
>>
>> Regards
>> JB
>>
>>
>> On 07/12/2017 01:53 PM, Jean-Baptiste Onofré wrote:
>>
>>> Hi guys,
>>>
>>> FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke
>>> the build (on master). It seems that the itests fail to resolve the
>>> artifacts. For instance:
>>>
>>> Error downloading mvn:org.apache.karaf.jms/org.a
>>> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>>>
>>> org.apache.karaf.features.internal.util.MultiException: Error:
>>>      Error downloading mvn:org.apache.karaf.jms/org.a
>>> pache.karaf.jms.pool/4.2.0-SNAPSHOT
>>>      at org.apache.karaf.features.internal.download.impl.MavenDownlo
>>> adManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
>>>      at org.apache.karaf.features.internal.download.impl.MavenDownlo
>>> adManager.createDownloader(MavenDownloadManager.java:72)
>>>
>>> I'm checking the other builds too.
>>>
>>> My plan is to fix all building issues asap.
>>>
>>> Regards
>>> JB
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Jenkins builds are failing

jbonofre
In reply to this post by jbonofre
Hi guys,

After the Jenkins upgrade, INFRA restarted Jenkins last night to increase the
max number of open files.

After that, our builds are green:

https://builds.apache.org/view/K/view/Karaf/

Regards
JB

On 07/12/2017 01:53 PM, Jean-Baptiste Onofré wrote:

> Hi guys,
>
> FYI, a recent commit (I'm suspecting on the karaf-maven-plugin) broke the build
> (on master). It seems that the itests fail to resolve the artifacts. For instance:
>
> Error downloading mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT
>
> org.apache.karaf.features.internal.util.MultiException: Error:
>      Error downloading
> mvn:org.apache.karaf.jms/org.apache.karaf.jms.pool/4.2.0-SNAPSHOT
>      at
> org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:86)
>
>      at
> org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)
>
>
> I'm checking the other builds too.
>
> My plan is to fix all building issues asap.
>
> Regards
> JB

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Loading...