[DISCUSS] Karaf distribution testing

[jgoodyear]

Hi All,

After encountering some release issues with the Apache Karaf 2.1.x
branch it has come to the forefront that we need to improve our
distribution testing. The main target of distribution testing we need
is to try out the various scripts we include in each distribution kit.
These tests would ideally cover un-archiving the kit, launch
bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.

To help build these tests we can make use of bin/client to help make
ssh connections (send commands), and admin/core process library to
control the various processed involved in each test. To uncompress the
different distributions we could use commons-compress or just ask
maven to do the work for us (this is done in the Servicemix NMR &
Features projects).

Regardless of which tools we use to build these tests a certain amount
of CI will be required. Most of us currently use unix based systems,
so Windows based issues for example tend to sneak by. Hopefully by
having such tests automated we'll be able to catch distribution issues
earlier.

Cheers,
Jamie

[Charles Moulliard]

Hi Jamie,

I completely agree with you. We should create some sub directories under
the itests directory of Karaf and take the time to create tests.

Like for the camel project, we could create a karafTemplate test that
will be used by all the sub tests.

We could create the following structure

itests
     windows
          commands
          features
          logging
          ....
     unix
          commands
          features
          logging
          ....

Regards,

Charles

On 22/11/10 21:39, Jamie G. wrote:

> Hi All,
>
> After encountering some release issues with the Apache Karaf 2.1.x
> branch it has come to the forefront that we need to improve our
> distribution testing. The main target of distribution testing we need
> is to try out the various scripts we include in each distribution kit.
> These tests would ideally cover un-archiving the kit, launch
> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>
> To help build these tests we can make use of bin/client to help make
> ssh connections (send commands), and admin/core process library to
> control the various processed involved in each test. To uncompress the
> different distributions we could use commons-compress or just ask
> maven to do the work for us (this is done in the Servicemix NMR&
> Features projects).
>
> Regardless of which tools we use to build these tests a certain amount
> of CI will be required. Most of us currently use unix based systems,
> so Windows based issues for example tend to sneak by. Hopefully by
> having such tests automated we'll be able to catch distribution issues
> earlier.
>
> Cheers,
> Jamie

[Guillaume Nodet]

Definitely something around that.  In details, 'm not sure we should
separate the windows / unix in separate folders, but that's a detail
(cygwin is in both categories, so that's a tricky one...).
As discussed with Jamie, I think we have some tools already that will
help us write such test, so we should start writing one asap and see
how it goes.  We should be able to refactor to adapt more use cases
over time.

On Tue, Nov 23, 2010 at 08:10, Charles Moulliard <[hidden email]> wrote:

> Hi Jamie,
>
> I completely agree with you. We should create some sub directories under the
> itests directory of Karaf and take the time to create tests.
>
> Like for the camel project, we could create a karafTemplate test that will
> be used by all the sub tests.
>
> We could create the following structure
>
> itests
>    windows
>         commands
>         features
>         logging
>         ....
>    unix
>         commands
>         features
>         logging
>         ....
>
> Regards,
>
> Charles
>
> On 22/11/10 21:39, Jamie G. wrote:
>>
>> Hi All,
>>
>> After encountering some release issues with the Apache Karaf 2.1.x
>> branch it has come to the forefront that we need to improve our
>> distribution testing. The main target of distribution testing we need
>> is to try out the various scripts we include in each distribution kit.
>> These tests would ideally cover un-archiving the kit, launch
>> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>>
>> To help build these tests we can make use of bin/client to help make
>> ssh connections (send commands), and admin/core process library to
>> control the various processed involved in each test. To uncompress the
>> different distributions we could use commons-compress or just ask
>> maven to do the work for us (this is done in the Servicemix NMR&
>> Features projects).
>>
>> Regardless of which tools we use to build these tests a certain amount
>> of CI will be required. Most of us currently use unix based systems,
>> so Windows based issues for example tend to sneak by. Hopefully by
>> having such tests automated we'll be able to catch distribution issues
>> earlier.
>>
>> Cheers,
>> Jamie
>

--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

[Łukasz Dywicki]

Hi,
We have many machines for hudson so we can run platform dependant tests
automatically on Apache infrastructure (Windows, Linux, Solaris, OSX).

Best regards,
Lukasz

-----Original Message-----
From: Guillaume Nodet [mailto:[hidden email]]
Sent: Tuesday, November 23, 2010 8:18 AM
To: [hidden email]
Subject: Re: [DISCUSS] Karaf distribution testing

Definitely something around that.  In details, 'm not sure we should
separate the windows / unix in separate folders, but that's a detail
(cygwin is in both categories, so that's a tricky one...).
As discussed with Jamie, I think we have some tools already that will
help us write such test, so we should start writing one asap and see
how it goes.  We should be able to refactor to adapt more use cases
over time.

On Tue, Nov 23, 2010 at 08:10, Charles Moulliard <[hidden email]>
wrote:
> Hi Jamie,
>
> I completely agree with you. We should create some sub directories under
the

> itests directory of Karaf and take the time to create tests.
>
> Like for the camel project, we could create a karafTemplate test that will
> be used by all the sub tests.
>
> We could create the following structure
>
> itests
>    windows
>         commands
>         features
>         logging
>         ....
>    unix
>         commands
>         features
>         logging
>         ....
>
> Regards,
>
> Charles
>
> On 22/11/10 21:39, Jamie G. wrote:
>>
>> Hi All,
>>
>> After encountering some release issues with the Apache Karaf 2.1.x
>> branch it has come to the forefront that we need to improve our
>> distribution testing. The main target of distribution testing we need
>> is to try out the various scripts we include in each distribution kit.
>> These tests would ideally cover un-archiving the kit, launch
>> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>>
>> To help build these tests we can make use of bin/client to help make
>> ssh connections (send commands), and admin/core process library to
>> control the various processed involved in each test. To uncompress the
>> different distributions we could use commons-compress or just ask
>> maven to do the work for us (this is done in the Servicemix NMR&
>> Features projects).
>>
>> Regardless of which tools we use to build these tests a certain amount
>> of CI will be required. Most of us currently use unix based systems,
>> so Windows based issues for example tend to sneak by. Hopefully by
>> having such tests automated we'll be able to catch distribution issues
>> earlier.
>>
>> Cheers,
>> Jamie
>

--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

[Guillaume Nodet]

I've just committed a first test to see what's possible.
It's really easy and only unzip the distribution, start karaf, wait
for it to be started, run the stop script, and wait for evrything to
shutdown.
We can build on that and add more tests for various scripts and
expected behavior.
We could even add some for demos if that makes sense ...
Feel free to hack on that ....

See http://svn.apache.org/viewvc?view=revision&revision=1039339
The actual test is available at
http://svn.apache.org/viewvc/karaf/trunk/itests/kittests/src/test/java/org/apache/karaf/kittests/KitTest.java?view=markup

On Mon, Nov 22, 2010 at 21:39, Jamie G. <[hidden email]> wrote:

> Hi All,
>
> After encountering some release issues with the Apache Karaf 2.1.x
> branch it has come to the forefront that we need to improve our
> distribution testing. The main target of distribution testing we need
> is to try out the various scripts we include in each distribution kit.
> These tests would ideally cover un-archiving the kit, launch
> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>
> To help build these tests we can make use of bin/client to help make
> ssh connections (send commands), and admin/core process library to
> control the various processed involved in each test. To uncompress the
> different distributions we could use commons-compress or just ask
> maven to do the work for us (this is done in the Servicemix NMR &
> Features projects).
>
> Regardless of which tools we use to build these tests a certain amount
> of CI will be required. Most of us currently use unix based systems,
> so Windows based issues for example tend to sneak by. Hopefully by
> having such tests automated we'll be able to catch distribution issues
> earlier.
>
> Cheers,
> Jamie
>

--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

[jbonofre]

Thanks for that Guillaume.

We start with this and extend it.

Regards
JB

On 11/26/2010 02:08 PM, Guillaume Nodet wrote:

> I've just committed a first test to see what's possible.
> It's really easy and only unzip the distribution, start karaf, wait
> for it to be started, run the stop script, and wait for evrything to
> shutdown.
> We can build on that and add more tests for various scripts and
> expected behavior.
> We could even add some for demos if that makes sense ...
> Feel free to hack on that ....
>
> See http://svn.apache.org/viewvc?view=revision&revision=1039339
> The actual test is available at
> http://svn.apache.org/viewvc/karaf/trunk/itests/kittests/src/test/java/org/apache/karaf/kittests/KitTest.java?view=markup
>
> On Mon, Nov 22, 2010 at 21:39, Jamie G.<[hidden email]>  wrote:
>> Hi All,
>>
>> After encountering some release issues with the Apache Karaf 2.1.x
>> branch it has come to the forefront that we need to improve our
>> distribution testing. The main target of distribution testing we need
>> is to try out the various scripts we include in each distribution kit.
>> These tests would ideally cover un-archiving the kit, launch
>> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>>
>> To help build these tests we can make use of bin/client to help make
>> ssh connections (send commands), and admin/core process library to
>> control the various processed involved in each test. To uncompress the
>> different distributions we could use commons-compress or just ask
>> maven to do the work for us (this is done in the Servicemix NMR&
>> Features projects).
>>
>> Regardless of which tools we use to build these tests a certain amount
>> of CI will be required. Most of us currently use unix based systems,
>> so Windows based issues for example tend to sneak by. Hopefully by
>> having such tests automated we'll be able to catch distribution issues
>> earlier.
>>
>> Cheers,
>> Jamie
>>
>
>
>

[Freeman-2]

Hi Guillaume,

About the kit and examples test, I think we can use XHarness[1] which  
is an extension to Ant that helps simplify the process of automating  
command line based task.
We've done it with FUSE ESB CPI test and it can verify the kit and  
examples works as expect before release.
The XHarness is really convenient for launching script and checking  
output both from stdio and log, it should be very suitable to do the  
task like distributing smoky test.

[1]http://svn.codehaus.org/xharness/trunk/docs/manual/index.html

Best Regards
Freeman
On 2010-11-26, at 下午9:08, Guillaume Nodet wrote:

> I've just committed a first test to see what's possible.
> It's really easy and only unzip the distribution, start karaf, wait
> for it to be started, run the stop script, and wait for evrything to
> shutdown.
> We can build on that and add more tests for various scripts and
> expected behavior.
> We could even add some for demos if that makes sense ...
> Feel free to hack on that ....
>
> See http://svn.apache.org/viewvc?view=revision&revision=1039339
> The actual test is available at
> http://svn.apache.org/viewvc/karaf/trunk/itests/kittests/src/test/java/org/apache/karaf/kittests/KitTest.java?view=markup
>
> On Mon, Nov 22, 2010 at 21:39, Jamie G. <[hidden email]>  
> wrote:
>> Hi All,
>>
>> After encountering some release issues with the Apache Karaf 2.1.x
>> branch it has come to the forefront that we need to improve our
>> distribution testing. The main target of distribution testing we need
>> is to try out the various scripts we include in each distribution  
>> kit.
>> These tests would ideally cover un-archiving the kit, launch
>> bin/karaf, bin/start, bin/stop, bin/client, and bin/admin.
>>
>> To help build these tests we can make use of bin/client to help make
>> ssh connections (send commands), and admin/core process library to
>> control the various processed involved in each test. To uncompress  
>> the
>> different distributions we could use commons-compress or just ask
>> maven to do the work for us (this is done in the Servicemix NMR &
>> Features projects).
>>
>> Regardless of which tools we use to build these tests a certain  
>> amount
>> of CI will be required. Most of us currently use unix based systems,
>> so Windows based issues for example tend to sneak by. Hopefully by
>> having such tests automated we'll be able to catch distribution  
>> issues
>> earlier.
>>
>> Cheers,
>> Jamie
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com

--
Freeman Fang

------------------------

FuseSource: http://fusesource.com
blog: http://freemanfang.blogspot.com
twitter: http://twitter.com/freemanfang
Apache Servicemix:http://servicemix.apache.org
Apache Cxf: http://cxf.apache.org
Apache Karaf: http://karaf.apache.org
Apache Felix: http://felix.apache.org