Rough Guide to CPAN Testing

Chris BinGOs Williams

CPAN Tester

Who the hell is this guy?

Who the hell is this guy?

• By day, a Network Services Specialist
• By night, perl hacker, POEvangelist and CPAN Tester
• CPAN ID and IRC nickname, 'BinGOs'
• email: chris@bingosnet.co.uk
• Using perl since 1995
• Using POE since 2003
• CPAN Tester since 2005

MANIFEST

MANIFEST

• What is CPAN Testing?
• CPAN Testing Community
• How CPAN Testing Works
• The tools of CPAN Testing
• The problems and dangers of CPAN Testing
• Scaled up CPAN Testing
• On the horizon

What is CPAN testing?

What is CPAN testing?

• Tests CPAN distributions:
  • fetch distribution
  • extract
  • perl Makefile.PL or perl Build.PL
  • resolve dependencies
  • make or ./Build
  • make test or ./Build test
• Many distributed testers
• Multiple platforms
• Multiple perls
• Structured feedback mechanism

CPAN Testing Community

Web Sites

• Reports - http://www.cpantesters.org/
• Statistics - http://stats.cpantesters.org/
• Wiki - http://wiki.cpantesters.org/
• Dependencies - http://deps.cpantesters.org/
• Matrix - http://matrix.cpantesters.org/
• Preferences - https://prefs.cpantesters.org/

Mailing Lists

• cpan-testers@perl.org - !
• cpan-uploads@perl.org
• cpan-testers-discuss@perl.org

IRC on irc.perl.org

• #cpantesters - "live" report feeds
• #cpantesters-discuss

Some Statistics

Some Statistics

• Nov: 13 different Operating Systems: Linux, FreeBSD, Solaris
• Nov: 77 different platforms: x86_64-linux-thread-multi
• Nov: 16 different Perl versions: 5.10.1, 5.8.9, 5.10.0, 5.11.2
• Nov: 164 different CPAN testers (171 in October)
• Distribution with most reports: DateTime-TimeZone
• There are more PASSes than FAILs

How CPAN Testing Works

How does CPAN testing work?

Author uploads a distribution to PAUSE (pause.perl.org)

How CPAN Testing Works

How does CPAN testing work?

How does CPAN testing work?

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)
• Tests results sent to CPAN Testers mailing list
• http://lists.cpan.org/showlist.cgi?name=cpan-testers
• nntp://nntp.perl.org/perl.cpan.testers
• Results tabulated on CPAN Testers Reports
• Centralised notification of reports (prefs)

Types of Reports

• PASS - distribution built and tested correctly
• FAIL - distribution failed to test correctly
• UNKNOWN - distribution failed to build, had no test suite or outcome was inconclusive
• NA - distribution is not applicable to this platform and/or version of Perl

Some tips for CPAN Authors

Author tips

• exit 0 from Makefile.PL/Build.PL
• Threads: use Config; $Config{'useithreads'}
• Devel::CheckLib / Devel::AssertOS
• $^X or Probe::Perl
• Yes, we set $ENV{AUTOMATED_TESTING}
• http://wiki.cpantesters.org/wiki/CPANAuthorNotes

The tools of CPAN Testing

Manual Testing

• CPAN + CPAN::Reporter
• CPANPLUS

Automated Testing

• CPAN::Reporter::Smoker
• CPANPLUS::YACSmoke
• minismokebox

CPAN::Reporter

CPAN::Reporter Installation

Super quick-start version...

 $ cpan

 cpan> install CPAN::Reporter

 cpan> reload CPAN

 cpan> o conf init test_report
    

CPAN::Reporter Benefits

• Adds CPAN test reporting to CPAN.pm
• Allows anyone who installs a distribution to contribute
• Allows testers to edit reports before sending
• Provides selective report submission based on configuration
• Keeps track of subject lines sent to CPAN Testers
• Subject line has grade, module name and perl version
• Avoids duplicate reports if re-testing

CPANPLUS

CPANPLUS Installation

 $ cpanp

 cpanp> s conf email bingos@cpan.org

 cpanp> s conf cpantest 1

 cpanp> s conf cpantest_mx smtp.somedomain.com

 cpanp> s conf verbose 1

 cpanp> s save

 cpanp> s selfupdate enabled_features

    

CPANPLUS Benefits

• Integrated CPAN test reporting
• Allows anyone who installs a distribution to contribute
• Allows testers to edit reports before sending
• Clean modular design, easy to extend functionality
• CPANPLUS::YACSmoke
• CPANPLUS::Dist::YACSmoke
• CPANPLUS::Dist::(RPM|MDV|Fedora|PAR|Deb)

Automated CPAN Smoke Testing

Dangers

Danger

• Running arbitary code
• Don't use root, use a sandbox account
• Sandbox the whole test machine using virtualisation
• Tests often assume unfettered network/Internet access
• Some distribution tests leave droppings:
  • Check the $HOME directory for '.' files/dirs
  • Check /tmp for files/dirs
• Stop rogue module installs:
  • If using CPANPLUS, chown ~/.cpan to root
  • chown to root the perl installation

Tools and Libraries

Tools and Libraries

• expat libxml2 libgd libgmp libmm
• libssh2 gtk2 glib gtk
• mysql postgresql
• curl rsync git wget
• zip unzip

Automated CPAN Smokers

• CPAN::Reporter::Smoker
• CPANPLUS::YACSmoke
• minismokebox

CPAN::Reporter::Smoker

• CPAN/CPAN::Reporter based
• Written to be 'turn-key' solution
• Use CPAN.pm distroprefs to handle/skip problem modules
• get latest CPAN.pm release
• set 'trust_test_report_history', 'build_dir_reuse'

CPANPLUS::YACSmoke

• CPANPLUS based
• Adds testing history recording
• Skip tests for previously passed prereqs

yactool --test

Problems

• Hanging distributions/tests
  • Not using prompt() in Makefile.PL/Build.PL
  • Hand rolled prompting BAD
  • Prompts that go into endless loops INSANE
  • Using CPAN::Shell in Makefile.PL WTF
  • Interactive tests
• Tests that use the group PID in kill

SmokeBox

SmokeBox

• Next generation POE based smoker framework
• Modular, flexible, extensible
• Common interface to smoker backends
• Numerous add-ons:
  • POE::Component::SmokeBox::Recent
  • POE::Component::SmokeBox::Dists
  • POE::Component::SmokeBox::Uploads::NNTP
  • POE::Component::SmokeBox::Uploads::RSS
  • POE::Component::SmokeBox::Uploads::CPAN::Mini
• Configurable minismokebox program

minismokebox

• Uses POE::Wheel::Run to launch smokes
• Launched smokes have a new process group and scrubbed %ENV
• Control of the smoke process
• Kill smokes that 'idle' for too long
• Kill smokes that run excessively
• Detect smokes that are trapped in loops and kill them off
• Separate from the test environment

minismokebox

minismokebox --perl /home/cpan/rel/perl-5.10.1/bin/perl

minismokebox --perl /home/cpan/rel/perl-5.10.1/bin/perl \
             --package '^POE'

minismokebox --perl /home/cpan/rel/perl-5.10.1/bin/perl \
             --author 'BINGOS'

minismokebox --perl /home/cpan/rel/perl-5.10.1/bin/perl \
             --backend CPAN::Reporter

Performance

Performance

• SMTP - use a local MTA/relay:
  • CPAN::Reporter: smtp_server = my.smtp.server
  • CPANPLUS: s conf cpantest_mx my.smtp.server
• Use a local CPAN mirror:
  • rsync a reliable CPAN mirror
  • CPAN::Mini::Devel mirror

Future Developments

Future Developments

• CPAN Testers 2.0
• HTTP submission transport
• Metabase
• Queryable API
• Shiny
• Soonish
• Encourage more CPAN Testers to test on a wider variety of platforms/perls
• https://admin.cpantesters.org/

CPAN Testing Community