Rough Guide to CPAN Testing

Chris BinGOs Williams

CPAN Tester

Who the hell is this guy?

• By day, a Network Services Specialist
• By night, perl hacker, POEvangelist and CPAN Tester
• CPAN ID and IRC nickname, 'BinGOs'
• email: chris@bingosnet.co.uk
• Using perl since 1995
• Using POE since 2003
• CPAN Tester since 2005

MANIFEST

• What is CPAN Testing?
• CPAN Testing Community
• How CPAN Testing Works
• The tools of CPAN Testing
• Scaled up CPAN Testing
• The problems and dangers of CPAN Testing
• Future developments

What is CPAN testing?

What is CPAN testing?

• Tests CPAN distributions
• Many distributed testers
• Multiple platforms
• Multiple perls
• Structured feedback mechanism
• Anarchic (Chaotic?)

CPAN Testing Community

• Web sites:
  • Reports - http://www.cpantesters.org/
  • Statistics - http://stats.cpantesters.org/
  • Wiki - http://wiki.cpantesters.org/
  • Dependencies - http://deps.cpantesters.org/
  • Matrix - http://matrix.cpantesters.org/
• Mailing lists:
  • cpan-testers@perl.org - !
  • cpan-uploads@perl.org
  • cpan-testers-discuss@perl.org
• Internet Relay Chat on irc.perl.org network:
  • #cpantesters - "live" report feeds
  • #cpantesters-discuss

How CPAN Testing Works

How does CPAN testing work?

How does CPAN testing work?

How does CPAN testing work?

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)
• Tests results sent to CPAN Testers mailing list
• http://lists.cpan.org/showlist.cgi?name=cpan-testers
• nntp://nntp.perl.org/perl.cpan.testers
• Results tabulated on CPAN Testers Reports
• Author notified of FAIL and UNKNOWN reports

Types of Reports

• PASS - all the tests in test suite passed
• NA - distribution not installable
• FAIL - one or more tests failed
• UNKNOWN - distribution has no test :(

The tools of CPAN Testing

Two styles of CPAN testing

• "Manual" -- during ordinary module installation
• CPAN + CPAN::Reporter
• CPANPLUS
• "Smoke" -- Automated, mass testing
• CPAN::Reporter::Smoker
• CPAN::YACSmoke
• CPANPLUS::YACSmoke
• POE::Component::CPAN::YACSmoke
• POE::Component::CPANPLUS::YACSmoke
• POE::Component::CPAN::Reporter

CPAN::Reporter

CPAN::Reporter Installation

Super quick-start version...

 $ cpan

 cpan> install CPAN::Reporter

 cpan> reload CPAN

 cpan> o conf init test_report
    

CPAN::Reporter Benefits

• Adds CPAN test reporting to CPAN.pm
• Allows anyone who installs a distribution to contribute
• Allows testers to edit reports before sending
• Provides selective report submission based on configuration
• Keeps track of subject lines sent to CPAN Testers
• Subject line has grade, module name and perl version
• Avoids duplicate reports if re-testing

CPANPLUS

CPANPLUS Installation

 $ cpanp

 cpanp> i YAML::Tiny

 cpanp> i Test::Reporter

 cpanp> s conf email bingos@cpan.org

 cpanp> s conf cpantest 1

 cpanp> s conf cpantest_mx smtp.somedomain.com

 cpanp> s conf verbose 1

 cpanp> s save

    

CPANPLUS Benefits

• Integrated CPAN test reporting
• Allows anyone who installs a distribution to contribute
• Allows testers to edit reports before sending
• Clean modular design, easy to extend functionality
• CPAN::YACSmoke
• CPANPLUS::YACSmoke
• CPANPLUS::Dist::YACSmoke

Automated CPAN Smoke Testing

Automated CPAN Smoke Frameworks

• CPAN::Reporter::Smoker
• CPAN::YACSmoke
• CPANPLUS::YACSmoke
• POE::Component::CPAN::YACSmoke
• POE::Component::CPANPLUS::YACSmoke
• POE::Component::CPAN::Reporter

CPAN::Reporter::Smoker

• CPAN/CPAN::Reporter based
• Written to be 'turn-key' solution
• Use CPAN.pm distroprefs to handle/skip problem modules
• get latest CPAN.pm dev release
• set 'trust_test_report_history', 'build_dir_reuse'
• 138,311 reports from David Golden (DAGOLDEN)!!!!!

CPAN(PLUS)::YACSmoke

• CPANPLUS based
• Adds testing history recording
• Skip tests for previously passed prereqs

perl -MCPAN::YACSmoke -e test

perl -MCPANPLUS::YACSmoke -e test

Problems

• Hanging distributions/tests
  • Not using prompt() in Makefile.PL/Build.PL
  • Hand rolled prompting BAD
  • Using CPAN::Shell in Makefile.PL BAD
  • Interactive tests
• Tests that use the group PID in kill
• $ENV{AUTOMATED_TESTING}

POE based smokers

• Uses POE::Wheel::Run to launch smokes
• Control of the smoke process
• Kill smokes that 'idle' for too long
• Kill smokes that run excessively
• Separate from the test environment

miniyacsmoker --perl /home/cpan/rel/perl-5.10.0/bin/perl

Dangers

Danger

• Running arbitary code
• Don't use root, use a sandbox account
• Sandbox the whole test machine
  • VMWare
  • Xen
  • Parallels
  • etc.
• Tests often assume network/Internet access

Future Developments

Future Developments

• Add more documentation to the CPAN Testers Wiki
• Encourage more CPAN Testers to test on a wider variety of platforms/perls
• CPAN Testers over HTTP
• SmokeBox - POE::Component::SmokeBox
• Auto-rolling of CPAN smokers

CPAN Testers BOF

• Birds of a Feather session
• Discussing CPAN testing
• @ 17:30 today in room 2.07