Rough Guide to CPAN Testing
Chris BinGOs Williams
Who the hell is this guy?
- By day, a Network Services Specialist
- By night, perl hacker, POEvangelist and CPAN Tester
- CPAN ID and IRC nickname, 'BinGOs'
- email: chris@bingosnet.co.uk
- Using perl since 1995
- Using POE since 2003
- CPAN Tester since 2005
MANIFEST
- What is CPAN Testing?
- CPAN Testing Community
- How CPAN Testing Works
- The tools of CPAN Testing
- Scaled up CPAN Testing
- The problems and dangers of CPAN Testing
- Future developments
What is CPAN testing?
- Tests CPAN distributions
- Many distributed testers
- Multiple platforms
- Multiple perls
- Structured feedback mechanism
- Anarchic (Chaotic?)
CPAN Testing Community
- Web sites:
- Reports - http://www.cpantesters.org/
- Statistics - http://stats.cpantesters.org/
- Wiki - http://wiki.cpantesters.org/
- Dependencies - http://deps.cpantesters.org/
- Matrix - http://matrix.cpantesters.org/
- Mailing lists:
- cpan-testers@perl.org - !
- cpan-uploads@perl.org
- cpan-testers-discuss@perl.org
- Internet Relay Chat on irc.perl.org network:
- #cpantesters - "live" report feeds
- #cpantesters-discuss
How CPAN Testing Works
How does CPAN testing work?
How does CPAN testing work?
How does CPAN testing work?
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Testers run tests (manual or automated)
- Tests results sent to CPAN Testers mailing list
- Results tabulated on CPAN Testers Reports
- Author notified of FAIL and UNKNOWN reports
Types of Reports
- PASS - all the tests in test suite passed
- NA - distribution not installable
- FAIL - one or more tests failed
- UNKNOWN - distribution has no test :(
The tools of CPAN Testing
Two styles of CPAN testing
- "Manual" -- during ordinary module installation
- CPAN + CPAN::Reporter
- CPANPLUS
- "Smoke" -- Automated, mass testing
- CPAN::Reporter::Smoker
- CPAN::YACSmoke
- CPANPLUS::YACSmoke
- POE::Component::CPAN::YACSmoke
- POE::Component::CPANPLUS::YACSmoke
- POE::Component::CPAN::Reporter
CPAN::Reporter
CPAN::Reporter Installation
Super quick-start version...
$ cpan
cpan> install CPAN::Reporter
cpan> reload CPAN
cpan> o conf init test_report
CPAN::Reporter Benefits
- Adds CPAN test reporting to CPAN.pm
- Allows anyone who installs a distribution to contribute
- Allows testers to edit reports before sending
- Provides selective report submission based on configuration
- Keeps track of subject lines sent to CPAN Testers
- Subject line has grade, module name and perl version
- Avoids duplicate reports if re-testing
CPANPLUS
CPANPLUS Installation
$ cpanp
cpanp> i YAML::Tiny
cpanp> i Test::Reporter
cpanp> s conf email bingos@cpan.org
cpanp> s conf cpantest 1
cpanp> s conf cpantest_mx smtp.somedomain.com
cpanp> s conf verbose 1
cpanp> s save
CPANPLUS Benefits
- Integrated CPAN test reporting
- Allows anyone who installs a distribution to contribute
- Allows testers to edit reports before sending
- Clean modular design, easy to extend functionality
- CPAN::YACSmoke
- CPANPLUS::YACSmoke
- CPANPLUS::Dist::YACSmoke
Automated CPAN Smoke Testing
Automated CPAN Smoke Frameworks
- CPAN::Reporter::Smoker
- CPAN::YACSmoke
- CPANPLUS::YACSmoke
- POE::Component::CPAN::YACSmoke
- POE::Component::CPANPLUS::YACSmoke
- POE::Component::CPAN::Reporter
CPAN::Reporter::Smoker
- CPAN/CPAN::Reporter based
- Written to be 'turn-key' solution
- Use CPAN.pm distroprefs to handle/skip problem modules
- get latest CPAN.pm dev release
- set 'trust_test_report_history', 'build_dir_reuse'
- 138,311 reports from David Golden (DAGOLDEN)!!!!!
CPAN(PLUS)::YACSmoke
- CPANPLUS based
- Adds testing history recording
- Skip tests for previously passed prereqs
perl -MCPAN::YACSmoke -e test
perl -MCPANPLUS::YACSmoke -e test
Problems
- Hanging distributions/tests
- Not using prompt() in Makefile.PL/Build.PL
- Hand rolled prompting BAD
- Using CPAN::Shell in Makefile.PL BAD
- Interactive tests
- Tests that use the group PID in kill
- $ENV{AUTOMATED_TESTING}
POE based smokers
- Uses POE::Wheel::Run to launch smokes
- Control of the smoke process
- Kill smokes that 'idle' for too long
- Kill smokes that run excessively
- Separate from the test environment
miniyacsmoker --perl /home/cpan/rel/perl-5.10.0/bin/perl
Dangers
Danger
- Running arbitary code
- Don't use root, use a sandbox account
- Sandbox the whole test machine
- VMWare
- Xen
- Parallels
- etc.
- Tests often assume network/Internet access
Future Developments
Future Developments
- Add more documentation to the CPAN Testers Wiki
- Encourage more CPAN Testers to test on a wider variety of platforms/perls
- CPAN Testers over HTTP
- SmokeBox - POE::Component::SmokeBox
- Auto-rolling of CPAN smokers
CPAN Testers BOF
- Birds of a Feather session
- Discussing CPAN testing
- @ 17:30 today in room 2.07