[Lazarus] Stable islands: testing stability

[Reinier Olislagers]

MML's post on stable islands has made me think.

Below some ideas for discussion:
If we want to know whether a certain combination of
OS+FPC+Lazarus+packages is stable, we:
1. should ask the user
2. don't trust him completely though and we should get measurements

Short-term action
=================
I suggest we can create a Lazarus GUI that
- asks the user how stable he thinks Lazarus is
- collects test measurement data (see below)
The GUI can then display the results, save them for the user's own use
(import in database etc), and upload to a central web page

The GUI could run these tests:
- compiler test suite. Custom test code; results available as XML, IIRC.
- FPC database tests (e.g. for bufdataset, memds, sdfdataset, perhaps
test Firebird embedded, sqlite, as well as any specified in the user's
database.ini). These are fpcunit tests; can spit out e.g. XML
- any other fpc tests written with fpcunit
- Lazarus non-GUI tests (can't remember what data format these have)

Medium-term action
==================
- Have FPC report not only version but also SVN revision (much like
lazarus does in help/about) so that data can be used to pinpoint exactly
what version is used if checked out from SVN (e.g. trunk)
- Write small cross platform database application to collect, read in
test results into database; useful for user's own use
- Similar but for a central web site. Web page with results could be
done much like the compiler test suite page
- Extend FPC fpcunit tests for packages (both in coverage and number of
packages tested)
- Extend Lazarus non-GUI tests (e.g. fill stringgrid, save as csv, load
from csv, compare grid contents). Perhaps even a policy/behaviour
change: no new units accepted without relevant tests (of course where
feasible)
- Extend packages so that there is a self-test possibility: the GUI
mentioned above) and/or IDE shoul run unit tests in the packages and get
the resutls. Example: Lazreport creates sample report, outputs to file,
and compares file contents with expected contents

Extending the test suites etc. would aid in regression testing as long
as attention is paid to the trend in test faults.

What do you think?

Thanks,
Reinier