Home / Software / GenStat / Quality Assurance

Quality Assurance

GenStat Quality Assurance

If you prefer you can read this as a pdf.

At VSN International, we recognise that your analyses depend on the accuracy and validity of our software. We therefore take great care to check that the results that we produce are reliable and correct. This document summarizes some of the methods that we use to achieve this.

Accuracy

We use algorithms that are designed to give numerical accuracy and stability. These are mainly produced in-house by our team of well-trained developers. We also make use of the renowned and carefully-validated algorithms produced by the Numerical Algorithms Group (one of our shareholders).

Correctness

We check the GenStat directives using an extensive suite of test programs whose results have been checked by

• reproducing analyses and comparing the results for representative data sets using alternative methods e.g. anova and regression, reml and anova, etc.
• by explicit calculation, using GenStat’s ability to perform general matrix and vector calculations
• by hand calculation where feasible
• comparing our results with those in published books and refereed journal papers
• by comparing with the results from other software systems

Procedures in the GenStat Procedure Library are assessed by an Editorial Board, who use similar methods to check that accepted procedures are useful, reliable and accompanied by clear documentation.

Quality

• Our technical staff have considerable experience of software development, dating back to the early 1970’s.
• They all have degree and post-graduate qualifications in statistics or computing.
• They enjoy close collaborative links with research statisticians at organisations including Rothamsted Research and New South Wales Department of Primary Industry.
• They also operate as members of the research community, attending and presenting papers at scientific meetings and conferences; this helps to ensure that they are up-to-date with current statistical research.
• All our staff subscribe to the concept of self-development to enhance our skills to the benefit of VSNi’s activities, and VSNi has recognised the importance of this activity by becoming certified under the UK Government’s Investors in People scheme.

Reliability

• The underlying material for our software (source code, test suites, documentation etc) is held in a secure archive that uses the source-control system Subversion. This keep a formal record of every change, with explanatory documentation to explain why it was made, and allows us to step back accurately and reliably to earlier versions to facilitate the diagnosis and correction of errors.
• To guard against mistakes, our extensive suite of test programs is run after every substantive change.
• Each new release undergoes a period of at least 2 months of beta-testing by committed users to check its reliability under “field conditions”. A formal reporting process is followed to ensure that errors are logged and corrected.
• Flaws and errors subsequently reported in the full release are also logged in a formal system, and patches and service packs are produced in a timely way to correct them.
• We should also place on record the fact that many of the core GenStat algorithms have been in intensive use for many years by a wide population of very disparate users. You can therefore base your confidence in GenStat, not only on our efforts but also on the accumulated efforts of your peers.

Support

We recognise the crucial safety-net that timely, well-informed and helpful support can provide for our supported users.

• We handle all our support calls in-house: you will not be sent ill-informed replies from an off-shore call centre.
• We enter support enquiries into a monitoring system to ensure that they do not become lost or forgotten.
• Our target, achieved with over 99% success, is to provide a reply within one working day.
• Our technical staff are fully involved in support, so you are guaranteed a well-informed answer. Their involvement also ensures that flaws that are reported through support receive appropriate priority in our development. Where a flaw can be corrected quickly, a software patch will be formed. Alternatively we endeavour to provide a work-around.
• We are happy to provide statistical advice and assistance as well as computing support. If your request is too complicated for a quick response, we will endeavour to offer you an acceptable consulting contract.
• In addition to our formal support service we also maintain web-based discussion lists for GenStat, GenStat Discovery and ASReml on the VSNi website. Although, these are intended mainly to provide a forum for users’ views and self-help, we do monitor the discussions so that we can answer any more-challenging queries.

Good statistical practice

Our technical development team includes statisticians with considerable experience of statistical consulting and teaching. We are thus very aware of the issues that can arise when statistical software is used by people with limited statistical experience or training. The menus, interfaced and algorithms in GenStat are designed to help these users to use statistics effectively and correctly. For example

• the menus for more sophisticated methods, such as generalized linear models, are designed to build naturally on more familiar methods, such as linear regression, to ease your transition into more advanced techniques
• we make a clear distinction between categorical variables (i.e. factors) and continuous numerical variables (variates) to help you to avoid using these in inappropriate ways
• the output window supports context-sensitive help to explain unfamiliar terms
• we use general algorithms, for example in our facilities in GenStat for anova, regression and reml, to avoid artificial constraints on the size of complexity of your analyses
• GenStat has a simple but powerful command language that allows you to access its full range of methods
• we have clear documentation and on-line help, and an extensive set of examples of the commands that are accessible through a specially designed menu
• the GenStat command language can be used as a general-purpose programming language so that you can develop or modify methods to match your own requirements
• development of new or customized methods is facilitated by the fact that the GenStat Procedure Library is open-source, with a menu to access the source code of any of the Library procedures
• we are not willing to provide questionable types of analysis even thought some of these may have achieved misguided popularity elsewhere!

History

VSNi was formed in 2000 as a spin-off company from Rothamsted Research and the Numerical Algorithms Group. This brought together the GenStat development group from Rothamsted with the statistical commercialisation group from NAG to provide a stronger collaboration of research and development with sales and marketing. This has enabled the development group to focus more closely on user needs while retaining their close links with the research community. So we believe that we are responsive to both user needs and research opportunities.

Pedigree

Rothamsted is probably the oldest agricultural research institute in the world, and is now recognised as a centre of excellence for science in support of sustainable land management and its environmental impact. Its scientific research ranges from studies of genetics, biochemistry, cell biology and soil processes to investigations at the ecosystem and landscape scale. Its interests in statistics began in 1919, when R.A. Fisher was appointed as its original statistician to study the accumulated results of the Rothamsted classical field experiments, which began in 1843 with the Broadbalk experiment on winter wheat. Fisher realised the need for improved statistical methods, and, over the next few years, laid the foundations of modern applied statistics. In fact, many of the standard methods that we now take for granted were originally devised by Fisher. Examples include analysis of variance, design of experiments, exact tests, discriminant analysis and maximum likelihood. Rothamsted became a pioneer of the use of computers in statistical analysis in 1954 when Frank Yates, Fisher’s successor as Head of Statistics at Rothamsted, obtained an Elliot 401 computer – one of the first computers to be used away from its manufacturing base. GenStat was the natural successor of this early work. Its development began at Rothamsted when John Nelder was appointed as the next Head of Statistics at Rothamsted, in 1968. Roger Payne took over its leadership in 1985, when John Nelder retired from Rothamsted, and continues in that role as VSNi’s Chief Science and Technology Officer. GenStat has benefited from the close relationship between statistical computing and statistical research and consulting at Rothamsted, and this relationship continues through the continuing links between VSNi and Rothamsted, where several of our technical staff have visiting positions. Other statistical theory and methods in GenStat that were originally developed by Rothamsted statisticians include generalized linear models (John Nelder), general balance (John Nelder originally, with further developments by Roger Payne), canonical variates analysis (John Gower) and REML analysis of mixed models (Robin Thompson). GenStat thus has a long history as the means of making available new statistical research that has been developed to address real biological research issues. We believe that this has given us unique insights into the needs of statisticians and scientists in biological research.

The Numerical Algorithms Group (NAG) is one of the pioneers of mathematical, statistical, data mining and visualisation software. The services offered by NAG have been used by academic, government and industrial institutions globally for over 30 years. In 1971 NAG developed the first mathematical software library, which has evolved over the years and become the largest commercially available library of mathematical and statistical algorithms.

We benefit from the expertise of our shareholders through their representatives on the VSNi Board. The Chairman of the Board is Jim Cooper, the Chief Executive Officer of Maplesoft, another very well-established company which can rightly be described as the world’s leading provider of high-performance software tools for engineering, science, and mathematics. They augment our scientific expertise and provide valuable advice and direction on our activities.

The future

We recognise the importance of keeping you up-to-date with new developments in statistics and computing. We thus have a regular schedule of new releases. GenStat itself has for many years followed an annual release schedule. Its new Editions generally include not only new statistical methodology but also enhanced interfaces, new documentation and updates to cater for any recent new releases of MS Windows.