Statistics Open For All

The SOFA Statistics project could go in a number of different directions. Ideally, it would:

• Add more chart types and more flexibility for graphical customisation
  (without compromising the SOFA goals of beautiful output and ease-of-use)
• Add a comprehensive array of the most important statistical tests
  (without compromising the ease-of-use and learn-as-you-go goals)
• Make it much easier to automate reporting
• Make publishing reports to the web, and office formats, seamless and simple.

I only have limited time to develop SOFA at the moment, so I have to choose the top priorities. Here is what I think I should do:

• Charting
  • Make it easy to export data so it’s ready for charting using spreadsheet charting tools
  • Provide brief documentation on how to use advanced tools like Matplotlib
• Statistical Tests
  • Make it really easy to export data from SOFA ready for analysis in R
• Report automation
  • Provide documentation so people can automate SOFA themselves using Python
• Publishing
  • Add a plug-in for exporting to a document format

I should also solve the remaining bugs preventing Mac users from being able to export output as images. What do people think about this direction? Drop me a line at grant@sofastatistics.com.

SOFA Statistics 1.3.4 is not an adventurous release but it squashes plenty of bugs – especially for MS Access and MS SQL Server users. Here are the features:

• Can make more complex charts and larger series of charts. It is now possible to override the conservative limits on charts e.g. the maximum number of series or charts or clusters. A warning is shown that you may not necessarily produce a viable chart or set of charts. But often it will work so now you get to try and see.

• Importing now copes with excessively long field names by shortening them automatically (without risking duplicates).
• MS SQL Server views can now be analysed, not just tables.

And here are the bug fixes:

• Fixed bug with calculation of mean with MS SQL Server data (now explicitly cast as float to avoid integer result).
• Fixed bug in ANOVA output for precise (as opposed to speed) – it used to try mixing Decimals and other numeric types unsuccessfully.
• Fixed bugs with chart data gathering queries so works in MS SQL Server properly. Also cleaner for the others in any case.
• Fixed bug in underlying code if unique=False ever applied with scatterplots (currently never but it was still, technically, a bug).
• Fixed bug in scatterplot SQL which affected MS Access and MS SQL Server (can’t use aliases in group by etc).
• Adjusted y-title position in dojo scatterplots to avoid it being cropped.
• Fixed bug when Postgresql date date being displayed as a category (couldn’t calculate a length of a datetime.datetime object).
• Fixed layout bug when report table resized after Add to Report checkbox hidden. Now freshens layout when checkbox reappears.
• Fixed bug preventing charts from being produced when linked to MS Access.
• Fixed bug adding large delay to display of output when linked to MS Access.

There are some important changes coming for SOFA but it was important to tidy up a bit first. Watch this space!

It is now easy to back up SOFA including data, reports, and any variable and project details. The backup button is on the main screen and can be made operational by installing the backup SOFA plug-in (available from www.sofastatistics.com/get_extensions.php).

The latest version brings lots of small improvements and one useful new feature – the ability to use sum as an option for charting e.g. a line chart showing total sales by country:

Here is the full list of improvements:

• Adding sum as an option for charting e.g. a line chart showing total income per month by product. And the interface has been simplified at the same time.
• Matplotlib scatterplots now have optimal min and max settings calculated for their x-axis.
• Added footnote to Wilcoxon output explaining that different statistics packages may report the test statistic differently.
• Misc fixes to chart layout including left margin offset.
• Easier to add new variable definition files from within dialog for choosing them.
• Modified recode column labels and help content to reduce confusion about which columns to enter range information into.

And bug fixes:

• Fixed code picking optimal min and max axis values for scatterplots and box plots to cope when value range is much smaller than gap to 0.
• CSV import now copes with new lines inside fields when gathering data for sample display.
• Extra settings for Line Charts now display when they should even if only changing data type.
• Fixed bug which allowed line breaks in field names.

SOFA 1.3.0 has plenty of small but important additions:

• Added Mode as an option to Row Stats report tables. Reports mode(s) with N of the mode value(s) e.g. mode weight 72.0, 76.0 (N=23)

  

• Line and area charts can show major labels only as an option.
  

  

• Pie charts now have option of displaying count and percentage (not just in the tooltips as at present).
  

• Histograms use consistent bins when charted by a second variable.
• Better placement of y-axis title when wide labels.
• Pie charts keep consistent colours even if sorted by count rather than value or label.

  

• Better message when adding new reports if required subfolder with javascript and background images is missing. Only show message now if a problem.
• When trying to export report, SOFA checks for expected subfolder as well (otherwise dojo fails for any charts and export fails).
• SOFA prevents attempt to export report if no report file (yet).
• No longer displays View or Export Report buttons on Projects dialog.

And there have also been some important bug fixes making it worth upgrading:

• Fixed bug in row stats where data should have explicitly filtered out None values.
• Fixed bug in setting of min and max values for y-axis for boxplots when min is below 0.
• Refactored code for running report in output module. Easier to understand and also made it easy to save copy of internal html output with absolute paths to images – very helpful when exporting images.
• Built more robust value quoting e.g. for sql statements.

I hope you like it.

• The latest SOFA, version 1.2.2, lets you import from Excel XLSX files (previously, Excel files had to be in the XLS format).

  

• Normality analyses can be included in reports, saved as output etc.

  

• And there is support for CUBRID databases. CUBRID is an open source relational database highly optimized for Web Applications. That brings to 6 the total number of SQL-type databases that SOFA can directly link to.
  

A few bugs were also fixed:

• Restored standard deviation option to row stats report tables.
• Fixed bug which meant row % was appended multiple times to config col dialog (until session closed).
• Restored PostgreSQL functionality by fixing faulty psycopg import statement.

The latest version of SOFA Statistics makes it easy (via a plug-in) to export to spreadsheet.

SOFA also makes it easier to select multiple variables when making report tables.

Additionally, there have been some important bug fixes – mainly for bugs which snuck in during the major change to 1.2.0.

• Fixed nasty bug breaking demo report tables. A casualty of the changes to the independent display of titles from report tables in 1.2.0 so wide titles didn’t mean wide table cells.
• Fixed bug with display of percent symbols in report tables. A missing not was the culprit – another casualty of the big refactoring for 1.2.0.
• Fixed bug with Data List reports – wouldn’t update display after changing sort order of a variable.
• Sort by value now works properly in Data List reports.
• Can now handle excessively long values being used as categories in report tables or charts etc. Checks are also now made for excessively long category variable values.
• If encoding problems, SOFA now tries to use the field encoding e.g. iso-8859-1.