SOFA website OK now on Mobile

June 24th, 2015

The SOFA Statistics website is now “responsive” – that is, it finally works well on mobile devices. I used Bootstrap to achieve this and had a major clean up of cruft. If there are any problems, let me know.

Responsive  design for SOFA website

Version 1.4.4 good for Mac Users

May 11th, 2015

Mac users can finally export output in PDF format including individual charts and report tables. And depending on the version of OS X Mac users will also be able to export as PNG images. Ensuring image exporting works across multiple versions of OS X is an ongoing project of work which users can help with if interested.

mac_export_output

mac_output

Users needing to produce monochrome output for publication will like the addition of a new monochrome theme.

monochrome

All New Features in 1.4.4:

  • Mac users can export output in PDF format (and PNG depending on version of OS X).
  • Added new monochrome theme.
  • Chi Square proportions output much easier to interpret successfully.
  • The name of the grouping variable is now displayed when running comparisons of groups e.g. Country if comparing Italy and Germany.
  • Exporting to spreadsheet detects if too many fields for xls output and informs user that only csv will be generated. Also truncates table name so worksheet name not too long.
  • Import dialog only displays file types suitable for importing.
  • Added message to let user know spreadsheet creation being skipped if no report tables to export.
  • More user help on need for raw data (not pre-summarised) and long-format vs wide-format data as appropriate.
  • Code reorganisation to make it possible for SOFA to be called in GUI form by external code GUI code.
  • Scripts are now easier to use for standalone purposes.
  • Added note about treatment of datetime data as categorical by SOFA for purposes of statistical tests.
  • When exporting to spreadsheet and csv changes reserved sofa_id field name to was_sofa_id so it is OK to reimport after changes.
  • More informative for larger range of potential problem e.g. database engine not functioning.

Bug Fixes:

  • Fix bug resulting in Pearson’s r being displayed instead of Spearman’s rho.
  • Fixed bug on some systems when saving a worksheet with spaces in name.
  • Prevented numerous bugs related to quoting table names, fully qualified file names etc.
  • Fixed bug with misuse of escape_pre_write on python code rather than normal content.
  • Skew, and normality test now cope with the nan issue better e.g. sqrt of a negative number. Just says unable to calculate instead of displaying nan (not a number). skewtest function now copes with negative number as input to square root.
  • Fixed bug when importing NaN text – now treated as a missing value in a numeric field.
  • Removed bug which sometimes prevented Mac users from being able to successfully change the report name.
  • Stopped making export folder if no output to export into it.

Calling Mac Users – Contact Me to Test Latest Greatest Version

March 14th, 2015

Over the last year, work on SOFA has been focused on a difficult packaging issue – enabling a Mac version to be built which allows Mac users to export their charts and reports as PNGs and PDFs. That functionality is now working on Snow Leopard and hopefully newer versions as well. But it would be nice to check with some people running Mac. If you’d like to try out the latest version of SOFA Statistics, please drop me a line via http://www.sofastatistics.com/contact.php.

SOFA Shows Support For Linux

April 10th, 2014

SOFA works on Windows, Mac, and Linux. But Linux is especially important for the project because SOFA is developed on Ubuntu. So it made sense to support the Linux ecosystem by signing up with the Open Invention Network. In an ideal world, it wouldn’t be necessary to have anything to do with software patents. For various reasons, they’re a bad idea and function more to inhibit innovation than encourage investment in software research and development. But the Open Invention Network plays a protective function in a world where people who create and actually make things can be preyed upon by parasites who have been granted monopolies on ideas – the so-called patent trolls.

The group was created to defend Linux from patent trolls and other attacks from patent holders. It tries to do this with its own patents which are then available royalty-free to any company, institution or individual that agrees not to assert its patents against Linux. While it hasn’t been done, these patents could also, in theory, be used by the OIN, or an OIN member, against a hostile company in a patent war.

Google joins Open Invention Network patent commons as a full member

Anyway, a range of companies and projects large and small (over 800 at present and growing) have signed up for the initiative including Google, Dropbox, IBM, Canonical, Mozilla, Twitter, Puppet Labs, Valve Software, Alfresco, NEC, Blender, OpenShot, Novell, Inkscape, Philips, Red Hat, CentOS, GNOME, Wikimedia, MariaDB Foundation, Rackspace, Moodle, Openstack, Slackware, Tor, and Sony. You get the idea.

PortableApps version of SOFA (alpha only)

April 2nd, 2014

J. David Eisenberg has kindly made a PortableApps version of SOFA for Windows. It is an alpha release only but it works and feedback/assistance is welcome. Here is his announcement as posted on the Google Group:

I have used the PortableApps guidelines (correctly, I hope!) to create a version of SOFA Statistics that can be installed on a USB drive and will retain your data and settings.

You can download the installer at http://evc-cit.info/SOFAStatisticsPortable_1.4.3_English.paf.exe ; this is Windows only.

Known problem: If you add the results of a statistical test to a report, any graphs for that statistical test will show up as a “missing image” icon. The image will be in the report; it just won’t show up on screen.

I have not tried scripting to see if that works properly.

Any comments are welcome at the SOFA statistics Google Group

Importing tab-separated files and more

March 15th, 2014

SOFA 1.4.3 now lets you import directly from tab-separated/tab-delimited files.

Importing tab-delimited text
Another change is less momentous but should really please people doing lots of row stats reports. As SOFA gained more measures it became increasingly more effort to select individual measures one-by-one, checkbox by checkbox. Now there is a toggle button for Select All/Deselect All. Much better :-).

Select or Deselect All

And the bonus themes are now part of the standard release making it easier than ever to make your charts and tables look good – I hope you like them.

New Themes

One change that most users will never notice is better support for running SOFA via scripts. An exciting automation project is currently under development using this functionality and I hope to have some news to share soon.

Here’s the full list of changes:

  • Can import tab-delimited data.
  • More options for attractive charts and reports. Three new themes available – sky, prestige (screen), and prestige (print).
  • Better support for automation (i.e. headless, running without GUI) esp in international context.
  • Exporting to spreadsheet now relies on more robust code library (xlwt)
  • Easy to select or deselect lots of row stats measures at once.
  • Faster opening in many cases.

And the bug fixes:

  • Minor tweak to PostgreSQL plug-in to handle timestamps without timezone.
  • Resolved bug when SQLite numbers are stored in a non-numeric field and processed for Chi Square test.
  • Importing csvs now copes better when only missing vals in sample of a field. Gives user the choice.
  • Fixed bug when doing a Row Stats table with a rows variable e.g. by Gender and some of the fields can’t be calculated for some of the row categories.
  • Headless importing now works in the event of inconsistent data types in fields.
  • Headless importing now reads entire dataset rather than a sample to avoid need for (human) decisions.
  • Scripts no longer rely on translated arguments. Much safer to use on other machines with different locales.
  • Fixed circular import bugs which only became visible when other bugs occurred.

Using SOFA alongside other statistics packages

February 15th, 2014

All statistics packages have their strengths and weaknesses so it is not uncommon for people to want to use more than one – even on the same project. SOFA is focused on making it easy to use some core statistical tests and producing attractive, high definition tabular and charting output. SOFA also makes it easy to link to, or import from, a wide range of formats: xls, xlsx, csv, google docs spreadsheets, MS Access, NySQL, MS SQL Server, PostgreSQL, SQLite, and, more recently, CUBRID.

But there is no point overcomplicating SOFA so it can do every statistical test that might be needed for a particular project. SOFA users have been routinely surveyed on what features they would like added and it has not consolidated into a clear list of priorities. People need lots of different things depending on their specific projects.

So a sensible goal for SOFA is to make it easy to import and export data, including metadata such as variable and value labels. This strategy has already resulted in the addition to SOFA (version 1.4.2) of built-in export-to-spreadsheet functionality. And it has already been improved for version 1.4.3 (not released yet at the time of this writing).

The question is, what packages should SOFA target as priorities for interoperability? Feel free to fire me off an email at grant@sofastatistics.com.

SOFA exports high-resolution results and more

February 9th, 2014

SOFA users can now export their output results as high-resolution images, PDFs, and spreadsheets ** without requiring additional plug-ins.

Easy high-quality exporting

It has never been easier to produce high-quality output ready to include in presentations, papers for publication etc.

High resolution output

It will also be possible to export table data. If there is a specialised analysis you can’t do in SOFA it will be much easier to export the data and import it into another stats package for that part of the process.

Export to spreadsheet from data

And the ability to backup your SOFA data and settings is built in.

Backup data and settings

So 1.4.2 is quite a major step forwards for the majority of users. I really hope you like it and spread the word.

** Doesn’t work for Macs currently – very sorry :-(. Any Mac users with Python experience are encouraged to contact the project – there are several ways you might be able to help SOFA resolve this problem. And get a little famous ;-).

1.4.1 Adds regression line for scatterplots

January 31st, 2014

Scatterplots can now be produced with a regression line and slope and intercept details:

Scatterplot regression lines

Additionally, the Export output plug-in (proprietary add-on) now gives option of exporting tabular data to spreadsheet. And there are some other minor improvements:

  • Better positioning of legend in scatterplots made by matplotlib.
  • Tweaked algorithm for getting optimal min and max axis values so more sensible when no variation.

The latest release also fixes a number of edge-case bugs:

  • Fixed bug in charting when users use variable names SOFA used in underlying SQL queries.
  • Fixed bug when refreshing database table dropdowns when no databases visible – was assuming the databases were included in the number of items in the sizer.
  • Fixed bug when print content redirected to output file in Windows and Mac – now coverted to utf-8 byte strings directly by overriding sys.stdout and sys.stderr with codecs.getwriter etc. Immediate impact is fix for bug when recoding a table and field names include non-ascii characters.
  • Fixed up fonts used so always look good on all systems.
  • Fixed bug which can occur when designing a new table. If we recode it before clicking the Update button, SOFA thinks we are trying to override another table of the same name. This is because SOFA started out thinking our table had no name and was never updated to tell it otherwise.
  • Minor changes to enable translation.
  • Fixed bug when importing empty pairs of double or single quotes. These were already being de-escaped (as a side-effect of the approach necessary to handle internal quote escaping in the csv module) and turned to solo quotes – and thus evading the check for blank raw vals which would have been turned to NULL.
  • Fixed bug giving error message for too many rows instead of too many columns when too many columns e.g. in Chi Square test.
  • SOFA now checks very early to see if you’ve installed SOFA under a local user folder instead of a program folder.
  • Fixed bug in PostgreSQL plug-in when working with a numeric field lacking a defined decimal points or numeric precision setting.
  • Show scatterplot minor axis ticks more readily so better when fewer distinct x values.
  • Scatterplots cope with absence of variability in an axis by forcing a different min and max for that axis.

SOFA releases 1.4.0 for Christmas :-)

December 15th, 2013

Ease of use is one of SOFA’s main goals (along with “learn as you go”, and “beautiful output”). Unfortunately, as new options were added to SOFA for exporting data, the simplicity of the output section of the user interface suffered slightly. New buttons were squeezed in one by one and the interface was getting more and more crowded. Something was going to have to change. And in version 1.4.0 it finally has! – I hope you like the change. It not only removes two interface items but it also adds room for more export options in the future. And there is more horizontal space in the drop-down control to describe each option more clearly and distinctly.

Here is the old design:

Old output layout

Lots of buttons

Not too bad, but lots of buttons, and more needed in the future. Here is the new design:

New output layout

It drops two items. There is also scope for adding more export options. Here are the current options available as displayed in the drop-down control:

Room for more

Obviously, it wouldn’t be hard to add a few more given it is a list. The Show Results button is larger, and its relationship with the report is more obvious by adding “Also” to “Also add to report”.

Large Show Results button

Only modest changes in many ways but hopefully another step in improving the user experience. Merry Christmas :-)