Archive for December, 2009

0.8.15 flexible data filtering options and support for US-style dates

Wednesday, December 23rd, 2009

Version 0.8.15 has the following new features:

* Data filtering: If you’re only interested in data for one nation, for example, you can add a filter so reports are derived from that nation’s subset of the data. It is now easy to add, remove, and alter filters to individual tables. This feature makes analysis much more fluid. It is arguably one of the most important changes in long time. Check it out. Almost anywhere you can see tables listed, you can right click on the table name and set, alter, rename, or remove a filter.
* Support for US-style dates: SOFA Statistics now allows the setting of date entry format preferences, e.g. if you are in the US, Canada, the Philippines, Belize, Palau, or the Federated Republic of Micronesia, you may wish to allow mm/dd/yy as a date format for data entry instead of dd/mm/yy. Over time, there will no doubt be other preference options available.
* Default data table now has a datetime field to experiment with.
* More tooltips and misc usability improvements.

There have also been miscellaneous bug fixes including:

* Removed bug which meant you couldn’t immediately open a freshly-created table.

In addition there has been further expansion of test coverage and a fix of sorts for the Karmic launcher. There is a problem (hopefully temporary) where users of GTK applications can click a button, have it depress, and have nothing actually happen (unless they then click on the Enter key). What this means for SOFA Statistics is that when a Karmic Ubuntu user clicked on the new Preferences button, a preferences dialog wouldn’t appear. A partial solution has been to change the launcher slightly so it sets the GDK_NATIVE_WINDOWS environment variable. So instead of:

python /usr/share/pyshared/sofa/start.py

the launcher now invokes:
bash -c “GDK_NATIVE_WINDOWS=1 python /usr/share/pyshared/sofa/start.py”

… which solves the problem in many cases. If it doesn’t, the work-around is to mouse click on the Preferences button then press the Enter key. Fortunately, the Preferences button will hardly ever be used. Ideally, the problem will be fixed properly in newer versions of SOFA Statistics as the underlying code libraries are fixed.

Adding features, fighting fires

Sunday, December 20th, 2009

The next version (0.8.15) will have two big new features:

  • The ability to apply filtering to tables. For example, if you are only interested in the data for one gender you could add a filter so that all the reports you make based on a given table are derived from the selected gender’s subset of the data. It is easy to add, remove, and alter filters to tables.
  • The ability to set preferences, e.g. if you are in the US, Canada, the Philippines, Belize, Palau, or the Federated Republic of Micronesia, you may wish to allow mm/dd/yy as a date format for data entry instead of dd/mm/yy. Over time, there will no doubt be other preference options available.

There have also been miscellaneous bug fixes, smaller features e.g. tooltips, and further expansion of test coverage. Now to the most recent fire I had to put out. There is a problem (hopefully temporary) where users of GTK applications can click a button, have it depress, and have nothing actually happen (unless they then click on the Enter key). What this means for SOFA Statistics is that when a Ubuntu user clicked on the new Preferences button, a preferences dialog wouldn’t appear. The solution was to change the launcher slightly so it set the GDK_NATIVE_WINDOWS environment variable. So instead of:

python /usr/share/pyshared/sofa/start.py

the launcher now invokes:
bash -c "GDK_NATIVE_WINDOWS=1 python /usr/share/pyshared/sofa/start.py"

… which solves the problem completely.

I made a similar change to the launcher for eclipse. See http://www.eclipse.org/forums/index.php?&t=msg&th=153842 and http://blog.export.be/2009/10/fixing-eclipse-for-ubuntu-karmic-koala-9-10/.

0.8.14 much more flexible management of data table design inside built-in database

Wednesday, December 9th, 2009

The latest release not only has the usual bug fixes and usability enhancements, it has a few new pieces of functionality.

  • Can modify the design of tables in the built-in SOFA database (variable names, types, order, table names).
  • Able to insert and delete rows when editing value labels.
  • Can delete tables in the built-in database.
  • The sofa id field is now read-only when designing tables and the user gets useful messages if they try to delete it or insert a new field before it.

The bug fixes were:

  • CSV importer copes with inconsistent new line characters anywhere they are encountered.
  • Fixed bug when a key is deleted in a value list and then OK is hit.
  • Removed bug where focus would move erratically when entering long lists of value labels.
  • Changing database or table when making table reports resets the column buttons as it should.
  • Numbers like 1000000000000.2 are now displayed as simple numbers instead of as 1e+12 etc.
  • Newly-created tables appear in the table dropdown list immediately.

IMPORTANT FOR ALL UPGRADING USERS – you must delete the /username/sofa folder first (after storing anything you wish to save). And inside project files, change conn_dets to con_dets.

Funding open source to benefit education, health etc

Tuesday, December 1st, 2009

Government investment in open source makes sense as a way of reducing the cost of government service delivery.

The model is this – a government agency invests a sum of money to benefit a government sector e.g. education, health, social service delivery etc. As long as the software is open source, the government sector gets a multiplier effect – any further development of the software in other countries will further benefit education, health, social service delivery etc in the original country.

“This has certainly been New Zealand’s experience with Moodle, a GPL licenced online learning management system. A few years ago Moodle was a nice platform, had a good development community and a few hundred sites. A New Zealand TEC fund was used to add a number of features, not least amongst these were some very solid enterprise and performance capabilities.

Moodle is now used in over 13,000 sites and must be the world’s most popular LMS. Investment from institutions such as the UK’s Open University has now well exceeded NZ’s initial investment. Yet because of the GPL New Zealand still gets to benefit from other peoples’ significant work.” (New Zealand Open Source Society 2007)

Now imagine if there was a hypothetical open source program ;-) that could be used throughout the education sector (senior secondary, polytechs, universities), across a myriad of smaller organisations that supply social services on behalf of government, and across the private sector as well. It enables everything from the teaching of statistics through to routine management reporting. A bit of assistance in its further development could have a large spin-off. Makes you think.

0.8.13 better support for inconsistent data and missing values

Tuesday, December 1st, 2009

Removing rough edges and handling less perfect incoming data have been the recent focus of attention. The main changes in 0.8.13 are:

  • Better support for inconsistent data and missing values.
  • The user is informed what happens to files being imported.
  • The user has the option of halting a large raw data table report.
  • CSV importing now asks user if file has header and offers to clean up files with mixed line separators.
  • New projects have the default SOFA database preconfigured ready to save tables to.
  • Program now returns useful message if SQLite table with median or std dev fails because of non-numeric values (in purportedly numeric field).
  • Attempts to make new tables without a connection to the default database now receive a useful message.
  • Users are given the opportunity to pull out of opening a large data table.

Also importantly, there have been a myriad of bug fixes:

  • Fixed bug when using SQLite database other than the SOFA default one.
  • Report tables now handle non-numeric values in numeric fields (SQLite).
  • Fixed simple but fatal bugs affecting raw and summary tables.
  • Fixed bug when user path includes international characters.
  • Fixed bug using MS SQL for making report tables.