July 10th, 2010
SOFA Statistics 0.9.13 has a number of exciting new features.
1) Easy recoding of data e.g. age to age group:
Recoding data e.g. age to age group
2) Better support when choosing statistical tests:
When a selection is made SOFA displays some helpful tips to affirm the choice made or suggest alternatives.
Helpful tips
The dialog also makes it easy to answer questions about your data which will help make a decision.
Answering questions about your data
3) Better support for importing CSV files with different data encodings:
Confirm encoding when importing csv files
4) Nicer Windows installer
New Windows installer
5) More flexible installation options. Anything containing ‘sofa’ is OK.
There have also been some important bug fixes:
- Recoding copes with REMAINING keyword properly and copes with varied field types properly.
- Generally copes better with system encodings like Chinese Traditional (big5).
- CSV importing handles non-English much better.
- Corrected handling of non-English characters when errors are encountered.
Posted in general, statistics | No Comments »
June 30th, 2010
June 2010 was easily the best month for SOFA Statistics downloads with 2,296 recorded. This is very encouraging for the project and suggests there is demand for a statistics package prioritising ease of use and aesthetics.
Sourceforge downloads for June 2010
The priority in coming releases is the addition of output charting. Some extra work will also go into the Statistical Test Selection Dialog to ensure the correct balance is struck between providing simple advice to beginners and recognising the multitude of factors which may be relevant to making a decision.
As always, users are encouraged to provide feedback on what they like/don’t like about the program. The goal is to make the best application possible so all feedback is welcome.
Posted in general, statistics | No Comments »
June 28th, 2010
SOFA Statistics now comes with integrated online help. Clicking on the “Online Help” button opens the SOFA Statistics User Guide in the user’s default browser.
Integrated Online Help
The goal is to make the user documentation as useful as possible – both to new users and to users trying to solve specialised problems. Over time, lots of gaps will be filled so it will be worth making repeat visits to the site.
The User Guide
Here is the full list of new features:
- Now able to look at and manipulate your SOFA data using free and open source tools like Sqlite Database Browser
- Improvements for connecting to Google Docs spreadsheets
- Major improvements to usability around data entry
- Now able to edit variable details from the table config dialog
- Much more useful help when errors encountered importing csv files
Version 0.9.12 also fixes numerous bugs with the Google Docs spreadsheet
Posted in general | No Comments »
June 12th, 2010
The latest version, 0.9.11, can import data from your on-line Google Docs Spreadsheets. I will supply more details later, but it is easy to conduct on-line surveys with Google Spreadsheets underneath. And now you can analyse those results using SOFA Statistics.
To connect to your Google account, there is a new button on the standard Import dialog.
New Google Spreadsheet button
This opens a simple dialog letting you sign into your Google account. The existing spreadsheets are listed and then you can select a worksheet. Once this is downloaded, it can be imported in the standard SOFA manner.
Download Google Spreadsheet
Version 0.9.11 also makes it easier to work with read-only columns in data tables.
Read-only Columns
The final improvement relates to filtering. If you have trouble with the flexible data filtering option, more helpful examples of correct syntax are supplied.
The main bug fix is that SQLite and MySQL now handle fields containing double quotes.
Posted in general | No Comments »
June 9th, 2010
In less than a week the Mac package for SOFA Statistics has been downloaded over 100 times, representing nearly a quarter of all downloads of version 0.9.10.
Example output on OS X
A positive response was expected given recent survey results at the main SOFA Statistics website – 35% of respondents wanted the Mac package as the next thing to add to the project. And about 14% of visitors to the main website are using a Mac OS. But it was still nice to see the response.
Posted in general | No Comments »
June 2nd, 2010
SOFA Statistics has been packaged for Mac OS X (Leopard and Snow Leopard). It only seems right that an application emphasising ease-of-use and beautiful output should be available for Apple users.
SOFA Statistics on OS X
The installer is supplied as a mpkg (metapackage) file inside a disk image. Open the mpkg to install.
SOFA mac installer
The same application now works much the same on Windows XP, Vista, and Windows 7, Ubuntu/Linux Mint, and Mac OS X Leopard and Snow Leopard.
SOFA controls on mac
Even if you are not a Mac user, there are a few extras in the new version:
- MySQL plugin now works with older versions of MySQL e.g. 4.1.
- MySQL now allows port details e.g. for remote connections.
- All database plugins allow empty default database or table configuration details (in which case they’ll use the first).
- Easier to fill in connection details in project. Tooltips (with examples) have been added to database configuration controls to make it more clear what is required from the user.
- Better placement of dialogs on smaller screens so nothing off-screen.
- Problems with connection details raise useful message for users.
- SOFA gives useful message if the SOFA application path is not found because of an unexpected folder name e.g. ‘sofa stats’ rather than ‘sofa’. The Windows installer was also upgraded to give better guidance.
- Better error messages if anything goes wrong with loading of initial images when starting SOFA.
There are also a few bug fixes:
- Fixed bug in calculation of total frequency across rows so that missing values are excluded.
- Background images for tables now available in Ubuntu install.
- Fixed some minor background bugs.
You can download version 0.9.10 now: SOFA downloads
NB 0.9.10 (zero point nine point ten) is newer than 0.9.9.
Posted in general | 2 Comments »
May 24th, 2010
SOFA Statistics has now reached the 10,000 download milestone on Sourceforge. And the rate of downloading has been steadily increasing. In May 2009 there were only 30 downloads whereas in May 2010 there are likely to be over 50 times that. The main area of work at the moment is creating a Mac package. Follow this blog or the twitter feed (http://twitter.com/sofastatistics) to stay informed.
10000 downloads!
Posted in general | No Comments »
May 1st, 2010
SOFA Statistics is growing in popularity, if SourceForge downloads are any indication.
Sourceforge Download Trend
Of course it is early days yet for the project, with version 1.0 not expected until later this year, but current trends are encouraging. It seems there is a gap in the market for an open source statistics, analysis, and reporting package aimed at non-specialist statisticians.
Posted in general | 2 Comments »
April 26th, 2010
R is an open source programming language and software environment for statistics. And it is not just any old programming language – it is the dominant system for open source statistics. So was it fair to call R an “epic fail” as Dr. AnnMaria De Mars did in her notorious blog post The Next Big Thing?
Clearly R has been a massive success and it has a vibrant and lively community, many of whom were galvanised into making a response by the Epic Fail blog (see An article attacking R gets responses from the R blogosphere – some reflections on the phenomenon and R and the Next Big Thing as an example). So on what terms could it be considered a failure? For De Mars, successful software will be usable by the vast majority of people – not just programmers and others comfortable with command line interfaces.
… if you even LOOK at R code – bug-free or not, compilable or not – it should be evident that this is not how the average person uses a computer. If we are talking about something that is going to be used by a large number of people, R is not it (Comment by De Mars on her own blog post – The Next Big Thing).
… If your target market is “People who own cars that drive from point A to point B” that is much BIGGER than “people who work on engines”. If you are looking for a job making things or selling things or providing services, the former is more likely to pay off for you than the latter.
Telling people that if they can’t appreciate an internal combustion engine they are too stupid to own a car probably won’t help, either.” (The Next Big Thing).
And in these terms, De Mars has a point. For many users, R needs a GUI. I like this quote tweeted by ravkalia (a big fan of R BTW): “Overheard at a computing meeting: ‘R is not a programming language, it’s a statistics package with the GUI missing.’” Of course there are various projects to provide a GUI interface for R but it can be argued there are limits to how far that can go given the inherent flexibility of R as an environment. Yihui Xie recently commented – “I prefer the command-line due to its flexibility. GUI cannot hold infinite components (buttons, drop-lists, check-boxes, …), whereas there are almost infinite possibilities in commands.” (r-is-an-epic-fail).
On her other points regarding R and data visualisation, and analysis of enormous quantities of unstructured data, De Mars is on shakier ground, but the observations about the mainstream preference for looking and clicking are valid.
So how does this relate to SOFA Statistics? SOFA stands for Statistics Open For All, which gives a strong hint as to where SOFA is aiming in terms of user interfaces and target audience. In practice this means:
- A simple GUI. In practice, this means trying hard to leave the right things out rather than adding in every possible option. Sometimes less is more. Think about your TV remotes.
Interface chaos
Some commentators have implied that a GUI is not important because the sorts of people who do statistics will also be comfortable with basic programming. But this is not always true. And lots more people, by several orders of magnitude, need to run basic statistical analyses than just specialist statisticians. Karen Grace-Martin put it especially well in her response to the Epic Fail post:
“I primarily help researchers, mainly in biology and social science, apply statistics to their research. They are not doing “business analytics,” do not have enormous databases, and really have no need to program anything beyond what SAS or SPSS syntax does. They are not programmers or statisticians, and they don’t have backgrounds in programming or math.
I believe they are the kinds of users of statistics that you are referring to and I agree with you wholeheartedly that they are probably the majority of statistics users and they have no need for a programming language. They don’t want to nor need to program new statistical procedures.
There are clearly people who do, but I agree they’re not the majority. At least not in the fields I work.” (The Next Big Thing).
Even full-time specialist statisticians may find it easier to use a simple GUI for basic data exploration e.g. generating simple frequency tables and cross tabs. It has been suggested that people should expect to use more than one package (SPSS, SAS, R, Stata, JMP? Choosing a Statistical Software Package or Two) SOFA Statistics may be a useful complement to R for many users.
And ease of use should not be premised on the assumption that people will be heavy users of the package – or of statistics in general, for that matter. The program needs to make it easy to become productive in a hurry.
- High priority on aesthetics. Output needs to look attractive; beautiful if possible.
Lucid spirals demo
Even the program itself needs to look good:
Form for selecting appropriate statistical test
- One True Way of Doing Things. It is not enough that there is a way of doing something – it can’t be buried somewhere obscure, and it has to clearly stand out as being correct and current (unlike some community technical advice).
* In the Zen of Python (type import this into your Python interpreter) there is this gem: “There should be one– and preferably only one –obvious way to do it.”
- Helping the user when errors occur. Ideally, there would never be any errors but given there are it is important to make them as useful as possible. This is an ongoing project in SOFA Statistics which is being given a high priority. Error messages are an important part of the interface and one of the most important to get right. The better the error messages, the less support people need and the happier they are (under the circumstances). Jon Peck commented on an unhelpful error message he receives from R:
Here is an error message that I get a lot from a popular R package.
‘Error in optim(0, f, control = control, hessian = TRUE, method = “BFGS”) :
non-finite finite-difference value [1]‘
I know what that means. Would an analyst?
(Jon Peck – in response to The Next Big Thing)
- Not relying on users to stitch together everything they need. Ordinary users benefit if their application bundles together related output. This is a balancing act and one which we want to get right for the target user group for SOFA Statistics. The following quote captures the tradeoffs well:
But one thing is clear to me: R aims at people who know what they are doing. Absolutely. You can see this with standard output in R which is very minimalistic. You must ASK R what you want from it. SAS and SPSS put everything out. And therefore you need to know how to program in R to use it, really. But if you do, you feel bound and limited with SAS or SPSS. (comment by mocianmomo in response to SAS v. R: Ease of learning).
- SOFA Statistics uses Python for Scripting. Python is a language consciously designed to be easy to learn. Many statisticians find it a pleasure to work with Python but the same is not always true of the syntax of many statistics packages, especially those with lots of historical cruft.
Example SOFA script in Python
Posted in general, python, statistics | 5 Comments »
April 24th, 2010
Although this latest release has many, many enhancements and fixes (full details below), the most pleasing change for most users will probably be the availability of some attractive new output styles. It is also easier to select and apply them.
The 3 new styles are:
- Grey Spirals
Grey spirals demo
- Lucid Spirals
Lucid spirals demo
- Pebbles
Pebbles demo
Output styles can be selected using a simple drop-down list.
Style selection
The full list of new features is:
- Added 3 attractive output styles – grey_spirals, lucid_spirals, and pebbles. These include background images.
- Styles can be selected and changed much more easily.
- Crosstabs with column totals and row percentages get a frequency in the total column even if frequency is not selected.
- Customised waiting message when making report tables according to what is required e.g. “Add and configure column”.
- Configuration settings (e.g. preferred output style) persist across all dialogs.
- The last configured row summary measures are the default for any new variables added to the row summary report table.
- The Expand button opens a much larger window to view content in.
- Better display of html messages in report tables dialog.
- Simplified standard names e.g. “SOFA_Default_db” became “sofa_db”.
- Can resize some dialogs smaller than initial display size.
- Application gives a useful error message, even if it fails very early.
- Faster production of report tables by avoiding duplicate queries.
- When adding row or col vars, can double click selections except for raw display tables or rows of row summary tables where still multichoice selections.
- Removed windows manager close buttons from dialogs so program close buttons used instead.
- Windows installer now installs Python 2.6.5 instead of 2.6.2.
- Misc UI changes to make setting up MS connection details easier.
There have also been numerous bug fixes:
- Debian package now uses desktop icon sofastats.desktop rather than sofa.desktop to prevent collision with sofa-apps.
- Fixed bug when selecting default project to edit after having selected another. Would set proj dropdown to point to the first proj (more generally, the last one that was saved and was not read-only).
- No longer possible to overwrite the default project with another of the same name.
- In the report table dialog, the Add Under button for columns only shows when a column variable has already been selected.
- Add Under button for rows disabled for Row Summaries.
- Fixed bug where right clicking on a row or column variable in the report tables dialog tree didn’t shift selection to it.
- Fixed bug where selection in report tables dialog row/column tree should shift to another item sibling but went nowhere.
- Fixed bug if MS Access database selected multiple times. Needed to properly clear resources before getting them again.
- Fixed bug running some statistical tests when variable includes % in name.
- Fixed bug where unable to change cell values in strangely named tables e.g. ‘demo;’.
- Fixed bug where some faulty values for DataDets would get through even if an error.
- When faulty database selection made, e.g. MS SQL Server model, reverts to last selected database.
Posted in general | No Comments »
