SOFA Statistics and the “R is an Epic Fail” blog

R is an open source programming language and software environment for statistics. And it is not just any old programming language – it is the dominant system for open source statistics. So was it fair to call R an “epic fail” as Dr. AnnMaria De Mars did in her notorious blog post The Next Big Thing?

Clearly R has been a massive success and it has a vibrant and lively community, many of whom were galvanised into making a response by the Epic Fail blog (see An article attacking R gets responses from the R blogosphere – some reflections on the phenomenon and R and the Next Big Thing as an example). So on what terms could it be considered a failure? For De Mars, successful software will be usable by the vast majority of people – not just programmers and others comfortable with command line interfaces.

… if you even LOOK at R code – bug-free or not, compilable or not – it should be evident that this is not how the average person uses a computer. If we are talking about something that is going to be used by a large number of people, R is not it (Comment by De Mars on her own blog post – The Next Big Thing).

… If your target market is “People who own cars that drive from point A to point B” that is much BIGGER than “people who work on engines”. If you are looking for a job making things or selling things or providing services, the former is more likely to pay off for you than the latter.
Telling people that if they can’t appreciate an internal combustion engine they are too stupid to own a car probably won’t help, either.” (The Next Big Thing).

And in these terms, De Mars has a point. For many users, R needs a GUI. I like this quote tweeted by ravkalia (a big fan of R BTW): “Overheard at a computing meeting: ‘R is not a programming language, it’s a statistics package with the GUI missing.’” Of course there are various projects to provide a GUI interface for R but it can be argued there are limits to how far that can go given the inherent flexibility of R as an environment. Yihui Xie recently commented – “I prefer the command-line due to its flexibility. GUI cannot hold infinite components (buttons, drop-lists, check-boxes, …), whereas there are almost infinite possibilities in commands.” (r-is-an-epic-fail).

On her other points regarding R and data visualisation, and analysis of enormous quantities of unstructured data, De Mars is on shakier ground, but the observations about the mainstream preference for looking and clicking are valid.

So how does this relate to SOFA Statistics? SOFA stands for Statistics Open For All, which gives a strong hint as to where SOFA is aiming in terms of user interfaces and target audience. In practice this means:

  1. A simple GUI. In practice, this means trying hard to leave the right things out rather than adding in every possible option. Sometimes less is more. Think about your TV remotes.
    Interface chaos

    Interface chaos

    Some commentators have implied that a GUI is not important because the sorts of people who do statistics will also be comfortable with basic programming. But this is not always true. And lots more people, by several orders of magnitude, need to run basic statistical analyses than just specialist statisticians. Karen Grace-Martin put it especially well in her response to the Epic Fail post:

    “I primarily help researchers, mainly in biology and social science, apply statistics to their research. They are not doing “business analytics,” do not have enormous databases, and really have no need to program anything beyond what SAS or SPSS syntax does. They are not programmers or statisticians, and they don’t have backgrounds in programming or math.

    I believe they are the kinds of users of statistics that you are referring to and I agree with you wholeheartedly that they are probably the majority of statistics users and they have no need for a programming language. They don’t want to nor need to program new statistical procedures.

    There are clearly people who do, but I agree they’re not the majority. At least not in the fields I work.” (The Next Big Thing).

    Even full-time specialist statisticians may find it easier to use a simple GUI for basic data exploration e.g. generating simple frequency tables and cross tabs. It has been suggested that people should expect to use more than one package (SPSS, SAS, R, Stata, JMP? Choosing a Statistical Software Package or Two) SOFA Statistics may be a useful complement to R for many users.

    And ease of use should not be premised on the assumption that people will be heavy users of the package – or of statistics in general, for that matter. The program needs to make it easy to become productive in a hurry.

  2. High priority on aesthetics. Output needs to look attractive; beautiful if possible.
    Lucid spirals demo

    Lucid spirals demo

    Even the program itself needs to look good:

    Form for selecting appropriate statistical test

    Form for selecting appropriate statistical test

  3. One True Way of Doing Things. It is not enough that there is a way of doing something – it can’t be buried somewhere obscure, and it has to clearly stand out as being correct and current (unlike some community technical advice).

    * In the Zen of Python (type import this into your Python interpreter) there is this gem: “There should be one– and preferably only one –obvious way to do it.”

  4. Helping the user when errors occur. Ideally, there would never be any errors but given there are it is important to make them as useful as possible. This is an ongoing project in SOFA Statistics which is being given a high priority. Error messages are an important part of the interface and one of the most important to get right. The better the error messages, the less support people need and the happier they are (under the circumstances). Jon Peck commented on an unhelpful error message he receives from R:

    Here is an error message that I get a lot from a popular R package.
    ‘Error in optim(0, f, control = control, hessian = TRUE, method = “BFGS”) :
    non-finite finite-difference value [1]‘
    I know what that means. Would an analyst?
    (Jon Peck – in response to The Next Big Thing)

  5. Not relying on users to stitch together everything they need. Ordinary users benefit if their application bundles together related output. This is a balancing act and one which we want to get right for the target user group for SOFA Statistics. The following quote captures the tradeoffs well:

    But one thing is clear to me: R aims at people who know what they are doing. Absolutely. You can see this with standard output in R which is very minimalistic. You must ASK R what you want from it. SAS and SPSS put everything out. And therefore you need to know how to program in R to use it, really. But if you do, you feel bound and limited with SAS or SPSS. (comment by mocianmomo in response to SAS v. R: Ease of learning).

  6. SOFA Statistics uses Python for Scripting. Python is a language consciously designed to be easy to learn. Many statisticians find it a pleasure to work with Python but the same is not always true of the syntax of many statistics packages, especially those with lots of historical cruft.
  7. Example SOFA script

    Example SOFA script in Python

5 Responses to “SOFA Statistics and the “R is an Epic Fail” blog”

  1. Tal Galili says:

    BTW, regarding “R is not a programming language”, I started a SO thread on this.
    People so “un-liked” it that the thread got closed:
    http://stackoverflow.com/questions/2707887/why-might-someone-say-r-is-not-a-programming-language-closed

  2. admin says:

    A good friend of mine is a big fan of R and so I have the greatest respect for it. A piece of trivia – I was born in the same small town (Waiuku – population at the time about 1500) that Dr Ross Ihaka comes from http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10551243. R and SOFA Statistics are focused on very different things. Another person I know uses both, albeit for different parts of the same research projects.

  3. thomas says:

    It’s unfortunate that the leading screenshot of the program looking good is also of it doing evil.

    The test-choosing wizard implies that you will know whether your data have a Normal distribution and should use this knowledge to decide whether to compare means of two groups or do a Mann-Whitney U test. But you won’t and you shouldn’t.

    Firstly, people who want to use the Mann-Whitney U test because of its resistance to outliers should always use it — you can’t reliably know that your data are Normal enough to prefer the t-test. After all, the reason you are doing the test is that you don’t even know if the means are different, so it’s unlikely you know the whole distribution accurately, and if the data are exactly from a Normal distribution the Mann-Whitney U test performs very well.

    Secondly, even non-Normal distributions have means. These are well-estimated by sample means (the Law of Large Numbers) and have a Normal distribution except in very small samples (the Central Limit Theorem). If you want to compare the means, you can use the t-test unless the sample size is tiny.

    Thirdly, the Mann-Whitney U test also has assumptions, which don’t go away with large samples, and strange things can happen if they are violated. For an example, Google for “efron non-transitive dice’ and think about what how the U statistic works.

  4. Thanks for the feedback Thomas. I take on board what you have said and some extra work will go into the Statistical Test Selection Dialog to ensure the correct balance is struck between providing simple advice to beginners and recognising the multitude of factors which may be relevant to making a decision. I am currently working on version 0.9.13 and it will provide better guidance when selecting either the independent t-test or the Mann-Whitney U.

  5. The screen shot of the Statistical Test Selector has been updated to show the new Tips section. One role of tips is to add caveats to the (first cut) guidance provided and to inform the user of what they can do to evaluate the suggested choice. The intention is to provide more accurate guidance to users (the “learn as you go” goal of SOFA) without unnecessarily confusing users (which would compromise the “ease-of-use” goal).