[Bf-committers] Blender roadmap article on code blog

[Vincent Akkermans]

On Monday, 1 July 2013 at 20:21, David Jeske wrote:
> On Sat, Jun 29, 2013 at 7:41 AM, Ton Roosendaal <ton at blender.org (mailto:ton at blender.org)> wrote:
>  
> > > 1) Include opt-in usage and automatic crash reporting in *every* blender
> > > build, and a web dashboard to live usage/crash stats to devs and the
> > > community.
> > >  
> >  
> >  
> > I always wondered what other projects/companies do with such reports. Is
> > there a public example where I can see the handling of such report logging,
> > with evidence for how this is being used well, the benefits? Or is it just
> > for statistics?
> >  
>  
>  
> Billrey's blender blog recently posted a link to this old but excellent
> talk on Firefox UI design, from 2011. At 38 minutes, it talks about their
> Usage Statistics Heat Map, and how they used it to streamline the option
> locations when redesigning menus.
>  
> http://www.youtube.com/watch?v=hMDBwa4huUY#at=302
>  
> The whole talk is really worth watching.. it's both an interesting
> visualization of some common UI design principles, and Firefox's approach
> to enabling open-source innovation by establishing clear design-rules to
> empower and enable more freedom within those design rules.
>  
>  

There are a few other examples that I know of.  

Firstly, there are some good papers from Autodesk Research that talk in detail about how they use data from their Customer Involvement Program (read: usage data collection) to prototype and verify new features. A recent work is Patina [1], where they use heatmaps, like in the Firefox talk by Faaborg. However, they present these heatmaps to the user so that the user becomes aware of the features that other people (from the same community) use. This allows them to discover features of complex software. Another example is CommunityCommands [2] in which a user is profiled through their command usage and by means of a collaborative filtering technique new commands are recommended to the user, based on the type of usage. Chronicle [3] is worth mentioning as well, because it uses this type of data to allow the user to query and retrieve information from the design process of a document.  

Then there's work by Terry [4] who instrumented The Gimp to collect usage data with an open source ethos. The analysis of it isn't fully explored though.

There's a good overview of how to employ usage data in usability studies by Hilbert and Redmiles [5]. Although the paper is from quite a while ago it introduces many aspects one has to think about when doing this kind of usability work.

Another good example is from Heer and others [6] where they describe how they used usage data to understand user behaviour and manage to turn this into concrete improvements of their visualisation software.

Microsoft also collects usage data [7], but I haven't found (also haven't looked too hard) many details.

I personally think there is a lot of value in remotely collected usage data for a) users, as can be seen from the Autodesk research, and b) developers, as can be seen from [7].

Apologies for the barrage of references.

Cheers,

Vincent

  
[1] http://autodeskresearch.com/publications/patina
[2] http://autodeskresearch.com/publications/communitycommands
[3] http://autodeskresearch.com/publications/chronicle
[4] Terry, M., Kay, M., Van Vugt, B., Slack, B., and Park, T. Ingimp: introducing instrumentation to an end-user open source application.  
[5] Hilbert, D.M. and Redmiles, D.F. Extracting usability information from user interface events. ACM Computing Surveys (CSUR) 32, 4 (2000), 384–421.
[6] Heer, J., Mackinlay, J., Stolte, C., and Agrawala, M. Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation. Visualization and Computer Graphics, IEEE Transactions on 14, 6 (2008), 1189–1196.
[7] https://www.microsoft.com/products/ceip/EN-US/default.mspx