The following dimensions of evaluation are a subset of the dimensions from the Cognitive Dimensions of Notations (CDs) framework which is “an approach to analysing the usability of information artefacts”. CDs has been used in a variety of papers to analytically investigate the usability of programming language features or an API. Often, CDs is only applied insofar as it makes sense for a particular information artifact. That is, some of the 14 dimensions are left out and some new are added possibly. In this way, CDs can be taken as a basis for the evaluation of different solutions to the 7GUIs benchmark. The following dimensions are thus a recommended subset of CDs which turned out to work well for the analysis of two different approaches to 7GUIs.
The list of dimensions is a recommendation to make it easier to get started with an analysis between different approaches to 7GUIs. Of course, you are free to use your own criteria as you see fit.
Does the system provide any way of defining new terms within the notation so that it can be extended to describe ideas more clearly? Can details be encapsulated? Does the system insist on defining new terms? What number of new high-level concepts have to be learned to make use of a system? Are they easy to use and easy to learn?
Each new idea is a barrier to learning and acceptance but can also make complex code more understandable. For example, Java Swing, the predecessor to JavaFX, employs a variation of the MVC design pattern in its general architecture and in particular for each of its widgets. Such being the case, there is a significant learning requirement to using the widgets reasonably well and often much boilerplate involved (“the system insists on defining new terms”) which does not pay off for simple applications. On the other hand, for very complex applications the MVC-architecture may make the code more understandable and manageable as details can be encapsulated in the new terms “Model, View and Controller”.
Another example is a function. A function has a name and, optionally, parameters as well as a body that returns a value following certain computational steps. A client can simply refer to a function by its name without knowing its implementation details. Accordingly, a function abstracts the computational process involved in the computation of a value. The learning barrier to the principle of a function is not great but it can still make a lot of code much more understandable by hiding unimportant details.
How closely related is the notation to the result it is describing resp. the problem domain? Which parts seem to be a particularly strange way of doing or describing something?
An example is the layout definition of a GUI. Languages that do not provide a way to describe the layout in a nested resp. hierarchical manner, and as such force the programmer to “linearize” the code with the introduction of meaningless temporary variables, make it hard to see how the structure of the layout definition relates to the resulting layout of the application. Not for nothing are XML-based view specifications widespread for GUI-toolkits in languages without native support for hierarchical layout expressions.
Are dependencies between entities in the notation visible or hidden? Is every dependency indicated in both directions? Could local changes have confusing global effects?
If one entity cites another entity, which in turn cites a third, changing the value of the third entity may have unexpected repercussions. The key aspect is not the fact that A depends on B, but that the dependency is not made visible. A well-known illustration of a bad case of Hidden Dependencies is the fragile base class problem. In (complex) class hierarchies a seemingly safe modification to a base class may cause derived classes to malfunction. The IDE in general cannot help discovering such problems and only certain programming language features can help preventing them. Another example are non-local side-effects in procedures, i.e. the dependencies of a procedure with non-local side-effects are not visible in its signature.
To what extent does the notation influence the likelihood of the user making a mistake? Do some things seem especially complex or difficult (e.g. when combining several things)?
In many dynamic languages with implicit definitions of variables a typing error in a variable name can suddenly lead to hard to find errors as the IDE cannot always point out such an error due to the language’s dynamicity. Java’s different calling semantics for primitive and reference types may lead to mistakes if the programmer mixes them up. Implicit null-initialization of variables can lead to null-pointer exceptions if the programmer forgets to correctly initialize a variable before its use.
How many symbols or how much space does the notation require to produce a certain result or express a meaning? What sorts of things take more space to describe?
Some notations can be annoyingly long-winded, or occupy too much valuable “real-estate” within a display area. In Java before version 8 in order to express what are lambdas today anonymous classes were employed. Compared to Java 8’s lambdas these anonymous classes used to be a very verbose way of encoding anonymous functions especially when used in a callback-heavy setting like traditional GUI programming.
Are there any inherent barriers to change in the notation? How much effort is required to make a change to a program expressed in the notation?
A viscous system needs many user actions to accomplish one goal. Changing the return type of a function might lead to many code breakages in the call sites of said function. In such a case an IDE can be of great help. Creating a conceptual two-way data-binding by means of two callbacks involves more repetition than a more direct way to define such a dependency.
This part is not so much a dimension but a place to mention everything else which is noteworthy and to give a conclusion. For instance, general observations that do not fit into the above dimensions, impressions during the development process, efficiency concerns of the resulting code and potential improvements can be addressed. In addition, the responsibilities of the other dimensions’ results are assigned to the paradigm, language, toolkit and the IDE.