Models for Quasi-Objective Rating of Digital Samples

A project for researching and developing quasi-objective models to perform a human-like rating of digital samples, more specifically, handwriting samples. The project covers data collection via observational studies, theory-building of machine learning models, system development of prototypes, and finally, a real-world application experiment. The project is currently in an initial phase of data collection.

Simultaneously doing research in machine learning and trying to learn Chinese, I came up with the idea of making an app that would help me training writing Chinese characters based on machine learning. It turned out to be far from trivial. The aesthetics of handwriting is an intrinsically subjective matter. However, most people would still agree about whether a particular handwriting sample was “good” or “bad,” which makes handwriting a good case study. Machine learning models to rate handwriting will at best be quasi-objective, meaning they have high external validity in that they align with the consensus of a large number of real humans.

This Ph.D. project is aimed at establishing data sets and researching quasi-objective machine learning models based on these data sets, for rating handwriting in real-time, to provide a user with immediate feedback on his handwriting.

During this project, two data sets will is collected and analyzed:

(1) an augmentation to the MNIST data set with real human ratings to establish a ground truth to train, evaluate, and test models

(2) a data set of the handwriting of Chinese characters with a temporal factor, that is, time-series of the strokes.

For each of these data sets, the following activities will take place:

(1) Analysis of how data sets should be collected and analyzed. (2) Collection of data sets.
(3) Theory building of machine learning models
(4) Development of prototype models

The two major activities in this project are generating data sets to establish a ground truth and the research and development of models around these data sets. A multi-methodological approach, as described by Nunamaker et al*., is therefore taken.

Observation, and more specifically, survey studies, are used to generate data sets. These data sets support theory building in form of machine learning models, and systems development of prototypes* (see figure).

Finally, a field experiment* will implement the selected best model into an app, that can be used by real users. App development is planned to happen in collaboration with a third party.