DTU Ph.D.-course (02930)

Analysis of Sensory and Consumer Data

at the

Technical University of Denmark,

Lyngby, Denmark


The course will NOT run in 2017 and there are currently no plan for when or if the course will run again




Organized by:


Per Bruun Brockhoff, DTU Compute

Technical University of Denmark





Material and lecture podcasts from the 2015 version are available below!!

If you would like to pursue some of this further:
  • Get support for the planning and analysis of your experiments
  • Get dedicated courses
  • Co-finance collaborative research projects e.g. Ph.D projects
  • Get support for dedicated implementations of some of the methodology - e.g. by personalized R Shiny web applications meeting your needs
  • Etc

Don't hesitate to contact us:
Per Bruun Brockhoff , Email: perbb@dtu.dk

Our Statistics and Data Analysis section here at DTU Compute has a large experience and interest in entering collaborative contracts - as well pure research contracts as research based consultancy contracts

2015 Material overview and access: (See podcast link table below - OR go directly to raw podcast channel )

Block Topic Slides R and Data Reading Material Exercises Solutions
Mon1 Intro to R

Discrimination basic, part 1

sensR_part1_examples.R eNote1fromCourse02402 (with R intro)


Chapter 7 from the book by Næs, Brockhoff and Tomic.

sensR_part1_exercises.pdf sensR_part1_exercises_AND_solutions.pdf
Mon2 Discrimination basic, part2

Similarity testing
Replicated data
sensR_part2HA.pdf sensR_part2.R The statistical power of Replications, FQP 2003

sensR Vignette on Methodology

The Design of Replicated Difference Tests, Meyners & Brockhoff (2003).
sensR_part2_exercises.pdf sensR_part2_solutions.R
Tue1 Intro to mixed models

Basic ANOVA in Panelcheck


eNote-1 from Course 02429: Introduction to mixed models

Selections from the book by Næs, Brockhoff and Tomic.
mixedmodelsinPanelcheck_exercises.pdf mixedmodelsinPanelcheck_exercises_and_solutions.pdf
Tue2 Mixed models in Consumercheck

SensMixed, part 1
SensMixed_ConsumerCheck_2015.pdf Ham_consumer_attributes.txt


Ph.D. thesis, Kuznetsova, A.(2015)

FQP Paper draft on the SensMixed package. (Kuznetsova et al)

JSS paper draft on ConsumerCheck (Tomic et al)

Exercises_Tuesday_afternoon.pdf Exercises_Tuesday_solutions.pdf
Wed1 PCA and Preference mapping in ConsumerCheck ConsumerCheck_PCA_2015.pdf Data_1_liking.xlsx

JSS paper draft on ConsumerCheck (Tomic et al)

eNote2 from Course 27411 - PCA intro with R
Exercises_Wednesday_morning.pdf Exercises_Wednesday_morning_solutions.pdf
Wed2 PCA, Tucker1 and panel performance in PanelCheck PanelCheck_August2015HA.pdf TV2.xls The book by Næs, Brockhoff and Tomic. Panelcheck_exercises.pdf
Thu1 Same-difference testing

Anot-A testing

Samediffpaper, FQP 2009.

Thurstonian models as GLMs, FQP paper, 2010
exercises_anota.pdf exercises_anota_solutions.pdf
Thu2 d-prime ANOVA

Likelihood methods

Ordinal data analysis




Statistical and Thurstonian models for the A-not A protocol with and without sureness, FQP 2011

Analysing Sensory ratings by CLMs. Christensen and Brockhoff, 2013

sensR Vignette: 2ACexamples

ordinal: Reference Manual

ordinal Vignette: Methodology

ordinal Vignette: tutorial on clm

ordinal Vignette: tutorial on clmm2
exercises_thursday_anova.pdf exercises_thursday_anova_solutions.pdf
Fri1 The MAM (scaling correction)

d-tilde plotting in ANOVA

Sensmixed, part 2

SensMixedTutorial.R tutorialSensMixed.pdf

d-tilde plotting paper, Brockhoff et al (2015)

Ph.D. thesis, Kuznetsova, A.(2015)

Automated Mixed ANOVA, FQP 2015, Kuznetsova et al.

The MAM paper, Brockhoff et al (2015)

The MAMCAP paper, Peltier et al (2014)

The Original Assessor model paper, Brockhoff (1994)

FQP Paper draft on the SensMixed package. (Kuznetsova et al)

Fri2 The lmerTest package lmerTest_summerschool_2015.pdf lecture_lmerTest_Rcode.R


Ph.D. thesis, Kuznetsova, A.(2015)

lmerTest Reference Manual


JSS paper on the lmerTest package, Kuznetsova et al(2015).

2015 Podcast link table:

Block Topic Podcasts
Mon1 Intro to R
Discrimination basic, part 1
Course software-an overview - Including a small R intro (13min)

The sensR package - An overview of package features (15min)

Basic sensory difference testing - And the first use of sensR (23min)

Five basic discrimination protocols - And recommendations for protocol choice (7min)

Proportion discriminators interpretation - And using sensR to jump between observed and discriminator interpretation (13min)

The Thurstonian model - d-primes, resolving Gridgeman's paradox and how to get it from sensR (23min)

What is the interpretation of the d-prime? As opposed to just looking at the p-value (4.5min)

Power and sample size for discrimination testing - And how to find these with sensR (26min)

Power and the five discrimination test protocols - And "first" versus "stable exact" sample sizes (7min)
Mon2 Discrimination basic, part2
Similarity testing
Replicated data
Similarity testing - And how to do it with sensR (23min)

Power and sample size for similarity testing - And how to do find these with sensR (10min)

Replicated difference testing - What is the challenge? Simulations and analysis by corrected beta-binomial with sensR (29min)
Tue1 Intro to mixed models
Basic ANOVA in Panelcheck
Mixed model software by the DTU Sensometrics group - An brief overview of PanelCheck, ConsumerCheck, lmerTest, SensMixed, sensR and ordinal (14min)

Introduction to mixed models - Motivating mixed models and explaining what they can do for you (14min)

The three ANOVA options in PanelCheck - A brief overview (16min)

The basic single observation 2-way ANOVA - And the link to paired t-testing (12min)

Two-way ANOVA with replications - Panelist-by-product interaction and the concept of a random effect (23min)

The proper product difference F-test - And the relation to the standard fixed effect ANOVA table (2min)

ANOVA with PanelCheck - A brief tutorial (9min)

The 3-way mixed model ANOVA in PaneclCheck - Taking also session and/or product batch effects into account (10min)

Tue2 Mixed
models in Consumercheck
SensMixed, part 1
SensMixed: The Shiny version. A tutorial for how to analyze more generally structured multi-attribute sensory data by the Shiny browser interface of the SensMixed package (29min)

Conjoint analysis in ConsumerCheck - A tutorial (20min)

Wed1 PCA and Preference mapping
in ConsumerCheck
PCA and Preference mapping in ConsumerCheck (26min)

Wed2 PCA, Tucker1 and panel performance
in PanelCheck
PanelCheck: What is it and what can you use it for - An overview (13min)

PanelCheck: Data structure and data import - And what about missing values? (4min)

PanelCheck: Workflow - How to use the various features of PanelCheck in your work with sensory panels (7min)

PanelCheck: Tucker 1 plots - How to investigate multivariate (dis)agreement between assessors (14min)

PanelCheck: Manhattan plots - Visualizing attribute wise explained variances from individually assessor-wise performed PCAs (15min)

PanelCheck: Export of plots - Very brief (1min)

PanelCheck: Plot based one individual one-way ANOVAs - F plots, MSE plots and p*MSE plots (10min)

PanelCheck: Statis - A few words: Weighted PCA based on agreement weights (7min)

Thu1 Same-difference testing
Anot-A testing
The A-not-A test - Procedure, Thurstonian model and sensR-tutorial (7min)

The same-different test - Procedure, Thurstonian model and sensR-tutorial (20min)

Measures of sensitivity - AUC (area under curve), overlap and d-primes. And how to with sensR and ordinal (12min)

The A-not-A with sureness protocol - And the soup data example and a little ordinal tutorial (23min)

Thu2 d-prime
Likelihood methods
Ordinal data analysis
Likelihood based confidence intervals - Motivation and examples (11min)

Likelihood based confidence intervals - The likelihood function and principle. (11min)

Likelihood based confidence intervals - Examples revisited (2min)

Likelihood based confidence intervals - Perspectives and more discussion based on sensR output (13min)

One-way ANOVA for d-primes - Principles, maximum likelihood and sensR tutorial. (16min)

Common multi-protocol d-prime - Estimating and testing the common d-prime (discrimination and/or similarity) (3min)

ANOVA for d-primes - Post-hoc tests and comparisons (21min)

The ordinal package - An overview of different types of ordinal data (15min)

Ordinal data analysis - The cumulative link model and the clm function of the ordinal package (8min)

Ordinal data analysis - Extensions of the clm-model, e.g. including differences of product variances (7min)

Ordinal data analysis - Mixed model extensions of clms (4min)

Ordinal data analysis - A few short examples (8min)

Fri1 The MAM (scaling correction)
d-tilde plotting in ANOVA
Sensmixed, part 2
MAM: Scaling correction of sensory data - Motivation, background and Sensobase meta investigation (26min)

MAM: Scaling correction of sensory data - Basis versus covariate version. Conditional and adjusted versions (4min)

MAM: Scaling correction of sensory data - What can R (SensMixed) do for you - including MAM-CAP table information (13min)

d-prime like interpretations of mixed model output - An improved alternative to multi-attribute F-barplots (23min)

MAM and d-prime in SensMixed - presentation on how to correct for the scaling effect and plot d-prime plots in SensMixed (22min)

Fri2 The lmerTest package

Course Objectives

To improve the ability of analysing human perception data. Some of the newest statistical methodologies will be covered using the open source software R. The packages sensR, ordinal, SensMixed and lmerTest will be used together with the free software packages PanelCheck and ConsumerCheck.


Learning Objectives

A student who has met the objectives of the course will be able to:


·        Work with R, Panelcheck and ConsumerCheck

·        Plan and analyze simple discrimination and similarity experiments using sensR

·        Analyze replicated discrimination data

·        Perform and understand simple Thurstonian modeling

·        Use mixed models for sensory profile data and consumer preference data by PanelCheck and ConsumerCheck

·        Analyze sensory profile data with the newest scale correction method (using R)

·        Use PanelCheck and ConsumerCheck for simple analysis as well as multivariate analysis (including Tucker-1)

·        Use the R-packages lmerTest and SensMixed to analyze non-standard data with mixed models

·        Analyze ordinal human perception data using the R-package ordinal

·        Visualize ANOVA results by the newest delta-tilde method using the SensMixed R shiny application

·        Know about the newest sensometrics research going on at the DTU Sensometrics group



The course is aimed at Ph.D-students within non-statistical areas such as sensory science, food science, marketing etc. with interest in data analysis and statistics. It is also well suited for sensory practitioners from industry and scientific institutions or for Ph.D. students within Statistics/Data analysis with interest in human perception data.


All lectures and activities will be given in English.



Professor Per Bruun Brockhoff, B324, R220, (+45) 2044 1711, perbb@dtu.dk



Programme, overview


·       Monday: Basic discrimination testing using R (package sensR)

·       Tuesday: Basic mixed models using PanelCheck, ConsumerCheck and R (package SensMixed).

·       Wednesday: Multivariate Analysis using PanelCheck and ConsumerCheck

·       Thursday: More advanced discrimination testing using R (replicates etc)(packages sensR and ordinal)

·       Friday: More Advanced mixed models using R-package lmerTest (and lme4)


We begin Monday morning at 9am and finish Friday at 3pm.



Professor Per Bruun Brockhoff

DTU Compute, Technical University of Denmark



Study Material:

1.     Brockhoff (2011), Sensometrics. In: International Encyclopedia of Statistical Science, Lovric, Miodrag (Ed.), Springer.

2.     Brockhoff P. B., Amorim I., Kuznetsova A., Søren Bech, Lima R. R. (2016). Delta-tilde interpretation of standard linear mixed model results, Food Quality and Preference, Vol 49, 129-139.

3.     T. Næs, P.B. Brockhoff and O. Tomic, (2010). Statistics for Sensory and Consumer Science, John Wiley & Sons.

4.     Brockhoff, P.B. and Christensen, R.H.B. (2010). Thurstonian models for sensory discrimination tests as generalized linear models. FQP, 21(3), 330-338.

5.     Brockhoff, P. B., Schlich, P., & Skovgaard, I. (2015). Taking individual scaling differences into account by analyzing profile data with the Mixed Assessor Model. FQP, 39, 156-166.

6.     Christensen, R. H. B., & Brockhoff, P. B. (2013). Analysis of sensory ratings data with cumulative link models. J. Soc. Fr. Stat. & Rev. Stat. App., 154(3), 58-79.

7.     Christensen, R.H.B. Cleaver, G. and Brockhoff, P.B. (2011). Statistical and Thurstonian models for the A-not A protocol with and without sureness, FQP 22(6), 542-54.

8.     Kuznetsova, A., Christensen, R.H.B., Bavay, C. and Brockhoff, P.B. (2015). Automated mixed ANOVA modeling of sensory and consumer data. FQP 40 (2015) 31-38


We will supplement with a number of guides/tutorials and newest papers.

Social activity

Coffee for coffee breaks, lunches and one dinner at a restaurant in Copenhagen is provided for everyone.  



You can find information on Visit Copenhagen's website.



Students are required to bring their own laptops for the computer exercises.


R software

We will use the software R throughout the course, and we ask all participants to install R on their laptop in advance of the course. We also ask that you install the graphical user interface (GUI) / Integrated development environment (IDE) RStudio. If you are a confident R user and acustumed to a different GUI, feel free to use that one (though we strongly discourage the standard GUI that comes with R). Both R and RStudio are open source and free of charge.


To install R go to http://cran.r-project.org/ and follow the installation instructions.

To install RStudio go to http://www.rstudio.com/products/rstudio/download/ and choose according to your operating system.

For further assistance, see the following short intro (with relevant links) to get started with R and RStudio:

R introduction - get started (incl. Rstudio). (New 2015 )(13 min)

If you want to watch a few very short videos on how to get started with
R then you can go to this Youtube Playlist:

(A number of short videos illustrating the use of R - shown within a Mac version,
but apart from the appearance of the sub-windows exactly the same works for RStudio
in any operating system)


Basic Statistics Brush-up??

We will assume that you have a background with (at least) an introductory statistics class. If p-values or the idea of confidence intervals etc is a  bit far away for you, you can find topic wise categorized links to podcasts on all topics of an introductory statistics class here:


You can also find a complete online textbook on statistics:



More advanced statistics
In the course we will be using the concepts of Mixed Models and also Principal Component Analysis. We do not assume that you are educated in these topics – so we will teach you as we go along. But having some idea of what these things are in advance could probably increase your outcome of the course:

Mixed models:

At the website of DTU course 02429:


you will find a complete online textbook on applied linear mixed models, and also a complete collection of small podcasts taking you through the course. You will find some introductory lectures there, that you might spend some minutes viewing.


At the website of DTU course 27411:


you will find some podcasts on e.g. Principal Component Analysis.  

Looking very much forward to welcoming you in Wonderful Copenhagen!