Work package 4: Standardization of manual scoring

Lead: Reykjavik University

The focus of WP4 is on the standardisation of current sleep study scoring and the development of state-of-the art machine learning solutions and a standardised European Sleep Scoring Manual for the scoring of the sleep studies. 

Scoring of biosignals during sleep, whether collected at home or in a laboratory, is ideally performed by trained sleep technologists following guidelines published by the American Academy of Sleep Medicine (AASM). 

There are significant variations across European sleep centres, which not only use different versions of the AASM’s sleep scoring guidelines, but in some cases have developed their own definition. This reflects a lack of agreed international standards on scoring as only the American standards are currently available as a reference manual. Additional the manual scoring process is expensive and laborious taking 2-3 hours for each sleep recording. The scoring also does not account for most of the information embedded in the recorded signals but simplistically reduces the information into one index, the apnea-hypopnea index (AHI) that simply counts the number of breathing events per hour, not their length or consequences. 

In this tasks, 10 certified expert sleep technologists from Iceland, Portugal, Germany, Finland, Italy, Sweden and Australia were appointed to a scoring group under the supervision of an RU expert sleep technologist.  

These 10 sleep technologists score the same 100 sleep studies using the same standardised AASM scoring rules to train and validate the automatic scoring. 

All scored events will be annotated as positive/uncertain/negative to find grey areas in manual scoring to further understand the interscorer reliability that exists in manual scoring and refine the current manual scoring rules. 

An event-by event agreement between expert scorers will be performed to highlight the areas of uncertainty for manual scoring. The uncertain areas in manual scoring will be discussed in detail by the scoring team to reach a consensus and suggestions made for updated scoring rules to increase scoring agreement.  

This forms a solid basis for machine learning algorithms to be developed. As the machine learning algorithms will be trained to score sleep studies using manually scored dataset, it is essential to have well-annotated sleep recordings. Also, by using the machine learning other signals can be used in addition of what the sleep technologist looks at in manual scoring.  

Half of the 100 studies are traditional polysomnography (PSG) sleep studies, and the other half are self-applied somnology (SAS) sleep studies where the participants sleep at home.    

The PSG studies must be prepared by a sleep technologist and it take around 1 to 1.5 hours with preparation and cleaning. 

The SAS study is self-applied, meaning that people put it on them self at home and use a video instruction for the setup which is used for three nights in a row at home.  It then only takes the sleep technologist 10- 20 min to prepare the study. Using the SAS increases the accessibility for patients for sleep studies and the quality of sleep is better in one’s own bed for three nights than in an hospital environment for one night. 

Today it takes an experienced sleep technologist around 2-3 hours to score each study. If we have a reliable automatic analysis, this process would take a lot less time. The specialist would then look at areas in the study that need special attention, for a quick manual review instead of scoring all the study.  

All this work aims to save time and money as well as creating a better sleep study for patients. This will be achieved by using self-applied studies, artificial intelligence for a reliable automatic analysis to quicken the manual scoring process. Also, the discrepancies between manual scorers can be lessened by improved scoring rules and fewer grey areas of scoring to review. 

Participants: