Aggression

Brief Description

Application to identify instances of aggressive behaviour in patients, including verbal, physical and sexual aggression.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“reported to be quite aggressive towards…”,

“violence and aggression, requires continued management and continues to reduce in terms of incidents etc”.

Also include verbal aggression and physical aggression.

Excludes negative and irrelevant mentions, e.g.:

“no aggression”,

“no evidence of aggression”

“aggression won’t be tolerated”.

etc. Definitions: Search term(case insensitive): *aggress*

Evaluated Performance

Cohen's k = 85% (50 un-annotated documents - 25 events/25 attachments, search term ‘aggress*’). Instance level (testing done on 100 random documents): Precision (specificity / accuracy) = 91%

Recall (sensitivity / coverage) = 75% Patient level (testing done on 50 random documents): Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated: xx

Agitation

Brief Description

Application to identify instances of agitation

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“Very agitated at present, he was agitated”,

“He was initially calm but then became agitated and started staring and pointing at me towards”,

“Should also include no longer agitated. “

Excludes negative and irrelevant mentions, e.g.:

“He did not seem distracted or agitated”,

“Not agitated”,

“No evidence of agitation”.

“a common symptom of psychomotor agitation” Definitions: Search term(case insensitive): *agitat*

Evaluated Performance

Cohen's k = 85% (50 un-annotated documents - 25 events/25 attachments, search term ‘agitat*’). Instance level (testing done on 100 random documents): Precision (specificity / accuracy) = 85%

Recall (sensitivity / coverage) = 79% Patient level: All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient) Precision (specificity / accuracy) = 82%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

Anergia

Brief Description

Application to identify instances of anergia

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“feelings of anergia…”

Excludes negative mentions, e.g:

“no anergia”,

“no evidence of anergia”,

“no feeling of anergia”. Definitions: Search term(case insensitive): *anergia*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘anergia*’). Instance level i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 89% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient): Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

Anhedonia

Brief Description

Application to identify instances of anhedonia (inability to experience pleasure from activities usually found enjoyable).

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“ X had been anhedonic”,

“ X has anhedonia”.

Excludes Negative mentions, e.g

“no anhedonia”,

“no evidence of anhedonia”,

“not anhedonic”,

Exclude ‘Unknown’ mentions e.g:

i) used in a list, not applying to patient (e.g. typical symptoms include …);

ii) uncertain (might have anhedonia, ?anhedonia, possible anhedonia);

iii) not clearly present (monitor for anhedonia, anhedonia has improved);

iv) listed as potential treatment side-effect;

v) vague (‘she is not completely anhedonic’, ‘appears almost anhedonic’) Definitions: Search term(s): *anhedon*

Evaluated Performance

Cohen's k=85% (50 un-annotated documents - 25 events/25 attachments, search term 'anhedon*’). Instance level, (testing done on 100 random documents):

Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

Anosmia

Brief Description

Application to extract and classify mentions related to anosmia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“Loss of enjoyment of food due to anosmia”,

“COVID symptoms such as anosmia”

Excludes Negative and Unknown mentions, e.g.:

“Nil anosmia”,

“Doctor mentioned they had anosmia so could not smell patient”,

“Anosmia related to people other than the patient”,

Unknown mentions: Annotations are coded as unknown when it is not clear if the patient has symptoms/experiences of anosmia

E.g.

“Mentions of medications for it”,

“Don’t come to the practice if you have any covid symptoms such as anosmia” etc. Definitions: Search term(case insensitive): Anosmia*

Evaluated Performance

Cohen's k = 83% (100 Random Documents). Instance level, (testing done on 100 random documents):Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 93%

Additional Notes

Run schedule– On demand

Other Specifications

Version 1.0, Last updated:xx

Anxiety

Brief Description

Application to extract and classify mentions related to (any kind of) anxiety.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Affirmed, Negated, and Irrelevant.

Output and Definitions

The output includes-

Affirmed examples of anxiety includes:

“ZZZZZ shows anxiety problems”

Exclude Negative mentions of anxiety includes:

“ZZZZ does not show anxiety problems”

Excludes Irrelevant examples of anxiety includes:

“If ZZZZ was anxious he would not take his medication”

Examples related to the patient or someone else (“experiencer”)

i. patient: “ZZZ shows anxiety problems”

ii. other: “nurse is worried about the patient”

iii. unknown: “he showed clear signs of anxiety” Search Terms (case insensitive)-Available on request

Evaluated Performance

Cohen’s k = 94% ( 3000 random documents) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Apathy

Brief Description

Application to extract the presence of apathy

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions includes:

any indication that apathy was being reported as a symptom:

“ continues to demonstrate apathy”

“ some degree of apathy noted”

“presentation with apathy”

“his report of apathy given”.

Exclude Negative mentions of apathy:

“denied apathy”

“no evidence of apathy”

Exclude ‘Unknown’ annotations of apathy:

“may develop apathy or as a possible side effect of medication”

“*apathy* found in quite a few names” Definitions: Search term(s): *apath*

Evaluated Performance

Cohen's k=86% (50 un-annotated documents - 25 events/25 attachments, search term ‘apath*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient): Precision (specificity / accuracy) = 73%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Arousal

Brief Description

Application to identify instances of arousal excluding sexual arousal.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes:

Positive mentions include:

“...the decisions she makes when emotionally aroused”,

“...during hyperaroused state”,

“following an incidence of physiological arousal”

Exclude Negative mentions include:

“mentions of sexual arousal”,

“no arousal”,

“not aroused”,

“denies being aroused”

Unknown mentions include:

“annotations include unclear statements and hypotheticals” Definitions: Search term(s): *arous*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - 25 events/25 attachments, search term ‘*arous*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 91%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Bad Dreams

Brief Description

Application to identify instances of experiencing a bad dream

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“ZZZZZ had a bad dream last night”,

“she frequently has bad dreams”,

“ZZZZZ has suffered from bad dreams in the past”,

“ZZZZZ had a bad dream that she was underwater”,

“ he’s been having fewer bad dreams”

Exclude Negative mentions:

“she denied any bad dreams”,

“does not suffer from bad dreams”,

“no other PTSD symptoms such as bad dreams”,

“he said the experience was like a bad dream”,

“ZZZZZ compared his time in hospital to a bad dream”

Exclude Unknown mentions:

“she said it might have been a bad dream”,

“he woke up in a start, as if waking from a bad dream”,

“ZZZZZ couldn’t remember whether the conversation was just a bad dream”,

“doesn’t want to have bad dreams” Definitions: Search term(s): bad dream*

Evaluated Performance

Cohen's k = 100% (100 unannotated documents- 50 events/50 attachments, search terms ‘dream’ and ‘dreams’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Blunted Affect

Brief Description

Application to identify instances of blunted affect

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“his affect remains very blunted”,

“objectively flattened affect”,

“states that ZZZZZ continues to appear flat in affect”

Exclude Negative annotations:

“incongruent affect”,

“stable affect”,

“no blunted affect”

Exclude Unknown annotations:

“typical symptoms include blunted affect”,

“slightly flat affect”,

“relative shows flat affect” Definitions: Search term(s):

*affect*

blunt* [0 to 2 words in between] *affect*

flat [0 to 2 words in between] *affect*

restrict [ 0 to 2 words in between *affect*

*affect* [0 to 2 words in between] blunt

*affect* [0 to 2 words in between] flat

*Affect* [0 to 2 words in between] restrict

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/24 attachments/1 mental health care plan). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 80% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Circumstantiality

Brief Description

Application to identify instances of circumstantiality

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“loose associations and circumstantiality”,

“circumstantial in nature”,

“some circumstantiality at points”,

“speech is less circumstantial”

Exclude Negative mentions:

“no signs of circumstantiality”,

“no evidence of circumstantial”

Exclude Unknown mentions:

“Such as a hypothetical cause of something else” Definition: Search term(s): *circumstan*

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/25 attachments). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 92% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Cognitive Impairment

Brief Description

Application to identify instances of cognitive impairment. The application allows to detect cognitive impairments related to attention, memory, executive functions, and emotion, as well as a generic cognition domain. This application has been developed for patients diagnosed with schizophrenia.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Affirmed and relating to the patient and Negated/Irrelevant.

Output and Definitions

The output includes-

Affirmed mentions:

“patient shows attention problems (positive) “

“ZZZ does not show good concentration “

“ZZZ shows poor concentration “

“patient scored 10/18 for attention “

“ZZZ seems hyper-vigilant “

Exclude Negated and Irrelevant mentions:

“patient uses distraction technique to ignore hallucinations “

“attention seeking”

“patient needs (medical) attention”

“draw your attention to…” Definition- Search term(s): attention, concentration, focus, distracted, adhd, hypervigilance, attend to

Evaluated Performance

Cohen’s k

Cognition – 66% (Done on 3000 random documents)

Emotion – 84% (Done on 3000 random documents)

Executive function – 40% (Done on 3000 random documents)

Memory – 68% (Done on 3000 random documents)

Attention – 99% (Done on 2616 random documents). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 96% Recall (sensitivity / coverage) = 92% Instance level – Patient with an F20 diagnosis (testing done on 100 random documents). Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 70%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Concrete Thinking

Brief Description

Application to identify instances of concrete thinking.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“text referring to ‘concrete thinking’”,

“speech or answers to questions being ‘concrete’”,

“the patient being described as ‘concrete’ without elaboration”,

“answers being described as concrete in cognitive assessments”,

“‘understanding’ or ‘manner’ or ‘interpretations’ of circumstances being described as concrete”

Exclude Negative annotations:

“no evidence of concrete thinking”

Exclude Unknown annotations:

“reference to concrete as a material (concrete floor, concrete house etc.)”

“no concrete plans”,

“delusions being concrete”,

“achieving concrete goals using concrete learning activities” Definitions: Search term(s): Concrete [word][word]think* think [word] [word] concret*

Evaluated Performance

Cohen's k = 83% (50 un-annotated documents - 25 events/25 attachments, search term ‘concrete*’) Instance level, i.e. for all specific mentions (testing done on 146 random documents):

Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 41% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Delusions

Brief Description

Application to identify instances of delusions

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“paranoid delusions”,

“continued to express delusional ideas of the nature”

“no longer delusional- indicates past”

Exclude Negative mentions:

“no delusions”,

“denied delusions”

Exclude Unknown mentions:

“delusions are common” Definitions: Search term(s): *delusion*

Evaluated Performance

Cohen's k = 92% (50 un-annotated documents - 25 events/25 attachments, search term ‘delusion*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 85% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Derailment

Brief Description

Application to identify instances of derailment.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“he derailed frequently”,

“there was evidence of flight of ideas”,

“thought derailment in his language”‘speech no longer derailed’.

Exclude Negative annotations:

“no derailment”,

“erratic compliance can further derail her stability”

“no evidence of derailment”,

“without derailment”,

“without derailing”,

“no evidence of loosening of association, derailment or tangential thoughts”

Exclude Unknown annotations:

“train was derailed” Definitions: Search term(s): *derail*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘derail*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 95% Patient level – Testing done on 30 random document, one document per patient :Precision (specificity / accuracy) = 74%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Disturbed Sleep

Brief Description

Application to identify instances of disturbed sleep.

Development Approach

Development approach: Rule-Based.

Classes produced: Positive

Output and Definitions

The output includes-

Instances of disturbed sleep:

“complains of poor sleep”,

“poor sleep”,

“sleep disturbed”,

“sleep difficulty”,

“sleeping poorly”,

“not sleeping very well”,

“cannot sleep”,

“sleep pattern poor”,

“difficulties with sleep”,

“slept badly last couple of nights” Definitions: Search term(s): Not poor*, interrupt*, disturb*, inadequat*, disorder*, prevent*, stop*, problem* , difficult*, reduc*, less*, impair*, erratic*, unable*, worse*, depriv* [0-2 token] sleep* or slep* , little sleep, sleepless night, broken sleep, sleep intermittently, sleep* or slep* [0-2 token] not, poor*, interrupt*, disturb*, inadequat*, disorder*, prevent*, stop*, problem*, difficult*, reduc*, less*, impair*, erratic* ,unable*, worse*, depriv* .

Evaluated Performance

Cohen's k = 75% (50 un-annotated documents - 25 events/25 attachments, search term ‘*sleep*’ or ‘slept’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 68%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Diurnal Variation

Brief Description

Application to identify instances of diurnal variation of mood

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

• ‘patient complaints of diurnal variation’

• ‘he reported diurnal variation in his mood’

• ‘Diurnal variation present’

• ‘some diurnal variation of mood present’

Exclude Negative examples:

• ‘no diurnal variation’

• ‘diurnal variation absent’

• ‘patient complaints of ongoing depression but no diurnal variation’

• ‘depressive symptoms present without diurnal variation’

Exclude Unknown examples:

• ‘diurnal variation could be a symptom of more severe depression’

• ‘we spoke about possible diurnal variation in his mood’

• ‘it was not certain if there were diurnal variation’ Definitions: Search term(s): diurnal variation

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (Testing done on 100 Random Documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Drowsiness

Brief Description

Application to identify instances of drowsiness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive examples:

“ZZZZZ appeared to be drowsy”,

“She has complained of feeling drowsy”

Exclude Negative examples:

“He is not drowsy in the mornings”,

“She was quite happy and did not appear drowsy”,

“ZZZZZ denied any symptoms of drowsiness”,

“Negative annotations should be when the patient denies drowsiness, or is described as not drowsy etc”

Exclude Unknown examples:

“In reading the label (of the medication), ZZZZZ expressed concern in the indication that it might make him drowsy”,

“Monitor for increased drowsiness and inform for change in presentation”,

“risk of drowsiness, instructions to reduce medication if the patient becomes drowsy” Definitions: Search term(s): drows*

Evaluated Performance

Cohen's k = 83% 1000 un-annotated documents, search term ‘drows*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) =100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Early Morning wakening

Brief Description

Application to identify instances of early morning wakening.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ patient complaints of early morning awakening”,

“he reported early morning wakening”,

“Early morning awakening present”,

“there is still some early morning wakening”

Exclude Negative annotations:

“no early morning wakening”,

“early morning wakening absent”,

“patient complaints of disturbed sleep but no early morning awakening”,

“sleeps badly but without early morning wakening”

Exclude Unknown annotations:

“early morning awakening could be a symptom of more severe depression”,

“we spoke about how to deal possible early morning wakening”,

“it was not certain if there were occasions of early morning awakening” Definitions: Search term(s): early morning wakening

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Echolalia

Brief Description

Application to extract occurrences where echolalia is present.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“no neologisms, but repeated what I said almost like echolalia”,

“intermittent echolalia”,

“some or less echolalia”

Exclude Negative annotations:

“no echolalia”,

“no evidence of echolalia”,

“cannot remember any echolalia or stereotyped utterances”

Exclude Unknown annotations:

“Echolalia is not a common symptom”,

“Include hypotheticals such as he may have some echolalia, evidence of possible echolalia” Definitions: Search term(s): *echola*

Evaluated Performance

Cohen's k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘echola*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 86% Patient level – Testing done on 30 random document, one document per patient, Precision (specificity / accuracy) = 74%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Elation

Brief Description

Application to identify instances of elation.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“mildly elated in mood”,

“elated in mood on return from leave”,

“she appeared elated and aroused”

Exclude Negative annotations:

“ZZZZZ was coherent and more optimistic/aspirational than elated throughout the conversation”,

“no elated behaviour" etc.

Exclude Unknown annotations:

“In his elated state there is a risk of accidental harm”,

“monitor for elation”,

“elation is a known side effect” and

“Statements were term is used in a list, not applying to patients (e.g. Typical symptoms include...)” Definitions: Search term(s): *elat*

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/25 attachments) 1) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 97% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Emotional Withdrawal

Brief Description

Application to identify instances of emotional withdrawal

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

To any description of the patient being described as withdrawn or showing withdrawal but with the following exceptions (which are annotated as unknown):

• Alcohol, substance, medication withdrawal

• Withdrawal symptoms, fits, seizures etc.

• Social withdrawal (i.e. a patient described as becoming withdrawn would be positive but a patient described as showing ‘social withdrawal’ would be unknown – because social withdrawal is covered in another application).

• Thought withdrawal (e.g. ‘no thought insertion, withdrawal or broadcast’)

• Withdrawing money, benefits being withdrawn etc.

Negative and unknown annotations are restricted to instances where the patient is being described as not withdrawn and categorised as unknown. Definition: Search term(s): withdrawn

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘withdrawn’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Eye Contact (Catergorisation)

Brief Description

Application to identify instances of eye contact and determine the type of eye contact.

Development Approach

Development approach:Rule-Based

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions: the application successfully identifies the type of eye contact (as denoted by the keyword) in the context (as denoted by the contextstring)

e.g., keyword: ‘good’; contextstring: ‘There was good eye contact’

Negative mentions: the application does not successfully identifies the type of contact (as denoted by the keyword) in the context (as denoted by the contextstring). The keyword does not related to the eye contact

e.g., keyword: ‘showed’; contextstring: ‘showed little eye-contact’. Definitions: Search term(s): Eye *contact*

Keyword: the term describing the type of eye contact

ContextString: the context containing the keyword in its relation to eye-contact

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91%

Recall (sensitivity / coverage) = 80%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Fatigue

Brief Description

Application to identify symptoms of fatigue.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ZZZZ has been experiencing fatigue”,

“fatigue interfering with daily activities”

Exclude Negative annotations:

“No mentions of fatigue”,

“her high levels of anxiety impact on fatigue”,

“main symptoms of dissociation leading to fatigue”

Exclude Unknown annotations:

“ZZZZ is undertaking CBT for fatigue”. Definitions: Search term(s): Fatigue, exclude ‘chronic fatigue syndrome’

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 95%

Additional Notes

Run schedule – Monthly

Other Specifications

Version: xx, Last updated:xx

Flight of Ideas

Brief Description

Application to extract instances of flight of ideas.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“Mrs ZZZZZ was very elated with by marked flights of ideas”,

“marked pressure of speech associated with flights of ideas”,

“Some flight of ideas”.

Exclude Negative annotations:

“no evidence of flight of ideas”,

“no flight of ideas”

Exclude Unknown annotations:

“bordering on flight of ideas”

“relative shows FOI” Definitions: Search term(s): *flight* *of* *idea*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘flight of’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 94% Patient level – Testing done on 30 random document, one document per patient.Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Fluctuation

Brief Description

The purpose of this application is to determine if a mention of fluctuation within the text is relevant

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-Positive annotations:

“Mrs ZZZZZ’s mood has been fluctuating a lot”,

“suicidal thoughts appear to fluctuate”

Exclude Negative annotations:

“no evidence of mood fluctuation”,

“does not appear to have significant fluctuations in mental state”

Exclude Unknown annotations:

“unsure whether fluctuation has a mood component”,

“monitoring to see if fluctuations deteriorate”,

“his mother’s responsibility fluctuated”,

“is the person’s risk likely to fluctuate.. yes/no…”

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Formal Thoughts Disorder

Brief Description

Application to extract occurrences where formal thought disorder is present.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“deteriorating into a more thought disordered state with outbursts of aggression”,

“there was always a degree thought disorder”,

“Include some formal thought disorder”

Exclude Negative annotations:

“no FTD”,

“no signs of FTD”,

“NFTD”

Exclude Unknown annotations:

“?FTD”,

“relative shows FTD”,

“check if FTD has improved”,

“used in a list, not applying to patient ‘typical symptoms include...” Definitions: Search term(s): *ftd*,*formal* *thought* *disorder*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘*ftd*, *formal**thought* *disorder*) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 83% Patient level – Testing done on 50 random document, one document per patient: Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Grandiosity

Brief Description

Application to extract occurrences where grandiosity is apparent.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ZZZZZ was wearing slippers and was animated elated and grandiose”,

“reduction in grandiosity”,

”No longer grandiose”

Exclude Negative annotations:

“No evidence of grandiose of delusions in the content of his speech”,

“no evidence of grandiose ideas”

Exclude Unknown annotations:

“his experience could lead to grandiose ideas” Definitions: Search term(s): *grandios*

Evaluated Performance

Cohen's k = 89% (50 un-annotated documents - 25 events/25 attachments, search term ‘grandio*’). Instance level: (testing done on 100 random documents)

Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 91% Patient level: Testing done on 30 random document, one document per patient,

Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Guilt

Brief Description

Application to identify instances of guilt.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she then feels guilty/angry towards mum”,

“Being angry is easier to deal with than feeling guilty”,

“Include feelings of guilt with a reasonable cause and mentions stating”,

“no longer feels guilty”

Exclude Negative annotations:

“No feeling of guilt”,

“denies feeling hopeless or guilty”

Exclude Unknown annotations:

“he might be feeling guilty”,

“some guilt”,

“sometimes feeling guilty” Definitions: Search term(s): *guil*

Evaluated Performance

Cohen's k = 92% (50 un-annotated documents - 25 events/25 attachments, search term ‘guil*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 83% Patient level – All patients with primary diagnosis code of F32* and F33*.Testing done on 90 random document, one document per patient: Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations (All)

Brief Description

Application to identify instances of hallucinations.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“her husband was minimising her hallucinations”,

“continues to experience auditory hallucinations”,

“doesn’t appear distressed by his hallucinations”,

“he reported auditory and visual hallucinations”,

“this will likely worsen her hallucinations”,

“his hallucinations subsided”,

“Neuroleptics were prescribed for her hallucinations”.

Exclude Negative annotations:

“denied any hallucinations”,

“no evidence of auditory hallucinations”,

“he reports it is a dream rather than hallucinations”,

“hears voices but denies command hallucinations”,

“did not report any further auditory hallucinations”,

“hallucinations have not recurred”,

“no longer appeared to have hallucinations”,

“has not had hallucinations for the last 4 months”,

“the hallucinations stopped”

Exclude Unknown annotations:

Statements contains probably/possibly/maybe/likely/unclear/unable to ascertain/unconfirmed reports of hallucinations/ experiencing hallucinations,

“pseudo(-) hallucinations”,

“hallucinations present?”,

“?hallucinations”,

“this is not a psychiatric symptom such as hallucinations”,

“abnormalities including hallucinations, derealisation etc”,

“rating scale including delusions, hallucinations, clinical domains”

“hallucinations is a sign of relapse”,

“it is unusual for hallucinations to present in this way”,

“CBT is effective for hallucinations” Definitions: Search term(s): hallucinat*

Evaluated Performance

Cohen's k = 83% (100 un-annotated documents - 50 events/50 attachments, search term ‘hallucinat*’). Instance level, i.e. for all specific mentions (testing done on 100 dandom documents): Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 98%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Hallucinations- Auditory

Brief Description

Application to identify instances of auditory hallucinations non-specific to diagnosis.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“Seems to be having olfactory hallucination”,

“in relation to her tactile hallucinations”

Exclude Negative annotations:

“denies auditory, visual, gustatory, olfactory and tactile hallucinations at the time of the assessment”,

“denied tactile/olfactory hallucination”

Exclude Unknown annotations:

“possibly olfactory hallucinations” Definitions: Search term(s): auditory hallucinat*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘auditory’ or ‘halluc*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 84%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations- Olfactory Tactile Gustatory (OTG)

Brief Description

Application to extract occurrences where auditory hallucination is present. Auditory hallucinations may be due to a diagnosis of psychosis/schizophrenia or may be due to other causes, e.g. due to substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“seems to be having olfactory hallucinations”,

“in relation to her tactile hallucinations”

Exclude Negative annotations:

“denies auditory, visual, gustatory, olfactory and tactile hallucinations at the time of the assessment”,

“denied tactile/olfactory hallucinations”

Exclude Unknown annotations

“possibly olfactory hallucinations” Definitions: Search term(s):

*olfactory* [0-10 words in between] *hallucin*

*hallucin* [0-10 words in between] *olfactory*

*gustat* [0-10 words in between] *hallucin*

*hallucin* [01-10 words in between] *gustat*

*tactile* [0-10 words in between] *hallucin*

*hallucin* [0-10 words in between] *tactile*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘olfact*’ or ‘gustat*’ or ‘tactile’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 68% Patient level – Testing done on 50 random document, one document per patient, Precision (specificity / accuracy) = 86%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations- Visual

Brief Description

Application to extract occurrences where visual hallucination is present. Visual hallucinations may be due to a diagnosis of psychosis/schizophrenia or may be due to other causes, e.g. due to substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“Responding to visual hallucination”,

“Experiencing visual hallucination”,

“history of visual hallucination”,

“distressed by visual hallucination”

Exclude Negative annotations:

“denied any visual hallucination”,

“not responding to visual hallucination”,

“no visual hallucination”,

“no current visual hallucination (with no reference to past)”

Exclude Unknown annotations:

“if/may/possible/possibly/might have visual hallucinations”,

“monitor for possible visual hallucination” Definitions: Search term(s): visual hallucinat*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘visual’ and ‘halluc*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Helplessness

Brief Description

Application to identify instances of helplessness.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“Ideas of helplessness secondary to her physical symptoms present”,

“ideation compounded by anxiety and a sense of helplessness”

Exclude Negative annotations:

“denies uselessness or helplessness”,

“no thoughts of hopelessness or helplessness”.

Exclude Unknown annotations:

“there a sense of helplessness”,

“helplessness is a common symptom” Definitions: Search term(s):*helpless*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘helpless*’).Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis of F32* or F33* in a structured field (random sample of 30 patients, one document per patient): Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hopelessness

Brief Description

Application to identify instances of hopelessness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“feeling very low and hopeless”,

“says feels hopeless”

Exclude Negative annotations:

“denies hopelessness”,

“no thoughts of hopelessness or helplessness”

Exclude Unknown annotations:

“there a sense of hopelessness”,

“hopelessness is a common symptom” Definitions: Search term(s):*hopeles*

Evaluated Performance

Cohen's k = 90% (50 un-annotated documents - 25 events/25 attachments, search term ‘hopeless*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 90% Recall (sensitivity / coverage) = 95% Patient level – All Patients with a primary diagnosis of F32* or F33* in a structured field (one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hostility

Brief Description

Application to identify instances of hostility.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“increased hostility and paranoia”,

“she presented as hostile to the nurses”

Exclude Negative annotations:

“not hostile”,

“denied any feelings of hostility”

Exclude Unknown annotations:

“he may become hostile”,

“hostility is something to look out for” Definitions: Search term(s): *hostil*

Evaluated Performance

Cohen's k = 94% (50 un-annotated documents - 25 events/25 attachments, search term ‘hostil*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 94% Patient level – Random sample of 30 patients (one document per patient): Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Insomnia

Brief Description

Application to identify instances of insomnia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“initial insomnia”,

“contributes to her insomnia”,

“problems with insomnia”,

“this has resulted in insomnia”,

“this will address his insomnia”

Exclude Negative annotations:

“no insomnia”,

“no evidence of insomnia”,

“not insomniac2

Exclude Unknown annotations:

“Typical symptoms include insomnia”,

“might have insomnia”,

“?insomnia”,

“possible insomnia”,

“monitor for insomnia”,

“insomnia has improved” Definitions: Search term(s): *insom*

Evaluated Performance

Cohen's k = 94% (50 un-annotated documents - 25 events/25 attachments, search term ‘insomn*’). Instance level, Random sample of 100 random documents:Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 94% Patient level – All patients with primary diagnosis of F32* or F33* in a structured field, random sample of 50 patients (one document per patient): Precision (specificity / accuracy) = 94%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Irritability

Brief Description

Application to identify instances of irritability.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“can be irritable”,

“became irritable”,

“appeared irritable”,

“complained of feeling irritable”

Exclude Negative mentions:

“no evidence of irritability”,

“no longer irritable”,

“no sign of irritability”

Exclude Unknown annotations:

“irritable bowel syndrome”,

“becomes irritable when unwell”,

“can be irritable if …[NB some ambiguity with positive ‘can be’ mentions, although linked here with the ‘if’ qualifier]”,

“less irritable” Definitions: Search term(s): *irritabl*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘irritabil*’ or ‘irritabl*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 83%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Loss of Coherence

Brief Description

Application to identify instances of incoherence or loss of coherence in speech or thinking.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“patient was incoherent”,

“his speech is characterised by a loss of coherence”

Exclude Negative annotations:

“patient is coherent”,

“coherence in his thinking”

Exclude Unknown annotations:

“coherent discharge plan”,

“could not give me a coherent account”,

“more coherent”,

“mood was coherent with speech”

“a few instances where coherence/incoherence was part of a heading or question” Definitions: Search term(s): coheren*, incoheren*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘incoheren*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 98%Recall (sensitivity / coverage) = 95% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient). Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Low energy

Brief Description

Application to identify instances of low energy.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“low energy”,

“decreased energy”,

“not much energy”,

“no energy”

Exclude Negative annotations:

“no indications of low energy”,

“increased energy”

Exclude Unclear annotations:

“..., might be caused by low energy”,

“monitor for low energy”,

“energy levels have improved”,

“fluoxetine reduces her energy”,

“some energy bars” Definitions: Search term(s): *energy*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - 25 events/25 attachments, search term ‘energ*’). IInstance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 82% Recall (sensitivity / coverage) = 85% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient). Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Mood instability

Brief Description

This application identifies instances of mood instability.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“she continues to have frequent mood swings”,

“expressed fluctuating mood”

Exclude Negative annotations:

“no mood fluctuation”

“no mood unpredictability”,

“denied diurnal mood variations”

Exclude Unknown annotations:

“she had harmed others in the past when her mood changed”,

“tried antidepressants in the past but they led to fluctuations in mood”,

“no change in mood”,

“her mood has not changed and she is still depressed” Definitions: Search term(s):

Chang* [0-2 words in between] *mood*

Extremes [0-2 words in between] *mood*

Fluctuat* [0-2 words in between] *mood*

Instability [0-2 words in between] *mood*

*labil* [0-2 words in between] mood

Rapid cycling [0-2 words in between] mood

*swings* [0-2 words in between] mood

*unpredictable* [0-2 words in between] mood

unsettled [0-2 words in between] mood

unstable [0-2 words in between] mood

*variable* [0-2 words in between] mood

*variation* [0-2 words in between] mood

*volatile* [0-2 words in between] mood

mood [0-2 words in between] chang*

mood [0-2 words in between] Extremes

mood [0-2 words in between] fluctuat*

mood [0-2 words in between] Instability

mood [0-2 words in between] *labil*

mood [0-2 words in between] Rapid cycling

mood [0-2 words in between] *swings*

mood [0-2 words in between] *unpredictable*

mood [0-2 words in between] Unsettled

mood [0-2 words in between] Unstable

mood [0-2 words in between] *variable*

Evaluated Performance

Cohen's k = 91% (50 un-annotated documents - 25 events/25 attachments, search term ‘mood’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 70% Patient level – All patients with random sample of 50 (one document per patient). Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Mutism

Brief Description

Application to identify instances of mutism.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she has periods of 'mutism”,

“he did not respond any further and remained mute”

Exclude Unknown annotations:

“her mother is mute”,

“muted body language” Definitions: Search term(s): *mute* , *mutism*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘mut*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 75% Patient level – All patients with random sample of 30 (one document per patient).

Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Negative Symptoms

Brief Description

Application to identify instances of negative symptoms.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she was having negative symptoms”,

“diagnosis of schizophrenia with prominent negative symptoms”

Exclude Negative annotations:

“no negative symptom”,

“no evidence of negative symptoms”

Exclude Unknown annotations:

“symptoms present?”,

“negative symptoms can be debilitating” Definitions: Search term(s): *negative* *symptom*

Evaluated Performance

Cohen's k = 85% (50 annotated documents - 25 events/25 attachments). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 95% Patient level – All patients with random sample of 30 (one document per patient).

Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Nightmares

Brief Description

Application to identify instances of nightmares.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she was having nightmares”,

“unsettled sleep with vivid nightmares”

Exclude Negative annotations:

“no nightmares”,

“no complains of having nightmares”

Exclude Unknown annotations:

“it’s been a nightmare to get this arranged”,

“a nightmare scenario would be….” Definitions: Search term(s): nightmare*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - events, search term 'nightmare*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Obsessive Compulsive Symptoms

Brief Description

Application to identify obsessive-compulsive symptoms (OCS) in patients with schizophrenia, schizoaffective disorder or bipolar disorder

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations of OCS include

• Text states that patient has OCD features/symptoms

• Text states that patient has OCS

• Text including hoarding, which is considered part of OCS, regardless of presence or absence of specific examples

• Text states that patient has either obsessive or compulsive or rituals or Yale-Brown Obsessive Compulsive Scale (YBOCS) [see keywords below] and one of the following:

o Obsessions or compulsions are described as egodystonic

o Intrusive, cause patient distress or excessive worrying/anxiety

o Patient feels unable to stop obsessions or compulsions

o Patient recognises symptoms are irrational or senseless

• Clinician provides specific YBOCS symptoms

• Text reports that patient has been diagnosed with OCD by clinician

Negative annotations of OCS include

• Text makes no mention of OCS

• Text states that patient does not have OCS • Text states that patient has either compulsions or obsessions, not both, and there is no information about any of the following:

o Patient distress

o Obsessive or compulsive symptoms described as egodystonic

o Inability to stop obsessions or compulsions

o Description of specific compulsions or specific obsessions

o Patient insight

• Text states that non-clinician observers (e.g., patient or family/friends) believe patient has obsessions or compulsions without describing YBOCS symptoms.

• Text includes hedge words (i.e., possibly, apparently, seems) that specifically refers to OCS keywords

• Text includes risky, risk-taking or self-harming behaviours

• Text includes romantic or weight-related (food-related) words that modify OCS keywords

Evaluated Performance

Cohen's k = 80% (600 annotated documents for interrater reliability). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Paranoia

Brief Description

Application to identify instances of paranoia. Paranoia may be due to a diagnosis of paranoid schizophrenia or may be due to other causes, e.g. substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“vague paranoid ideation”,

“caused him to feel paranoid”

Exclude Negative annotations:

“denied any paranoia”,

“no paranoid feelings”

Exclude Unknown annotations:

“relative is paranoid about me”,

“paranoia can cause distress” Definitions: Search term(s): *paranoi*

Evaluated Performance

Cohen's k = 92% (100 annotated documents - 25 events/69 attachments/1 mental state formulation/3npresenting circumstances/2 progress notes). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 94% Patient level – All patients, random sample of 50 (one document per patient), Precision (specificity / accuracy) = 82%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Passivity

Brief Description

Application to identify instances of passivity.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“patient describes experiencing passivity”,

“patient has experienced passivity in the past but not on current admission”

Exclude Negative annotations:

"denies passivity",

"no passivity".

Exclude Unknown annotations:

“passivity could not be discussed”,

“possible passivity requiring further exploration”,

“unclear whether this is passivity or another symptom” Definitions: Search term(s):passivity

Evaluated Performance

Cohen's k = 83% (438 unannotated documents – search term ‘passivity’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Presecutory Ideation

Brief Description

Application to identify instances of ideas of persecution.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“she was having delusions of persecution”,

“she suffered persecutory delusions”,

“marked persecutory delusions”,

“paranoid persecutory ideations”,

“persecutory ideas present”

Exclude Negative annotations:

“denies persecutory delusions”,

“he denied any worries of persecution”,

“no persecutory delusions”,

“no delusions of persecution”,

“did not report persecutory ideas”,

“no persecutory ideation present”

Exclude Unknown annotation:

“this might not be a persecutory belief”,

“no longer experiencing persecutory delusions” Definitions: Search term(s): [Pp]ersecu*

Evaluated Performance

Cohen's k = 91% (50 un-annotated documents - 25 events/25 attachments, search term ‘persecut*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Appetite

Brief Description

Application to identify instances of poor appetite (negative annotations).

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations( applied to adjectives implying a good or normal appetite):

“Appetite fine”

“Appetite and sleep OK”,

“Appetite reasonable”,

“appetite alright”,

“sleep and appetite both preserved”

Exclude Negative annotations:

“loss of appetite”,

“reduced appetite”,s

“decrease in appetite”,

“not so good appetite”,

“diminished appetite”,

“lack of appetite”

Exclude Unknown annotations:

“Loss of appetite as a potential side effect”,

“as an early warning sign, as a description of a diagnosis (rather than patient experience)”, “describing a relative rather than the patient, ‘appetite suppressants’” Definitions: Search term(s): *appetite* within the same sentence of *eat* *well*, *alright*, excellent*, fine*, fair*, good*, healthy, intact*, not too bad*, no problem, not a concern*.

Evaluated Performance

Cohen’s k = 91% (Done on 50 random documents). Instance level, Random sample of 100 random documents: Precision (specificity / accuracy) = 83%

Recall (sensitivity / coverage) = 71% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 30 (one document per patient), Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Concentration

Brief Description

Application to identify instances of poor concentration.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“my concentration is still poor”,

“she found it difficult to concentrate”,

“he finds it hard to concentrate”

Exclude Negative annotations:

“good attention and concentration”,

“participating well and able to concentrate on activities”

“concentrate is adequate or reasonable”

Exclude Unknown annotations:

“‘gave her a concentration solution”,

“talk concentrated on her difficulties”,

“urine is concentrated”, “he is able to distract himself by concentrating on telly”. Definitions: Search term(s): *concentrat*

Evaluated Performance

Cohen's k = Cohen's k = 95% (100 annotated documents – 45 attachments/3 CAMHS events/1 CCS correspondence/35

mental state formulation/1POSProforma/10 ward progress note). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 60% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient), Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Eye Contact

Brief Description

Application to identify instances of poor eye contact.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“looked unkempt, quiet voice, poor eye contact”,

“eye contact was poo”,

“she refused eye contact”,

“throughout the conversation she failed to maintain eye contact”,

“unable to engage in eye contact”,

“eye contact was very limited”,

“no eye contact and constantly looking at floor”

Exclude Negative mentions:

“good eye contact”,

“he was comfortable with eye contact”,

“she made eye contact whilst talking”,

“excessive eye contact was made throughout our conversation”,

“ZZZZZ made occasional eye contact with me”,

“eye contact was inconsistent”,

“Mr ZZZZZ made reasonable eye contact”,

“low voice, average eye contact”

Exclude Unknown mentions:

“she showed increased eye contact”,

“I noticed reduced eye contact today” Definitions: Search term(s): Available on request

Evaluated Performance

Cohen’s k = 92% (100 annotated documents). Patient level – All patients, random sample of 100 Random Documents (one document per patient) Precision (specificity / accuracy) = 81% Recall (sensitivity / coverage) = 65%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Insight

Brief Description

Applications to identify instances of poor insight.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotation:

(An instance is classed as positive if the patient’s insight is minimal or absent)

“Lacking/ Lack of insight”

“Doesn’t have insight”

“No/ None insight”

“Poor insight”

“Limited insight”

“Insightless”

“Little insight”

Exclude Negative annotations (An instance is classed as negative if the patient displays a moderate or high degree of insight into their illness).

“Clear insight”

“Had/ Has insight”

“Improving insight”

“Partial insight”

“Some insight”

“Good insight”

“Insightful”

“Present insight” Exclude Unknown annotation:

“There is a lengthy and unclear description of the patient’s insight, without a final, specific verdict”

“Insight was not assessed”

“The word ‘insight’ is not used in a psychiatry context, rendering it irrelevant”

“The record does not refer to the patient’s current level of insight, perhaps mentioning predicted/ previous levels instead”

“It doesn’t contain the above keywords, despite the general conclusion that can be drawn from it, as this would decrease the overall accuracy of the app”

“Lack of insight not suggestive of psychotic illness, e.g. ‘lack of insight into how his drinking affects his son’ or ‘lack of insight into how she repeats the same cycles with romantic partners” Definitions: Search term(s): insight

Evaluated Performance

Cohen’s k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘insight*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 70% Patient level – Random sample of 50 (one document per patient)

Precision (specificity / accuracy) = 83%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Motivation

Brief Description

This application aims to identify instances of poor motivation.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive’ annotations:

“poor motivation”,

“unable to motivate’ self”,

“difficult to motivate’ self”,

“struggling with motivation”

Exclude Negative annotations:

“patient has good general motivation”,

“participate in alcohol rehabilitation”,

“improving motivation”

Exclude Unknown annotations:

“tasks/groups designed for motivation”,

“comments about motivation but not clearly indicating whether this was high or low”,

“plans to ascertain motivation levels”,

“other use of the word (e.g. ‘racially motivated’)” Definitions: Search term(s): Motivate* with in the same sentence of lack*, poor, struggle*, no

Evaluated Performance

Cohen's k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘motiv*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 45% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poverty Of Speech

Brief Description

Application to identify poverty of speech.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“he continues to display negative symptoms including blunting of affect, poverty of speech”,

“he does have negative symptoms in the form of poverty of speech”

“less poverty of speech”

Exclude Negative annotations:

“no poverty of speech”,

“poverty of speech not observed”

Exclude Unknown annotations:

“poverty of speech is a common symptom of…, “

“?poverty of speech” Definitions: Search term(s): speech within the same sentence of poverty, impoverish.

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 12 events/32 attachments/5 CCS_correspondence, 1 discharge notification summary). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 85% Patient level – Random sample of 30 (one document per patient), Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poverty Of Thought

Brief Description

Application to identify instances of poverty of thought.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“poverty of thought was very striking”,

“evidence of poverty of thought”,

“some poverty of thought”

Exclude Negative mentions:

“no poverty of thought”,

“no evidence of poverty of thought”

Exclude Unknown mentions:

“poverty of thought needs to be assessed”,

“poverty of thought among other symptoms” Definitions: Search term(s): *poverty* *of* *thought*

Evaluated Performance

Cohen's k = 90% (50 annotated documents). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 95%

Recall (sensitivity / coverage) = 93% Patient level – Random sample of 30 (one document per patient), Precision (specificity / accuracy) = 73%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Psychomotor Activity (Catergorisation)

Brief Description

Application to identify instances of psychomotor activity and determine the level of activity

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

Positive/correct mentions identifies the level of psychomotor activity (as denoted by the keyword) in the context (as denoted by the contextstring). In addition, psychomotor_activity column correctly states whether the reference to abnormal levels of psychomotor activity in the contextstring.

For example: Keyword: ‘psychomotor agitation’; Contextstring: ‘patient showed psychomotor agitation’;

Negativity: ‘No’; psychomotor_activity: ‘psychomotor agitation’

Negative/incorrect/irrelevant mentions do not successfully identify the level of activity (as denoted by the keyword) in the context (as denoted by the contextstring). Or an instance of psychomotor activity is noted as negated.

For example: Keyword: ‘psychomotor activity’; Contextstring: ‘normal psychomotor activity’; Negativity: ‘yes’;psychomotor_activity’: ‘psychomotor activity’

Keyword: ‘psychomotor activity’; Contextstring: ‘change in psychomotor activity’; Negativity: ‘yes’; psychomotor_activity: ‘psychomotor activity’

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 92% Recall (sensitivity / coverage) = 92%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Smell

Brief Description

Application to identify symptoms of loss of smell.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“ she has not recovered her sense of smell since she contracted COVID-19 in May 2021”,

“ Complains of loss of smell and loss of tastes”

Exclude Negative annotations:

“Negative annotations include denies any symptoms of loss of smell”,

” Her mother could not smell the food she made”.

Exclude Unknown annotations:

“ no one else could smell it either”,

“ she was unsure whether her smell had been affected” Definitions: Search term(s): Loss of smell/ Lack of smell

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Social Withdrawal

Brief Description

Application to identify instances of social withdrawal.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“she is withdrawn socially from friends and family”,

“Mr ZZZZZ became very isolated and socially withdrawn”,

“ some social withdrawal”

Exclude Negative annotations:

“not being socially withdrawn”,

“no evidence of being socially withdrawn”

Exclude ‘Unknown’ annotations:

“social withdrawal is common in depression”,

“need to ask about social withdrawal”. Definitions: Search term(s): Social within the same sentence of withdraw.

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘withdraw*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 60% Recall (sensitivity / coverage) = 86% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Stupor

Brief Description

Application to identify instances of stupor. This includes depressive stupor, psychotic stupor, catatonic stupor, dissociative stupor and manic stupor.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“ZZZZ presented in a psychotic stupor”,

“man with stuporous catatonia”,

“he is in a depressive stupor”,

“his presentation being a schizoaffective stupor”,

“periods of being less responsive/stuporous”,

Exclude Negative annotations:

“not in the state of stupor”,

“presentation not suggestive of depressive stupor”,

“reported not feeling stuporous”

Exclude Unknown annotations:

“?manic stupor”,

possible psychotic stupor however need to exclude medical cause and stupors induced by substance abuse such as:

“drink himself to stupor”,

“drinking heavily and ending up stuporific”,

“drinking to a stupor”,

“drunken stupors”.

Definitions: Search term(s): Stupor*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘aggress*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 87%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Suicidal Indeation

Brief Description

Application to identify instances of suicidal ideation - thinking about, considering, or planning suicide.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“Her main concerns were his low mood QQQQQ suicidal ideation”,

“He has recently sent a letter to mom describing suicidal ideation”,

“QQQQQ then advised of suicidal ideation”

Exclude Negative annotations:

“There was no immediate risk in relation to self-harm or current suicidal ideation”,

“There has been no self-harm and no suicidal ideation disclosed to QQQQQ”,

“She denies having self-harming or suicidal ideation although sometimes would rather sleep and not get up in the morning”

Exclude ‘Unknown’ annotations:

“Suicidal ideation is a common symptom in depression”,

“It wasn’t certain if she was experiencing suicidal ideation” Definitions: Search term(s): *suicide* ideat*

Evaluated Performance

Cohen's k = 92% (50 un-annotated documents - 25 events/25 attachments, search term ‘ideation’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 81% Recall (sensitivity / coverage) = 87%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Tangentiality

Brief Description

Application to identify instances of tangentiality.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“ he was very tangential lacked goal directed thinking”,

“ there was evidence of tangential speech”

Exclude Negative mentions:

“ no evidence of formal thought disorder or tangentiality of thoughts”,

“there was no overt tangentiality or loosening of associations”

Exclude ‘Unknown’ annotations:

“there can be tangentiality”,

“FTD is characterised by tangentiality”,

“ go off on a tangent” -Definitions: Search term(s): *tangent*

Evaluated Performance

Cohen's k = 81% (50 un-annotated documents - 25 events/25 attachments, search term ‘tangent*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 99% Recall (sensitivity / coverage) = 90% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Taste

Brief Description

Application to identify symptoms of loss of taste within community populations

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“the patient reported loss of enjoyment of food due to loss of taste”,

“COVID symptoms present such as loss of taste”

Exclude Negative mentions:

“the patient denied loss of taste”,

“patients’ mother reported loss of taste due to COVID”

Exclude ‘Unknown’ annotations( when there is reference of loss of taste in terms of an automated letter or email between colleagues or when it is not clear if the patient has symptoms/experiences of loss of taste):

“the patient is not sure if he has lost his taste”,

“don’t come to the practice if you have any COVID symptoms such as loss of taste etc Definitions: Search term(s): Loss of taste*, lack of taste*

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Tearfulness

Brief Description

Application to identify instances of tearfulness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“appeared tearful”,

“was tearful (including was XX and tearful; was tearful and YY)”,

“became tearful”,

“moments of tearfulness”,

“a bit tearful”

Exclude Negative mention:

“not tearful”,

“no tearfulness”,

“denies feeling tearful”,

“no tearful episodes”

Exclude ‘Unknown’ annotations:

“less tearful”,

“couldn’t remember being tearful” and

Statements applying to another person (e.g. mother was tearful) or a person who was not clearly enough the patient.Definitions: Search term(s): *tearful*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘tearful*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 94% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Thought Block

Brief Description

Application to identify instances of thought block.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“showed some thought block”,

“thought block”,

“paucity of thought”

Exclude Negative mentions:

“denies problems with thought block”,

“no thought block elicited”

Exclude ‘Unknown’ annotations:

“thought block can be difficult to assess”,

“ …among thought block and other symptoms” Definitions: Search term(s): *thought* *block*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘thought block*’) Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 75% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Throught Broadcast

Brief Description

Application to identify instances of thought broadcasting.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

"patient describes experiencing thought broadcasting",

"patient has experienced thought broadcasting in the past but not on current admission".

Exclude Negative annotations:

"denies thought broadcasting",

"no thought broadcasting".

Exclude Unknown annotation:

" thought broadcasting could not be discussed",

"possible thought broadcasting requiring further exploration",

"unclear whether this is thought broadcasting or another symptom". -Definitions: Search term(s): Though* within the same sentence of broadcast*

Evaluated Performance

Cohen's k = 94% (95 unannotated documents – search term ‘thought broadcast*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 92%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Thought Insertion

Brief Description

Application to identify instances of thought insertion.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

"patient describes experiencing thought insertion" or "patient has experienced thought insertion in the past but not on current admission".

Exclude Negative mentions:

"denies thought insertion",

"No thought insertion".

Exclude Unknown annotations:

"thought insertion could not be discussed",

"possible thought insertion requiring further exploration",

"Unclear whether this is thought insertion or another symptom". Definitions: Search term(s): Though* [0-2 words] insert*

Evaluated Performance

Cohen's k = 97% (96 unannotated documents – search term ‘thought insert*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 81% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – On Request

Other Specifications

Version 1.0, Last updated:xx

Thought Withdrawal

Brief Description

Application to identify instances of thought withdrawal.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

"patient describes experiencing thought withdrawal" ,

"patient has experienced thought withdrawal in the past but not on current admission".

Exclude Negative mentions:

"denies thought withdrawal",

"no thought withdrawal".

Exclude Unknown annotations:

"thought withdrawal could not be discussed",

"possible thought withdrawal requiring further exploration",

"unclear whether this is thought withdrawal or another symptom" Definitions: Search term(s): Though* [0-2 words] withdraw*

Evaluated Performance

Cohen's k = 95% (76 unannotated documents – search term ‘thought withdraw*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 90% Recall (sensitivity / coverage) = 88%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Waxy Flexibility

Brief Description

Application to identify instances of waxy flexibility. Waxy flexibility is a psychomotor symptom of catatonia as associated with schizophrenia, bipolar disorder, or other mental disorders which leads to a decreased response to stimuli and a tendency to remain in an immobile posture.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“she presents as catatonic with waxy flexibility”,

“exhibiting waxy flexibility”

Exclude Negative mentions:

“no waxy flexibility”,

“no evidence of waxy flexibility”

Exclude ‘Unknown’ annotations:

“his right pre-tibial region was swollen and waxy and slightly pink”,

“waxy flexibility is a very uncommon symptom” Definitions: Search term(s): *waxy*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘waxy’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 86% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Weight Loss

Brief Description

Application to identify instances of weight loss.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“significant weight loss”,

“pleased with his weight loss”

Exclude Negative mentions:

“no weight loss”,

“denies weight loss”.

Exclude Unknown annotations:

“maintain adequate dietary intake and avoid weight loss”,

“the latter reduced in line with weight loss” Definitions: Search term(s):

Loss [0-2 words in between] *weight*

Lost [0-2 words in between] *weight*

Weight* [0-2 words in between] loss

Weight* [0-2 words in between] lost

Evaluated Performance

Cohen’s k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘weight* loss’, ‘loss* weight’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 90% Recall (sensitivity / coverage) = 88%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Worthlessness

Brief Description

Application to identify instances of worthlessness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“feeling worthless”,

“feels hopeless and worthless”

Exclude Negative mentions:

“no worthlessness”,

“denies feelings of worthlessness”

Exclude Unknown annotations:

“his father had told him that he was worthless”,

“would call them worthless” Definitions: Search term(s): *worthless*

Evaluated Performance

Cohen's k = 82% (50 un-annotated documents - 25 events/25 attachments, search term ‘worthless*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 86% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Asthma

Brief Description

Application to identify patients with diagnosis of asthma.

Development Approach

Development approach: Sem-EHR

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

‘past medical history: eczema, asthma’,

‘diagnosed with asthma during childhood’,

‘uses inhaler to manage asthma symptoms’,

‘suffered from an asthma attack’,

‘ZZZZZ suffers from severe asthma’,

‘Mrs ZZZZZ has mild asthma’. Definitions: Search term(s): Ontology available on request

Evaluated Performance

Cohen’s k = 98% (50 patients from patient level testing, 50 documents from annotation level testing, search term ‘asthma’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 84%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Bronchitis

Brief Description

Application to identify patients with diagnosis of bronchitis

Development Approach

Development approach: Sem-EHR

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

‘Recently had COPD (chronic obstructive pulmonary disease’,

‘ZZZZ had chronic bronchitis,

‘Past diagnosis: chronic obstructive airway disease’,

‘physical health history: asthma, bronchitis’,

‘centrilobular emphysema’. Definitions: Search term(s): Ontology available on request

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 48% Patient level – Random sample of 30 (one document per patient), Precision (specificity / accuracy) = 94%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Cough

Brief Description

Application to identify instances of coughing.

Development Approach

Development approach: Machine- learning

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“She has been experiencing a cough for the last week and is going to call her GP”,

“ZZZ called ahead of today’s session reporting a cough so we agreed to move the session to over the phone due to current COVID guidance”,

“He has been to the GP due to coughing up sputum”

Exclude Negative mentions:

“she denied any coughing or shortness of breath”,

“He stated he was unwell with a cold last week, no cough or cough reported”,

“Fever, cough, shortness of breath: Nil”

Exclude ‘Unknown’ annotations:

“She is feeling very distresses because people were coughing near her on the bus”,

“Her son is currently off school with bad cough”

Definitions: Search term(s): Cough*

Evaluated Performance

Cohens k = 79% (150 un-annotated documents, search terms ‘cough*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 80%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Crohn's Disease

Brief Description

Application to identify patients with diagnosis of Crohn’s disease.

Development Approach

Development approach: Sem-EHR

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Include Positive mentions:

“recently been diagnosed with crohn’s disease”,

“ZZZZ has crohn’s disease”,

“she has a history of crohn’s disease”,

“has been hospitalised due to severe crohn’s disease”,

“physical health history: asthma, diabetes, hypertension, crohn’s disease” Definitions: Search term(s): Ontology available on request

Evaluated Performance

Cohen’s k = 98% (50 patients from patient level testing, 50 documents from annotation level testing, search term

‘crohn*’). Patient level – Random sample of 50 (one document per patient) Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 78%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Falls

Brief Description

Application to identify instances of falls or falling.

Development Approach

Development approach: Rules-Based

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Output:

• Fall_single_episode, any reference to a single fall (regardless of when it happened) e.g., ‘he fell last night’, ‘he had one fall 10 years ago’).

• Fall_recurrent: any reference to more than one fall (regardless of when they happened), e.g. ‘he reported recurrent falls’, ‘she had a couple of falls.

• Not_relevant: to capture irrelevant mentions or false positives, e.g. ‘in the fall’, ‘falling in love’ or any other fall mention such as risk of falling, side effect of this medication is risk of falling.

Note 1: positive annotations must refer to the patient and not someone else.

His mother had one fall > NOT_RELEVANT

Note 2: hypothetical statements should not be counted

If she took this medication, she might be at risk of falling > NOT_RELEVANT

Note 3: classes should be chosen on an annotation level: “She had a fall 10 months ago and then had another fall yesterday” should end up as two single-episode annotations, but “she had a couple of falls: 10 months ago and yesterday” would end up as a FALL_RECURRENT

Note 4: accidental falls are to be considered relevant

He fell from the bed > FALL_SINGLE_EPISODE

Note 5: mentions where a fall is "suggested" but not explicitly written (e.g. 'Fall pendant','Falls clinic', 'Falls referral', 'Falls prevention advice') should be considered as NOT_RELEVANT Definitions: Search term(s): Fall*, fell

Evaluated Performance

Cohen's k = xx Patient level – Random sample of 50 (one document per patient) Precision (specificity / accuracy) = 77% Recall (sensitivity / coverage) = 58%

Additional Notes

Run schedule – On Request

Other Specifications

Version 1.0, Last updated:xx

Fever

Brief Description

Application to identify patients with any symptom of fever developed within the last month.

Development Approach

Development approach: Machine-learning

Classification of past or present symptom: Past.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“She informed me on the phone she has had a fever all week”,

“ZZZ has been taking paracetamol for a fever”,

“Attended A&E reporting fever”,

“She felt feverish”

Exclude Negative mentions:

“I asked if she had any symptoms, such as fever, which she denied”,

“Temperature was checked for signs of fever, none observed”

“Cough, fever, shortness of breath: Nil”

Exclude Unknown annotations:

“Her son had a fever last night and she can’t make it to today’s session”

“She reported worrying over what to do if the baby developed a fever”

“I have informed her that if symptoms worsen, or she develops a fever, to attend A&E”

Definitions: Search term(s): fever*

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 86%

Additional Notes

Run schedule – On Request

Other Specifications

Version 1.0, Last updated:xx

Hypertension

Brief Description

Application to identify patients with diagnosis of hypertension or high blood pressure

Development Approach

Development approach: Sem-EHR

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“Recently been diagnosed with hypertension”,

“ZZZZ has high blood pressure”,

“she has a history of hypertension”,

“physical health history: asthma, diabetes, high blood pressure” Definitions: Search term(s): Ontology available on request

Evaluated Performance

Cohen’s k = 91% (50 patients from patient level testing, 50 documents from annotation level testing, search term

‘hypertension*’, ‘high blood pressure*’). Instance level, Random sample of 200 Random Documents: Precision (specificity / accuracy) = 94% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 94%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Multimorbidity- 21 Long Term Conditions (Medcat)

Brief Description

Application to identify patients with diagnosis of physical health conditions (21 conditions in total, including hypertension, diabetes, epilepsy etc.)

Development Approach

Development approach: Machine-learning

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“He reported that he suffers from diabetes and hypertension”,

“Ms ZZZZ has a history of atopy including asthma”,

“Physical health history: asthma, diabetes, high blood pressure: Nil” ,

“Physical health: lung disease confirmed”

Evaluated Performance

Cohen's k = 91% (Done on 50 random documents)

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Pain

Brief Description

Application to determine if a mention of pain (or related words, such as sore, ache, *algia, *dynia etc.) within the text is relevant i.e. associated with the patient and refers to physical pain.

Development Approach

Development approach: Machine-learning

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

‘she is in constant pain’;

‘he suffers from severe headaches’;

he is taking pain killers due to a pulled muscle’ Definitions: Search term(s): %dynia%, '%algia%, %burn%', % headache%, % backache%, % toothache%, % earache%, % ache%, %sore%, %spasm%, % colic%, % cramp%, % hurt%, % sciatic%, % tender%, % pain %, % pains%, % painful%

Evaluated Performance

Cohen’s k = 86% for Attachment (based on 865 annotations)

Cohen’s k = 91% for Event (based on 458 annotations). Patient level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 78%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Rheumatoid Arthiritis

Brief Description

Application to identify patients with diagnoses of rheumatoid arthritis.

Development Approach

Development approach: Sem-EHR

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“ZZZZZ has been in pain due to her rheumatoid arthritis”,

“she has been bedbound with rheumatoid arthritis this week”,

“medication for her rheumatoid arthritis”,

“physical health comorbidities: hypertension, rheumatoid arthritis”,

“diagnosed with rheumatoid arthritis is 1988” Definitions: Search term(s): Ontology available on request

Evaluated Performance

Cohen’s k = 98% (50 patients from patient level testing, 50 documents from annotation level testing, search term ‘rheumatoid arthritis’) Patient level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 86%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

HIV

Brief Description

Application to identify instances of HIV diagnosis.

Development Approach

Development approach: Machine-learning

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

ZZZZZ was diagnosed with HIV (only include cases where a definite HIV diagnosis is present in the text) Definitions: Search term(s): hiv

Evaluated Performance

Cohen's k = Cohen's k = 98% (Done on 50 random documents). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 70% Recall (sensitivity / coverage) = 100% Patient level – Precision (specificity / accuracy) = 64%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

HIV Treatment

Brief Description

Application to identify instances of HIV treatment.

Development Approach

Development approach: Machine-learning

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-Positive mentions:

Include any positive references to the search terms above. Definitions: Search term(s): Anti-retroviral, antiretroviral, ARV, HAART, cART , ART, CD4, Undetectable, Abacavir, Lamivudine, Zidovudine, Aptivus, Atazanavir, Atripla, Celsentri, Cobicistat, Combivir, Darunavir, Didanosine, Dolutegravir, Edurant, Efavirenz, Elvitegravir, Emtricitabine, Emtricitabine, Emtriva, Enfuvirtide, Epivir, Etravirine, Eviplera,

Fosamprenavir.

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 98% Recall (sensitivity / coverage) = 100% Patient level – Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Amphetamine

Brief Description

To identify instances of amphetamine use.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Include Positive mentions:

“denies current use of amphetamine, however last reported using 3 months ago”,

“first took amphetamines at the age of 15”,

“UDS: +ve amphetamine”,

“ZZZZZ has been trying to give up amphetamine for the last 2 months”,

“ZZZZZ was found in possession of large quantities of amphetamines”,

“She admitted to having bought amphetamine 2 days ago” ,

“amphetamine-psychosis”

Exclude Negative mentions:

“ZZZZZ denies use of alcohol and amphetamine”,

“ZZZZZ has not used amphetamine for the last week”,

“-ve: amphetamine”

Exclude ‘Unknown’ mentions:

“ZZZZZZ’s mother has a history of amphetamine abuse” – subject other than patient,

“ZZZZZ is planning on taking amphetamine this weekend” – future or conditional event,

“We discussed the dangers of amphetamine” Definitions: Search term(s): Amphetamin*

Evaluated Performance

Cohen's k = 84% (50 un-annotated documents - 25 events/25 attachments, search term ‘amphetamine*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 84% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version xx, Last updated:xx

Cannabis

Brief Description

To identify instances of cannabis use.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“He is a cannabis smoker”,

“she smoked cannabis when at uni”

“she stopped using cannabis 3 years ago”

Exclude Negative mentions:

“denied taking any drugs including cannabis”,

“no cannabis use”

Exclude ‘Unknown’ annotations:

“she stated in hash voice”,

“pot of yoghurt”,

“father cannabis user”,

“pot for UDS” Definitions: Search term(s): cannabis ,skunk, weed, Pot, marijuana, grass ,THC, hash, cannabinoids, resin, hashish, weeds, Cannabis- ,spices, Spice, ganja, CBD, cannabis-induced, Cannabinoid, cannabies, grasses, Cannaboids, marijuana, cannabbase, cannabis-free, skunk- cannabis, Hashis, cannabis-related, cannabi, cannabise, cannabinoids, cannabis-use, marijuana, cannabus, cannabiss, weed- skunks

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search terms ‘cannabis ‘, ‘marijuana’, ‘weed’, ‘pot’, ‘hash’, ‘skunk’, ‘resin’, ‘spice*’). 1) Instance level (Overall) , Random sample of 100 Random Documents:Precision (specificity / accuracy) = 77% Recall (sensitivity / coverage) = 93% Instance level (Current) – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 72% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version xx, Last updated:xx

Cocaine or Crack Cocaine

Brief Description

To identify instances of cocaine or crack cocaine use.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“denies current use of cocaine, however last reported using 3 months ago”,

“first smoked cocaine at the age of 15”,

“UDS: +ve cocaine”,

“ZZZZZ has been trying to give up cocaine for the last 2 months”,

“ZZZZZ was found in possession of large quantities of cocaine”,

“She admitted to having bought cocaine 2 days ago” ,

“He has stopped taking cocaine”.

Exclude Negative annotations:

“ZZZZZ denies use of street drugs such as cocaine”,

“ZZZZZ has not used cocaine for the last week”,

“Crack N” – form style.

Exclude ‘Unknown’ annotations:

“ZZZZZZ’s mother has a history of crack abuse” – another subject other than the patient,

“ZZZZ is planning on taking cocaine this weekend” – future or conditional events,

“When cooking he decided to crack the eggs open” – irrelevant ,

“ZZZZZ believes cocaine isn’t good for people” – irrelevant,

“We discussed the dangers of crack”. Definitions: Search term(s): Cocaine*, crack

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - 25 events/25 attachments, search term ‘cocaine*’). Instance level (Overall) , Random sample of 100 Random Documents:Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 97% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version xx, Last updated:xx

MDMA

Brief Description

Application to identify instances of MDMA use

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“denies current use of MDMA, however last reported using 3 months ago”,

“first took MDMA at the age of 15”,

“UDS: +ve MDMA”,

“ZZZZZ has been trying to give up MDMA for the last 2 months”,

“ZZZZZ was found in possession of large quantities of MDMA”,

“She admitted to having bought MDMA 2 days ago”,

“He has stopped taking MDMA”

Exclude Negative annotations:

“ZZZZZ denies use of street drugs such as MDMA”,

,“ZZZZZ has not used MDMA for the last week”,

“UDS -ve: MDMA”

Exclude ‘Unknown’ annotations:

“ZZZZZZ’s mother has a history of MDMA abuse” – another subject other than the patient,

“ZZZZ is planning on taking MDMA this weekend” – future or conditional events,

“ZZZZZ believes MDMA isn’t good for people” – irrelevant,

“We discussed the dangers of MDMA”. Definitions: Search term(s): mdma

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘mdma’). Instance level (Overall) , Random sample of 100 Random Documents:Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 99% Patient level – Random sample of 30 (one document per patient)

Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version xx, Last updated:xx

Smoking

Brief Description

This application distinguishes between people who are a) current smokers, b) current non-smokers (ever smoked) and c) non-smokers. This application may at times bring back contradictory information on the same patient since patient may start smoking and stop smoking and because of the varied level of information available to the clinician.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes-

Status:

One of the following must be annotated in the status feature:

Never: clearly not smoking currently or just a general message that the subject does NOT smoke. Ex: “…is a non-smoker”, “… was/is not a smoker”, “… doesn’t smoke”, “ZZZZZ denies ever smoking”, or “… is currently not smoking”

Current: a clear message that the subject is currently smoking

Ex: “…smokes 20 cigarettes a day”, “… has been smoking for 10 years”, “…is a smoker”, “ZZZZZ smokes in the ward”, “…went to garden for a smoke”, “ZZZZZ is stable when smoking”, “…has a history of heavy smoking”, “Consider stopping smoking”, “ZZZZZ found smoking in her room” or “… is a tobacco user”)

Past: any hint that the subjects was smoking

Ex: “… used to smoke”, “… has quitted smoking”, “… stopped smoking”, “ZZZZZ is an ex-smoker” or “…was a smoker”)

Evaluated Performance

Cohen's k = xx Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 89% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 95%

Additional Notes

Run schedule – Weekly

Other Specifications

Version xx, Last updated:xx

Education

Brief Description

Application to identify the highest level of education at patient level.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes- Output:

Group 1: A level group

Rule Stage of course

Accepted Accepted for A-level course or equivalent (course or institution)

Ongoing Started course but not (yet) completed (including evidence of attending relevant institution)

Dropped Out Started course but not completed - dropped out

Expelled Started course but not completed - expelled

Failed Completed course – failed all exams

Completed Completed course

Passes Passed at least one exam

Applied_undergrad Applied for university / course

Note: aspirations, plans, application only are not accepted.

Group 3: University

Rule Stage of Course

Accepted Accepted for course / institution

Ongoing Started course but not (yet) completed

Dropped out Started course but not completed - dropped out

Expelled Started course but not completed - expelled

Failed Completed course – failed

Completed Completed course

Passed Passed / graduated

Applied_University Applied for University

Group 4: unqualified group

Rule Definition

Unqualified A specific reference in notes describing as having left school without any qualifications.

GSCE_Dropped_out Started GCSE course but not completed - dropped out

GSCE_Expelled Started GCSE course but not completed - expelled

GSCE_Failed Completed GCSE course – failed all exams

School leaving age

Examples

He left school at the age of 16 years

Was 19 years old when she left school

Mrs ZZZZZ left school at 15 without any qualifications

Group 2: GCSE group

Rule Stage of Course

Ongoing Started GCSE course (or equivalent) but not (yet) completed

Completed Completed GCSE course or equivalent

Passed Passed at least one exam (GSCE or equivalent)

Applied_A-level Applied for 6th form (college) / A-level

Evaluated Performance

GCSE – Cohen's k = 90% (50 annotated documents – 25 events, 25 attachments)

No qualifications - Cohen’s k = 100% (50 annotated documents – 25 events, 25 attachments)

Additional Notes

Run schedule – On request

Other Specifications

Version xx, Last updated:xx

Occupation

Brief Description

Application to identify occupations/work descriptions and who these refer to.

Development Approach

Development approach: Machine- learning and Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes-

Output:

There are two parts to each annotation: Firstly, the occupation feature is annotated - this could be a job title, for example a ‘builder’; or a job description, for example ‘working in construction’. Secondly, the occupation relation is annotated: who the occupation belongs to, for example the patient or their family member.

Unpaid occupational categories were included (e.g. student, unemployed, homemaker, volunteer). Depending on the text available, extractions can state a specific job title (e.g. head-teacher) or a general occupational category (e.g. self-employed).

Work aspirations were excluded from annotations. Frequently extracted health/social care occupations (e.g. psychiatrist) are not annotated as belonging to the patient, in order to maximise precision.

Occupation feature (text) – the job title (e.g. ‘hairdresser’)

Occupation relation (text) – who the occupation belongs to (e.g. ‘patient’)

The full annotation guideline document is available on request. Definitions: Search term(s): Gazetteer available on request

Evaluated Performance

Cohen’s k = 77% for occupation feature (200 ‘personal history’ documents)

Cohen’s k = 72% for occupation relation (200 ‘personal history’ documents). Instance level (Overall) , Random sample of 200 Personal History Documents:

Precision (specificity / accuracy) = 77% Recall (sensitivity / coverage) = 79% Instance level (Overall) , Random sample of 82 Personal History Documents from records of patients aged >= 16 years Precision (specificity / accuracy) = 96% Patient level – Random sample of 82 Personal History Documents from records of patients aged >= 16 years Precision (specificity / accuracy) = 96%

Additional Notes

Run schedule – On request

Other Specifications

Version xx, Last updated:xx

Lives Alone

Brief Description

Application to identify instances of living alone.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes-

Output:

The application identifies the following:

“Lives on her own”, Who- none ,

“She lives alone”, Who- She

“He presently lives alone on 7th floor”, Subject – He

“His father lives alone”, Subject – Father Definitions: Search term(s): Lives alone, Lives by himself, Lives by herself, Lives on his own, Lives on her own

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘lives on his/her own’),

‘lives by him/herself’, ‘lives alone’). Instance level (Overall) , Random sample of 100 Random Documents:Precision (specificity / accuracy) = 77%

Recall (sensitivity / coverage) = 83% Precision (Subject) = 61%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Domestic Violence

Brief Description

Application to identify instances of domestic violence.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“ZZZZZ was a victim of DV”;

“X has a history of domestic violence”;

“he experienced DV in the past”.

Exclude Negative mentions:

“ZZZZZ denied that DV took place”,

“denies any domestic violence”.

Exclude ‘Unknown’ annotations(where the term “DV” or “domestic violence” did not describe any form of domestic violence):

“saw X on a DV yesterday” Definitions: Search term(s): domestic violence DV

Evaluated Performance

Cohen’s k = 87% (180 unannotated documents, search terms ‘dv’ and ‘domestic violence’). Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Loneliness

Brief Description

Application to identify instances of loneliness.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“the patient is lonely”,

“the patient confirms they have a sense/feeling of loneliness”,

“preventing further loneliness”

Exclude Negative mentions:

“Patient is not lonely”,

“denies being lonely”.

Exclude ‘Unknown’ mentions:

“the patient’s family member is lonely”;

“they are participating in an activity on a ward to prevent boredom/loneliness”,

“EHR discusses the prevention of loneliness”,

“Instances where a clinician suspects loneliness, or if there “might be loneliness/lonely” but it is not declared by, or agreed by the patient, would be classified as unknown”

“Loneliness if indicated on a form as a heading or question would be classified as unknown”. Definitions: Search term(s): lonely loneliness

Evaluated Performance

Cohen’s k= 81% (100 unannotated documents, search terms ‘lonely’, ‘loneliness’). Instance level (Overall) , Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Violence

Brief Description

Application to determine if mentions related to violence are being described as clearly relevant to the patient,

and if they are also indicating that the patient is the perpetrator, or the victim; and if the type of violence

relates to physical, domestic, or sexual. This is in essence 6 applications: violence status, domestic violence,

sexual violence, physical violence, violence perpetration, violence victimisation.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: 1. affirmed/negated/irrelevant (“status”),

2. when affirmed: related to the patient being the perpetrator or victim (could be both)

3. when affirmed: domestic, sexual or physical (could also be more than one type)

Output and Definitions

The output includes-

Detailed annotation guidelines are available: https://docs.google.com/document/d/19OxdB0kMYsinPG-4xh-Ft0DZi-FJpgPQkBm4T_VkMXc/edit?usp=sharing Definitions:

Search term(s): % abus%, % assault% , % attack%, % beat%, % chok% , % fight%, % fought%, % hit%, % punch%, % push%,

% rape%

Evaluated Performance

Cohen’s k

Status = 85% (2652 un-annotated documents)

Victimisation = 70% (2652 un-annotated documents)

Perpetration = 80% (2652 un-annotated documents)

Domestic = 74% (2652 un-annotated documents)

Physical = 60% (2652 un-annotated documents)

Sexual = 68% (2652 un-annotated documents)

Additional Notes

Run schedule – On Request

Other Specifications

Version xx, Last updated:xx

Cognitive Behavioural Therapy (CBT)

Brief Description

An application to identify instances of delivered sessions of Cognitive Behavioural Therapy (CBT).

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes- Definitions:

Search term(s):

1.1 Inclusions:

A session of CBT is defined as an event (excluding ward progress notes) having “CBT” or

“Cognitive Behavioural Therapy” or

“Cognitive Therapy” followed by “session”, “assessment” or “follow up” plus the following variations specified below:

1.2 Assessment session:

“Attended for CBT”

“LICBT” & “session”

“CBT appointment”

“CBT appt”

“saw ZZZZZ for CBT”

“CBT: Seen”

“CBT: Reviewed”

“Session X of CBT”

“X CBT”

“Xst CBT”

“CBT #X”

“CBT #X”

“SX CBT”

“session of CBT”

“continued with CBT” “CBT psychology session”

“session X of CBT”

“Met with ZZZZZ to continue the CBT work.”

“MIND WORKOUT (CBT GROUP)“

1.3 Follow Up:

“CBT follow up appointment”,

“CBT 12-month follow-up”

1.4 Alternative terms for CBT

“SX HICBT”

“SX LICBT”

Evaluated Performance

Cohen's k = xx 1) Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 86%

Additional Notes

Run schedule – Weekly

Other Specifications

Version 1.0, Last updated:xx

Family Intervention

Brief Description

The application identifies instances of family intervention delivery.

Development Approach

Development approach: Rule-Based.

Classes produced:

The application will produce the following 6 features for each annotation: -

FI Session: Y/N

Session n: Session number

Stage: Assessment, first session, last, treatment, follow-up,

Subject: Both patient and carer/Carer/Patient but patient only relevant FI intervention for Behavioural Family Therapy (BFT). – Note if a single subject + patient then annotate as both (“ZZZZZ and carer”) and if more than one other attendee then annotated as family (“ZZZZ, mum and sister”).

Delivery: Individual Family/Multi Family – note Multi family groups are not generally practiced in the psychosis services but will be in the eating disorders service

Outcome: Attended, DNA, cancelled

Output and Definitions

The output includes- Definitions:

FI Session

Inclusions

A session of FI is defined as an event having “FI” or equivalent terms ("family intervention", “FI”, "family therapy", "family work","family workshop","systemic work","systemic therapy", "family session", “FTWS”, “Behavioural Family Therapy”, “BFT”, ”BFI”, “FIP”) followed by “session” or equivalent terms (“appt”, “Appointment”, “Ass”, “Assessment”, “Reviewed”, “Seen”) and additional terms specified below.

Exclude “family meeting” and “carer” from NLP app but include in the heading section – exclude at the combined_view stage.

Note - FIP refers to Family Intervention in Psychosis

Assessment session:

Other terms that should be included

“FI Assessment” Assessment

“FI: Ax” Assessment

“Assessment and FI in the same sentence” Assessment

Treatment session Other terms that should be included:

“Attended for FI”

“FI appointment”

“FI appt”

“saw ZZZZZ for FI”

“FI: Seen”

“FI: Reviewed”

“Session X of FI”

“X FI”

“Xst FI”

“FI #X”

“FI #X”

“SX FI”

“session of FI”

“continued with FI”

“session X of FI”

“Met with ZZZZZ to continue the FI work.”

Follow up

“FI follow up appointment”

“FI 12-month follow-up” Exclusions:

The following combinations below with FI in the same sentence are considered as exclusions. Note if the above inclusion criteria are met then this would be considered a positive hit independently of below but if only “next session” and FI were present in the same sentence this wouldn’t be annotated as a positive hit: -

“next session -/-” (day/month)

“next session 2nd”

“next session _._._” (day/month/year)

“Next session _._” (day/month)

“next appointment -/-” (day/month)

“next appointment 2nd”

“next appointment _._._” (day/month/year)

“Next appointment _._” (day/month)

“next appt -/-” (day/month)

“next appt 2nd”

“next appt _._._” (day/month/year)

“Next appt _._” (day/month)

Session n

Where a FI session has been indicated record the session number where specified. Note include first and last. Think about proximity – usually “Session x” but also examples of 1st session of FI, etc…

Other terms

“Final FI session” “last FI Session”

“Final session of FI”

“last session of FI”

Stage

Assessment terms:

“FI Assessment”

“FI: Ax”

“Assessment” and “FI” in the same sentence

Some services e.g picup service has mid therapy assessment

Follow-up terms

“FI Follow up appointment”

“FI Follow up appt”

Subject

Inclusions

Both patient and carer

Carer/

Patient but patient only relevant for Behavioural Family Therapy (BFT) (only in psychosis services)

Delivery

Inclusions

Group or individual therapy 253

Outcome

Attended, DNA, cancelled by carer, cancelled by patient, cancelled by staff

Evaluated Performance

Cohen's k = 88% (50 annotated documents - 25 events/25 attachments). Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 77% Recall = 87%

Additional Notes

Run schedule – Weekly

Other Specifications

Version 1.0, Last updated:xx

Social Care- Care Package

Brief Description

Application to identify instances of receiving current, recommended or planned general care package. This is a generic term relating to any social care intervention. This could be a specific type of social care (e.g. Meals on wheels, regular home care) or general mention of a package. ‘Status’ states whether patient currently has a care package, will get one in the future or that there is potential to receive it. ‘Subject’ states who the receiver of the care package is.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

‘carer came twice that day as per the care package’,

‘should receive a package of care from next week’,

‘we have recommended the care package to be increased’.

Exclude Negative mentions:

‘refused the care package’,

‘tried to discuss having a package of care after discharge, but refused to converse’.

Anything with suggested uncertainty of recommendation should be classed as negative.

Exclude ‘Unknown’ mentions:

vague mentions that do not suggest that a care package is actually recommended.

Status definition

Current- is currently receiving a care package (present tense mentions).

Potential- recommended or discussions relating to possible care package. No exact plan to have it in the future but future discussions to take place.

Future- planned or arranged meals care package.

Subject definition

The individual who the care package relates to e.g. ‘his wife’, ‘he’, ‘patient’ or ‘none’ if the mention does not directly specify the individual. Definitions:

Search term(s): ‘care package’ ‘package of care’

Evaluated Performance

Cohen’s k = 95% (100 un-annotated documents, search term ‘care package’, ‘package of care’). Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 72.8%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Social Care- Home Care

Brief Description

Application to identify instances of home care/help. This is help by someone who comes to assist the patient with activities of daily living.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

‘patient gets home help 3x daily’,

‘carers admit he can be agitated during the day at home’,

‘home care plan to start next week’,

‘meeting to discuss potential home care’,

‘suggested home help which the family are considering’.

Exclude Negative mentions:

forms, informal care not provided by a care service (informal carers, such as family members, friends or neighbours),

ward round notes referring to care coordinator,

‘home care FORMCHECBOX’,

‘carer support to be given to sister of ZZZ’,

‘refused home help’,

‘home help discussed but ZZZ does not want at this time’,

‘care coordinator contacted’,

‘ward round: care coordinator responded with…’. Exclude Unknown mentions (where it is unclear if current/potential home care is being given, this may be questions, mention of home treatment team without reference to home care or mention of past care without current/future care mentioned)

‘Does the patient have home care support?’,

‘home treatment team called’,

‘used to have home care support’. Status definition

Current- is presently receiving home care/help.

Future- plans have been made for home care to occur.

Potential- home care has been suggested but there has been no planning of it being used for certain in the future.

Subject definition

The individual who the care package relates to e.g. ‘his wife’, ‘he’, ‘patient’ or ‘none’ if the mention does not directly specify the individual.

Time definition

This is how often the home care is received, outputs include day, week, month or NULL if not specified in the text.

Frequency definition

This refers to how often homecare occurs within the time frame (numeric output or NULL if not specified). For example, if homecare was received 3 times daily, the output would be: time= day, frequency=3. Definitions: Search Term:‘home care’ ,‘carer visits’ ,‘carer support’ ,‘home carer’ ,‘home help’ ,‘home helper’, ‘home carer had visited’

Evaluated Performance

Cohen’s k = 95% (100 un-annotated documents, search ‘home care’, ‘carer visits’, ‘carer support’,‘home carer’, ‘home help’, ‘home helper’, ‘home carer had visited’). Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 79% Recall (sensitivity / coverage) = 65%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Social Care- Meals on Wheels

Brief Description

Application to identify instances of receiving current or recommended meals on wheels (food delivery, usually from a private firm).

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

‘receiving MOW’,

‘recommended for MOW’,

‘planning to have meals on wheels from Monday’,

‘will arrange for ZZZ to have MOW’,

‘isn’t eating his MOW’,

‘prefers the Wiltshire farms to the original MOW’,

‘discussed MOW’,

‘would be happy to have MOW’.

Exclude Negative mentions:

‘mow the lawn’,

spelling error ‘he is mow in mood’,

‘refuses MOW’,

‘stopped having meals on wheels as he did not like the taste’,

‘used to have MOW, now stopped’,

‘FORMCHECKBOX’ forms.

Exclude Unknown mentions:

‘MOW?’, ‘consider MOW’, ‘need to consider a plan that could partly include meals on wheels’, ‘will discuss MOW’. Status definition

Current- is currently receiving MOW (present tense mentions). Current use relates to MOW delivery, if the individual refuses to eat/ignores the food, this would still be classed as a positive instance.

Potential- recommended, planned or arranged meals on wheels. Anything with suggested uncertainty of recommendation should be classed as negative. Definitions: Search term(s): ‘MOW’,‘meals on wheel*’

Evaluated Performance

Cohen’s k = 95% (100 un-annotated documents, search ‘MOW’, ‘meals on wheel*’). Instance level (Overall) , Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 71% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – On Request

Other Specifications

Version 1.0, Last updated:xx

Blood Pressure (BP)

Brief Description

Application to identify instances of blood pressure scores in the format of overall score, systolic blood pressure

score and diastolic blood pressure score.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes-

Output:

BP: 126/85 diastolic:85, systolic:126

blood pressure was 126/73 diastolic:73, systolic:126 Definitions: Search term(s): ‘blood pressure’,‘bp’

Evaluated Performance

Cohen's k = 98% (Done on random 100 documents). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) (Overall) = 98%

Precision (Systolic) = 98%

Precision (Diastolic) = 98%

Precision (Full Score) = 98%

Precision (Same day) = 92%

Precision (One Week) = 98%

Precision (One Month) = 98%

Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Body Mass Index (BMI)

Brief Description

Application to identify body mass index (BMI) scores.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Output and Definitions

The output includes- Output:

Inclusions

Criteria Examples

BMI and number in the same sentence Bmi 45, bmi:46, Body Mass Index is 22.9, 16 BMI

BMI, number and units in the same sentence Bmi 45 kg/m2, BMI 47 Kg/m2 , BMI 22.8 kg/m 2

Exclusions Criteria Examples

BMI and number in a sentence that indicates centile Bmi centile 46, Bmi centile 77, He is on the 34th centile for BMI, BMI above 96th centile

BMI, number and units in the same sentence, bmi units are indicated wrong in the sentence Her BMI is 48 kg, BMI: 22 kg, BMI/Weight : 103.2 kg

There is no score in the sentence, but there is an indication of the word BMI. Record her weight to find out her BMI, BMI indicated that she was obese, Raised BMI, stable weight and BMI

BMI indicates as BMI range BMI between 20.0 and 25.0, BMI within the healthy range 25.0 to 27.0 Definitions: Search Terms (case insensitive)

bmi

Bmi

BmI

BMI

BMi

Body Mass Index

Body mass index

body mass index

Units for BMI: Kg/m2

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) (Overall) = 89%

Precision (Same Day) = 66%

Precision (One Week) = 71%

Precision (One Month) = 72%

Precision (Three Months) = 73%

Recall (sensitivity / coverage) = 78%

Additional Notes

Run schedule – Weekly

Other Specifications

Version 1.0, Last updated:xx

Brain MRI report volumetric Assessment for dementia

Brief Description

Application for automated extraction of mentions of dementia-related volumetric assessments from plain text brain MRI reports

• This model has only been trained on brain MRI reports, where the clinical indication was for the investigation of dementia

• This had not been validated for use on other imaging reports, and for other clinical indications would be unlikely to return many results as the terms of interest are unlikely to be mentioned

• The terms of interest are unlikely to appear outside of the radiologists report, other than if they are copied into the patient notes and/or a clinical letter

Development Approach

Development: Machine-learning, based on spaCy python library

Output and Definitions

The output includes- Output:

• GVL = Global Volume Loss – Present

• NO_GVL = Global Volume Loss – Absent

• RVL = Regional Volume Loss – Present

• NO_RVL = Regional Volume Loss – Absent

• HVL = Hippocampal Volume Loss – Present

• NO_HVL = Hippocampal Volume Loss – Absent Definitions: Search term(s): Patient referred for an MRI from memory clinic. Takes the plain text MRI report as input and identifies and returns labelled spans or token that are predicted as belonging to any of the below 6 classes which is given in Output section.

For each span:

Text (string), label (String), score (float)

E.g., ‘brain volume is normal’ = NO_GVL

‘Severe bilateral hippocampal atrophy’ = HVL

The underlying span categorization approach allows multiple labels to be applied to the same span, e.g., ‘no regional predominant or hippocampal atrophy’ = NO_RVL and NO_HVL

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents:

GVL: F-score = 0.95, P= 95%, R=95%

NO_GVL: F-score = 0.85

RVL: F-score = 0.68

NO_RVL: F-score = 0.92

HVL: F-score = 0.89

NO_HVL: F-score = 0.93

F1 score averaged over all 6 categories on holdout test set was 0.89

Additional Notes

Run schedule – Yearly

Other Specifications

Version 1.0, Last updated:xx

Cholesterol

Brief Description

To identify total cholesterol score of a patient, in clinical notes. Total cholesterol referred to as total cholesterol or serum cholesterol

Development Approach

Development: Rule-based.

Classification of past or present symptoms: Both.

Classes produced: In clinical notes, cholesterol level is referred as totals cholesterol score and serum cholesterol.

Output and Definitions

The output includes- Definitions:

Output:

Examples:

1) Total cholesterol 06-Sep-2022 4.8 mmol/L

2) Serum cholesterol level 08-March-2001 3.8 mmol/L

3) Serum total cholesterol level 14-June-1998 6.3 mmol/L

4) Total cholesterol level 19-April-2010 5.3

5) 19-March-2020 6 mmol/L serum cholesterol level

Negative Examples:

1) Serum cholesterol > 4.0 mmol/L

Evaluated Performance

Cohen's k = xx Instance level (Overall), Random sample of 100 Random Documents:Precision (specificity / accuracy) (Overall) = 82% Recall (sensitivity / coverage) = 95%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

HBA1C

Brief Description

The application will use a structured code to identify instances where HbA1c* and its results are found within CRIS from non-structured fields (i.e. case notes). This will help provide a clearer indication of how HbA1c is being recorded within CRIS.

*HbA1c can be obtained from a routine blood test and refers to glycated haemoglobin. It develops when haemoglobin, a protein within red blood cells that carries oxygen throughout your body joins with glucose in the blood, becoming 'glycated'. By measuring glycated haemoglobin (HbA1c), clinicians are able to get an overall picture of what our average blood sugar levels have been over a period of weeks/months. For people with diabetes, this is important as the higher the HbA1c, the greater the risk of developing diabetes-related complications. Therefore, it is important to ensure that this is being recorded and monitored effectively within South London and Maudsley as we know that those with psychosis are at a greater risk of diabetes.

Development Approach

Development approach: Rule-Based

Classification of past or present symptom: Both.

Output and Definitions

The output includes- Inclusion criteria:

1) HbA1c score is present in format of e.g.

a. HbA1c was 40, HbA1c 40, HbA1c was 40mmol/mol, HbA1c was 40mmol

b. HbA1c was 15%

2) Decimals are allowed (e.g. 13.6)

3) HbA1c score relates to the patient only

Exclusion criteria:

1) HbA1c is mentioned without score e.g.

a. HbA1c was measured and found to be within normal range

b. HbA1c was measured on 11/11/19

c. HbA1c 10/10/18

Post Processing Notes: N.B: The application was not developed with upper or lower score limits. However, during testing anything lower than 3% or 9mmol and anything higher than 24% or 238mmol was excluded.

HbA1c mmol/mol %

Normal Below 42 mmol/mol Below 6.0%

Prediabetes 42 to 47 mmol/mol 6.0% to 6.4%

Diabetes 48 mmol/mol or over 6.5% or over

Evaluated Performance

Cohen's k = xx Instance level (Overall), Random sample of 100 Random Documents:

Precision (specificity / accuracy) (Overall) = 92% Recall (sensitivity / coverage) = 76%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Mini-Mental State Examination (MMSE)

Brief Description

This app identifies MMSE scores and returns information on:

- MMSE score (overall and subdivided into numerator and denominator)

- Associated date

Development Approach

Development approach: Machine-learning.

Output and Definitions

The output includes- Numerator should be a number from 0 to 30 and denominator should always be 30. Date is identified in the format of DD/MM/YYYY.

Evaluated Performance

Cohen's k = 90% (50 un-annotated documents - 25 events/25 attachments, search term ‘*MMSE*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) (Numerator) = 97%

Precision (Denominator) = 98%

Precision (Date- Same Day) = 68%

Precision (Date- One Week) = 76%

Precision (Date- Two Weeks) = 81%

Precision (Date- One Month) = 84% Instance level (Overall), Random sample of 100 Random Documents:Precision (specificity / accuracy) (Overall) = 93%

Recall (sensitivity / coverage) = 94%

Additional Notes

Run schedule – Weekly

Other Specifications

Version 1.0, Last updated:xx

Diagnosis

Brief Description

Application to extract instances of diagnosis.

Development Approach

Development approach: Rule-Based

Classification of past or present symptom: Both.

Output and Definitions

The output includes-

The main aim is to look for a standard or as close as possible to a definitive standard diagnosis:

1.) When reading through document, if you come across phrase(s) similar to the examples below:

……Diagnosis: Fxx.x diagnosis name……(this could be with or without the colon, or could even have several colons and/or other punctuation marks before they diagnosis name, following each

……Diagnosis Fxx.x diagnosis name……

……Diagnosis: diagnosis name……

……Diagnosis: Fxx.x……

Highlight this as ‘Diagnosis’ – please label the annotation just as I have specified it (i.e. with a CAPITAL D).

2.) The following features have been added under the Diagnosis annotation:

ICD10: if there is a name of a diagnosis, but no ICD10 code, find the ICD10 code and fill in under the feature ICD10

Diagname: if there is a diagnosis name then please copy this in the annotation feature. Please copy the exact diagnosis name even if it varies from the official ICD10 name.

Diffdiag – add this only if there is a differential diagnosis. This kind of diagnosis is often mentioned because usually most documents are trying to find out what the diagnosis is and in the process give a possible diagnosis which is vague or will not be the correct one eventually.

Nonpsychdiag – any definite diagnosis where the annotation does not come under the F group diagnosis. For example, COPD. Definitions: Search term(s):

Gazetteer of diagnoses and ICD10 codes.

Evaluated Performance

Cohen's k = xx Instance level (Lifetime), Sample of 50 Random Documents:

Precision (F20/ Schizophrenia) – 96%

Recall (F20/ Schizophrenia) – 63%

Precision (F20) - 100%

Recall (F20) - 65%

Precision (SMI) - 95%

Recall (SMI) - 43%

Precision (Schizoaffective) - 80%

Recall (Schizoaffective) - 29%

Precision (Depression) - 100%

Recall (Depression) - 40%

Precision(All patients with primary diagnosis of learning disability (f7 or learning dis*) in a structured field or unstructured text) – 93%

Additional Notes

Run schedule – Weekly

Other Specifications

Version 1.0, Last updated:xx

Treatment- Resistant Depression

Brief Description

Application to identify instances of treatment-resistant depression.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“X year history of treatment resistant depression”,

“problems with low mood (resistant depression)”,

“diagnosis: treatment resistant depression”,

“resistant endogenous depression, suffers from chronic treatment resistant depression”,

“referred for management of treatment resistant recurrent depression”

Exclude Unknown mentions:

“talked about ways in which they might resist allowing each other’s depression to …”,

“has a diagnosis of treatment resistant schizophrenia and depression”,

“we discussed him enrolling for a study of treatment resistant depression”,

“we talked about medication for treatment resistant depression”,

“resisted antidepressant therapy for a number of years”,

“needs an assessment to rule out treatment resistant depression”,

“assess whether depression was resistant to mirtazapine”,

“accepts that ECT is a strategy for treatment resistant depression” Definitions: Search term(s): depression [0-8 words in between] *resist* *resist* [0-8 words in between] depression

Evaluated Performance

Cohen's k = 85% (50 un-annotated documents - 25 events/25 attachments, search term ‘resistant depression’). Instance level , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 77% Recall (sensitivity / coverage) = 95% Patient level , All patients with primary diagnosis code of F32* or F33* in a structured field (Random sample of 50 Random Documents) Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Bradykinesia (Dementia)

Brief Description

To identify instances of bradykinesia in the context of dementia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

e Positive mentions:

“presence of bradykinesia”,

“motor symptoms – moderate bradykinesia L>R”.

Exclude Negative mentions:

“absence of bradykinesia”,

“he was moving easily in bed and transferring independently with no bradykinesia or tremor”

Exclude Unknown annotations:

“bradykinesia is a symptom of dementia”,

“difficult to assess if it has caused any bradykinesia”,

“SHO to look out for bradykinesia” Definitions: Search term(s): Bradykine*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term 'bradykine*’). Instance level , Random sample of 100 Documents:

Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 84%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Trajectory

Brief Description

Application to identify evidence from clinical text that something is getting better or getting worse.

Development Approach

Development approach: Rule-Based

Classification of past or present symptom: Both.

Output and Definitions

The output includes- Definitions:

Search term(s):

A1. The trajectory

Evidence from text that, at the time of writing, something has improved or deteriorated when compared with the past. Given the nature of clinical text the subject of change will normally relate in some way to the patient’s health; either directly (e.g. “Hallucinations have improved”) or indirectly (e.g. “living conditions are worse than they were’) .

The app does not restrict output to these, i.e. references necessarily relating to the patient and their health. References to other types of change therefore may be included, e.g. “Her piano playing has improved’; “His mother’s eye-sight is better”.

Exclusion:

- Conditional change: e.g., “His drinking is better when attends the session”; “If she smokes she feels worse”

- Modals suggesting uncertainly or potential: e.g., “she may have improved”; “his knee could deteriorate”

- Past change, i.e. not likely to be recent or relating to ‘now’ (the time of writing): e.g., “Symptoms improved when he was younger”; “The relationship with her mother deteriorated…” ; “He had been feeling worse…”

Evaluated Performance

Cohen's k = xx 1) Instance level , Random sample of 100 Documents: Precision= 97% Recall= 76%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Tremor (Dementia)

Brief Description

Application to identify instances of tremor in patients with dementia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“There was evidence of a tremor when writing…”,

“…with a degree of resting tremor…”

Exclude Negative mentions:

“There are no reports of any noticeable motor symptoms such as tremor…”,

“No dystonic movement or tremor”.

Exclude ‘Unknown’ annotations:

“ZZZZ will be reviewed with regards to side effects and if there is no tremor then can have another 75mg of Paliperidone”,

“there is a family history of tremor”. Definitions: Search term(s): *Tremor*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘tremor*’). Instance level (Overall) , Random sample of 100 Documents: Precision (specificity / accuracy) = 63%

Recall (sensitivity / coverage) = 96%

Precision (Random Sample of 100 documents with dementia diagnosis)- 83%

Recall – 92% Patient level (Overall) , Random sample of 30 Documents:

Precision (specificity / accuracy) = 83%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

QT

Brief Description

Application to identify instances of QT interval or Corrected QT interval, QTc.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“QTc now 497”,

“QTc 450”

“Uncorrected QT 384ms”

“QTc – 404”

“QTc: 421”

“QTC interval of 430ms”

“QTc= 442ms”

Excluded Positive mentions:

“increase the QTc by 1.3ms. (Excluded since the patient’s actual QTc not stated.)”,

“QTc interval was less than 440ms. (Excluded since the patient’s actual QTc not stated.)”,

“QT was given 7 days. (Excluded since QT is referring to the patient.)”,

“Date of birth: - QT- 1R "i '^n^^jSTtv^eN (Excluded since a random mix of letters not clinically relevant.)”,

“recommended QTC interval less than 440ms in men and less than 470ms in women. (Excluded since it is stating the recommended ranges.)” Definitions: Search term(s):

QT

QTc

Followed by

=, <, >, -

Followed by

number

Evaluated Performance

Cohen's k = xx Instance level (Overall) , Random sample of 100 Documents: Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – On Request

Other Specifications

Version 1.0, Last updated:xx

Forms

Brief Description

Application to identify documents that include form structures such as yes/no questions and other checklists.

Development Approach

Development approach: Rule-Based

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

Manual annotators look for forms as stand-alone documents or as part of the wider document text and annotate either positive (that text corresponds to a form and therefore the document contains a form or is a form), or negative (that text doesn’t correspond to a form and therefore the document neither contains a form nor is a form).

A form is identified based on:

1) Presence of the term ‘form’ within document heading

2) Presence of yes/no questions

3) Presence of checkboxes

Forms are identified as such even if they are not filled in, are part of a letter or email or correspond to symptom checklists.

The following rules were applied by the app to determine the presence of a form.

-Parse the HTML Text, identifying the text within tags.

-Identify the term “Form” within heading, paragraph of emboldened tags with lengths of less than 80

- Identify the presence of yes/no questions

-Identify the presence of check boxes

-Identify unique text to particular forms (this is an evolving part of the app that is updated as more information becomes available).

-Identify terms to exclude the document from being a form. These are terms that indicate the document may be a letter or an email (such as a greeting to open or close a letter, or terminology to indicate an email reply, such as “re:”.

Evaluated Performance

IRR = 92% - Cohen’s k could not be computed. Instance level (Overall) , Random sample of 100 Documents:

Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 76%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Quoted Speech

Brief Description

Application to extract text within quotation marks

Development Approach

Development approach: Rule-Based

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

Rules:

Regular expression matching is used to identify text occurring within matching quotation mark pairs [(" & "), (' & '), (“ & ”), (‘ & ’)] in the EHR. To avoid mistaking apostrophes used in contractions for the start of quoted phrase, a quote followed by a sequence (‘c’, ‘d’, ‘e’, ‘m’, ‘n’, ‘s’, ‘t’ , ‘ve’, ‘re’, ‘s’, ‘ll’, ‘all’ ) was treated as an apostrophe not a quote. A similar check was performed for end quotes. Once a quoted phrase was identified, any sub-quotations occurring within that quote were assumed to be part of the larger quotation. The length of quoted phrases was allowed to vary from one word to more than a paragraph; however, a maximum length of 1,500 characters was applied to avoid extracting the entire text where a quote was not properly closed. Phrases that consisted only of emails or starting with “https” were removed using standard regular expression pattern matching and substitution procedures.

Annotation guidelines:

Manual annotation involves identifying the full scope of human identifiable quoted speech.

Manual annotations that are deemed to be quoted speech if they include:

• Matched pairs of quotation marks as specified by the quotation algorithm [(" & "), (' & '), (“ & ”), (‘ & ’)].

• Mismatched quotation mark pairs e.g. “hello’

• Quotation mark pairs that start with the end quote e.g. ”goodbye“

• Incorrect/unusual quotation mark types that are not specified in the algorithm e.g. backticks in `x`

• Identifiable quotations that have no end quotation mark e.g. “I am always sad. The patient’s general mood was… Manual annotations that are not deemed to be quoted speech if they include:

• Emails or URLs (https)

Evaluated Performance

Cohen's k = xx Instance level (Overall) , Random sample of 100 Random Documents: Precision (specificity / accuracy) = 82%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx