Location:Home >> Detail
This work is licensed under aCreative Commons Attribution 4.0 International License
Journal of Psychiatry and Brain Science 2018; 3(5): 9; https://doi.org/10.20900/jpbs.20180009
Department of Psychiatry, University of Alberta, Edmonton, Alberta T6K 4C1, Canada.
*Corresponding Author: Yakov Shapiro. Email: firstname.lastname@example.org.
Marijuana is the most widely consumed recreational drug in the world. In Canada, physicians are experiencing increasing pressure to prescribe medical marijuana, with proposed legalization coming in late 2018. The use of marijuana in the psychiatric population is increasing, and the prescribing process is largely unregulated. In spite of several medicinal indications, chronic marijuana use is associated with serious consequences including early-onset psychosis, addiction, persistent psychosocial dysfunction, and neuropsychological abnormalities.
In this paper, we present the first spectral electroencephalography (EEG) study of brain changes during cannabis-induced psychosis coupled with a comprehensive review of the literature on medicinal and recreational marijuana use. The findings suggest that psychotic symptoms following cannabis are distinct from schizophrenic and affective psychoses, and occur as the consequence of a generalized shift to right hemispheric dominance. This is coupled with abnormal activation sources in the excitatory beta and gamma bands in the left temporo-parietal region, with impaired engagement of the relevant networks in both cognitive and spatial goal-directed tasks. Detailed recommendations for public education and prescribing process of medicinal marijuana use are discussed.
“Insanity is the dream of the man who is awake.”
Jacques-Joseph Moreau de Tours, author of “Du hachisch et de l’aliénation mentale: études psychologiques”, 1845
In the Western medical tradition, medicinal marijuana has been used for pain, vomiting, convulsions, spasticity and analgesia since the 1830s. While its use waned in the 20th century, the Institute of Medicine report  documents its utility in stimulating appetite in HIV-related wasting syndrome and chemotherapy-induced nausea/vomiting; chronic pain of various etiologies; and spasticity, such as in Multiple Sclerosis (MS). A recent comprehensive review by Volkow and others  documents marijuana efficacy in glaucoma; nausea/anorexia; chronic pain; inflammatory disease; MS; and epilepsy. Limited studies support its use in Tourette’s syndrome, Crohn’s disease, and amyotrophic lateral sclerosis (ALS). In Canada, medicinal marijuana is presently indicated for patients with HIV/AIDS; severe arthritis, spinal cord injuries; cancer-related pain and anorexia; epilepsy; and multiple sclerosis spasticity .
Unfortunately, the evidence for marijuana effectiveness in treating psychiatric syndromes is presently lacking. Even though limited studies indicate the potential usefulness of synthetic cannabinoids (nabilone) in post-traumatic stress disorder (PTSD) related sleep dysfunction  marijuana use has been shown to exacerbate the course of PTSD  as well as psychotic and anxiety disorders . There is no evidence that marijuana-induced euphoria translates into any sustained antidepressant effects, and marijuana use worsens the course of depressive and bipolar disorders and leads to higher suicide risk . A recent review of marijuana effects on sleep  concluded that cannabis does not offer any hypnotic benefits outside of pain reduction. In the United States, of the 24 states with medicinal marijuana laws no jurisdiction approves its use for psychiatric conditions except PTSD . Nevertheless, over 25% of medicinal marijuana prescriptions in California were given for mood disorders, with insomnia and anxiety being the next most common indications .
The prevalence of cannabis use as a recreational drug is staggering. According to the 2015 World Drug Report compiled by the United Nations Office on Drugs and Crime (UNODC) , 181.8 million adults (age 15 to 64) worldwide resort to cannabis use, which represents up to 4.9% of the adult world population, higher than all the other illicit drugs combined. In the US, the annual prevalence of marijuana use among adults was at 11.6%; among high-school students at 25.8%; and among grade 12 students up to 36.4% in 2013. Men are 3 times more likely to use cannabis than women, and some 17% of men and 10% of women become regular users once initiated. When strict criteria for drug dependence are used, the 10–year risk of marijuana dependence among new users is estimated at 8–9%, compared to 12–13% for alcohol, and 15–16% for cocaine . However, the risk of dependence goes up to 17% among teen marijuana users, and up to 50% among daily users. According to the 2012 US Survey on Drug Use and Health, 2.7 million Americans (12 years and older) met the DSM-IV criteria for marijuana dependence, which represented over 50% of all drug dependence in the country excluding alcohol . Cannabis addiction accounts for over 40% of the drug rehabilitation population in North America, and over 30% in Western Europe, higher than any other drug type. These findings are consistent with the animal models in rodents, dogs and rhesus monkeys documenting a distinct cannabinoid withdrawal syndrome with self-administration paradigm .
The chemical composition of common marijuana (Cannabis sativa) includes over 600 substances, more than 70 of which are cannabinoids . The most common is Δ9–Tetrahydrocannabinol (THC), which binds to endocannabinoid CB1 receptors responsible for the psychotropic effects of marijuana such as euphoria, perceptual disturbances, cognitive changes, as well as potential for anxiety and psychotic reactions. The three main forms of natural cannabis products are leaves and flowers, resin (hashish), and oil. The UNODC estimates that cannabis leaves contain up to 5% THC, resin up to 20% THC, and oil up to 60% THC. However, cannabidiol (CBD), a cannabinoid marijuana ingredient, is a 5–HT1A receptor agonist, the mechanism that may contribute to its anti-anxiety, antipsychotic and anticonvulsant properties [14,15]. Cannabis potency as well as its potential to cause harmful effects is measured in terms of its THC content, which is highly lipophilic with a half-life of 30 h. It has a bioavailability of up to 0.25 when smoked and 0.05 to 0.2 when ingested . In contrast to inhaled THC-containing products, which take effect within minutes and last around 2 to 3 hours, oral consumption has a delayed effect of between 30 and 90 min and lasts from 4 to 12 h. Significantly, THC accumulates in the brain, and serum concentrations do not always correlate with psychoactive effects.
Higher THC intake has been associated with anxiety, depression, increased risk of dependence, and psychotic symptoms among others . A recent study published in The Lancet Psychiatry  suggests that the risk of psychosis is three times higher in users of high-potency herbal cannabis (“skunk”) than in non-users. It is therefore concerning that THC potency in street marijuana samples has been increasing in the past two decades, particularly with the use of sinsemilla (unfertilized female flowers) characterized by high THC/low CBD ratio . The THC content of recreational cannabis in the United States increased from 3.7% in 1993 to 12.6% in 2013, whereas CBD content remains low at 0.4%. Correspondingly, the US data showed an increase in cannabis-related treatments from 6.9% of all drug-related visits in 1993 to 17.5% in 2012, with increasing cannabis-related hospital admissions .2.2 Neuropsychiatric and EEG changes in chronic cannabis users
There is extensive evidence that chronic heavy marijuana use, defined as using the drug daily or on most days, is associated with serious clinical consequences, both neuropsychiatric, medical and psychosocial. Regular adolescent exposure is particularly problematic because it correlates with higher risk of psychotic illness, higher suicide rates, psychosocial maladjustment, educational underachievement, persistent neuropsychological deficits, and impaired learning ability with declines in overall IQ . Cannabinoids readily cross the placental barrier and accumulate in breast milk, and prenatal exposure has been linked to executive deficits in cohort studies . Finally, smoked cannabis increases clearance of some antipsychotics and antidepressants through CYP 1A2 induction, leading to decreased antidepressant and antipsychotic drug efficacy, and may cause CNS depression in combination with anxiolytics .
Some of the established adverse consequences of chronic marijuana use include:
Persistent neuropsychological and neurophysiological changes following heavy cannabis use have been documented in a number of studies. Meier and his team  followed a cohort 1,037 individuals from birth to age 38, and assessed IQ at the age of 13 before the onset of use, and at age 38. Persistent users showed evidence of dose dependent neuropsychological impairment in both verbal and performance domains irrespective of the highest educational level achieved. Significantly fewer persistent cannabis users obtained post-secondary education. Adolescent-onset users showed greater decline by as much as 8 IQ points, and cessation of cannabis did not fully restore neuropsychological function.
A structural study by Matochik and others  demonstrated lower gray matter density in the right parahippocampal gyrus and higher white matter density in the left parahippocampal and fusiform gyri in heavy marijuana users compared to non-users. These results correlated with an impairment on a decision-making task and PET findings of hypo-perfusion in the right orbitofrontal and dorsolateral prefrontal cortex in a dose dependent relationship to the amount of marijuana used. The authors speculated that white matter density increases may be compensatory to marijuana-induced grey matter damage in neighboring regions. Similarly, Yucel and his team  showed bilateral decrease in hippocampus and amygdala volumes in chronic marijuana users, reductions in left hippocampal volume showing linear association with both cumulative cannabis exposure during the previous 10 years and subthreshold positive psychotic symptoms.
A functional MRI study by Eldreth and others  demonstrated persistent deficits in executive functioning in heavy marijuana users abstinent for 28 days in a dose dependent relationship with the number of joints consumed per week. These were associated with hypoactivity of the left anterior cingulate and dorsolateral prefrontal cortex coupled with hyperactivity in hippocampal areas bilaterally. The results provide evidence for persistent cognitive impairment in heavy marijuana users compensated by recruiting additional brain networks for cognitively demanding tasks. Similar findings were reported by Nestor and others , chronic cannabis users showing deficits in learning, short- and long-term memory performance, which were associated with hypoactivity in the frontal and temporal cortices and relative hyperactivity in the right parahippocampal area. Finally, Grubera and others  found a decrease in both anterior cingulate and amygdala activity during masked affective stimuli in chronic heavy marijuana users suggesting differences in affective processing due to altered activation of the frontal and limbic networks.
In the area of electrophysiology, specific EEG changes following THC injection has been observed in rhesus monkeys, with biphasic slow-to-fast high voltage spikes in fronto-temporal lobes and hypothalamus correlating with behavioral depression and secondary agitation . Ilan and others  showed that acute marijuana smoking in normal human subjects has been implicated in impaired working and episodic memory function correlating with disruptions in the normative alpha power decrease/beta power increase in the fronto-central regions in response to increasing working memory load. In a quantitative follow up study the same team investigated differential effects of THC, CBD, and cannabichromene (CBC), replicating their previous findings and suggesting that “systematic variations in the levels of two non-THC constituents, CBC and CBD, did not significantly change any of the subjective, physiological or performance effects of marijuana” . They concluded that impaired task performance was attributable to the effects of THC consumption.
An EEG study by Struve and his team demonstrated that compared to controls, daily cannabis users show increased alpha power and coherence over frontal and prefrontal areas bilaterally, as well as significantly higher levels of delta and theta coherence over frontal cortical areas, suggesting functional hypofrontality not limited to acute THC effects. The authors proposed that “when cumulative daily THC exposure becomes prolonged, the acute transient THC induced EEG changes become permanent or quasi-permanent” (p. 175) . In keeping with these findings, a study by Herning and others  showed sustained EEG changes in a dose-dependent relationship to the amount of marijuana use (more than 8 years), with decreased high alpha and high beta power in posterior (occipital) areas.2.3 Cannabis and psychosis
The specific focus of this study is on the association between cannabis and psychotic states. While the randomized controlled trial data to establish a causal relationship between cannabis use and the onset of chronic psychosis is still lacking, numerous controlled studies (cross-sectional and longitudinal cohort) established the cannabis/psychosis association and controlled for confounding factors such as reverse causality (prodromal psychosis leading to cannabis use), socioeconomic influences, pre-existing trauma, and concomitant illicit drug use. A 15–year follow up of 45,570 Swedish conscripts  found the relative risk of schizophrenia to be 6.0 in high users (50 times or more) than in non-users. The same group  later expanded the sample to 50,087 subjects followed for over 27 years. The overall risk ratio was 6.7 for heavy users fell to 3.1 when adjusted for concomitant drug use, personality and social factors, but showed a dose-dependent relationship with the amount of cannabis used. Based on their data, the authors estimated that up to 13% of all new schizophrenia cases could be prevented by avoiding marijuana use.
A longitudinal study in the Netherlands  followed 4,045 psychosis-free adults (age 18 to 64) for 3 years; it concluded that cannabis use increased the risk of psychosis both for patients with or without lifetime psychotic vulnerability. The results showed a dose-dependent relationship, with occasional cannabis use resulting in 1.23 odds ratio (OR) of developing psychotic symptoms, which increased to OR of 6.81 in heavy cannabis users. Significantly, the subgroup of subjects with severe psychosis showed OR of 7.9 with occasional use and as high as 74.67 in the heaviest users, even after adjustments for age, sex, ethnic group, marital status, educational level and concurrent drug use.
A prospective study of 759 New Zealand teens  similarly showed a dose-dependent relationship between marijuana use and the onset of schizophreniform symptoms. Specifically, earlier consumption (age 15) was associated with higher risk, with 10% of early users presenting with psychotic symptoms by the age of 26 compared to 3% of controls. Similar findings were reported by Fergusson and others  in a cohort of 1,265 children (50% males; 50% females) followed from birth. By the age of 18, the rate of psychotic symptoms was 3.7 times higher in those meeting DSM-IV criteria for cannabis dependence. Sibling pair analysis  in 228 twin pairs over 14 years of follow up demonstrated the adjusted risk of developing non-affective psychosis to be 2.2 times in a cannabis-using twin compared to the abstinent sibling.
Not only the overall risk but the age of onset of psychosis shows a direct relationship to marijuana use. An Australian study of 997 participants  showed a linear association between the initial age of cannabis use and psychosis onset, while DiForti’s team in Britain  demonstrated that earlier age of initial use (before the age of 15) and high-THC marijuana use (skunk) advanced the onset of the first psychotic episode by 3 and 5 years respectively. Finally, a comprehensive meta-analysis by Large and others  suggested a causal role of cannabis in the development of psychosis, with onset 2.7 years earlier in heavy users.
The critical research findings reviewed by Pierre  can be summarized as follows:
There are few available studies on EEG changes in cannabis-induced psychosis. An early study by Campbell  identified excessive theta wave activity with diffuse dysrhythmia most prominent in fronto-temporal areas bilaterally. Morrison and others at King’s College London  documented decreased theta coherence in the frontal lobes bilaterally following THC administration, which correlated with positive psychotic symptoms. More recently, the same team  utilized a double blind design to demonstrate a shift towards higher frequency bands following IV THC administration, correlating with positive symptom score on Positive and Negative Symptom Scale (PANSS), a standard measure of psychotic presentations.
The purpose of this pilot study was to look for any cannabis-specific electrophysiological findings across the spectrum of psychotic presentations, and to build on the limited data on cannabis-induced psychosis available in the literature. Specifically, laterality data on the differential involvement of the right- versus left-hemispheric systems in cannabis-induced psychosis is presently lacking. Our study examined a group of cannabis-induced psychosis patients with spectral EEG and BEAMFORMER analysis in order to determine sources of abnormal brain activity during rest, verbal and spatial cognitive tasks compared to normal controls.
A comprehensive database compiled at the Alberta Hospital Edmonton Clinical Diagnostic and Research Centre (CDRC) over a period of 20 years has been utilized in this pilot study. Normal volunteers and patients admitted to the acute psychiatric unit underwent comprehensive clinical interview to exclude any illicit drug use, medication, alcohol abuse, migraine or other neurological disorders, comorbid psychiatric disorders, birth complications, and head trauma. In addition, The Diagnostic Interview Schedule, version IV , Basic Personality Inventory , and Multidimensional Aptitude Battery , which provides verbal, performance and full scale IQ, we administered to all controls and psychiatric inpatients included in the database.
For the purpose of this study, sixteen consecutive dextral male subjects (average age 27) hospitalized with cannabis induced psychosis and characterized as a mixed psychotic state (schizophrenia spectrum, schizomanic, bipolar manic, bipolar mixed, bipolar depressed, and delusional hypomanic components) according to the DSM-IV criteria  were compared to 62 healthy male controls (average age 30.5, t = 1.3288, df = 86, p < 0.187, ns). Both groups were investigated with a 48 channel spectral EEG system and BEAMFORMER source localization. All subjects were drug-free and unmedicated at the time of their data recordings (more than 5 half-lives of any antipsychotic, antidepressant, anxiolytic, or mood stabilizer agent that had been prescribed). All subjects were completely right handed as determined by the handedness questionnaires.
EEGs were recorded from all control subjects during a passive condition (eyes open, EO or eyes closed, EC), a spatial cognitive task (dot localization—DL), and a verbal activation task (WF) as developed by Miller and others  The cannabis group (except for one patient) was only examined under passive conditions because the patients were too disorganized to be able to complete DL and WF tasks. During the passive condition, the subjects were told to relax and either keep their eyes closed or fix their gaze on a point across the room. The DL task contained 25 items, each consisting of two vertically arranged rectangles, the top rectangle containing a pair of dots and the bottom rectangle, slightly offset to either the right or the left, containing an array of numbers. The offset of the bottom rectangle differed randomly among items with half of the items offset to the right or the left side. Item difficulty was determined by the size of the number arrays, which varied in five levels, from 8 to 50 numbers. The subjects were asked to estimate which numbers align with the pair of dots above. In the WF task, subjects were asked to correctly identify an object when presented with a written definition. For example, the correct answer for the item: “a very large piece of floating ice” is iceberg. The task consisted of 30 differing items in ascending order of difficulty.
The research protocol as outlined above was reviewed by the Ethics Review Committee of Alberta Hospital Edmonton and found to be acceptable within the limitations of human experimentation.3.2 Recordings, spectral EEG and BEAMFORMER analysis
For technical details on the recording procedures please refer to our previous work with CDRC database .
In the normal dextral population, the right/left alpha power ratio increases during verbal tasks, indicating dominant left hemisphere activation, and correspondingly decreases during non-verbal tasks (Fig. 1A). The R/L alpha power ratio [(right − left)/(right + left)] was significantly smaller in the patients with cannabis-induced psychosis, achieving significance with negative values in 2 out of 18 locations: fronto-central at FC6/FC5, and central at C2/C1 compared to positive values in the controls (Table 1).
These findings suggest a greater degree of right hemisphere activation and relative left hypofunction in the cannabis group. The reversal of the alpha power ratios indicates that the normal pattern of hemispheric dominance is altered in the direction of the right hemisphere in the resting EC condition.
Outside of the findings in the alpha power ratio, significant differences on the quantitative EEG analysis were observed, which included increased activation in the low beta band (14–20 Hz) at C1/C2 (p < 0.02) and in the high beta (21–50Hz) at CP5/CP6 (p < 0.03); P3/P4 (p < 0.02); and P1/P2 (p < 0.04), suggesting abnormal activation in the central and central-parietal regions bilaterally. There were no significant differences in the theta (4–7 Hz) and gamma (40 Hz) bands.
The source analysis showed increased sources in the eyes open condition at 14–20, 21–50 Hz and 40 Hz.
For the cannabis patient who completed the verbal and spatial activation tasks, the data is presented on Fig. 2. In the passive (EC) condition, the subject’s D-scores that provide statistical separation between the mean distributions fall outside the control range (red vs. orange circles) and clusters between the spatially activated vs. cognitively activated controls, suggesting abnormal overactivation bilaterally in the resting state. By contrast, under the conditions of either spatial or cognitive activation (DL or WF), the subject fell fully within the resting domain for the normal population. This finding may have implications for the capacity of cannabis-impaired subjects to engage relevant brain circuitry in goal-directed tasks, whether verbally or spatially based, although further studies would be required to corroborate this finding.
The BEAMFORMER source analysis by the power band distribution shows that in the alpha frequency (8–13 Hz, EC condition), the cannabis group had decreased power over the whole of the left hemisphere and posteror-frontal and temporo-parietal aspects of the right hemisphere, suggesting generalized overactivation in these regions (Fig. 3A–D).
In the low beta frequency (14–20 Hz, EO condition), there is an area of increased sources in the left temporo-parietal region (Fig. 4A(1,2)). A similar pattern of increased sources in the left temporo-parietal region is evident in the high beta (21–50 Hz, EO condition, Fig. 4B(1,3)), and in the gamma band (38–42 Hz, EO condition, Fig. 5A,C).
In spite of its small size, this pilot study demonstrates for the first time that patients with cannabis-induced psychosis may carry a unique psychophysiological signature, which includes:
In comparison to our spectral EEG findings in non-cannabis psychosis that utilized the same control and inpatient database, schizophrenia patients show significant increases in the beta band source-current density in the left fronto-temporal region during both resting and cognitively challenged conditions, suggesting primary disorganization of left-hemispheric networks in schizophrenia . By contrast, affective psychosis patients (depressive and manic-depressive) demonstrate primary right hemispheric disorganization . These comparisons are relevant because all 3 studies utilized the same protocol and age-matched control population, and none of the CDIC database data were confounded with psychoactive medication, concomitant recreational drug use or neurological conditions. This allows for functional differentiation between relevant diagnostic clusters of psychotic sub-populations.
Our pilot study of cannabis-induced psychosis suggests increased right brain activation leading to secondary left hemisphere disorganization through contralateral disinhibition, the mechanism similar to affective psychoses rather than schizophrenia-spectrum presentations. However, less extensive involvement of the left hemispheric systems in cannabis patients may imply that they belong to a distinct subcategory with potentially better functional outcomes given ongoing marijuana abstinence.
While only one cannabis subject was able to complete the active cognitive protocol, the findings of resting overactivation and marked impairment to activate relevant brain circuitry on both verbal and spatial tasks bilaterally would suggest executive difficulties differentiating between internal versus relevant environmental stimuli, which is consistent with published data documenting functional hypofrontality, absence of normal hippocampal lateralization on attention and memory tasks, and lower activation in the right orbitofrontal and dorsolateral prefrontal cortex in decision making tasks in both current and abstinent cannabis users . However, our findings have to be interpreted with caution since they are based on a single subject who was able to complete the active portion of the protocol (WF and DL tasks).
Marijuana-induced alterations in the individual’s sense of self, time perception, and deficits in verbal/analytical reasoning are typical of other drug- and meditation-induced states. We recently described brain changes during a self-induced shamanic trance in a normal subject  and postulated a general neurophysiological model for altered states of consciousness (ASCs), which involves a shift from the left hemisphere-dominant “causal operator” mode of self-awareness in ordinary states (OSCs) to the right hemisphere-dominant “holistic operator” mode (Fig. 1B). This is coupled with a shift from the left prefrontal dominance (“anterior self” mode that allows for an autobiographical experience of a subjective observer interacting with others and outside world), to the posterior “sensorimotor self” mode characterized by the dissolution of ego boundaries and synesthetic experiences.
Our findings of right hemispheric dominance and more posterior left-hemispheric activation sources in cannabis-induced psychosis suggest that we may be dealing with a more generalized phenomenon common to altered and psychedelic experiences. This is in keeping with Carhart-Harris’ and others  magnetoencephalography and fMRI data that suggests a decoupling between the default mode network and the medial temporal lobes in psychedelic-induced states. While contextual set and setting effects can impact on psychedelic drug experience in general and cannabis effects in particular, such effects are unlikely to serve as a significant factor in clinically psychotic population.
Limitations of this pilot study include small sample size and the fact that only one psychotic subject was able to complete DL/WF tasks; limited information about the psychotic subjects’ history of concomitant recreational drug use; and lack of non-psychotic cannabis user comparison group to separate background cannabis effects. However, our findings are consistent with previously demonstrating functional hypofrontality effects in acute and chronic THC users [27,35].
This study adds to the considerable body of evidence that chronic cannabis use can be associated with serious psychiatric consequences that are not limited to its acute intoxicating effects. While there is preliminary data that cannabinoids are beneficial in a number of medical conditions, the perception of marijuana as a benign recreational drug or harmless “natural medicine” needs to be challenged given the very real burden of addiction, mental illness, and psychosocial maladjustment that its heavy use entails. While marijuana use is not associated with acute mortality seminal in the current opioid crisis in North America, high-THC preparations carry a heavy burden of psychosocial and psychiatric morbidity, and high addiction rates among daily users.
Unfortunately, prescribing marijuana as a medicine in both medical and psychiatric populations is driven largely by economic and political factors, with little attention to the individual’s mental health history and the substantial health risks involved. Several Canadian, British, and American studies suggest that a majority of medicinal marijuana users are recreationally familiar with the drug, and medicinal use has largely “occurred within a context of chronic use” . Medicinal marijuana prescribers also tend to pay insufficient attention to their patients’ psychiatric history and concurrent psychotropic medication use, including predisposition to psychotic breakdowns.
Based on the results of this review and pilot study, the authors would suggest the following psychiatric recommendations for medicinal marijuana prescribers and legislators:
P.F-H. conceived and carried out the experiments. Y.S. took the lead in writing the manuscript. Both authors discussed the results and their implications.
The authors declare no conflict of interest.
Flor-Henry P, Shapiro Y. Brain Changes during Cannabis-Induced Psychosis: Clarifying the Marijuana Medicine/Harm Dichotomy. J Psychiatry Brain Sci. 2018; 3(5): 9; https://doi.org/10.20900/jpbs.20180009