This is an intersection of many difficult aspects:
* LLM Chatbots are currently perceived as a safe space where suicidal thoughts can be discussed, without fear that a welfare check and involuntary commitment is going to be triggered. Attach liability to this and the safest/easiest thing for the AI companies to do is always refuse the prompt if suicide is discussed. If they are required to inform the authorities, your chat account will need location services.
* Involuntary holds are a good solution to some kinds of suicide risk, particularly those that are either acute or treatable with medication. But they actually make some patients worse off by destroying trust, disturbing life patterns, and saddling with debt.
* Mental health care is famously expensive, and not so accessible. LLMs are available 24/7 and are cheap. There's both a huge danger to unqualified advice and opportunity here for accessibility.
* Even if we claim that LLMs should not act as a therapist, they still need training/evaluation on mental issues. AI psychosis will keep happening unless we start training and evaluating AI on intervening appropriately.
* There are probably more subtle forms of AI psychosis that are not being detected yet. AI sycophancy plus lack of context means AI can be highly reinforcing in areas in which someone would actually benefit from being challenged.
This is an intersection of many difficult aspects:
* LLM Chatbots are currently perceived as a safe space where suicidal thoughts can be discussed, without fear that a welfare check and involuntary commitment is going to be triggered. Attach liability to this and the safest/easiest thing for the AI companies to do is always refuse the prompt if suicide is discussed. If they are required to inform the authorities, your chat account will need location services.
* Involuntary holds are a good solution to some kinds of suicide risk, particularly those that are either acute or treatable with medication. But they actually make some patients worse off by destroying trust, disturbing life patterns, and saddling with debt.
* Mental health care is famously expensive, and not so accessible. LLMs are available 24/7 and are cheap. There's both a huge danger to unqualified advice and opportunity here for accessibility.
* Even if we claim that LLMs should not act as a therapist, they still need training/evaluation on mental issues. AI psychosis will keep happening unless we start training and evaluating AI on intervening appropriately.
* There are probably more subtle forms of AI psychosis that are not being detected yet. AI sycophancy plus lack of context means AI can be highly reinforcing in areas in which someone would actually benefit from being challenged.
Full Title Too Long:"She told no one about her agony except ChatGPT. What her death reveals about AI risks"