Participants in the SST program receive a comprehensive blend of behavioral health, medical and social supports delivered by a coordinated multidisciplinary team. The team conducts mobile diagnostic assessments and facilitates communication with and access to medical and behavioral health facilities and providers. Staff also offer health education and medication management and ensures coordinated risk assessment and crisis response.

In order to be eligible for the SST program, a person must:

• Be an Allegheny County resident, age 18 and older.
• Be currently unhoused/experiencing street homelessness. 
• Demonstrate severe psychiatric symptoms related to mental health or substance use issues. 
• Be unwilling or unable to engage in traditional services due to severity of symptoms.
• Be assessed as highly vulnerable based on a street assessment and/or risk model. 

Because SST-eligible clients are managing severe and unmet behavioral health needs and are routinely disengaged from human services programs, no single referral process can be expected to identify all eligible individuals; rather, SST referrals are made through two channels: provider referrals and data-driven referrals. 

After an individual is identified through either referral pathway, the SST enrollment process begins with a case conference in which the team reviews available information to determine whether the person’s needs align with the program’s eligibility criteria. This review process is augmented with summaries provided utilizing a Large Language Model (LLM) based on data from the Homeless Management Information Systems (HMIS) and a number of Electronic Health Record (EHR) systems.

The data-driven referral list is created using a composite of four data-driven models and filtered based on the set of eligibility criteria previously described. The composite is a deterministic function of four data-driven models that predict binary outcomes (0,1)

SST staff and clients are never shown the raw composite rating or component-model outputs. An individual excluded from the data-driven referral list for any reason is still eligible for a provider referral.

The composite is used as a proxy for co-occurring severe psychiatric symptoms, chemical dependency, and acute risk of morbidity and mortality. The referral list includes the subset of people who have the highest composite ratings based on the model assessments after filtering for eligibility.

Considering outcomes for the top 50, 100 and 200 people on the list (after filtering for eligibility) helps describe the severity of the situation of the people who are referred. We expect more than 90% of people on the referral list to experience one of the four predicted outcomes in the year after their referral date. In most cases, they will experience more than one.

In addition to performance, we used standard measures to assess the fairness of the data-driven model referrals. The key metric, the sensitivity ratio, is the ratio of recall among demographic subgroups. Recall is the fraction of all cases among the cohort referred by the model in which one or more outcomes occur. We used the Sensitivity Ratio to judge whether the model assigned output fairly based on subgroup demographic. As no process will have a Sensitivity Ratio of exactly 1.0, we consider models with Sensitivity Ratios between 0.7 and 1.3 as performing well according to this fairness diagnostic.

To make this comparison, we ran the referral process once per month on historical data for each month of 2024. The model returned 1,200 referrals, representing 284 unique individuals.

Race

With regards to race, the composite showed slightly higher precision and slightly lower recall for non-White individuals compared to a reference group of White individuals.

Gender

When comparing gender, we used binary male and female labels because gender data are tracked as a binary in the Allegheny County data warehouse. The composite showed slightly higher precision and slightly lower recall among females than among males.

The summaries act as a starting point for case conceptualization. They provide plain text descriptions of a client’s interactions with County services and institutions. Below is an example report that includes synthetic data on housing services and evictions as well as behavioral health services and diagnoses.

The goal of the LLM is to format structured data to be more legible and easily accessible to a member of the care team. It should not make choices about what information to include or not include. We used an LLM-as-judge approach with a private GPT 4.1 deployment in our cloud tenant to evaluate performance on a sample of 5,000 summaries. Additionally, we used human review to assess the quality of the LLM judge.

We employed the following LLM-as-judge metrics:

• Recommendations: The fraction of summaries flagged by a GPT 4.1 evaluator to contain a recommendation or call to action for the reader
• Information Modified: The fraction of summaries flagged by a GPT 4.1 evaluator to contain event descriptions that are rewritten in a way that changes the meaning of an event from the input artifact
• Information Excluded: The fraction of summaries flagged by a GPT 4.1 evaluator that omitted an event in the input artifact

The final metric is event miscounts, the fraction of summaries featuring event counts that do not correspond to the count in the input artifact. Event miscounts are computed using regex.

Beginning in late 2023, ACDHS gathered input from stakeholders and practitioners in community mental health and homelessness services to inform and refine the development of the SST design.

The full list of stakeholder engagement conversations, including the purpose and timing of each event, is captured in the appendix of Introducing the Street Stabilization Team Report (PDF, 1MB)

Introducing the Street Stabilization Team Report (PDF, 1MB)