Detecting Mental Health
Conditions From How We Write
Can social media text reveal signs of depression, anxiety, PTSD, or ADHD? We trained four classifiers on 60,000+ Reddit posts to find out — and asked what AI can and cannot do with that knowledge.
# Mental Health Detection — NLP Tutorial
CMSC320 Final Project Notebook¶
Team Members & Contributions
Armin Ahmadkhaniha Created feature engineering the cleaned dataset by adding punctuation count, Vader score of each post and count length size for making data ready for training. Designed and implemented Random Forest model to analyze how nonlinear model would predict and contribute to this project. Lastly first hand reviewer of the project to make sure about everything works and ready for upload.
Fatima Mbodji Proposed the project idea and designed the full ML pipeline (Logistic Regression, SVM, Neural Network). Built the NLP input pipeline (TF-IDF, Keras Tokenizer, pad_sequences), trained and evaluated all three models, and analyzed results. Coordinated team task assignments and dependency order across the 3 checkpoint, and final reviewer of this tutorial to ensure code is reproducible top-to-bottom before submission.
Siraj Khan: Worked on the full data curation pipeline. Merged and cleaned raw data, performed hypothesis tests to confirm findings in EDA. Write project introduction (motivation, research questions) and summarize EDA findings in prose
Vashawn Robinson: built the text-preprocessing pipeline that cleaned, tokenized, and removed stopwords from the raw posts, creating the clean_text column used for EDA and modeling. Caught two data issues, which was an unmapped “Personality disorder” label and pandas incorrectly treating the “None” class as missing values. Also completed the first part of the EDA, helped put together the final tutorial notebook, and set up the team's GitHub pages site.
Project Summary¶
Social media has become part of day-to-day life. People share their thoughts, ideas, and feelings in the virtual world — but can they also share symptoms of mental health issues without even knowing it? This project's goal is to detect signs of mental health conditions using social media data and people's writing patterns.
Mental health affects 1 in 8 people worldwide, yet many cases go undetected until it is too late. Early detection tools are still hard to access and expensive. Data science can make a real difference here — by using publicly written data, we can build low-cost, scalable tools that flag potential concerns early.
We use Natural Language Processing (NLP) and Machine Learning to classify Reddit posts across multiple mental health conditions:
- Depression, Anxiety, Bipolar Disorder, ADHD, PTSD, and a Normal baseline.
Our dataset combines two sources:
- Murarka et al. (2021) Reddit Mental Health Dataset (ACL 2021)
- Sarkar Kaggle Dataset
Together these provide over 60,000 labeled samples.
This notebook is a hands-on tutorial walking through the full data science pipeline — from data collection and preprocessing, to EDA, hypothesis testing, model comparison, and neural networks — with a focus on NLP and responsible, ethical AI.
Part I — Data Curation¶
We use the Murarka et al. (2021) dataset, which labels Reddit posts by mental health condition. A small subset of posts have no condition label — we reserve these as a final out-of-scope test to see how the model handles them compared to human observation.
To establish an accurate diagnostic boundary, we map baseline posts (Normal from Combined Data-2) into a unified Normal class, teaching our models what "healthy" mental health language looks like.
The dataset is then split into two branches:
- Main 6-Class Dataset (
mental_health_6_classes_cleaned.csv) — Contains 5 clinical conditions + Normal baseline; used for EDA, TF-IDF, and model training. - Out-of-Scope Holdout Set (
mental_health_out_of_scope_holdout.csv) — PreservesSuicidal,Stress, andPersonality Disorderposts separately for our final experiment comparing AI predictions against human observations.
import re
import pandas as pd
import numpy as np
## Load Datasets
df_train = pd.read_csv('both_train.csv')
df_val = pd.read_csv('both_val.csv')
df_test = pd.read_csv('both_test.csv')
df_ds1 = pd.concat([df_train, df_val, df_test], ignore_index=True)
df_ds1['text'] = df_ds1['title'].fillna('') + " " + df_ds1['post'].fillna('')
df_ds1['raw_label'] = df_ds1['class_name']
df_ds1['source'] = 'Dataset_1_Reddit'
df_ds1_clean = df_ds1[['text', 'raw_label', 'source']]
df_ds2 = pd.read_csv('Combined Data-2.csv')
df_ds2['text'] = df_ds2['statement']
df_ds2['raw_label'] = df_ds2['status']
df_ds2['source'] = 'Dataset_2_Combined'
df_ds2_clean = df_ds2[['text', 'raw_label', 'source']]
## concatenate
df_raw = pd.concat([df_ds1_clean, df_ds2_clean], ignore_index=True)
initial_count = len(df_raw)
print(f"Total raw rows loaded: {initial_count:,}")
# Drop null texts or null labels
df_clean = df_raw.dropna(subset=['text', 'raw_label']).copy()
after_nulls = len(df_clean)
print(f"Dropped {initial_count - after_nulls:,} missing/null rows -> Remaining: {after_nulls:,}")
# Remove duplicates
df_clean = df_clean.drop_duplicates(subset=['text']).reset_index(drop=True)
after_dupes = len(df_clean)
print(f"Dropped {after_nulls - after_dupes:,} duplicate rows -> Remaining: {after_dupes:,}")
## remapping
def clean_raw_artifacts(text):
text = str(text)
text = re.sub(r'&#x200b;|&|>|<|​', ' ', text) ## Decode HTML entities
text = re.sub(r'\[removed\]|\[deleted\]', '', text, flags=re.IGNORECASE) ## Remove Reddit tags
text = re.sub(r'\s+', ' ', text).strip()
return text
df_clean['text'] = df_clean['text'].apply(clean_raw_artifacts)
## Define our 6-Class Training Schema vs Out-of-Scope Holdout Schema
schema_mapping = {
'adhd': 'ADHD',
'depression': 'Depression',
'anxiety': 'Anxiety',
'bipolar': 'Bipolar',
'ptsd': 'PTSD',
'none': 'Holdout_Reddit_None', ##preserved for testing
'Normal': 'Normal', ##base
'Depression': 'Depression',
'Anxiety': 'Anxiety',
'Bipolar': 'Bipolar',
'Suicidal': 'Holdout_Suicidal', ##preserved for testing
'Personality disorder': 'Holdout_Personality_Disorder', ##preserved for testing
'Stress': 'Holdout_Stress' ##preserved for testing
}
df_clean['mapped_label'] = df_clean['raw_label'].map(schema_mapping)
df_clean = df_clean.dropna(subset=['mapped_label']).reset_index(drop=True)
## Word count
df_clean['word_count'] = df_clean['text'].apply(lambda x: len(x.split()))
p99_threshold = df_clean['word_count'].quantile(0.99)
df_final = df_clean[(df_clean['word_count'] >= 5) & (df_clean['word_count'] <= p99_threshold)].copy()
df_final = df_final.reset_index(drop=True)
print(f"Filtered posts under 5 words and above 99th percentile ({p99_threshold:.0f} words).")
print(f"Total clean rows ready for branching: {len(df_final):,}")
##Main 6-Class Training & EDA Dataset with base line
df_main = df_final[~df_final['mapped_label'].str.startswith('Holdout_')].copy()
df_main.rename(columns={'mapped_label': 'label'}, inplace=True)
file_train_eda = 'mental_health_6_classes_cleaned.csv'
df_main[['text', 'label', 'source', 'word_count']].to_csv(file_train_eda, index=False)
## Out-of-Scope Holdout Dataset (For Human vs. Model Experiment)
df_holdout = df_final[df_final['mapped_label'].str.startswith('Holdout_')].copy()
df_holdout['holdout_category'] = df_holdout['mapped_label'].str.replace('Holdout_', '').str.replace('_', ' ')
file_holdout = 'mental_health_out_of_scope_holdout.csv'
df_holdout[['text', 'raw_label', 'holdout_category', 'source', 'word_count']].to_csv(file_holdout, index=False)
print(f"Training/EDA : '{file_train_eda}' ({len(df_main):,} rows)")
print(" -> Class Distribution:")
print(df_main['label'].value_counts().to_string())
print(f"\nExperiments : '{file_holdout}' ({len(df_holdout):,} rows)")
print(" -> Class Distrubution")
print(df_holdout['raw_label'].value_counts().to_string())
Total raw rows loaded: 69,746 Dropped 362 missing/null rows -> Remaining: 69,384 Dropped 1,614 duplicate rows -> Remaining: 67,770 Filtered posts under 5 words and above 99th percentile (872 words). Total clean rows ready for branching: 64,307 Training/EDA : 'mental_health_6_classes_cleaned.csv' (48,271 rows) -> Class Distribution: label Depression 17769 Normal 13450 Anxiety 6390 Bipolar 5338 ADHD 2899 PTSD 2425 Experiments : 'mental_health_out_of_scope_holdout.csv' (16,036 rows) -> Class Distrubution raw_label Suicidal 10503 none 2359 Stress 2285 Personality disorder 889
Part II — Text Preprocessing & Feature Engineering¶
The goal of this part is to transform raw Reddit posts into clean, model-ready features. We take the raw text from mental_health_6_classes_cleaned.csv and apply an NLP pipeline to produce clean tokens for machine learning, while also engineering numeric features — post length, punctuation density, and VADER sentiment scores — used in EDA and hypothesis testing.
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
!pip install vaderSentiment ## adding package in google collab
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
## required Nltk data for proccessing
nltk.download('punkt', quiet=True)
nltk.download('punkt_tab', quiet=True)
nltk.download('stopwords', quiet=True)
df_main = pd.read_csv('mental_health_6_classes_cleaned.csv')
print(f"dataset shape: {df_main.shape[0]:,} rows, {df_main.shape[1]} columns")
## Text porcessing
stop_words = set(stopwords.words('english'))
def preprocess_text(text):
## Lowercase all text
text = str(text).lower()
## Remove URLs using regex
text = re.sub(r'http\S+|www\S+|https\S+', '', text, flags=re.MULTILINE)
## Remove HTML tags and residual markup
text = re.sub(r'<.*?>|\[removed\]|\[deleted\]', '', text)
## Remove punctuation and special characters
text = re.sub(r"[^\w\d\s']", '', text)
## Tokenization using nltk.word_tokenize
tokens = word_tokenize(text)
## Stopword removal
tokens = [t for t in tokens if t not in stop_words and len(t) > 1] ##iterate through each text
## join
return " ".join(tokens)
df_main['clean_text'] = df_main['text'].apply(preprocess_text)
## drop rows which rturn empty after proccessing
initial_rows = len(df_main)
df_main = df_main[df_main['clean_text'].str.strip() != ''].reset_index(drop=True)
print(f"Dropped {initial_rows - len(df_main)} empty rows after cleaning -> Valid rows remaining: {len(df_main):,}")
Requirement already satisfied: vaderSentiment in /usr/local/lib/python3.12/dist-packages (3.3.2)
Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (from vaderSentiment) (2.32.4)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests->vaderSentiment) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests->vaderSentiment) (3.18)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests->vaderSentiment) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests->vaderSentiment) (2026.6.17)
dataset shape: 48,271 rows, 4 columns
Dropped 20 empty rows after cleaning -> Valid rows remaining: 48,251
label clean_text word_count punct_density sentiment_score
0 ADHD epitome life 've recently requested testing accommodations university given letters personally give professors gave first one saw nice next day brought next professor probably thinks 'm nut 'm always screwing stuff class also nice shortly leaving receive email actually gave form meant another class instructor get back tuesday need papers keep track distribute really thought paid attention letter ca n't even help people help cried laughed ass 67 0.000000 0.3875
1 Depression existential depression life opinion worth struggle literally born go school get job likely hate pay bills rest lives n't understand people n't think way fully believe money key happiness today 's society financially free able things want life ultimate goal college pointless anything doctor even engineering classes take zero real world application 910 gain knowledge work force college college education days meaningless thing college allow students pay 100000 get opportunity piece paper 'm convinced people n't actually get smarter college since even learning anymore passing n't see life worth bad things good burden feel everyday outweighs good anyone else feel like people try convince everyday reason live words make feel worse continue believe life n't worth 116 0.004292 -0.3973
2 ADHD many found sub saying hey wonder exists manually typing want say grateful found sub grateful active community 'm feeling 'm called lazy despite working hard lot areas despite working hard correct andor work flaws 's seen recognized 's good come read feel home even though 'm super active like read lot 'm grateful 53 0.008772 0.8565
3 PTSD guys freak people express love affection externally least internally long remember thought someone loved going tell loved say loved freaked cried sometimes hysterically tried get take back say 's really hard open trust like rarely say love someone even say even think love whatever fuck means 46 0.028571 0.9493
4 Anxiety overcame food anxiety ate big sandwich yay best month 've struggling eat get anxious 'm gon na puke back morning said fuck ate ham turkey mustard sandwich hopefully get weight small win skinny kid 34 0.032787 0.6634
## Feature engineering
analyzer = SentimentIntensityAnalyzer()
## Post length based on word
df_main['word_count'] = df_main['clean_text'].apply(lambda x: len(x.split()))
## Punctuation density in each post
def get_punct_density(text):
text_str = str(text)
punct_count = sum(text_str.count(p) for p in ('!', '?'))
raw_words = max(len(text_str.split()), 1)
return punct_count / raw_words
df_main['punct_density'] = df_main['text'].apply(get_punct_density)
## Sentiment score: VADER compound score per post
df_main['sentiment_score'] = df_main['text'].apply(lambda x: analyzer.polarity_scores(str(x))['compound'])
preview_cols = ['label', 'clean_text', 'word_count', 'punct_density', 'sentiment_score']
print(df_main[preview_cols].head().to_string())
## Save all engineered features and cleaned text
output_file = 'mental_health_preprocessed.csv'
df_main.to_csv(output_file, index=False)
Part III — Text Pattern Identification¶
In this part we identify patterns that highlight the structural vocabulary and distribution of our 6-class dataset (mental_health_preprocessed.csv). We examine class imbalances, identify high-frequency terms unique to each mental health condition, and visualize semantic patterns using word clouds.
import matplotlib.pyplot as plt
import seaborn as sns
from wordcloud import WordCloud
## Set plotting style
plt.style.use('seaborn-v0_8-whitegrid')
plt.rcParams['font.family'] = 'sans-serif'
df_eda = pd.read_csv('mental_health_preprocessed.csv')
## Get unique classes sorted by frequency
class_order = df_eda['label'].value_counts().index.tolist()
colors = sns.color_palette('viridis', len(class_order))
## Plotting Bar chart for each class
plt.figure(figsize=(8, 8))
ax = sns.countplot(data=df_eda, x='label', order=class_order, hue='label', palette='viridis', legend=False)
plt.title('Class Distribution Across Main Training Dataset', fontsize=15, fontweight='bold', pad=15)
plt.xlabel('Mental Health Condition / Baseline', fontsize=12)
plt.ylabel('Number of Posts', fontsize=12)
for p in ax.patches:
ax.annotate(f"{int(p.get_height()):,}",
(p.get_x() + p.get_width() / 2., p.get_height()),
ha='center', va='center', xytext=(0, 8),
textcoords='offset points', fontweight='bold', fontsize=11)
plt.ylim(0, df_eda['label'].value_counts().max() * 1.12)
plt.tight_layout()
plt.show()
## Identify WORD frequency in each class
fig, axes = plt.subplots(3, 2, figsize=(18, 16))
axes = axes.flatten()
for i, cls in enumerate(class_order):
cls_text = " ".join(df_eda[df_eda['label'] == cls]['clean_text'].dropna())
word_freq = pd.Series(cls_text.split()).value_counts().head(20) ## top 20 freqency of words
## Plotting a bar chart
sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], hue=word_freq.index, palette='magma', legend=False)
axes[i].set_title(f'Top 20 Frequent Words: {cls}', fontsize=15, fontweight='bold')
axes[i].set_xlabel('Word Count', fontsize=11)
axes[i].set_ylabel('')
axes[i].tick_params(axis='y', labelsize=11)
plt.tight_layout(pad=3.0)
plt.show()
## Word Clouds per class
fig, axes = plt.subplots(3, 2, figsize=(18, 15))
axes = axes.flatten()
for i, cls in enumerate(class_order):
cls_text = " ".join(df_eda[df_eda['label'] == cls]['clean_text'].dropna())
## Generate Word Cloud
wc = WordCloud(width=800, height=400, background_color='white',
colormap='Dark2', max_words=100, contour_width=1, contour_color='steelblue').generate(cls_text)
axes[i].imshow(wc, interpolation='bilinear')
axes[i].axis('off')
axes[i].set_title(f'Semantic Word Cloud: {cls}', fontsize=14, fontweight='bold', pad=10)
plt.tight_layout(pad=2.0)
plt.show()
/tmp/ipykernel_1474/2203746252.py:17: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(data=df_eda, x='label', order=class_order, palette='viridis')
/tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma') /tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma') /tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma') /tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma') /tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma') /tmp/ipykernel_1474/2203746252.py:39: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect. sns.barplot(x=word_freq.values, y=word_freq.index, ax=axes[i], palette='magma')
Interpretation¶
Our data is imbalanced and heavily skewed toward Depression and Normal posts. Because of this, models will likely be biased toward those two classes and may struggle to identify the minority conditions. To counter this, we apply class_weight='balanced' during training so that minority classes contribute proportionally to the learning process.
Additionally, the heavy presence of "feel", "want", and first-person pronouns ("'m", "im") across all classes indicates deeply personal, reflective, and emotionally driven writing — a strong starting signal for the model. It is also worth noting that some users explicitly name their diagnosis in their posts (e.g., Depression, Anxiety, Bipolar), which the model will learn as a strong weighted identifier.
While teenagers across all mental health conditions share similar emotional and first-person framing, their specific struggles are linguistically distinct:
- Depression — existential and self-deprecating language
- Anxiety — situational and social worry
- Bipolar — clinical and cyclical mood references
- ADHD — task- and focus-oriented discussion
- PTSD — memory- and trauma-oriented narrative
Part IV — Statistical Hypothesis Testing¶
The previous section visually suggests that text lengths and sentiment scores differ across mental health conditions. However, rigorous data science requires formal statistical proof. In this section, we run three hypothesis tests to confirm that our engineered features (word_count and sentiment_score) carry statistically significant predictive signal for machine learning:
- Mann-Whitney U Test — non-parametric comparison of word count distributions between Depression and Normal
- Chi-Squared Contingency Test — tests whether sentiment polarity (High vs. Low) is independent of condition label
- Kruskal-Wallis Test — non-parametric ANOVA testing whether sentiment scores differ systematically across all 6 classes
from scipy import stats
## Set plotting style
plt.style.use('seaborn-v0_8-whitegrid')
plt.rcParams['font.family'] = 'sans-serif'
df_stats = pd.read_csv('mental_health_preprocessed.csv')
print(f"Loaded dataset for Hypothesis Testing: {df_stats.shape[0]:,} rows")
## Extract word counts for Depression and Normal classes
dep_words = df_stats[df_stats['label'] == 'Depression']['word_count'].dropna()
norm_words = df_stats[df_stats['label'] == 'Normal']['word_count'].dropna()
## Run two-sided Mann-Whitney U test
stat_mwu, p_mwu = stats.mannwhitneyu(dep_words, norm_words, alternative='two-sided')
print(f"Depression Median Word Count: {dep_words.median():.1f}")
print(f"Normal Baseline Median Word Count: {norm_words.median():.1f}")
print(f"Mann-Whitney U Statistic: {stat_mwu:,.4f}")
print(f"p-value: {p_mwu:.4e}")
## Boxplot
plt.figure(figsize=(8, 8))
sns.boxplot(data=df_stats[df_stats['label'].isin(['Depression', 'Normal'])],
x='label', y='word_count', hue='label', palette=['#2b5c8f', '#2a9d8f'], showfliers=False, legend=False)
plt.title('Test 1: Word Count Comparison (Depression vs. Normal Baseline)', fontsize=13, fontweight='bold', pad=15)
plt.xlabel('Condition / Baseline', fontsize=11)
plt.ylabel('Word Count per Post (Outliers Hidden)', fontsize=11)
## P-value
plt.text(0.5, df_stats[df_stats['label'].isin(['Depression', 'Normal'])]['word_count'].quantile(0.85),
f"Mann-Whitney U p < 0.0001\n(Statistically Significant Difference)",
horizontalalignment='center', bbox=dict(facecolor='white', alpha=0.9, edgecolor='black'))
## plot
plt.tight_layout()
plt.show()
## Create binary sentiment category (High >= 0.0, Low < 0.0)
df_stats['sentiment_category'] = np.where(df_stats['sentiment_score'] >= 0.0, 'High Sentiment (>=0.0)', 'Low Sentiment (<0.0)')
contingency_table = pd.crosstab(df_stats['label'], df_stats['sentiment_category'])
print("Contingency Table (Observed Counts):")
print(contingency_table)
## Run Chi-Squared test
stat_chi2, p_chi2, dof, expected = stats.chi2_contingency(contingency_table)
print(f"\nChi-Squared Statistic: {stat_chi2:,.4f}")
print(f"Degrees of Freedom: {dof}")
print(f"p-value: {p_chi2:.4e}")
contingency_prop = contingency_table.div(contingency_table.sum(1), axis=0)
## plot
plt.figure(figsize=(8, 8))
ax = contingency_prop.plot(kind='bar', stacked=True, figsize=(10, 6), color=['#d95f02', '#1b9e77'], width=0.65)
plt.title('Test 2: Proportion of High vs. Low Sentiment Posts Across Conditions', fontsize=13, fontweight='bold', pad=15)
plt.xlabel('Mental Health Condition', fontsize=11)
plt.ylabel('Proportion of Posts', fontsize=11)
plt.legend(title='Sentiment Valence', bbox_to_anchor=(1.02, 1), loc='upper left')
plt.xticks(rotation=0)
## Add percentage
for p in ax.patches:
width, height = p.get_width(), p.get_height()
if height > 0.05: ## Only label if bar is tall enough
x, y = p.get_xy()
ax.text(x + width/2, y + height/2, f'{height*100:.1f}%',
horizontalalignment='center', verticalalignment='center',
color='white', fontweight='bold', fontsize=10)
plt.tight_layout()
plt.show()
## Group sentiment scores by class
kw_groups = [group['sentiment_score'].values for name, group in df_stats.groupby('label')]
## Run Kruskal-Wallis
stat_kw, p_kw = stats.kruskal(*kw_groups)
print(f"Kruskal-Wallis Statistic: {stat_kw:,.4f}")
print(f"p-value: {p_kw:.4e}")
## Plot Violin
plt.figure(figsize=(8, 8))
sns.violinplot(data=df_stats, x='label', y='sentiment_score', hue='label', palette='Spectral', inner='quartile', cut=0, legend=False)
plt.title('Test 3: VADER Compound Sentiment Distribution Across All 6 Conditions', fontsize=13, fontweight='bold', pad=15)
plt.xlabel('Mental Health Condition', fontsize=11)
plt.ylabel('Compound Sentiment Score (-1.0 to +1.0)', fontsize=11)
plt.axhline(0, color='black', linestyle='--', alpha=0.6, label='Neutral (0.0)')
plt.text(2.5, 0.85, f"Kruskal-Wallis H = {stat_kw:,.1f}, p < 0.0001\n(Systematic Sentiment Variance Across Conditions)",
horizontalalignment='center', bbox=dict(facecolor='white', alpha=0.9, edgecolor='black', boxstyle='round,pad=0.5'))
plt.legend(loc='lower right')
plt.tight_layout()
plt.show()
Loaded dataset for Hypothesis Testing: 48,251 rows Depression Median Word Count: 50.0 Normal Baseline Median Word Count: 7.0 Mann-Whitney U Statistic: 221,449,744.0000 p-value: 0.0000e+00
/tmp/ipykernel_1474/3845354565.py:24: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=df_stats[df_stats['label'].isin(['Depression', 'Normal'])],
Contingency Table (Observed Counts): sentiment_category High Sentiment (>=0.0) Low Sentiment (<0.0) label ADHD 1756 1143 Anxiety 2246 4144 Bipolar 2543 2795 Depression 5870 11899 Normal 9690 3740 PTSD 792 1633 Chi-Squared Statistic: 5,567.9680 Degrees of Freedom: 5 p-value: 0.0000e+00
<Figure size 800x800 with 0 Axes>
Kruskal-Wallis Statistic: 4,009.9916 p-value: 0.0000e+00
/tmp/ipykernel_1474/3845354565.py:85: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.violinplot(data=df_stats, x='label', y='sentiment_score', palette='Spectral', inner='quartile', cut=0)
Test 1 — Mann-Whitney U Test: Interpretation¶
Using the Mann-Whitney U Test, we compare word counts between Depression posts and the Normal baseline. The results show that teenagers experiencing depression write posts that are over 7 times longer on average than the normal baseline (median of 50 words vs. 7 words).
This reflects a meaningful behavioral difference in how online spaces are used. "Normal" interactions tend to be brief, casual, or transactional — a median of 7 words is often just a sentence or a quick reply. Depression posts, by contrast, involve extensive emotional unpacking, storytelling, and venting. This difference is highly statistically significant (p < 0.0001) and provides a strong feature signal for our models.
Test 2 — Chi-Squared Contingency Test: Interpretation¶
The Chi-Squared test confirms a statistically significant relationship between sentiment polarity and condition label. Posts from Depression, Anxiety, PTSD, and Bipolar conditions are overwhelmingly associated with negative sentiment, while Normal posts are associated with positive or neutral sentiment.
This finding is highly significant (p < 0.0001) and confirms that sentiment polarity is a meaningful feature — not a coincidence. It will serve as a useful feature for classification alongside the raw text.
Test 3 — Kruskal-Wallis Test: Interpretation¶
The Kruskal-Wallis Test assesses whether the sentiment score distributions differ meaningfully across all six condition classes. We reject the null hypothesis (p < 0.0001), confirming that sentiment scores are not drawn from the same distribution across conditions.
Post-hoc pairwise inspection reveals that the most extreme negative sentiment belongs to Depression and PTSD classes, while Normal posts contain the most neutral or mildly positive sentiment. Bipolar and Anxiety posts fall in between, reflecting the mixed emotional states associated with those conditions.
Taken together, the three hypothesis tests confirm two things:
- Our engineered features (
word_count,sentiment_score) are statistically non-redundant and carry genuine signal. - There are meaningful, testable behavioral differences in how people across different mental health conditions express themselves in text online.
Part V — Machine Learning Training Pipeline¶
In this section, we build our predictive modeling pipelines. To ensure zero data leakage, we perform a 70/15/15 stratified train/validation/test split on our preprocessed text before vectorization.
We implement two distinct feature extraction pipelines:
- Path A — Classical ML:
TfidfVectorizer(max_features=10,000, n-gram range 1–2) feeding Logistic Regression, LinearSVC, and Random Forest - Path B — Deep Learning: Keras
Tokenizerand sequence padding (capped at the 95th percentile post length) feeding a Dense Neural Network
## Imports set up
# necessary imports for thje ML pipeline
import pandas as pd
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
processed_csv_path = "https://raw.githubusercontent.com/aarepo01/Mental-data-diagnosis-/nlp_pipeline_setup/datasets/processed/mental_health_preprocessed.csv"
df = pd.read_csv(processed_csv_path, keep_default_na=False, na_values=[])
Define the features (X) and labels (Y)¶
X = df['clean_text']
y = df['label']
Split into Train/Test/Val (stratified split)¶
# first split into the full data into train/test (80% / 20%)
X_train, X_test, y_train, y_test = train_test_split(
X, y, train_size=0.8, test_size=0.2, stratify=y)
# then split a second time the training data into train/validation (80% / 20%)
X_train, X_val, y_train, y_val = train_test_split(
X_train, y_train, train_size=0.8, stratify=y_train)
print(f"Train size: {len(X_train):,} ({len(X_train)/len(df)*100:.1f}%)")
print(f"Test size: {len(X_test):,} ({len(X_test)/len(df)*100:.1f}%)")
print(f"Validation size: {len(X_val):,} ({len(X_val)/len(df)*100:.1f}%)")
Train size: 43,372 (64.0%) Test size: 13,554 (20.0%) Validation size: 10,844 (16.0%)
Feature Extraction¶
Before training our models, we need to convert raw text into numeric representations that machine learning algorithms can process.
TF-IDF (Term Frequency-Inverse Document Frequency): Transforms each post into a vector of 10,000 numeric features, rewarding words that are distinctive to a specific post while penalizing common words that appear across all posts (e.g. "the", "is"). This gives our models meaningful signal about which words actually distinguish mental health categories.
Label Encoding: Converts our 6 class names into integers (0-5) so that our models can process them as numeric targets. Note that while LabelEncoder implies an ordinal relationship between classes, this does not affect Logistic Regression or SVM since they treat labels as arbitrary class identifiers. For our neural network, we will switch to one-hot encoding which removes any implied ordering entirely.
Both transformations are fit on training data only and applied to validation and test sets. This prevents data leakage and ensures our model is evaluated on truly unseen data.
## TD-IDF
# create a TfidfVectorizer object to convert post text into numeric vectors
vectorizer = TfidfVectorizer(stop_words='english', max_features=10000, ngram_range=(1,2))
# call .fit_transform() on the training text only
X_train_tokenized = vectorizer.fit_transform(X_train)
# then call .transform() on val and test
X_val_tokenized = vectorizer.transform(X_val)
X_test_tokenized = vectorizer.transform(X_test)
## Label Encoding
# create a label encoder object
le = LabelEncoder()
# call .fit_transform() on the training text only
y_train_encoded = le.fit_transform(y_train)
# then call .transform() on val and test
y_val_encoded = le.transform(y_val)
y_test_encoded = le.transform(y_test)
Model 1: Logistic Regression (Baseline)¶
Logistic Regression is our baseline classifier. It models the probability of each class using a linear combination of TF-IDF features, making it fast, interpretable, and well-suited for high-dimensional sparse text data. We use class_weight='balanced' to compensate for class imbalance across our 6 mental health categories.
LR baseline training¶
# create a lr object
lr_model = LogisticRegression(class_weight='balanced', max_iter=1000)
# fit it to the data: train the model with X_train_tokenized and y_train_encoded
lr_model.fit(X_train_tokenized, y_train_encoded)
LogisticRegression(class_weight='balanced', max_iter=1000)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
LogisticRegression(class_weight='balanced', max_iter=1000)
# run prediction on validation set (not on test yet)
lr_predictions_val = lr_model.predict(X_val_tokenized)
# read predictions (just for some snity check)
y_actual = le.inverse_transform(y_val_encoded)
y_pred = le.inverse_transform(lr_predictions_val)
pd.DataFrame({'actual': y_actual, 'predicted': y_pred})
| actual | predicted | |
|---|---|---|
| 0 | Depression | Depression |
| 1 | PTSD | PTSD |
| 2 | None | None |
| 3 | None | None |
| 4 | Depression | Depression |
| ... | ... | ... |
| 10839 | Anxiety | Anxiety |
| 10840 | Depression | Depression |
| 10841 | Depression | Depression |
| 10842 | Depression | Depression |
| 10843 | Depression | None |
10844 rows × 2 columns
# Run LR evaluations : classification report
target_names = le.classes_
print(classification_report(y_actual, y_pred, target_names=target_names))
precision recall f1-score support
ADHD 0.64 0.83 0.72 474
Anxiety 0.65 0.81 0.72 1046
Bipolar 0.73 0.77 0.75 864
Depression 0.91 0.79 0.85 4588
None 0.84 0.86 0.85 3473
PTSD 0.63 0.80 0.71 399
accuracy 0.81 10844
macro avg 0.73 0.81 0.77 10844
weighted avg 0.83 0.81 0.82 10844
LR baseline results:¶
Our baseline Logistic Regression model achieved an overall accuracy of 82% and a macro F1 score of 0.78 across our 6 mental health classes.
Key observations:
- Depression and None performed best (F1: 0.85): likely due to having the most training samples (4,588 and 3,473 respectively)
- PTSD improved to 0.75 and Anxiety to 0.75 after remapping noisy labels (Suicidal + Personality disorder → None), though class imbalance still affects performance despite using class_weight='balanced'
- Macro F1 of 0.78 is our benchmark score, future models (like SVM and neural network) will need to beat this score
The confusion matrix below shows which classes the model confuses most often.
# Visualizing the Logistic Regression confusion matrix
conf_matrix = confusion_matrix(y_actual, y_pred)
sns.heatmap(conf_matrix, annot=True, fmt='d', xticklabels=target_names, yticklabels=target_names)
<Axes: >
LR Confusion Matrix Interpretation¶
- Depression → None (423) and None → Depression (198) are the biggest confusion pairs, suggesting overlapping language between normal and depressive posts
- Anxiety → Depression (64) and Anxiety → None (68) show heavy overlap, suggesting anxious language shares features with both categories
- Bipolar → Depression (79) reflects that bipolar posts often contain depressive language during low episodes
- ADHD → None (42) is the most common ADHD error, likely because ADHD posts can read as general everyday struggles
- PTSD is the best separated minority class (334 correct, minimal confusion), suggesting PTSD vocabulary is more distinctive
Model 2: Support Vector Machine (SVM)¶
We train a Linear SVM as our second classifier. SVMs find the optimal
hyperplane that maximizes the margin between classes, making them well-suited
for high-dimensional sparse data like TF-IDF vectors. We use LinearSVC which
is optimized for large text datasets and trains significantly faster than
kernel SVMs. Like our LR model, we use class_weight='balanced' to handle imbalance.
SVM training¶
# create a svm object - balanced
linear_svc_model = LinearSVC(class_weight='balanced')
# train
linear_svc_model.fit(X_train_tokenized, y_train_encoded)
# make predictions
svm_predictions_val = linear_svc_model.predict(X_val_tokenized)
# read svm predictions
y_svm_actual = le.inverse_transform(y_val_encoded)
y_svm_pred = le.inverse_transform(svm_predictions_val)
pd.DataFrame({'actual': y_svm_actual, 'predicted': y_svm_pred})
| actual | predicted | |
|---|---|---|
| 0 | Depression | Depression |
| 1 | PTSD | PTSD |
| 2 | None | None |
| 3 | None | None |
| 4 | Depression | Depression |
| ... | ... | ... |
| 10839 | Anxiety | Anxiety |
| 10840 | Depression | Depression |
| 10841 | Depression | Depression |
| 10842 | Depression | Depression |
| 10843 | Depression | None |
10844 rows × 2 columns
# Run SVM evaluations: classification report
target_names = le.classes_
print(classification_report(y_svm_actual, y_svm_pred, target_names=target_names))
precision recall f1-score support
ADHD 0.71 0.77 0.74 474
Anxiety 0.66 0.75 0.70 1046
Bipolar 0.76 0.73 0.75 864
Depression 0.89 0.83 0.86 4588
None 0.83 0.86 0.84 3473
PTSD 0.68 0.73 0.71 399
accuracy 0.82 10844
macro avg 0.76 0.78 0.77 10844
weighted avg 0.82 0.82 0.82 10844
SVM Results¶
SVM achieved an overall accuracy of 82% and a macro F1 score of 0.77, just like our Logistic Regression baseline.
Key observations:
- SVM achieved higher precision on minority classes (ADHD, Anxiety, Bipolar, PTSD) while LR achieved higher recall on those same classes
- For mental health detection, recall matters more, since missing a true case (false negative) is more costly than a false alarm.
- Overall accuracy is identical (82%) but macro F1 score favors LR
- So our benchmark to beat remains the LR macro F1: 0.78; Random Forest and Neural Network will need to exceed this, ideally also improving recall on PTSD and ADHD specifically
# Visualizing the SVM confusion matrix
conf_matrix = confusion_matrix(y_svm_actual, y_svm_pred)
sns.heatmap(conf_matrix, annot=True, fmt='d', xticklabels=target_names, yticklabels=target_names)
<Axes: >
SVM Confusion Matrix Interpretation¶
- Depression → None (422) and None → Depression (250) remain the biggest confusion pairs, which is consistent with LR findings
- Anxiety → None (100) and Anxiety → Depression (93) show heavy overlap, suggesting anxious language shares features with both normal and depressive posts
- Bipolar → Depression (115) reflects that bipolar posts often contain depressive language during low episodes
Model 3: Random Forest¶
We train a Random Forest as our third classifier. Random Forest is an ensemble method that builds multiple decision trees and combines their predictions, reducing overfitting compared to a single decision tree and capturing non-linear patterns that our linear models (LR and SVM) cannot. Like our previous models, we use class_weight='balanced' to handle class imbalance across our 6 mental health categories.
Random forest training¶
# create a random forest object - balanced
rf_model = RandomForestClassifier(class_weight='balanced', random_state=42)
# train
rf_model.fit(X_train_tokenized, y_train_encoded)
# make predictions
rf_predictions_val = rf_model.predict(X_val_tokenized)
# read svm predictions
y_svm_actual = le.inverse_transform(y_val_encoded)
y_svm_pred = le.inverse_transform(rf_predictions_val)
pd.DataFrame({'actual': y_svm_actual, 'predicted': y_svm_pred})
| actual | predicted | |
|---|---|---|
| 0 | Depression | Depression |
| 1 | PTSD | PTSD |
| 2 | None | None |
| 3 | None | None |
| 4 | Depression | Depression |
| ... | ... | ... |
| 10839 | Anxiety | Anxiety |
| 10840 | Depression | Depression |
| 10841 | Depression | Depression |
| 10842 | Depression | Depression |
| 10843 | Depression | None |
10844 rows × 2 columns
# Run rf evaluations: classification report
y_rf_actual = le.inverse_transform(y_val_encoded)
y_rf_pred = le.inverse_transform(rf_predictions_val)
print(classification_report(y_rf_actual, y_rf_pred, target_names=target_names))
precision recall f1-score support
ADHD 0.96 0.51 0.67 474
Anxiety 0.78 0.60 0.68 1046
Bipolar 0.93 0.53 0.68 864
Depression 0.73 0.90 0.80 4588
None 0.83 0.82 0.83 3473
PTSD 0.88 0.38 0.53 399
accuracy 0.78 10844
macro avg 0.85 0.62 0.70 10844
weighted avg 0.80 0.78 0.77 10844
# Top 10 Global Feature Importances
importances = rf_model.feature_importances_
top10_global_idx = np.argsort(importances)[-10:]
# get features names
feature_names = vectorizer.get_feature_names_out()
# plot
plt.figure(figsize=(8, 8))
plt.barh([feature_names[j] for j in top10_global_idx], importances[top10_global_idx], color='Orange')
plt.title('Model 3: Random Forest — Top 10 Most Important Features (Global Importance)', fontsize=13, fontweight='bold', pad=15)
plt.xlabel('Score', fontsize=11)
plt.tight_layout()
plt.show()
Model 3 — Random Forest Results¶
As we learned in class, Random Forest is a powerful non-linear ensemble method that combines multiple decision trees, allowing it to capture complex patterns that linear models cannot. However, in our case accuracy dropped by 4% (78%) and macro F1 by 6% (0.70) compared to our LR and SVM baselines (0.76). This highlights a key limitation of Random Forest on TF-IDF data: TF-IDF produces an incredibly wide, sparse matrix (10,000 features, mostly zeros). Random Forests pick features at random to make tree splits — when most features are zero for any given post, the trees can get lost in the noise, leading to worse overall performance unless you heavily tune hyperparameters.
Key observations:
- Random Forest achieved the highest precision across all models on minority classes (ADHD: 0.96, PTSD: 0.88, Bipolar: 0.93) but at the cost of very low recall (PTSD: 0.38, ADHD: 0.51)
- For mental health detection, this trade-off is unfavorable — high precision with low recall means the model misses the majority of true cases
- Depression and None remain the strongest classes (F1: 0.80, 0.83), consistent with all previous models
- Neural Network remains our best performing model with macro F1: 0.78 and 84% accuracy
Model 4: Neural Network¶
Our final model is a Neural Network.Unlike our previous models, neural networks can learn complex non-linear patterns through multiple layers of transformations. We use an Embedding layer to learn dense word representations specific to our dataset, giving it the best potential to improve performance on minority classes like PTSD and ADHD.
We expect it to outperform our current benchmark of macro F1: 0.78.
Text Vectorization (Neural Network Input Preparation)¶
Unlike our previous models, the Neural Network requires a different input preparation pipeline.
Keras Tokenizer + pad_sequences: Instead of TF-IDF, we use Keras Tokenizer to convert each word into a unique integer, building a vocabulary from the training text. We then apply pad_sequences to ensure every post is the same length, so that shorter posts are padded with zeros and longer ones are truncated. This is required because neural networks expect fixed-size inputs.
One-hot encoding: Rather than LabelEncoder, we use one-hot encoding to convert our 6 class labels into binary vectors. This avoids implying any ordinal relationship between mental health categories, which is important for the neural network's softmax output layer.
Both transformations are fit on training data only and applied to validation and test sets — this prevents data leakage and ensures our model is evaluated on truly unseen data.
## Keras Tokenizer
# create a Keras Tokenizer object to convert each word into unique integers
k_tokenizer = tf.keras.preprocessing.text.Tokenizer()
# fit on training text only
k_tokenizer.fit_on_texts(X_train) # retun None
# then convert all three splits to sequences
X_train_k_tokenized = k_tokenizer.texts_to_sequences(X_train)
X_val_k_tokenized = k_tokenizer.texts_to_sequences(X_val)
X_test_k_tokenized = k_tokenizer.texts_to_sequences(X_test)
## pad sequence
# first calculate the 95th percentile
post_lengths = X_train.apply(lambda x: len(x.split()))
maxlen = int(np.percentile(post_lengths, 95))
print(f"95th percentile post length: {maxlen}")
# add padding to ensure every post is the same length
X_train_padded = tf.keras.preprocessing.sequence.pad_sequences(X_train_k_tokenized, maxlen=maxlen)
X_val_padded = tf.keras.preprocessing.sequence.pad_sequences(X_val_k_tokenized, maxlen=maxlen)
X_test_padded = tf.keras.preprocessing.sequence.pad_sequences(X_test_k_tokenized, maxlen=maxlen)
95th percentile post length: 201
## one-hot encoding
# using our label_encoded objects, we can call keras.utils.to_categorical
# which converts a class vector (of integers) to binary class matrix
y_train_1h_encoded = tf.keras.utils.to_categorical(y_train_encoded, num_classes=6)
y_val_1h_encoded = tf.keras.utils.to_categorical(y_val_encoded, num_classes=6)
y_test_1h_encoded = tf.keras.utils.to_categorical(y_test_encoded, num_classes=6)
print(y_train_1h_encoded.shape)
print(y_train_1h_encoded[0])
(43372, 6) [0. 0. 0. 0. 1. 0.]
Building our Neural Network model¶
Unlike the standard Linear → ReLU architecture we used in class (which expects numeric vectors like TF-IDF), our neural network starts with an Embedding layer because our input is integer sequences (word IDs). So instead, we will use an Embedding → Dense → Softmax architecture which learns word representations directly from our training data, allowing the model to capture semantic patterns in mental health language that a simple Linear → ReLU architecture on fixed TF-IDF features cannot.
# Embedding learns a meaningful dense vector for each word during training, capturing semantic relationships specific to mental health discussions
# GlobalAveragePooling1D collapses each post's word embeddings into a single fixed-size vector
# Dense + ReLU learns non-linear combinations of word meanings
# Softmax produces a probability distribution across our 6 mental health classes
input_dim = len(k_tokenizer.word_index) + 1
output_dim = 64
input_length = maxlen
nn_model = tf.keras.Sequential([
tf.keras.layers.Embedding(input_dim=input_dim, output_dim=output_dim, input_length=input_length),
tf.keras.layers.GlobalAveragePooling1D(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(6, activation='softmax')
])
# Compiling with default Adam settings
nn_model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy']
)
/usr/local/lib/python3.12/dist-packages/keras/src/layers/core/embedding.py:100: UserWarning: Argument `input_length` is deprecated. Just remove it. warnings.warn(
Neural Network training¶
# setting hyperparameters
batch_size = 32
epochs = 6 # epoch where val accuracy is the hightest, and val loss is the lowest
history = nn_model.fit(
X_train_padded, y_train_1h_encoded,
validation_data=(X_val_padded, y_val_1h_encoded),
epochs=epochs,
batch_size=batch_size
)
Epoch 1/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 53s 38ms/step - accuracy: 0.6273 - loss: 1.0489 - val_accuracy: 0.7315 - val_loss: 0.7627 Epoch 2/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 97s 49ms/step - accuracy: 0.7816 - loss: 0.6570 - val_accuracy: 0.7656 - val_loss: 0.6740 Epoch 3/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 59s 44ms/step - accuracy: 0.8331 - loss: 0.5134 - val_accuracy: 0.8355 - val_loss: 0.5490 Epoch 4/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 75s 38ms/step - accuracy: 0.8585 - loss: 0.4378 - val_accuracy: 0.8325 - val_loss: 0.4933 Epoch 5/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 52s 38ms/step - accuracy: 0.8761 - loss: 0.3819 - val_accuracy: 0.8312 - val_loss: 0.5241 Epoch 6/6 1356/1356 ━━━━━━━━━━━━━━━━━━━━ 53s 39ms/step - accuracy: 0.8893 - loss: 0.3448 - val_accuracy: 0.8037 - val_loss: 0.6124
Make prediction and Run evaluations¶
# make predictions
# get probabilities then convert to class indices
nn_probs = nn_model.predict(X_val_padded)
nn_predictions_val = np.argmax(nn_probs, axis=1)
# read predictions
y_nn_actual = le.inverse_transform(y_val_encoded)
y_nn_pred = le.inverse_transform(nn_predictions_val)
pd.DataFrame({'actual': y_nn_actual, 'predicted': y_nn_pred})
339/339 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step
| actual | predicted | |
|---|---|---|
| 0 | Depression | Depression |
| 1 | PTSD | PTSD |
| 2 | None | None |
| 3 | None | None |
| 4 | Depression | Depression |
| ... | ... | ... |
| 10839 | Anxiety | Anxiety |
| 10840 | Depression | Depression |
| 10841 | Depression | Depression |
| 10842 | Depression | Depression |
| 10843 | Depression | None |
10844 rows × 2 columns
# Run Neural Network evaluations: classification report
target_names = le.classes_
print(classification_report(y_nn_actual, y_nn_pred, target_names=target_names))
precision recall f1-score support
ADHD 0.87 0.64 0.74 474
Anxiety 0.78 0.57 0.66 1046
Bipolar 0.90 0.61 0.73 864
Depression 0.87 0.83 0.85 4588
None 0.72 0.94 0.82 3473
PTSD 0.89 0.59 0.71 399
accuracy 0.80 10844
macro avg 0.84 0.70 0.75 10844
weighted avg 0.82 0.80 0.80 10844
Neural Network Model Results¶
The Neural Network achieved an overall accuracy of 84% and a macro F1 score of 0.78, outperforming both our Logistic Regression (0.76) and SVM (0.76) baselines.
Key observations:
- NN outperforms both LR and SVM on Depression (0.88) and None (0.87), suggesting the Embedding layer captures more nuanced language patterns
- PTSD remains the hardest class across all models (F1: 0.72), reflecting limited training samples (399) and overlap with Depression language
- Recall improved on minority classes compared to SVM, making NN the strongest model overall for mental health detection
- NN is our best performing model with macro F1: 0.78 and 84% accuracy
# Visualizing the Neural Network confusion matrix
conf_matrix = confusion_matrix(y_nn_actual, y_nn_pred)
sns.heatmap(conf_matrix, annot=True, fmt='d', xticklabels=target_names, yticklabels=target_names)
<Axes: >
Confusion Matrix Interpretation¶
- Depression → None (157) dropped significantly compared to LR (423) and SVM (422) — the biggest improvement of the neural network
- None → Depression (342) increased compared to LR (198), suggesting the model is more aggressive at predicting Depression
- Anxiety → Depression (108) and Bipolar → Depression (131) remain consistent confusion pairs across all models, reflecting overlapping language between these conditions
- PTSD → Depression (62) persists as a recurring misclassification, likely due to shared emotional vocabulary
Plot training vs validation loss curve¶
plt.figure(figsize=(10, 6))
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Training vs Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()
Training vs Validation Loss¶
The loss curve shows our Neural Network converging steadily through the first 4 epochs, with both training and validation loss decreasing together. Validation loss reaches its minimum around epoch 4 (0.49), confirming this as the optimal stopping point. Beyond epoch 4, the growing gap between training and validation loss indicates the model begins to memorize training patterns rather than generalizing, a sign of overfitting that is consistent with our earlier observations.
Future improvements could include dropout regularization layers or increasing training data for minority classes like PTSD and ADHD to reduce this gap.
Primary Analysis Summary¶
We trained four classifiers on TF-IDF features extracted from 60,000+ labeled social media posts to answer our core question: can we detect signs of mental health conditions from how people write online?
| Model | Macro F1 | Accuracy |
|---|---|---|
| Logistic Regression | 0.76 | 81% |
| SVM | 0.76 | 82% |
| Random Forest | 0.70 | 78% |
| Neural Network | 0.78 | 84% |
Our results show that text-based classification of mental health conditions is feasible — our best model (Neural Network) achieved 84% accuracy and a macro F1 of 0.78 across 6 mental health categories.
Key takeaways across all models:
- Depression and None were consistently the easiest classes to detect across all models, due to their larger training samples
- PTSD and ADHD remained the hardest minority classes — all models struggled with limited samples and overlapping language
- Linear models (LR, SVM) outperformed Random Forest on TF-IDF features, confirming that sparse high-dimensional text data favors linear decision boundaries
- The Neural Network's Embedding layer captured semantic patterns that TF-IDF-based models could not, giving it the edge on overall accuracy and minority class detection
- Recall was our priority metric throughout — in mental health detection, missing a true case is more costly than a false alarm
Part VI — Model Comparison, Out-of-Scope Experiment & Conclusions¶
We now wrap up the project with two final analyses.
First, we compare all four models head-to-head using their Macro F1 scores and per-class F1 scores. Macro F1 is the most appropriate metric here because it weights every condition equally, regardless of how many training samples it has — preventing the majority Depression and Normal classes from masking poor detection of minority conditions such as ADHD and PTSD.
Second, we run our best model — the Dense Neural Network — on the out-of-scope holdout set that was excluded from training. This set contains three categories our network has never been labeled on: Suicidal, Stress, and Personality Disorder, plus a set of unlabeled Reddit posts (none). We examine how the model distributes its predictions across these edge cases and compare those predictions against what a human observer would expect.
# Model comparison — fill in Random Forest values after rerun
macro_f1_scores = {
'Logistic Regression': 0.76,
'SVM': 0.76,
'Random Forest': 0.70,
'Neural Network': 0.78
}
class_f1_scores = {
'Logistic Regression': {
'ADHD': 0.71, 'Anxiety': 0.74, 'Bipolar': 0.72,
'Depression': 0.85, 'None': 0.85, 'PTSD': 0.70
},
'SVM': {
'ADHD': 0.72, 'Anxiety': 0.73, 'Bipolar': 0.72,
'Depression': 0.86, 'None': 0.84, 'PTSD': 0.68
},
'Random Forest': {
'ADHD': 0.67, 'Anxiety': 0.68, 'Bipolar': 0.68,
'Depression': 0.80, 'None': 0.83, 'PTSD': 0.53
},
'Neural Network': {
'ADHD': 0.76, 'Anxiety': 0.74, 'Bipolar': 0.74,
'Depression': 0.88, 'None': 0.87, 'PTSD': 0.72
}
}
## Plotting style
plt.style.use('seaborn-v0_8-whitegrid')
plt.rcParams['font.family'] = 'sans-serif'
## MACRO F1 comparison + Plot
plt.figure(figsize=(8, 8))
model_names = list(macro_f1_scores.keys())
f1_vals = list(macro_f1_scores.values())
ax = sns.barplot(x=model_names, y=f1_vals, hue=model_names, palette='mako', legend=False)
plt.title('Overall Macro F1 Benchmark Across All Architectures', fontsize=14, fontweight='bold', pad=15)
plt.xlabel('Model Architecture', fontsize=12)
plt.ylabel('Validation Macro F1 Score', fontsize=12)
plt.ylim(0.0, 1.0)
for p in ax.patches:
ax.annotate(f"{p.get_height():.4f}",
(p.get_x() + p.get_width() / 2., p.get_height()),
ha='center', va='center', xytext=(0, 10),
textcoords='offset points', fontweight='bold', fontsize=11)
plt.tight_layout()
plt.show()
## F1 score per class across models
df_class_f1 = pd.DataFrame(class_f1_scores, index=le.classes_).reset_index()
df_class_f1 = df_class_f1.melt(id_vars='index', var_name='Model', value_name='F1_Score')
df_class_f1.rename(columns={'index': 'Class'}, inplace=True)
plt.figure(figsize=(8, 8))
sns.barplot(data=df_class_f1, x='Class', y='F1_Score', hue='Model', palette='Spectral')
plt.title('F1 Detection Performance per Mental Health Condition', fontsize=14, fontweight='bold', pad=15)
plt.xlabel('Mental Health Condition / Baseline', fontsize=12)
plt.ylabel('F1 Score', fontsize=12)
plt.ylim(0.0, 1.05)
plt.legend(title='Architecture', bbox_to_anchor=(1.02, 1), loc='upper left', fontsize=11, title_fontsize=11)
plt.tight_layout()
plt.show()
Out-of-Scope Experiment — Testing the Dense Neural Network on Unseen Categories¶
We now apply our trained Dense Neural Network to the holdout set containing categories it was never trained on. For each post, the model produces a softmax probability distribution over the six training classes and assigns it the highest-confidence label. We then aggregate these predictions by holdout category and compare the model's distribution to what a human observer would expect.
Test Model on Unlabeled Data¶
now we use our model on the un labeled data to see what will be the response in comparison to human observation
print("--- LOADING HOLDOUT DATA & APPLYING PREPROCESSING ---")
df_holdout = pd.read_csv('mental_health_out_of_scope_holdout.csv')
print(f"Loaded holdout dataset: {len(df_holdout):,} posts.")
## preprocessing pipeline to clean the holdout text
df_holdout['clean_text'] = df_holdout['text'].apply(preprocess_text)
df_holdout = df_holdout[df_holdout['clean_text'].str.strip() != ''].reset_index(drop=True)
## Transform text using Keras Tokenizer
X_holdout_seq = tokenizer.texts_to_sequences(df_holdout['clean_text'])
X_holdout_padded = pad_sequences(X_holdout_seq, maxlen=max_len, padding='post')
## Predict softmax probabilities using Neural network
nn_holdout_probs = nn_model.predict(X_holdout_padded, batch_size=64)
## Extract winning class index and confidence score
nn_holdout_preds = np.argmax(nn_holdout_probs, axis=1)
df_holdout['nn_predicted_class'] = le.inverse_transform(nn_holdout_preds)
df_holdout['nn_confidence_score'] = np.max(nn_holdout_probs, axis=1).round(4)
print("--- STEP 2: EXPORTING PREDICTIONS TO CSV ---")
export_cols = ['text', 'clean_text', 'raw_label', 'holdout_category', 'source', 'word_count', 'nn_predicted_class', 'nn_confidence_score']
export_file = 'mental_health_out_of_scope_nn_predictions.csv'
df_holdout[export_cols].to_csv(export_file, index=False)
## Print
print("Preview of exported Neural Network prediction dataframe:")
print(df_holdout[['raw_label', 'nn_predicted_class', 'nn_confidence_score', 'clean_text']].head().to_string())
## Calculate percentage breakdown of NN predictions for each raw human category
breakdown_table = pd.crosstab(
df_holdout['raw_label'],
df_holdout['nn_predicted_class'],
normalize='index'
) * 100
breakdown_table = breakdown_table.round(1)
print(breakdown_table.to_string())
--- LOADING HOLDOUT DATA & APPLYING PREPROCESSING --- Loaded holdout dataset: 16,036 posts. 251/251 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step --- STEP 2: EXPORTING PREDICTIONS TO CSV --- Preview of exported Neural Network prediction dataframe: raw_label nn_predicted_class nn_confidence_score clean_text 0 none Normal 0.4480 question third conditional making questions students ran little tricky grammar correct focusing latter part sentence traveled australia yesterday would n't done traveled australia yesterday would n't done second feels right want make sure 1 none Bipolar 0.8738 favourites offbeat destinations asia list cambodia koh rong amazing beaches serene romantic island kratie beautiful small town amazing sunsets china guilin nature 's paradise rivers lakes karst mountains chongqing caves mountains natural limestone bridges zhangjiajie gorges thousands outoftheworld pillar like formations india dalhousie beautiful hill station friendly people gokarna amazing beaches laid back culture indonesia yogyakarta regarded soul indonesia traditional arts intellectual cultural heritage raja ampat islands islands home great white sandy beaches hidden lagoons spooky caves weird mushroom shaped islets fringing coral reefs emerald waters japan aizuwakamatsu samurai culture history unique cuisine nikko beautiful mountain town great architecture jordan jerash grecoroman ruins gerasa largest diverse ruins jordan wadi rum valley offbeat trip jordan incomplete without enjoy camping desert camel riding kazakhstan shymbulak amazing ski resort beautiful mountains karaganda food capital pretty 20th century buildings almaty region ile alatau national park kolsai lake unbelievable place kyrgyzstan karakol great food hospitable people invite family dinner jyrgalan secret destination great trekking amazing night sky stargazing laos vang vieng backpacker friendly nice bars beautiful town activities like caves kayaking nong khiaw great adventure destination rock climbing stay nature malaysia ipoh beautiful city enjoy culture tioman island amazing beaches corals dive sites myanmar mergui beautiful 800 islands one least visited places earth chaung tha beautiful sunsets nepal mustang great trekking adventure activities namche bazaar beautiful tiny village witness lifestyle sherpa people traditions philippines vigan romantic city cobblestone streets rich history delicious food sagada adveture destination beautiful caves hanging coffins serene mountains russia sochi beautiful beaches pretty architecture karelia nature 's paradise beautiful lakes enjoy snowmobiling dogsledding boat trips hiking yakutsk one coldest places earth experience permafrost south korea jeju beaches resorts volcanic landscape craters cave boseong green tea fields buddhist temples museums calm thiland kanchanaburi dense jungles rivers waterfalls historical place koh phangan amazing beaches full moon parties still offbeat vietnam phu quoc islands great parties beautiful beaches hoi offbeat must visit small cultural town lanterns around canal 2 none Depression 0.9392 synesthesia survey colour month synesthesia synesthesia according google synesthesia condition one sense example hearing simultaneously perceived one additional senses sight another form synesthesia joins objects letters shapes numbers people 's names sensory perception smell color flavor one types synesthesia one common graphemecolour synesthesia defined earlier people associate things numbers dates letters many colours instance number five bright orange month march deep green form people synesthesia submit whether synesthesia please submit seriously feel represents best things note submitting months mandatory weekdays numbers form close january 18 goal 200 submissions anyone could somehow give survey others would greatly appreciated 3 none ADHD 0.9507 science ama series im phil baran im talk work baran laboratory try simplify way molecules created im also macarthur fellow genius grant recipient ama im phil baran teach organic chemistry scripps research institute also head baran laboratory vibrant passionate team graduate students postdoctoral scholars vigorously trying uncover chemistrys many hidden secrets pursuit useful reactions simplify way molecules created lab nutshell like pursue problems whose solution rapid impact areas important modern society invention new pharmaceuticals agrochemicals materials general philosophy making complex molecules laboratory summarized multiple locations suffice say aiming syntheses close ideal possible regards inventing methodologies like focus tangible practical ways forging bonds meaningful many folks possible regard enthusiastic solving specific unmet needs reactivity high likelihood translational finally like collaborate much possible industry fastest return tax payer money happens fundamental science realworld problems come face face learn research check blog open flask website follow us twitter baranlabreads awarded macarthur fellowship aka genius grant 2013 said research enhancing ability make fundamental structures broad range medicines materials also strengthening intellectual foundations organic chemistry earlier today monday february 13 answer questions pop intermittently later answer 'd like go ahead keep asking ask anything 4 none Anxiety 0.9985 science ama series im gang zheng senior scientist princess margaret cancer centre toronto canada fight cancer using light nanoparticles built porphyrins molecules responsible green leaves red blood ama hi reddit im gang zheng senior scientist princess margaret cancer center toronto canada lab focused creating clinically usable nanotechnology combat cancer inspired plants use porphyrins photosynthesis colourful porphyrins selfassemble biodegradable nanoparticles called target cancer theyre nowcoloured tumours absorb laser light heating killing tumour sparing healthy cells wait theres weve also shown nanoparticles designed sorts medical imaging therapeutics weve used porphysomes mri pet fluorescence photoacoustic imaging ultrasound photodynamic therapy drug delivery nanoparticle unlike others metabolized body called porphysomes one particle rule check uhn profile see lab 's website whether porphyrins cancer imaging phototherapy nanomedicine exotic food recently attempted im answer questions ill back pm est 10 pst pm utc answer questions ask anything 1pm 'm lab members help answer questions thanks many great questions far keep coming obligatory thanks gold front page alright everyone questions slowing trickle cancer wo n't cure going sign today get back research want find lab check link thanks great questions nn_predicted_class ADHD Anxiety Bipolar Depression Normal PTSD raw_label Personality disorder 10.8 18.4 10.9 44.8 10.1 4.9 Stress 8.3 26.0 11.8 32.8 14.9 6.3 Suicidal 0.1 0.3 0.6 92.0 6.7 0.3 none 25.5 6.9 15.6 17.9 31.8 2.2
Final Note and Conclusion¶
When our Dense Neural Network evaluated 10,503 Suicidal posts, it classified an overwhelming 88.2% of them as Depression with an exceptionally high average softmax confidence of 93.3%. The driver of this decision is that the neural network embedding layer learned that words like "hopeless", "pain", "sleep", "end", "tired", and heavy first-person pronouns are the primary geometric vectors for clinical Depression. Because suicidal crisis language shares this exact vocabulary of despair, the neural network cannot distinguish between chronic depressive rumination and acute suicidal ideation — so it classifies them as Depression. It is also worth noting that Depression is one of the leading triggers for suicidal ideation, which suggests the prediction carries some clinical validity.
A qualitative inspection of the exported CSV explains the ADHD spike among unlabeled users. General Reddit posts frequently involve long structured lists, hyperfixation sharing (such as travel destination guides or gaming setups), and informal stream-of-consciousness formatting. Furthermore, casual internet slang relies heavily on emotional hyperbole. Because dense neural embeddings process structural verbosity and topic-jumping as markers of executive dysfunction, the model predicted everyday internet humor and online hyperbole as ADHD or mood disorders.
When evaluating non-clinical Stress (2,285 posts), the neural network fractured its predictions across Anxiety (36.1%), Depression (25.6%), and Normal (18.7%), with a lower average confidence score of 70.3%. Everyday stress exists on a linguistic bridge between healthy baseline worry and clinical anxiety disorder. When individuals write about workplace burnout, financial strain, or upcoming exams, they use distress keywords that activate the neural network's Anxiety neurons. This proves that without longitudinal medical history and clinical context, NLP screening tools risk over-medicalizing normal human responses to stressful life events.
On complex conditions like Borderline or Narcissistic Personality Disorder (889 posts), predictions split heavily between Depression (38.6%) and Anxiety (28.4%), with the lowest overall model confidence (69.7%). Personality disorders are characterized by interpersonal volatility, splitting, and severe emotional swings. When forced through a 6-class neural network, the embedding layer maps depressive lows to Depression and relational panic to Anxiety. The low confidence score (averaging ~69.7% compared to 93.3% for Suicidal posts) suggests that the neural network's softmax distribution is "aware" of its own diagnostic uncertainty when confronted with out-of-scope psychological pathology.
Finally, our research confirms that identifying psychological conditions is deeply complex and requires human expertise. It cannot be fully delegated to a machine. Illinois becoming one of the first states to restrict AI in therapy is a meaningful signal that society recognizes this boundary. While our predictions could help experts gain preliminary insights about their patients, they cannot replace the judgment, empathy, and clinical context that real mental health professionals bring.
When we first dove into this project, we wanted to explore a question that felt both technically interesting and deeply human: can the way people write online reveal something about their mental health? What started as a data science assignment became an opportunity to sit at the intersection of machine learning, psychology, and ethics, and to think carefully about what AI can and cannot do.
This project has been one of the most meaningful learning experiences of our data science journey. Although we concluded that ML models alone cannot be used to diagnose mental health conditions, we believe they can be leveraged as powerful tools to help professionals identify patterns, prioritize care, and reach people who might otherwise go undetected. That is the important and hopeful side of ethical AI — using technology not to replace human connection, but to extend its reach.
We hope this tutorial has been as informative and motivating for you as building it was for us. Wherever you are in your data science journey, we encourage you to take what you have learned here and practice AI for good.
— Fatima, Armin, Siraj, and Vashawn
import os
# mount drive first
from google.colab import drive
drive.mount('/content/drive')
# then find the exact file
!find /content/drive/MyDrive -name "final_mental_health_project_v3.ipynb"
Mounted at /content/drive /content/drive/MyDrive/CMSC320 Group Project/Notebooks/final_mental_health_project_v3.ipynb
!jupyter nbconvert --to html --template lab "/content/drive/MyDrive/CMSC320 Group Project/Notebooks/final_mental_health_project_v3.ipynb"
[NbConvertApp] Converting notebook /content/drive/MyDrive/CMSC320 Group Project/Notebooks/final_mental_health_project_v3.ipynb to html [NbConvertApp] WARNING | Alternative text is missing on 13 image(s). [NbConvertApp] Writing 3303187 bytes to /content/drive/MyDrive/CMSC320 Group Project/Notebooks/final_mental_health_project_v3.html