Predicting Peace: Machine Learning and NLP for Peace Agreement Success

Sun, 01 Jun 2025 00:00:00 +0000

Note: This is a very early prototype — work in progress. All local work will be pushed to GitHub once it is complete.

— data, scripts, trained models, and Streamlit app
— explore predictions and SHAP explanations for any agreement configuration

Project Overview

This project develops a machine learning and natural language processing framework to forecast whether a peace agreement will successfully end active armed conflict. Grounded in computational social science, it integrates structured metadata, contextual conflict characteristics, and the full text of peace agreements to build interpretable predictive models, bridging empirical conflict research with state-of-the-art AI methods.

The project uses the (University of Edinburgh), covering over 2,000 peace agreement texts with extensive metadata on conflict type, negotiation stage, institutional provisions, and implementation outcomes.

Beyond academic contribution, the project is deployed as a live, interactive web application that enables researchers and practitioners to input agreement characteristics and receive real-time success predictions with AI-generated explanations.

Research Questions

Can AI systems reliably forecast whether a peace agreement will end armed conflict — and explain why?

Which features of a peace agreement — structural, institutional, or textual — are most predictive of durability?

Methods

Feature Engineering

Metadata encoded into binary and categorical indicators (conflict type, negotiation stage, security arrangements, human rights provisions, inclusion of social groups)
Peace agreement texts preprocessed via tokenization, lowercasing, stopword removal, and TF-IDF representation
Hybrid feature matrix combining structured metadata with textual information

Classic Machine Learning Models

Trained across metadata and text combinations:

Model	Notes
Logistic Regression	Baseline linear classifier
Support Vector Machines	Margin-based classification
Random Forest	Best stable performance on metadata
Gradient Boosting / XGBoost	Ensemble boosting methods
AdaBoost	Adaptive ensemble

Key finding: text-only classic ML performed poorly; adding metadata and contextual variables substantially improved performance.

Transformer-Based Models

Fine-tuned using Hugging Face:

Model	Accuracy	Precision	Recall	F1
DistilBERT-base-uncased	—	—	—	—
DeBERTa-v3-Large	0.888	0.824	0.893	0.824

DeBERTa-v3-Large substantially outperforms classic ML for text classification, particularly on the class imbalance present in peace agreement outcomes.

Best-performing model performance summary and top SHAP-identified predictive signals across 1,508 PA-X peace agreements.

Explainable AI (SHAP)

The deployed Random Forest integrates SHAP-based interpretability, enabling users to understand why a specific agreement is predicted to succeed or fail.

Most influential features identified by SHAP:

Negotiation stage
Conflict type
International missions and enforcement mechanisms
Human rights provisions
Security guarantees

This explainability layer allows practitioners and researchers to interpret model predictions in a theory-informed way — connecting computational output to established conflict studies literature.

Interactive Application

The is a deployed Streamlit app that enables interactive exploration of model predictions.

Users can:

Input metadata features describing any peace agreement
Receive a predicted outcome (SUCCESS / FAILURE) with probability estimates
Explore SHAP feature contributions explaining the specific prediction

Research Contribution

This project contributes to computational social science and AI-for-policy research by demonstrating that transformer-based NLP, combined with structured conflict metadata, can produce accurate and interpretable forecasts of peace agreement outcomes. Unlike prior work relying on static quantitative models, this framework integrates textual content, provides local explainability, and is deployed as a live tool accessible to non-technical users.

Work in Progress

Survival analysis extending the framework to model peace duration rather than binary success (Cox proportional hazards, Random Survival Forests, DeepHit)
Natural-language explanation layer via OpenAI API to describe not only what is likely to fail, but what provisions could strengthen a treaty before signing
Full interactive web application integrating the OpenAI API for real-time, human-readable prediction explanations accessible to researchers and policymakers
Research manuscript in preparation for journal submission

Computational Social Science |