Measuring Bias and Hallucination in Large Language Models

Large Language Models (LLMs), such as GPT-4, represent a groundbreaking advancement in artificial intelligence, providing transformative capabilities for diverse applications, from customer service automation to scientific research. Despite their potential, LLMs pose significant ethical challenges due to inherent biases in their outputs. These biases—stemming from training datasets, model objectives, and fine-tuning processes—can perpetuate stereotypes, suppress viewpoints, and align with particular ideologies, raising concerns about fairness, transparency, and societal equity.

This project focuses on developing a robust methodology for auditing and mitigating biases in LLMs. Unlike traditional direct-prompting techniques, which often fail against advanced LLM guardrails, this research will employ indirect prompting methods and role-based simulations to reveal implicit biases. By engaging LLMs in organic discussions through various contextual roles (e.g., scientist, journalist), the project aims to bypass evasive model behaviours and extract meaningful insights into bias manifestations.

The proposed framework will integrate both qualitative and quantitative evaluation metrics to systematically analyse biases. This approach will ensure scalability, reproducibility, and alignment with ethical standards and regulatory mandates, such as the EU AI Act. The ultimate goal is to enhance the transparency and accountability of LLMs, fostering inclusivity and trust in their applications across industries.

This research holds significant implications for ethical AI practices and aims to establish a scalable methodology for ensuring that LLMs remain fair, responsible, and beneficial tools in modern society.

Expected Outcomes

1. A Scalable Framework for Bias Auditing in LLMs

o  Development of a novel methodology using indirect prompting and role-based simulations to uncover and evaluate implicit biases in Large Language Models (LLMs). This framework will be designed for scalability and reproducibility, addressing limitations of existing bias detection techniques.

1. Actionable Insights for Ethical AI Development

o  Detailed analysis of bias manifestations across various contexts and roles, providing actionable recommendations for mitigating these biases during training, fine-tuning, and deployment of LLMs. These insights will guide both developers and policymakers in aligning LLMs with ethical and societal standards.

1. Compliance with Ethical and Regulatory Standards

o  Creation of tools and guidelines that facilitate adherence to ethical AI principles and regulatory requirements, such as the EU AI Act. The outcomes will enhance the transparency, accountability, and fairness of LLM applications across diverse industries.

Keywords: Large Language Models, Artificial Intelligence, Bias, Hallucination, Harms Detection, Ethics, Auditing Methodology.

Requirements:

Essential Requirements

• Bachelor’s or Master’s degree (2:1 or above) in Computer Science, Artificial Intelligence, Machine Learning, or a closely related field.
• Proficiency in programming (e.g., Python) and familiarity with Machine Learning frameworks (e.g., PyTorch, TensorFlow).
• Understanding of Natural Language Processing (NLP) techniques and models (e.g., Transformers, LLMs).
• Demonstrated ability to conduct independent research and analyse complex datasets.
• Strong analytical and critical thinking skills, with an ability to address ethical challenges in AI.
• Excellent communication skills, both written and verbal, for documentation and publication of research findings.

Desirable Requirements

• Experience in Bias Detection or fairness in Machine Learning.
• Familiarity with ethical AI frameworks and regulatory guidelines (e.g., EU AI Act).
• Knowledge of qualitative and quantitative evaluation methods in AI research.
• Hands-on experience with fine-tuning and auditing machine learning models.

How to Apply:

Interested candidates should email Dr. Yashar Moshfeghi (yashar.moshfeghi@strath.ac.uk) and include the following attachments:

• Cover letter detailing contact information, motivation, background, and proposed research direction (max 3 pages).
• Up-to-date CV.
• Transcripts and certificates of all degrees.
• Two references, one academic.

Contact Dr. Yashar Moshfeghi to express interest. Applications will be processed on a ‘first come, first served’ basis, and the hiring process will conclude as soon as a suitable candidate is identified.

We are committed to inclusion across race, gender, age, religion, identity, and experience, and we believe that diversity makes us stronger by bringing in new ideas and perspectives. The University of Strathclyde was established in 1796 as “the place of useful learning”. This remains at the forefront of our vision today for Strathclyde to be a leading international technological university that makes a positive difference in the lives of its students, society and the world. Strathclyde was the first institute to win the coveted Times Higher Education “University of the Year” award twice, in 2012 and 2019, and has since been voted the Scottish University of the Year in 2020.