Evaluating AI-assisted Social Engineering

We are looking for both technical and governance students. Below are some representative tasks one can expect during the fellowship period.

Technical Candidates (ML Engineer)

• Expand our AI agents to include voice and SMS-based phishing simulations.
• Improve the OSINT functionality of our scam-training tool to accommodate diverse user demographics (e.g., students and seniors).
• Review and refine our current AI tool’s persuasion techniques, and conduct experiments on how to measure and mitigate different persuasion attack types..
• Create a blue team AI agent that tests the effectiveness of our scam-generating (red team) agents.
• Investigate how social-engineering techniques integrate into the broader chain of automated cyberattacks, including technical intrusion methods such as those documented in Anthropic’s recent AI-espionage case: https://www.anthropic.com/news/disrupting-AI-espionage

Governance, Policy, and Legal Candidate s

• Create a plan for how to better align ScamBench with AISI’s guidelines for evaluation benchmarks (https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluation).
• Map relevant compliance, legal, and ethical requirements for scaling and working with ScamBench (e.g., data protection, privacy, and consent frameworks).
• Map out and implement strategies to recruit participants for ScamBench to scale its data and reach.
• Review and improve our metrics for different types of persuasion techniques and scams (such as the grandchild-in-distress, EZ Toll, and Mastercard verification scams).
• Develop and implement a plan to ensure comprehensive demographic representation.

Relevant links: https://arxiv.org/abs/2412.00586 https://ieeexplore.ieee.org/document/10466545 https://www.reuters.com/investigates/special-report/ai-chatbots-cyber/ https://hbr.org/2024/05/ai-will-increase-the-quantity-and-quality-of-phishing-scams

AI-powered cyberattacks, social engineering, and manipulation are widely considered some of the greatest risks of our time. This project conducts cutting-edge research on detecting, preventing, and mitigating these threats by systematically evaluating AI-enabled attack capabilities and designing safeguards to protect individuals, institutions, and critical infrastructure.

Multiple weekly check-in calls = Detailed Guidance [2+ hr/week/mentee]: I won't be working directly with my mentees but will dedicate substantial attention to checking on their progress in one or more meetings where I will be able to engage with the details of their work.

Technical Candidate (ML Engineer) – Qualifications Required

• Background in computer science.
• Familiarity with AI development, equal to having completed Arena (https://github.com/callummcdougall/ARENA\_3.0). Bonus
• Web development experience.
• Interest in cybersecurity, social engineering, and human-computer interaction.
• Experience with Docker, Kubernetes, and API integrations.

Governance and Policy Candidate – Qualifications Required

• Prior experience in community engagement or nonprofit outreach, and building strategic partnerships.
• Ability to conduct structured interviews, including calling or meeting with seniors, caregivers, or other target groups to gather qualitative data.
• Experience in gathering user feedback, such as designing and conducting short interviews, surveys, or feedback sessions with users (including seniors and caregivers) to assess the effectiveness of our AI-phishing tool and ScamBench, identify usability issues, and collect insights to improve training materials and benchmarks
• Comfort in learning the basics about legal compliance for data management. For example, ensuring we maintain compliance with data protection and privacy requirements across different US states, streamlining verification processes, ensuring ethical consent collection, secure data storage, and anonymization practices, and staying up to date on relevant laws and ethical standards for handling sensitive user information. Bonus
• Experience from working with vulnerable population groups.
• Organizational and project management skills, such as scheduling participant sessions, keeping track of datasets, and coordinating research tasks.

We're based in Cambridge US and San Francisco