Enhancing User Interface Design Tools with AI-Driven Evaluation

Peitong Duan

EECS Department, University of California, Berkeley

Technical Report No. UCB/

May 1, 2025

User interface (UI) design is an essential domain that shapes how humans interact with technology and digital information. Feedback is critical in the design process, guiding designers toward improving their UIs. Traditionally, UI feedback has come from humans, such as expert evaluations and user testing. However, human feedback is not always available and can be costly to obtain. Automating UI evaluation offers significant benefits by providing instant feedback, enabling designers to quickly iterate on their designs.

In this dissertation, I introduce a set of techniques that utilize developments in AI to automatically evaluate UI designs, and study how design tools built on these (sometimes imperfect) AI-driven methods fit into design practice. This work includes several key contributions. First, it introduces a set of guidelines and metrics that bring semantic awareness to automated UI evaluation. Second, it investigates the performance of text-only LLMs in providing feedback on UI mockups and explores how a tool built on this approach can be integrated into existing design practices. Third, it presents UICrit, a targeted dataset of human-annotated UI design feedback, and examines its application in enhancing the feedback quality of general-purpose LLMs. Finally, it introduces an approach to improve the feedback quality generated by multimodal LLMs through various prompting techniques. This research establishes methods, guidelines, benchmarks, and datasets for AI-driven UI design evaluation, and explores novel applications of LLMs in design practice. These contributions create a foundation for future AI-driven systems that provide more effective design feedback, helping designers create engaging and user-friendly interfaces.

Advisors: Björn Hartmann

BibTeX citation:

@phdthesis{Duan:31646,
    Author= {Duan, Peitong},
    Title= {Enhancing User Interface Design Tools with AI-Driven Evaluation},
    School= {EECS Department, University of California, Berkeley},
    Year= {2025},
    Number= {UCB/},
    Abstract= {User interface (UI) design is an essential domain that shapes how humans interact with technology and digital information. Feedback is critical in the design process, guiding designers toward improving their UIs. Traditionally, UI feedback has come from humans, such as expert evaluations and user testing. However, human feedback is not always available and can be costly to obtain. Automating UI evaluation offers significant benefits by providing instant feedback, enabling designers to quickly iterate on their designs.

In this dissertation, I introduce a set of techniques that utilize developments in AI to automatically evaluate UI designs, and study how design tools built on these (sometimes imperfect) AI-driven methods fit into design practice. This work includes several key contributions. First, it introduces a set of guidelines and metrics that bring semantic awareness to automated UI evaluation. Second, it investigates the performance of text-only LLMs in providing feedback on UI mockups and explores how a tool built on this approach can be integrated into existing design practices. Third, it presents UICrit, a targeted dataset of human-annotated UI design feedback, and examines its application in enhancing the feedback quality of general-purpose LLMs. Finally, it introduces an approach to improve the feedback quality generated by multimodal LLMs through various prompting techniques. This research establishes methods, guidelines, benchmarks, and datasets for AI-driven UI design evaluation, and explores novel applications of LLMs in design practice. These contributions create a foundation for future AI-driven systems that provide more effective design feedback, helping designers create engaging and user-friendly interfaces.},
}

EndNote citation:

%0 Thesis
%A Duan, Peitong 
%T Enhancing User Interface Design Tools with AI-Driven Evaluation
%I EECS Department, University of California, Berkeley
%D 2025
%8 May 1
%@ UCB/
%F Duan:31646