Teaching Our Agent to Judge Design

We built an internal tool that picks the best graphic design from two options so the Glif agent can produce better outputs. Tested against the TASTE benchmark.

Try GlifBrowse the TASTE benchmark research

We started by looking at the TASTE benchmark study (May, 2026). This is a designer-annotated preference dataset, meaning that in the study five professional designers ranked AI-generated designs.

To generate the dataset, they made large number of prompts, and fed those into various text2image models. Then, the five professional designers ranked the outputs on nine criteria in two tracks.

  1. The aesthetics track contains: typography, color harmony, mood, hierarchy, and overall preference.
  2. The brief-adherence track contains: color accuracy, spatial accuracy, typography, and preference.

This produced a dataset of 4,314 pairs with five raters each and resulted in:

  • 747 unanimous pairs (5–0) (17%)
  • 1,555 majority pairs (4–1) (36%)
  • 2,012 contested pairs (3–2) (47%) — This large contested number points to the subjective nature of design taste.

The paper’s conclusion:

"We benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel."

Basically, the takeaway is that the current best off-the-shelf judges (VLM judges, T2I scorers) don’t reliably capture what professional designers actually care about in generated designs.

  • The best off-the-shelf judge scores 0.543.
  • A human designer agrees with the panel majority at 0.741.
  • The paper’s best result is a purpose-trained MLP head (small neural network trained specifically for this one job) with a score of 0.611.

Approach

We started by making a 200-pair subdataset, which has a similar distribution of splits to the full dataset. Each pair also got a fixed flip sequence to circumvent positional bias. (Often, models just like to pick the first image they parse as the better one.)

Next, we picked a handful of off-the-shelf LLMs and ran them against the subset with a shared basic starting prompt. We went with LLMs specifically because they allow for flexible input/output formats and, more importantly, they can explain why they chose one design over the other. For the top LLM model candidates, we did a few rounds of prompt iteration by getting a frontier model to look at the results and suggest changes each time.

What came out the other end is a pretty solid 0-shot LLM judge.

Results

Our tool scores 0.045 above the paper’s best off-the-shelf judge under the paper’s own protocol.

SystemAgreement
Random
0.500
Best off-the-shelf judge (paper)
0.543
Our tool
0.588
Trained MLP head (paper)
0.611
Human single-rater ceiling
0.741
The anchor ladder on the full pool. Our tool is above every off-the-shelf judge.
The anchor ladder on the full pool. Our tool is above every off-the-shelf judge.
Our tool scores 0.045 above the paper’s best off-the-shelf judge. The trained head keeps a small lead on the macro.
Per-bucket scores on the full pool. Where designers agree, our tool is better than the trained head.
Per-bucket scores on the full pool. Where designers agree, our tool is better than the trained head.

The buckets show where each one wins:

  • Unanimous pairs: our tool 0.693, trained head 0.653.
  • 4–1 pairs: our tool 0.618, trained head 0.600.
  • Contested 3–2 pairs: our tool 0.524, trained head 0.602.

Where designers clearly agree, our tool wins. On contested pairs, the head that was fitted to this panel’s tendencies wins. No rater does well on contested pairs, humans included. That’s because the subjectivity of taste is more pronounced in contested designs.

So our tool is weaker where experts cannot agree (3-2 splits), but scores well above the paper’s trained MLP head on 4-1 and 5-0 splits. It is very good at separating the wheat from the chaff, something we think is a useful skill for an agent to have.

Use It in the Glif Agent

This judge lives in the Glif agent’s analyzeMedia tool. The tool looks at one image at a time, so the agent first lays both candidates side by side on a single canvas, then sends that canvas together with the brief and the design rubric and asks for a winner on each criterion.

Because we are using an LLM to select the best image, we can also query why it thinks it’s better. The tool gives this back to the agent so it knows in which direction it needs to iterate.

The same tool also critiques a single image on its own, which is how the agent checks an output against the brief before showing it to you.

Our Tool Applied to Glif’s Own Work

Check out what our tool does in your session. It gets given two candidates for the same brief, it picks the best design, and explains why.

Editorial Poster

analyzeMedia · Scored in both orders · Winner B

Brief

A cutting-edge promotional poster for an end-of-year critique event at an art school, using bold typographic design as the primary visual language, referencing contemporary graphic design studios such as Zak Group and OK-RM. Bold, editorial, typographically driven, referencing Swiss International Style filtered through contemporary brutalist graphic design. The color palette is taken from a New York Magazine cover — salmon coral pink as the dominant background, deep burgundy red for the primary headline, yellow as a sharp accent block, and black for supporting information. The headline 'END OF YEAR CRIT' is set in a massive, high-contrast condensed serif that dominates the entire poster, with letterforms bleeding off the edges to create tension and energy. A small black-and-white photograph of a student presenting work at a crit wall is inset into the composition, with type overlapping or weaving through it. Supporting event information is grouped in a yellow block or set in clean sans-serif below. 'STRATUM SCHOOL OF ART' appears in tight spaced caps at the top, and the footer reads 'ALL DISCIPLINES WELCOME' in small monospaced type. The composition is asymmetric and intentionally tense, with scale contrast doing the primary design work.

Gemini 3.1 Flash

Headline confined to one cramped, hard-to-read line; supporting details sized almost as large as the headline; type doesn’t overlap the photo

GPT Image 2

Headline stacks across the full poster height with real edge bleed, supporting details stay clearly subordinate in the yellow block, and the type physically overlaps the crit-wall photo as briefed

Verdict

Image B (GPT Image 2) wins because it builds the scale hierarchy the brief is built around: 'END OF YEAR CRIT' stacks across four lines of massive condensed serif type that genuinely dominates the poster and bleeds off its edges, while the event details stay contained and subordinate inside a small yellow block, and the final 'T' physically overlaps the crit-wall photograph as the brief specifies. Image A (Gemini 3.1 Flash) crams the same headline into a single dense, hard-to-read line that never achieves the same commanding scale, then undercuts its own hierarchy by setting the venue details almost as large as the headline itself, so the two compete instead of one dominating. Its type also never touches the inset photo, missing the 'type overlapping or weaving through it' instruction entirely.

See more examples

References

  1. Zhu, H., Hirsch, E., Minetti, A., Nulty, A., Mehta, P. (2026). TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design. arXiv:2605.20731. Dataset: purvanshi/TASTE.
  2. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 D&B. arXiv:2306.05685.
  3. Wang, P. et al. (2023). Large Language Models are not Fair Evaluators. ACL 2024. arXiv:2305.17926.