Jack Puccini

If we assume AI will lack 'taste' for some time going forward (think: does this marketing copy read like slop, is this generated image cheesy), then we'll need systems to compensate. You can give skills and optimise prompts to refine what the model does at the generation stage, but my feeling is that investment in the review stage matters more.

A composer doesn't immediately produce their ideal melody. They try many, and it's their expert judgement that selects from the candidates. Correcting taste might work the same way: an adequately proficient reviewer, working in tandem with the generator, yields a better eventual output. The system would involve finetuning the reviewer - not necessarily in the technical weights sense - to align its judgement with a reference standard, presumably an expert human who carries that judgement.

There's much to discuss here:

• How much signal is actually present in the human judgement? If the human judged twice, what would the correlation between trials be? This upper-bounds the potential ability of any reviewer.
• What methodologies best suit this analysis? Pairwise comparisons and Bradley-Terry?
• What if the output requires a translation layer? For example, if the thing to be judged is a video production, AI can't adequately judge it as a whole (for now) - it must first be parsed into some LLM-understandable structure, which itself may be lossy.

Anyway - these were supposed to be short. Fuller post later (hopefully).