Perceptual optimization is primarily driven by the fidelity objective, which
enforces both semantic consistency and overall visual realism, while the
adversarial objective provides complementary refinement by enhancing perceptual
sharpness and fine-grained detail. Despite their central role, the correlation
between their effectiveness as optimization objectives and their capability as
image quality assessment (IQA) metrics remains underexplored. In this work, we
conduct a systematic analysis and reveal an unanticipated asymmetry between
perceptual optimization and assessment: fidelity metrics that excel in IQA are
not necessarily effective for perceptual optimization, with this misalignment
emerging more distinctly under adversarial training. In addition, while
discriminators effectively suppress artifacts during optimization, their
learned representations offer only limited benefits when reused as backbone
initializations for IQA models. Beyond this asymmetry, our findings further
demonstrate that discriminator design plays a decisive role in shaping
optimization, with patch-level and convolutional architectures providing more
faithful detail reconstruction than vanilla or Transformer-based alternatives.
These insights advance the understanding of loss function design and its
connection to IQA transferability, paving the way for more principled
approaches to perceptual optimization.