Learning Representations to Guide
Pixel Diffusion Transformers

Abstract

Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as externally aligned semantics can be progressively lost during pixel synthesis. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling, without necessarily yielding representations better suited to generation. These insights suggest that a useful generative representation must be both properly formed and actively exploited. Motivated by these insights, we introduce Representation Guidance (RG), which directly forms and exploits internal representations while retaining a plain Transformer architecture. A low-dimensional bottleneck forces generation-relevant information into a compact representation, which is then explicitly used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset. RG learns clearer semantic and spatial structure while scaling efficiently in both parameters and computation. On ImageNet $256\times256$, RG-L/16 with only 459M parameters outperforms substantially larger pixel diffusion transformers, while RG-XL/16 achieves an FID of 1.55. RG further reaches an FID of 1.74 at $512\times512$ and extends to text-to-image generation, achieving 0.89 on GenEval and 87.1 on DPG-Bench.

Motivation

Why pixel space and a plain Transformer?

• Pixel space. Representation learning and image synthesis take place end-to-end from raw pixels, rather than through a representation space predefined by an external autoencoder.
• Plain Transformer. Without specialized architectural design, it allows efficient scaling and provides a simple setting for examining how internal representations emerge and interact with pixel synthesis.

Together, they offer a fully end-to-end and scalable setting, in which representations for generation are learned rather than predefined.

Two Key Insights

Our analysis of this setting reveals two key insights.

1Better representation quality does not necessarily translate into greater generative utility.

• Baseline
• REPA-4
• REPA-6
• REPA-8

2Architectural separation mainly redistributes semantic and spatial modeling rather than yielding a representation better suited to generation.

• Vanilla
• PixelDiT
• DiP
• DeCo

Representation Guidance

The bottleneck and guidance play complementary roles: the former governs how the representation is formed, while the latter ensures that generation actually uses it.

• Implicit latent space. A low-dimensional bottleneck between the encoder and decoder forms a compact implicit latent space, encouraging generation-relevant information to be organized into a concise internal representation.
• Representation guidance. We compare decoder predictions with and without the representation and use the resulting representation-induced velocity offset to modify the flow-matching target. This increases the model's dependence on the representation, while gradients through it simultaneously shape the encoder toward information useful for synthesis.

Results

Without introducing specialized pixel modules, RG achieves a substantially stronger parameter–quality and compute–quality trade-off, scales consistently with model size, and develops markedly clearer semantic and spatial organization in its internal representations.

Faster convergence

Layer-wise representations

ImageNet $256\times256$

Bold = best in column.

Samples

ImageNet

Text-to-image, $1024\times1024$

BibTeX

@article{pan2026learning,
  title   = {Learning Representations to Guide Pixel Diffusion Transformers},
  author  = {Pan, Zihao and Chen, Mengting and Shi, Cheng and Hu, Taihang and
             Yue, Zhengrong and Wang, Zikang and Wang, Yuan and Li, Wei and Yang, Sibei},
  journal = {arXiv preprint arXiv:2610.12978},
  year    = {2026}
}