We propose an optically coherent self-heterodyne transceiver structure for differential quadrature amplitude modulation (QAM) transmission. The scheme is based on phase modulation detection that potentially simplifies the front-end of a coherent optical receiver. The scheme has been demonstrated via extensive simulations for 16-QAM at 40 Gb/s and 256-QAM at 100 Gb/s, together with a novel phase noise compensation scheme that improves the system performance. © 2012 IEEE.