This paper considers the issue of optimal subcarrier allocation in OFDMA. We show, via a counter example, that water-filling based subcarrier allocation policies, contrary to conventional wisdom, fail to provide rate-stability for an otherwise stabilizable OFDMA system. Water-filling is too myopic when considering long-time average performance, e.g. delay, queue lengths, and even long-run throughput. This is because such policies ignore variable state (queue length) information, while, in fact, such an information is necessary to guarantee rate stability and/or to minimize average delay. In this paper, we identify an optimal non-idling policy which balances the queue lengths, when the channel follows an ON/OFF model. In such case, we show that such a policy achieves the minimum holding cost (mean response time) at any time.