This article reports from a study of reliability in the Swedish national reading test in ninth grade. Based on a representative sample of 500 students, the study investigates internal consistency and power to discriminate between different levels of reading ability. Data was analyzed using both classical test theory and Rasch modelling. Results from the study suggest that neither the subscales nor the whole scale of items included in the test reach desirable levels of internal consistency. Three out of four subscales include items with strikingly low degree of internal correlation (.23–.46). Furthermore, results from the Rasch analysis suggest that the test is capable of reliable separation between two groups of test-takers only: lower performing students and higher performing students. The empirical support for categorizing student performances into six separate grade levels is therefore questionable. Implications for future development of the national test of reading are discussed.