This  article  discusses  the  validity  of  reading  comprehension  measurements  in  the  9th grade  national  assessments  of  Swedish.  It  argues  for  the  necessity  of  establishing  suitable   construct   definitions   of  reading   comprehension   before  any   valid   inferences   are  to   be   made   from  students’  test  results.   Theory   of  assessment  and  theory  of  literary  response  are  drawn  upon  in  order  to  analyze,  on  item  level,  the  way  in  which  good  reading   comprehension  is  repre-­‐sented  in  the  test.  The  results  demonstrate  two  different  types  of  shortcomings  related  first  of  all  to  inadequate  construct  definitions.  The  first  problem  is  the  narrow  perspective  from  which  acceptable  interpretations  in  student’s  literary  responses   are   defined.   The  second   problem   is  the   priority   given  to  quantitative   over   qualitative   facets   when   assessing   students’   constructed  responses.