ACL 2010: The 48th Annual Meeting of the Association for Computational Linguistics Review form for LONG CHALLENGE papers This review form is appropriate for papers that present a challenge to the field of computational linguistics. APPROPRIATENESS (1-5) Does the paper fit in ACL 2010? (Please answer this question in light of the desire to broaden the scope of ACL.) 5: Certainly. 4: Probably. 3: Unsure. 2: Probably not. 1: Certainly not. CLARITY (1-5) For the reasonably well-prepared reader, is it clear what the challenge is, why it is important, and what would be regarded as successfully addressing the challenge? Is the paper well-written and well-structured? 5 = Very clear. 4 = Understandable by most readers. 3 = Mostly understandable to me with some effort. 2 = Important questions were hard to resolve even with effort. 1 = Much of the paper is confusing. ORIGINALITY / INNOVATIVENESS (1-5) Does the paper present a challenge that is new to the field? Does the challenge require innovative research or new approaches? 5 = Seminal: Significant new challenge for the field -- addressing the challenge would require a change in research direction or resource allocation. 4 = Novel: An intriguing challenge to the field that is substantially different from current research goals. 3 = Respectable: A minor challenge that would require some new research effort. 2 = Pedestrian: Current research will likely fulfill the challenge. 1 = Nothing new: Previous research has already satisfied the challenge. JUSTIFICATION / FORESIGHT (1-5) Does the paper present cogent arguments for the importance and timeliness of the challenge? Does the paper lay out a reasonable timeframe for satisfying the challenge? 5 = The paper presents solid evidence for the importance of the challenge and for current research effort being allocated toward it. 4 = Generally solid arguments for addressing the challenge, but the paper does not totally convince me that it is reasonable to focus current research on addressing the challenge. 3 = An interesting challenge that may be important to tackle at some time in the future, but seems inappropriate at the current time. 2 = Troublesome. There are some ideas worth considering here, but the presentation of the challenge has serious deficiencies. 1 = Fatally flawed. The challenge would be impossible to address in the suggested time-frame given the current state-of-the-art in computational linguistics. MEANINGFUL COMPARISON (1-5) Does the paper clearly describe the current state-of-the-art relevant to the challenge? Is it clear how addressing the challenge would differ from existing research? Are the references adequate? 5 = Precise and complete consideration of existing work. Good job given the space constraints. 4 = Mostly solid bibliography and discussion of existing work, but it could be improved. 3 = Bibliography and discussion of existing work are somewhat helpful in situating the challenge, but it could be hard for the reader to determine exactly how current research efforts are failing to address it. 2 = Only partial awareness and understanding of existing work, or a flawed consideration of it. 1 = Little awareness of existing work. SUBSTANCE (1-5) Does this paper have enough substance, or would it benefit from further work? Note that this question mainly concerns the amount of work; its quality is evaluated in other categories. 5 = Does a better job of laying out the challenge, justifying it, and situating it with respect to existing research than would be expected of a challenge paper; goes the extra mile. 4 = Represents an appropriate amount of work for a challenge paper at this conference. (most submissions) 3 = Leaves open one or two natural questions regarding the challenge that should have been pursued within the paper. 2 = Work in progress. There are some good ideas, but perhaps not enough substance in the paper yet. 1 = Seems thin. Not enough ideas here for a challenge paper. IMPACT OF THE CHALLENGE (1-5) How significant is the challenge? If research is directed toward tackling the challenge, will it have a significant positive impact on the field of computational linguistics? 5 = Will affect the field by altering other people's choice of research problem or direction. 4 = The challenge will affect some aspects of future research. 3 = Interesting but not too influential. The challenge may be mentioned and cited, but it is unlikely to have an impact on research at this time. 2 = Marginally interesting. 1 = Will have no impact on the field. RECOMMENDATION (1-6) There are many good submissions competing for slots at ACL 2010; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented? In deciding on your ultimate recommendation, please think over all your scores above. But remember that no paper is perfect, and remember that we want a conference full of interesting, diverse, and timely work. If a paper has some weaknesses, but you really got a lot out of it, feel free to fight for it. If a paper is solid but you could live without it, let us know that you're ambivalent. Remember also that the author has a few weeks to address reviewer comments before the camera-ready deadline. Should the paper be accepted or rejected? 6 = Exciting: I'd fight to get it accepted; probably would be one of the best papers at the conference. 5 = Strong: I'd like to see it accepted; it will be one of the better papers at the conference. 4 = Worthy: A good paper that is worthy of being presented at ACL. 3 = Ambivalent: OK but does not seem up to the standards of ACL. 2 = Leaning against: I'd rather not see it in the conference. 1 = Poor: I'd fight to have it rejected. REVIEWER CONFIDENCE (1-5) 5 = Positive that my evaluation is correct. I read the paper very carefully and am familiar with related work. 4 = Quite sure. I tried to check the important points carefully. It's unlikely, though conceivable, that I missed something that should affect my ratings. 3 = Pretty sure, but there's a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper's details and evidence. 2 = Willing to defend my evaluation, but it is fairly likely that I missed some details, didn't understand some central points, or can't be sure about the novelty of the challenge. 1 = Not my area, or paper is very hard to understand. My evaluation is just an educated guess. RECOMMENDATION FOR BEST LONG PAPER AWARD (1-3) 3 = Definitely. 2 = Maybe. 1 = Definitely not. ACCEPTANCE AS A SHORT PAPER (1-4) If this submission is rejected as a long paper, could it be turned into a reasonable short paper? (It is possible that some of the long paper submissions will be accepted as short papers.) Submissions to which this might apply include long papers in which the work is too preliminary, or a challenge paper which really doesn't need 8 pages of content, but which would make a good 4-page paper. In making this judgement please consider only whether the paper could be reasonably reduced to 4 pages, rather than its quality. (If some long papers are accepted as short papers, then your evaluation above will also be taken into account.) 4 = Could easily be turned into an excellent short paper. 3 = Could be turned into a reasonable short paper with some effort. 2 = Could be turned into a short paper with a lot of effort, but key points of the paper may be lost. 1 = Not at all suitable for a short paper.