---------- Forwarded message ---------- From: ICMT 2009 Date: Sat, Mar 21, 2009 at 6:01 PM Subject: ICMT 2009 notification To: Ralf Laemmel Dear authors, Thank you for your submission to ICMT 2009. Unfortunately, we were unable to accept your submission. The reviewing process was extremely competitive, and we were only able to accept 22% of the papers submitted. You should find the reviews for your paper below. I hope that you will be able to join us in Zurich nevertheless, for what promises to be an excellent conference. Thanks again for your interest in ICMT 2009. Best wishes, Richard Paige ICMT'09 Program Committee Chair --------------------------------------------- Paper: 33 Title: Consistency of the Java Language Specification -------------------- review 1 -------------------- PAPER: 33 TITLE: Consistency of the Java Language Specification OVERALL RATING: -2 (reject) ----------------------- REVIEW -------------------- This paper reports experiences on comparing the three published versions of the Java Language Specification (JLS), each of which contains two versions of the Java grammar. The authors compare the changes over each 3 JLS's, and also the relationships between the two grammars in each. Exactly what the final results are is, to me, slightly unclear: the results on Page 13 don't fully make sense to me in the sense that I don't really understand why the data is interesting. I think this might just be a lack of explicit analysis on the authors part in Section 5 (I'm sure they've done this internally, but it doesn't come across in the prose). I didn't really understand how much of the "grammar convergence" is automatic or manual. Initially it sounded automatic; by Section 4.2 I thought it was semi-automated; and by the conclusion I realised it was mostly manual. I don't think this confusion is deliberate on the authors part, but it does need fixing. The fact that the whole study took 10 man months is really scary. To be frank, the results in the paper don't justify 10 months worth of effort. The authors seem to realise this also, judging by comments in the conclusion. This suggests to me that, unless the amount of effort required can be substantially lessened, this approach will remain a curiou, and probably a dead-end. I was also very surprised not to see any mention in the paper of the cock-ups that are present in the JLS grammars. It's fairly well known that the readable grammar in the JLS3 spec (at least) contains not only nasty typos, but also incorrect rules that means it doesn't actually parse most Java code. The required fixes aren't that difficult (for someone who knows what they're doing), but I did wonder if the authors measurements were based on the known incorrect version?! There is a fundamental lack of motivation in this paper. Personally I can vaguely imagine academic interest in analysing grammars in this way, but I can't think of a killer example. Is there one? If so, put it in the paper - your readers will greatly appreciate a reason for reading the whole paper! I also felt it odd that the paper started off in a very formal style (too formal probably), gradually getting less formal as the paper went on; this seemed the wrong way around to me. In summary, this paper is the result of real work. It's fairly well written (if too formal at the beginning). If the authors can fix the motivational problem, the lack of analysis, and be more explicit about exactly how their proposed process works, then I think this will be a good paper. However I do think the authors need to think very carefully about what an appropriate venue is for this paper. Minor comments It would be very helpful if the paper stated, in the introduction, what "more implementable" means. After all, both grammars are implied to be CFGs, so they're both implementable, by definition. If one is more easily implemented in limited parsing algorithms, that's different. Personally I wasn't that interested in Section 2.3 or Section 3. This is incredibly specific to the way the Java grammar is stored that I can't see that it contains a single useful lesson for readers. However, I do appreciate that it makes reproducability of results easier; so perhaps these sections should be present but just much smaller? -------------------- review 2 -------------------- PAPER: 33 TITLE: Consistency of the Java Language Specification OVERALL RATING: 1 (weak accept) ----------------------- REVIEW -------------------- This paper presents a mechanized approach to representing the correspondences between large grammars. In particular, this work is done in the context of various versions of the Java Language Specification. The work is very carefully described and seems technically sound. The main issue with this work is that it requires quite a level of expertise to be reproduced by somebody else than the authors. It is hard to assess whether this "tour de force" could be made more accessible in the future. Also, it would be interesting to have a discussion on how this approach extends beyond programming language syntax and applies to meta-modelling languages. Still, the work should inspire the model-driven engineering community. -------------------- review 3 -------------------- PAPER: 33 TITLE: Consistency of the Java Language Specification OVERALL RATING: 0 (borderline paper) ----------------------- REVIEW -------------------- The paper describes an approach to aligning different grammars in the Java Language specification using grammar transformations. Dealing with grammar transformations, the paper is certainly different than most ICMT papers, but still relevant to the conference. The paper presents the first large-scale application of a method on grammar convergence. This is interesting. However, I am concerned with the case study and some of its conclusions, because it seems that the authors "make up" the problem to be solved, rather than solving a declared problem: - The authors state that "one would expect the two grammars to be formally equivalent" in the introduction section. Is this expectation based on a claim in the JLS? Oftentimes "more readable" language introductions abstract from certain details and are not meant to be formally equivalent. In fact, the JLS quote on Page 3 seems to indicate that the JLS folks acknowledge that the two grammars are not equivalent (i.e., "similar"). On Page 11, the authors actually state that "language equivalence is too strict of a requirement for a pair of 'more readable' and 'more implementable' grammars, as we assumed in the introduction." It seems that the authors have realized half way that their initial requirement was unrealistic. - A second expectation stated in the introduction is "language inclusion". Again, is this expectation based on a claim made in the JLS? There have been languages that developed without satisfying this inclusion property. In fact, the observation stated on Page 3 that " the grammars of JLS2 and JLS3 are nowhere related explicitly..." seems to indicate that the JLS folks do not see language inclusion as a declared consistency objective - The authors claim that the above two conditions are necessary for the consistency of the Java Standard. However, if these properties are not in scope of the JLS objectives, it is not justified to say the JLS is inconsistent (or "flawed" as said in the introduction). - I understand that the grammar transformations have been performed manually. This seems to contradict the initial claim that the approach is a mechanical (meaning automatic) method. - Are the metrics reported in Fig. 7 inter-subjective? I.e., if the approach is human-driven, are the resulting numbers guaranteed to be the same if the experiment is carried out by different individuals. If not, what meaning can be assign to these metrics? - Why is 68/2 a safe lower bound for the correctness issues? - The authors report that the study has taken them 10 man months. How much effort was spent on which part of the study. How many of the results can be generalized to future similar projects? Presentation comments: - Avoid the use italics for entire paragraphs of text, e.g., the last paragraph on page 2 and the last paragraph of section 4.2. This makes the text harder to read.