The purpose of this document is to define the Review Process for the MICCAI conference. The document has been initiated and is endorsed by the MICCAI 2017, 2018, 2019 and 2020 organizing committees, to be passed on each year, from successive MICCAI organizing committees, with the goal to refine and include feedback from the full set of future conference organising committees to create a "sliding window” and consistency between years. Updated by the 2021 committee in consultation with the MICCAI board and the 2022 and 2023 organizing committees.
The policies set within are born of a desire to enhance the previous process, by specifically:
As in previous MICCAI conferences, the goal of the MICCAI conference review process is to select the best papers in each discipline. This selection should be fair, taking into consideration the specialized nature of our discipline and the size of our community; efficient, in not wasting valuable time, effort and funding from our peers; and just, in exploiting the consensus of peer comments.
This summary is made public so that all participants understand the review process and can plan accordingly.
The MICCAI 2021 review process will be overseen by the following individuals:
Marleen de Bruijne
Philippe Cattin
Stéphane Cotin
Nicolas Padoy
Stefanie Speidel
Yefeng Zheng
Kitty Wong
Caroline Essert
The program committee of MICCAI 2021 consists of two bodies of participants that are essential to the review process:
The following documents supplement this guide, and will be made available on the MICCAI 2021 conference website:
The MICCAI Society maintains a database of past Area Chairs and reviewers. Excellent ACs and reviewers should be identified after each event. The MICCAI 2021 organizing committee will build on the list provided by earlier editions. As in the previous three years, MICCAI 2021 will use CMT to maintain this database. The information about the number of times a member has been an AC will be kept indefinitely. The information on excellence of reviewers or ACs will be maintained in the system.
MICCAI 2021 also will issue a call for participation in the program committee via the newsletter and mailshots. Individuals will submit relevant information regarding their past participations in MICCAI and other similar conferences, as well as other biographical details.
Individuals will be chosen from the database, the received applications and other sources, and invited to join the MICCAI 2021 program committee. When they apply, ACs will need to update their Toronto Paper Matching System entries, create/update their CMT profiles and associate themselves as being either MIC, CAI or MICCAI. They will need to use their institutional email address. The MICCAI Program Chairs will then select the program committee using the following guidelines:
A webinar may be held to explain the review process to ACs and secure their commitment early in the process.
A webinar reviewer tutorial may be held to explain to the reviewers what is expected of them and how they can write high quality reviews.
Details and schedule for AC duties include the following:
You will be asked to update your subject areas in CMT and upload your relevant papers to TPMS by the end of February. An initial teleconference will be held in March to explain the full review process and to discuss reviewing criteria. After the full paper submission deadline on 3rd March 2021, the first phase of your work will run from 11-17 March 2021 for the initial reviewer assignment period and it is essential that you are available during this week. The paper review period will run from 31 March - 20 April during which time you are expected to monitor the progress of the reviewers and communicate with them to request for further information in reviews where needed. Between 20 April-7 May you will need to provide feedback for authors and recommend papers for accept, reject or rebuttal. The second phase will then start once the rebuttals have been submitted and between 22 May - 4 June you will be asked to provide final rankings on your original paper allocations. In addition, you will be asked to act as a metareviewer and rank a further 15-20 papers. These are time intensive tasks and it is essential that you will be available during these crucial periods. Your responsibilities will end with the program committee teleconferences in June 2021. A detailed timeline is posted on the Conference webpage.
By accepting to serve on the program committee, you agree to this timeline.
The list of potential reviewers is primarily based on the list of active reviewers of MICCAI 2019 and 2020.
In addition, all members of the MICCAI Society and all authors of MICCAI 2020 will be invited to apply to become a reviewer. Volunteering reviewers will be vetted by the Program Chairs. Student reviewers must be at least enrolled at Ph.D. level and have two or more published articles in a related field to qualify as reviewers. ACs will contribute to updating and expanding the list of reviewers.
Each potential reviewer will be invited to review for MICCAI. By accepting to review:
Authors will provide an intention to submit ten days prior to the paper submission deadline. Authors must provide: i) a list of all co-authors and their affiliations and email addresses, ii) Google Scholar, Semantic Scholar and DBLP ids if available and publication email addresses, as requested in the CMT User Profile iii) conflict of interest domains for all authors, iv) the title of the paper, and v) the abstract of the paper. All co-authors will be asked to fill out their CMT User Profile including Google Scholar, Semantic Scholar and DBLP ids if available and publication email addresses. This information will be used to start building the assignment of papers to ACs and potential reviewers. Authors and co-authors must use their institutional/work email address whenever possible.
Authors will submit papers in CMT. The MICCAI Conference review process will be double-blind, i.e., the names of the authors will be hidden from the Area Chairs and Reviewers, and the names of Reviewers and Area Chairs will not be revealed to the authors. To achieve this, papers must be properly anonymized before submission. At the discretion of the Program Chairs, a paper may undergo an outright rejection when it is in blatant breach of these anonymization rules. Area Chairs and Reviewers shall bring concerns about such a breach to the attention of the Program Chairs.
Each paper must be submitted with Primary and Secondary areas selected from the CMT system. Authors also must identify to which stream the paper belongs, i.e., either MIC, CAI, or MICCAI. These areas, the stream, and the paper itself, will be used to generate suggested reviewers using the automated TPMS paper matching system embedded in the CMT system (see Stages 5 and 6).
The goal of this step is to ascertain the breadth and depth of expertise required within the program committee and to increase the number of ACs when needed in response to the initial submissions from authors. Given that it is almost impossible to accurately predict which domains will be most represented in any conference edition, this step will allow for the adjustment of the program committee composition.
The Program Chairs may then invite approximately 10 additional AC members having an identified expertise from the database of potential ACs and reviewers.
Each paper will be assigned to an AC automatically by the CMT system, based on the TPMS and CMT subject areas, while balancing the load across ACs. Note that the AC remains blind to the paper authorship. The Program Chairs will check the assignments to make sure all papers received a good assignment.
The essential role of the AC is to move the paper through the review process, up until the decision by the program committee. ACs will use their knowledge of the topic and of the appropriate reviewers to ensure the best (most informative) reviews.
For MICCAI 2021, 20-25 papers will be allocated to each AC. This is necessary to ensure a proper statistical distribution in the ranking to follow (cf. Stage 7). The number of ACs will be adjusted accordingly. Thus, for approximately 80ACs, MICCAI 2021 is expecting to handle between 1600 and 2000 submitted papers.
The goal of the paper allocation is to find the most appropriate reviewers in terms of expertise for a given paper. This step is achieved in two phases using the CMT system:
In phase 1, the CMT system will provide a list of potential reviewers for each paper to the AC. This ordered list will be generated based on keywords and TPMS scores. Using their expertise and judgement, the AC will create a ranked list of 10-15 suggested reviewers for each paper, avoiding reviewers who already have more than 25 papers to bid. In phase 2, reviewers will bid for papers by categorising them into "Eager to review”, "Willing to review”, "In a pinch”, "Not Willing”. Reviewers will be asked to bid on all the papers for which the ACs suggested them as possible reviewer.
The CMT system will then optimize matching of reviewers to papers based on the ranked list provided by the ACs, reviewer bids, TPMS scores, and keywords, while load balancing across all papers, reviewers and ACs. Three reviewers are assigned for each paper.
Note that the reviewers and ACs remain blind to the paper authorship at all times. The Program Chairs will ensure ACs have suggested reviewers on time. The Program Chairs and Platform Manager will check reviewer assignments and manually adjust where needed.
The goal of the review step is to provide constructive feedback of each submitted paper.
The reviewers will:
MICCAI 2021 will use the 10-point reviewing scale as introduced by the 2019 team, with slightly altered wording compared to the 2020 version.
The AC will log in frequently to check review progress and ask for a more detailed or fair review if needed. This is critical. Should the AC be unsatisfied with the quality of a review, and failing to get further feedback from the reviewer, then the AC can ask for additional reviewer(s) input on the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified in the reviewer database.
The Area Chairs will provide a ranking of the papers they handle as Primary AC, identify borderline papers for rebuttal, and recommend early acceptance or rejection of papers based on consistent reviews and scores.
ACs will provide metareviews explaining their recommendation and indicating for borderline papers sent to the rebuttal phase what aspects the authors should address in their rebuttal. ACs will also be asked to rate the usefulness of each review, both in Stage 8 and Stage 9.
The Program Chairs recommend that:
The Program Chairs will check all AC recommendations - they will follow AC recommendations in most cases but may make some adjustments to ensure consistency across ACs.
In line with previous editions, MICCAI 2021 expects to reach an acceptance rate of ~30%.
We anticipate that after the initial review in this stage, ~15% will be accepted outright; ~ 50% will be rejected, and the remaining borderline papers will be sent for rebuttal. Approximately 40% of these borderline papers will ultimately be accepted.
Reviews and Primary AC metareviews of the papers are sent to the authors. Authors will have one week to submit their rebuttal/response.
The goal of the rebuttal/response process is
New in MICCAI 2021 is that we intend to make reviews and rebuttals of accepted papers publicly available (without disclosing the reviewers names) on the MICCAI website. This holds for metareviews as well. By accepting to be an AC for MICCAI, you give us permission to publish your metareviews.
Borderline papers will enter Stage 9 (below). At this point, all borderline papers will be assigned 2 Secondary ACs based on TPMS matching and CMT subject areas, balancing load over ACs. Secondary AC assignment will be checked, and if necessary adjusted, by the Program Chairs.
The goal of this step is to provide a more reliable ranking and acceptance decision for borderline papers by combining the assessments of multiple ACs. After rebuttals are entered, each AC will rank all their borderline papers (both primary and secondary AC papers) and provide a recommendation to either "accept” or "reject”. ACs will further provide a metareview (for both their primary and secondary assignments) for each assigned paper, consistent with their recommendation and based on their assessment of the reviews and rebuttal.
New in MICCAI 2021 is that we intend to make reviews and rebuttals of accepted papers publicly available (without disclosing the reviewers names) on the MICCAI website. This holds for metareviews as well. By accepting to be an AC for MICCAI, you give us permission to publish your metareviews.
In this stage, ACs can consult the Primary ACs and/or the reviewers for clarifications via the CMT system.
In addition, each AC will:
The goal of the decision process is to establish the final list of accepted papers.
The final decision for rebuttal papers is in principle determined by majority voting, i.e. the paper is accepted if at least two ACs recommend "Accept”. Taking into account possible restrictions on the number of papers that can be accepted to the conference, in consultation with the Organizing Committee and the MICCAI Board, the Program Chairs prepare the final list of accepted papers.
We will hold a set of dedicated teleconferences (in different time zones) between the MICCAI 2021 Program Chairs and the ACs to inform about the overall results and statistics of the review process to discuss and resolve any arising issues, and to collect their feedback and recommendations for any further improvements of the process.
The goal of this step is to select the best submitted papers that justify an oral presentation.
The number and format of the oral sessions will be determined by the Organizing Committee, in consultation with the MICCAI Board and will reflect the overarching objectives of the conference. The recommendations of the ACs will be taken into consideration when deciding which papers to select for full oral presentation. There will be an emphasis on providing Young Investigators with the opportunity to present and there will minimally be 1 CAI oral session.
Final decisions of oral versus poster presentations may be made and communicated up to two months after acceptance decision.
The goal of this step is to inform authors of borderline papers about the outcome of the review process.
The Program Chairs will issue the following, via email, to all authors:
A complete list of acceptance and orals will be drafted for dissemination as a program. The Program Chairs will record each complaint and follow up.