A modified Delphi survey was utilised to seek consensus on LOs suitable for incorporation into UK medical school curricula. The original delphi method was developed by Dalkey and Helmer and is an iterative process designed to determine consensus through exposing the expert panel to multiple iterations of data, in this case learning outcomes. [14,15] There are a variety of observed ways of delivering a delphi however its overall distinct features including the use of an expert panel and a round based, iterative approach. Delphi methodology is used extensively in curriculum development.  This delphi was defined as modified and therefore variant methodology due to the creation of the draft curriculum by the research group rather than the use of the expert group to create the original curriculum.  This methology was used to avoid multiple rounds and therefore the risk of poor response rates with progressive rounds.  This study involved the creation of the original draft curriculum by the research team, which was put through 2 rounds of review by the expert panel before being finalised by the research team. In keeping with the principles of Delphi methodology, contributions given by the expert panel were kept anonymous to the research group throughout the process.  The methodology used in this study was also used for a Delphi study conducted by members of the research team for postgraduate SEM curricula, the same expert panel was utilised for both studies. .
Establishing the research group
The research group was made up of DV, AI, KM, TP, DS, AP, CN, PB and GF. This group was formed to incorporate individuals with experience in a wide variety of related topics including medical education, delivering SEM education, experience undertaking and/or teaching on undergraduate medical degrees, SEM Masters courses and Delphi methodology. In addition, individuals were included due to their experience with the Undergraduate Sport and Exercise Medicine Society (USEMS) and their interest in the specialty of SEM. USEMS is a UK based, non-profit society aimed at promoting the specialty of SEM for undergraduates.  All decisions regarding content were finalised by the research group such as reviewing and amending learning outcomes based on comments from the expert panel following the first round.
Expert Delphi Panel
Individuals suitable for joining the expert panels are defined as individuals with knowledge and experience in the subject area. [21,22] All members of the British Association of Sport and Exercise Medicine (BASEM) and the Faculty of Sport and Exercise Medicine (FSEM) were emailed invitations to express interest in joining this expert panel. Invitations to submit interest were also shared on Twitter by the research group.
Individuals expressing interest in joining the expert panel were asked demographic information and questions selected by the research panel to determine eligibility. The eligibility criteria were selected to ensure the expert panel was made up of individuals with adequate knowledge in the field of SEM. It was not determined whether the participants had prior experience of writing LOs.
The eligibility criteria used consisted of:
Doctors that have completed their Foundation Training.
Achieved a higher qualification in SEM: specifically, either a SEM masters degree/diploma, or membership/fellowship of the FSEM (MFSEM/FFSEM).
Graduated more than 5 years prior to the start of the study.
Working in the United Kingdom at the time of the study.
Individuals that did not meet the eligibility criteria were removed by the research group. A panel size of at least 30 was aimed for as the quality of a Delphi study has previously not been found to be improved by a panel size greater than 30. [18,23]
Development of the initial curriculum
A documentary analysis was performed and LOs included in previously published UK-based SEM specialty training and SEM Masters curricula were combined by the research group. Each learning outcome was then discussed by the research group at a meeting. Learning outcomes that were duplicated or deemed inappropriate for an undergraduate medical student were either removed or amended based on research group consensus. Following the research group review and consensus an initial draft of the curriculum was created. [2,23,24,25] The LOs in these curriculums were grouped into relevant themes as determined by the research group. All LOs were reviewed by the research group before being amended or removed if required. Bloom’s taxonomy was used to establish an appropriate level for undergraduate medical students as agreed by the research group (Fig. 1).  Reducing the Bloom’s taxonomy level was the main reason for amending a learning outcome at the draft curriculum phase by the research group. The Bloom’s taxonomy levels used are given in Fig. 1.
Google Forms (Google Inc. USA) was used to create the initial survey for those wishing to express interest in joining the expert group, and demographic information was collected via this form. Qualtrics software was used to create the electronic surveys for the Delphi.  This survey was emailed to all individuals deemed eligible to join the expert panel. Participant information sheets were emailed to all eligible individuals. Written informed consent to participation was gained by all members of the expert panel via a mandatory question asked at the start of the survey. The survey instructions stated clearly that experts should consider the relevance of individual LOs for medical student level when answering the survey.
Data were collected from October 2020 to November 2020. Members of the expert panel were given 12 days to respond to each phase of the Delphi. A system-generated email was sent after day 8 and day 10 to non-responders to act as reminders. In addition, members received a text message if no response had been received by day 10. The research group aimed for the response rate to not fall below 70% as expert panel engagement is vital for Delphi studies. .
Phase 1: First review of learning objectives.
Members of the expert panel were asked to either accept, reject or modify each LO in the proposed curriculum. To ensure standardisation of approach, the expert panel were asked to focus not only on the content or topic conveyed in the learning outcome but also on the level of Bloom’s taxonomy used. An explanation of Blooms taxonomy as well as links to further resources were provided to the expert panel. A consensus of opinion was defined by 75% agreement, as reported previously by Keeney et al.  Ranges from 70 to 100% have been reported in previous literature as appropriate for consensus. [19,29] Participants had the option of providing anonymous comments after reviewing each LO. For each LO the percentage of agreement was calculated and any anonymous comments regarding that LO were reviewed by the research group. All comments for all LOs were reviewed regardless of the level of agreement. The research group then accepted, rejected or altered each LO depending on the responses given to create a second draft of the curriculum. If a level of agreement of 75% or above was received with no comments given, the LO was accepted outright with no further need for review. If comments were given, the research group discussed the comments and, if appropriate, amended the LO and included it in the second draft of the curriculum for further review. If the level of agreement was below 75% and no comments were given, the LO was rejected. If comments were given, the research group reviewed the comments and, when appropriate, amended the LOs and included them again in the second draft of the curriculum for further review.
Phase 2: Second Review of learning objectives.
The second draft of the curriculum, based on responses given in phase 1, was sent via email only to members of the expert panel that had completed phase 1. In phase 2 members of the expert panel were asked to either accept or reject each LO and again there was the option to provide anonymous comments this time after review of each theme rather than each LO. The LOs accepted outright in phase 1 did not require a response but were included for reference. Again, percentages of agreements and comments were reviewed by the research team and decisions were made to accept or reject each LO. A level of agreement of 75% of above was again utilised. Phase 2 would be repeated until a final consensus on each LO was reached.