Score Administration Management

Thalamus allows programs to build customized Scorecards to support the applicant selection process. By creating a scorecard, your program can define specific scoring criteria—structured through competencies and their associated questions—and automatically deliver evaluations to your reviewers within Cortex. Reviewers can then score applicants directly and consistently using the scorecard.

Programs can create multiple scorecards to support different stages or aspects of applicant evaluation. Each scorecard can also be configured with customized weighting to prioritize competencies based on your program’s goals.

Note: Thalamus Holistic Scoring includes a prebuilt set of Competencies, Questions, and Scorecards to simplify setup. However, you can also create and tailor your own Competencies, Questions, and Scorecards to meet the specific needs of your program.

Competencies & Questions

To build a scorecard, Programs must first create Competencies & Questions. Competencies are your foundation for evaluating applicants at all stages of the recruitment process and across Thalamus products, including Cortex (for selecting applicants to interview) and Thalamus Core (for evaluating applicant interviews).

When creating competencies, think of answers to these questions:

• Who are you looking for?

• What qualities do they possess?

• What are the characteristics of trainees who flourish in your program?

Examples of competencies may include academic performance, medical knowledge, professionalism, communication, voluntarism, geographic connection, and more.

Competencies and questions are then used to build scorecards in the next step, allowing reviewers to submit applicant evaluations throughout application review/screening, interviews, and/or ranking.

Create a New Competency

1. Navigate to the Settings tab from your top menu bar. Select “Score” from the drop-down.

2. Click the “+ Add Competency” button at the bottom of the “1 Competencies & Questions” page.

3. The New Competency editor module will appear. Click the pencil and paper icon to edit a new or existing competency. Click the trash can icon to remove the competency or cancel your edits.
   NOTE: A competency may not be deleted if there are questions added to that competency.

4. Click the “New Competency” box and type your competency name.

5. Click the “Competency Description” box and write a short description of the competency.

6. Click the “Save” button to save your new competency.

7. A green notification box will appear in the upper right-hand corner of your screen when the competency is saved successfully.

Add Questions to a Competency

1. Click the pencil and paper icon of the competency you wish to add the question to. The competency editor module will appear.

2. Click the “Add Question” button and the New Question module will appear underneath the competency name and competency description.

3. Click the “New Question” box and type your question or statement.

4. Click the “Question type” Dropdown menu to choose which type of question you wish to create. For all question types, the option for Unable to Assess allows the reviewer to select and exclude this question from the score rollup. Additionally, all question types provide the option to enable a note field adjacent to the question.

   • Score offers configurable upper and lower limits setting the allowed range. Reviewers will choose a number within the allowed range.

   • Dropdown supports selection of one response from a predefined list of choices. A custom score value may be assigned to each choice.

   • Levels offer a range of 2-10, a title and description for each level and also supports a response between levels. The configurable scores are not visible to the reviewer. 

   • Likert offers a range of 2-10 with customizable labels. The configurable scores are not visible to the reviewer.

   • Yes/No offers a binary choice. The option for Unable to Assess enables a third, non-binary response when needed.

   • NOTE: Question Tooltip = supports a rubric in the tooltip adjacent each question. This tooltip can contain a couple of paragraphs of text if desired.

5. After choosing the Question type, users may customize the available answers to be selected by reviewers.

6. When all settings are configured for the question, click “Save Question” in the bottom right corner of the new question module.

7. A green notification box will appear in the upper right hand corner of your screen when the question is saved successfully.

Create a Scorecard

After creating competencies and questions, users may build Scorecards.

To build a Scorecard:

1. Click the “2 Scorecards” step at the top of your Score Administration page.

2. Click the “+ Add Scorecard” button on the bottom right side of the page.

3. The Scorecard editor will appear, allowing you to customize:

   • Scorecard name

   • Scorecard description

   • Where the Scorecard will be used (in Cortex, Thalamus Core, or Disabled)

   • If the Scorecard should appear first in the list of available scorecards if there are multiple

   • Any question from any competency can be added to any scorecard

4. To add questions to the Scorecard, click the “Manage Questions” button, and the “Manage Questions for Selection” module will appear.

5. The Manage Questions module will present all available competencies and their associated questions. You may also search for questions in the by typing in the Search bar on the top of the module. For example, typing “communication” will pull all available questions with the word “communication.”

6. Select the box(es) next to the question(s) to include in this Scorecard.

7. The sum/average button on the scorecard configuration controls how the scorecard score is created. The aggregation page controls how the scorecards are aggregated together.

8. Normalize Scores (optional)
   • To normalize your questions' scores, enable the Normalize option next to the question you want to use as the reference. The other questions' scores will be adjusted to match this question's scale. 

   • Normalization adjusts question scores so they use the same scale. This ensures results can be compared or averaged fairly, even if the original questions have different scoring ranges. For example, if some questions are scored from 1-3 and others from 1-5, their raw scores can't be compared directly. A "3" on a 3-point scale represents a different level than a "3" on a 5-point scale. Normalization converts all scores to a single, consistent range (such as 0-1 or 1-5) so that each question contributes equally to the overall evaluation. 

     

9. After selecting all of your desired questions and the Scorecard Score type, click the “Save Changes” button on the bottom right side of the Manage Questions module

10. The Scorecard will now appear as an option for reviewers when scoring applicants in Cortex and Thalamus Core.

Manage Reviewer Weighting

You can customize how individual scores are aggregated across your team. By default, scores are weighted equally (100% each) and averaged. Reviewer weighting applies to all program scorecards and score types, including Overall Score.

How to Assign Review Weights:

1. Select "3 Reviewer Weighting" at the top of the screen

2. Click “Add Reviewer”

3. Click in the text box labeled “Reviewer” to select the assigned reviewer

4. The role will automatically populate once you add a Reviewer

5. Adjust “Weight” if applicable

6. Click “Save Changes”

Aggregate Scores

Several score types are calculated at different stages of the recruitment process. Aggregating scores allows your program’s score managers to customize how the Overall Score, Interview Score, and Cortex Score are calculated.