How to Build a Scoring Rubric and Run a Fair Judging Panel

In short

A scoring rubric is a short list of criteria, each with a clear description of what a high and low score looks like, that every reviewer uses to assess every application. The best rubrics are simple: three to five scored criteria, a short scale with each point described, and eligibility checked separately before scoring starts. Pair that with a brief calibration session, at least two independent reviews per application, and a separate field for feedback to applicants, and you have a fair, defensible process that your judges will actually enjoy using.

What is a scoring rubric for grants and awards?

A scoring rubric is the shared yardstick your reviewers use to judge applications. It breaks a big, subjective question (“is this good?”) into a few smaller ones (“does it fit our mission?”, “can they deliver it?”) and defines what each score means.

A good rubric does three jobs:

Consistency. Every application is judged on the same things, whoever reviews it.
Defensibility. When an applicant or trustee asks why one project was funded over another, you have an answer.
Speed. Reviewers spend less time working out what you want and more time reading applications.

There are two broad types. An analytic rubric scores each criterion separately and combines them into a total. A holistic rubric gives one overall score against a single description. Most grant and award programmes use analytic rubrics, sometimes with one holistic “overall recommendation” vote at the end.

Why should you keep your rubric simple?

Because complicated rubrics make worse decisions, not better ones. Every extra criterion is another judgement call a tired reviewer has to make, and another place for two reviewers to read the same words differently.

The clearest example is NIH. In January 2025, NIH simplified peer review for most research project grants to tackle the complexity of the process and the risk of reputational bias. Its five scored criteria (Significance, Investigators, Innovation, Approach, Environment) were reorganised into three factors, and the third, covering expertise and resources, is now rated “sufficient” or “not sufficient” rather than given a number.

If the world’s largest public research funder decided its rubric was asking reviewers too much, yours probably is too. Simple doesn’t mean shallow. It means each question you ask actually moves the decision.

How many criteria should a scoring rubric have?

Three to five scored criteria is the sweet spot for most grants and awards. Fewer than three and you can’t tell strong applications apart. More than five and reviewers stop weighing each one carefully and start scoring on gut feel, which defeats the point of the rubric.

To get there, cut anything that is really one of these:

An eligibility check. “Is the applicant based in the UK?” or “Is the budget under £10,000?” are yes/no questions. Check them before review, not in the rubric.
A duplicate. “Innovation” and “Originality” are usually the same criterion under two names.
Something you can’t see in the application. If applicants weren’t asked about it, reviewers can’t score it fairly.

A typical grant rubric might be: Fit with our mission, Quality of the idea, Feasibility, and Impact. A typical arts award might be: Originality, Technical skill and Strength of the body of work.

What scoring scale should you use?

Use the shortest scale that still separates your applications. For most programmes that’s a 1–5 scale, with every point described.

Scale	Best for	Watch out for
Yes / No / Maybe vote	Early sift rounds, eligibility, shortlists	Too blunt for ranking finalists
1–5	Most grant and award criteria	Reviewers clustering on 3 if points aren’t described
1–10	Large, competitive pools where you need fine ranking	Reviewers can’t reliably tell a 6 from a 7
Sufficient / Not sufficient	Criteria that are thresholds, not scales (e.g. capacity to deliver)	Not usable for ranking

Longer scales feel more precise but rarely are. If two reviewers can’t agree on what separates a 6 from a 7, that extra precision is noise. It’s also worth keeping the same maximum across criteria so scores are easy to compare at a glance.

How do you write rubric descriptors that reviewers agree on?

Describe evidence, not adjectives. “Excellent”, “good” and “adequate” mean different things to different people. “Has a timeline and a budget that match” means the same thing to everyone.

Here’s an example for a Feasibility criterion on a 1–5 scale:

Score	What it looks like
1	No plan or budget; unclear how the work would happen
2	Some plan, but major gaps (no timeline, or a budget that doesn’t add up)
3	Plan and budget present; one or two obvious risks not addressed
4	Clear timeline and budget that match; main risks named
5	As 4, plus evidence they’ve delivered similar work before

Write these before applications open, and test them by scoring two or three past applications yourself. If you find yourself hesitating between two numbers, the descriptors need tightening. Publish your criteria to applicants too: people write better applications when they know what’s being judged.

Should some criteria count more than others?

Weight criteria only when one genuinely matters more, and give the most weight to the criterion that should decide close calls. For most funders that’s mission fit; for most awards it’s the quality of the work itself.

Weighting works by turning each score into a percentage, multiplying by its weight, then averaging. For example, with three criteria scored out of 5:

Mission fit: 4/5 = 80, weight 3
Quality of idea: 5/5 = 100, weight 2
Feasibility: 3/5 = 60, weight 1

(80×3 + 100×2 + 60×1) ÷ 6 = 83.3 out of 100

Keep weights simple: 1, 2 and 3 are plenty. If your weighting needs a spreadsheet to explain, it’s too complicated for reviewers to trust.

When should reviewers vote, score or comment?

Each type of question has a different job. Using the wrong one is a common source of clutter.

Use a…	When you need…	Example
Score	To rank applications against each other	“Quality of the idea, 1–5”
Vote	A decision or category, not a ranking	“Recommend for shortlist: Yes / No / Discuss”
Internal comment	The reviewer’s reasoning, for your team	“Strong idea, but budget relies on unconfirmed match funding”
Applicant feedback	Something the applicant will read	“Your timeline was clear; the budget needed more detail on artist fees”

Votes are underused. A single “overall recommendation” vote alongside your scores is often more useful in a panel meeting than a decimal-point total, because it shows you what each reviewer actually thinks should happen.

How do you get reviewers to score consistently?

Run a short calibration session before scoring starts. Give every reviewer the same two or three sample applications, have them score independently, then compare and talk through where they differ.

This makes a measurable difference. In a 2015 study of grant reviewers, reviewers who watched an 11-minute training video agreed with each other far more closely than those who didn’t: inter-rater reliability of 0.89 versus 0.61. The improvement held for both new and experienced reviewers.

Calibration also shouldn’t be a one-off. Standards drift as reviewers work through a round, especially when they score lots of applications in a row. On a large round, check in halfway through and look at the spread of scores per reviewer. If one judge is consistently two points harsher than everyone else, a quick conversation fixes it.

Other simple fairness controls:

At least two independent reviews per application, more for finalists.
Hide names and identifying details where the work can be judged on its own.
Let reviewers declare conflicts of interest and step away from an application.
Score before discussing, so the loudest voice in the room doesn’t set everyone’s numbers.

How do you run a judging panel meeting?

Use the panel meeting for disagreements, not for re-reading every application. Scores do the sorting; the meeting decides the close calls.

A simple running order:

Agree the clear yeses and clear nos quickly based on scores and votes.
Spend most of the time on the middle band applications near the funding line, or where reviewers disagreed sharply.
Ask the reviewers who disagreed to explain their reasoning referring back to the rubric.
Record the reason for each final decision in a sentence, which becomes the basis for applicant feedback.

Watch for mismatches between what reviewers write and how they score. A study of NIH panels, aptly titled “Your comments are meaner than your score”, found that talk about each other’s scoring habits drove much of the variation between panels. If a reviewer’s comments are critical but their score is high, ask which one reflects their real view.

Should you give feedback to unsuccessful applicants?

Yes, if you can make it specific. Applicants consistently say feedback is what they most want from funders. When the Paul Hamlyn Foundation surveyed its applicants, more useful feedback on unsuccessful applications was one of the most common requests from declined applicants. The Ellerman Foundation heard the same and began giving written feedback to first-stage rejections from October 2020.

Your rubric makes this far easier, as long as you keep two things separate:

Internal comments are candid notes for your team.
Applicant feedback is written for the applicant: one or two sentences per application, tied to the criteria, with something they can act on next time.

Ask reviewers to write applicant feedback in a separate field while they score, rather than trying to rewrite internal notes weeks later. Then check it before it goes out. If you can’t give individual feedback to everyone, a short summary of common reasons applications fell short is still far better than a template rejection.

How does Dapple handle rubrics, voting and feedback?

Dapple is built around the simple approach in this guide. Each Review Round is attached to a stage of your process and has its own scoring, guidance and settings, so an early sift round can use a quick vote while your finalist round uses a full rubric.

Judging results with average scores and votes in Dapple

Rubric scoring lets you add Score criteria with a description, max score and weight. Dapple converts everything to a weighted average out of 100, so criteria on different scales stay comparable.
Vote and Text criteria sit alongside scores without affecting the total, with fully customisable voting options.
Applicant feedback is a separate field on the review form, kept apart from internal notes, and sent to applicants in bulk with each person receiving only their own feedback.
Fairness controls include hidden form fields, anonymous review, conflict-of-interest declarations, minimum and maximum review counts, and reviewer pools that share applications out automatically.
Dapple AI can flag eligibility issues against criteria you set and write a summary of each application before reviewers open it, so judges spend their time judging.

See how reviews and judging work in Dapple.

Where to go next

Sources

Last updated November 2026.

How to Build a Scoring Rubric and Run a Fair Judging Panel (Without Overcomplicating It)