How Olive reports are built, and how to check every score
Every Olive report is a structured evaluation: one requirement set written for a specific buyer, the same questions put to every vendor, and scores that trace back to their source. Here is the full process, what it measures, and where it stops.
In four lines
The short version
Requirements first
Every report starts from what a specific kind of buyer needs, not from a list of vendors.
Same questions for everyone
Every vendor in a report is scored against the same requirement set.
Scored mechanically
Positions come from answers to the requirements, not from anyone's opinion of a vendor.
Sourced, score by score
Every score shows where its answer came from.
How a report is made
Four steps, the same for every report
Each step depends on the one before it, and the same four apply whether Olive or an Olive consultant partner authors the report.
Step 1
Define the buyer and their requirements
Each report is scoped to a buyer audience, such as national restaurant chains evaluating ERP. Olive, or an Olive evaluation consultant, then writes the requirement set the way a real evaluation is scoped: grouped into areas, with the ones that buyer cannot compromise on marked Must Have. When a consultant authors a report, the report says so.
Step 2
Collect answers from three sources
Every answer comes from public product documentation, a direct vendor response, or an evaluation run by buyers and consultants in Olive. Where a vendor has not responded, its answers are inferred from public documentation and marked as such.
Step 3
Score mechanically, rank by use case
Each answer is scored against its requirement. A vendor’s total is the share of scored requirements it meets; Must Have requirements are also scored on their own, and an unverified answer earns no credit. One category can have several leaders, so each report shows the overall ranking plus rankings by segment and capability, where the order changes and why.
Step 4
Publish with sources attached, refresh on demand
Every score in a published report carries the source class of the answer behind it: inferred, vendor confirmed, or unverified. Reports with active reader demand are refreshed about once a year, sooner when demand justifies it. Vendors can respond at any time, and new responses are included in the next edition.
Reading a report
What a score tells you, and what it does not
A score measures fit to the report's requirements
A score of 86% means the product meets 86% of the requirements set for that report's buyer. It is not a measure of overall quality, popularity or market share.
A position is only as close to yours as the requirements are
If your requirements look like the report's, its ranking is a strong starting point. If they differ, especially in your Must Haves, the order can change. Every report shows its segment, requirement count and Must Haves so you can judge the fit before you read.
Scores describe what a product does, not what it is like to live with
Some reports include customer stories verified or conducted by Proofmap, a third party. These stories are labelled, and they never affect scores or positions.
Alternatives
Where Olive reports fit alongside other vendor research
Each type of research answers a different question. Olive reports answer one: which vendors fit a defined set of requirements, and why.
| What it measures | Who sets the criteria | Can you trace a score? | Fit to your requirements | |
|---|---|---|---|---|
| Olive reports | Fit to a defined requirement set | Written for the report's buyer, and published | Yes, score by score | Direct when the requirements match yours. Exact when you run your own. |
| Peer review sites | User satisfaction and sentiment | Reviewers, individually | Partly, through the individual reviews | Indirect. It depends on whose reviews you read. |
| Analyst reports | Market position and vision | The analyst firm | Rarely. The criteria are published; the scoring usually is not. | Broad, typically for large enterprise buyers |
| "Best of" lists | Editorial or commercial selection | The publisher | Usually not | Generic |
| AI chat answers | A summary of what is already published | Whatever the model draws from | Only as far as its citations go | Depends on the prompt |
Peer reviews are the best signal for day-to-day experience, and analyst research is valuable for market context. Use them together. Olive reports are for the question neither was built to answer.
Your requirements
The most exact ranking is built on your requirements.
A report ranks vendors against a buyer close to you. To rank them against your own requirements, run the same evaluation in Olive.
Questions
The rules, and the limits
Same rules for every report, including the ones we write, and the limits stated plainly.
Can a ranking be bought?
No. Inclusion and position come from answers to the requirements, scored mechanically. They are not purchased by vendors, at any price.
Do customer stories affect scores?
No. Customer proof features never affect scores, position or inclusion.
Are consultant-authored reports scored differently?
No. Consultant-authored reports are labelled as such, and the same scoring rules apply. Vendor responses to a consultant-authored report are shared with that consultant.
Is a vendor response an independent test of the product?
No. A vendor response describes the vendor’s own product and is marked vendor confirmed. It is not an independent test.
Why might a product be missing from a report?
A report reflects the vendors that responded or had enough public documentation to score. Some products in a category may not appear.
How current are the scores?
Scores reflect a report’s last edition, and the edition date is shown on every report. Reports with active reader demand are refreshed about once a year, sooner when demand justifies it.