Results
How often PanelProof is right, and how often it is wrong.
Every number here comes from pictures the app had never seen. Each test was fixed in writing before its pictures were opened, and each was read once.
A plant of 1,000 panels
Say 20 of your 1,000 panels are faulty. This is what the app does.
It sorts every batch into three piles: suspected faults, panels that need review, and panels it did not flag. A person looks at the first two piles. Here are the piles, panel by panel.
3,002 pictures held back from the public set the app learned from.
246
panels go to a person: 103 as suspected faults and 144 more to review.
19.4 of 20
faulty panels are among them. 7.9 of every 100 panels sent are really faulty. The rest are false alarms.
0.6
faulty panel stays among the 754 that are not flagged. That is 0.8 in every 1,000 of them.
A fault with a patternAn evenly warm panelSent as a suspected faultSent for review
Read it with care. Most panels the app sends to a person are not faulty. That is the price of missing few. On the other camera it misses nearly every evenly warm panel, because a picture of one panel alone cannot show one. Full photos are checked for that in another way.
The fault finder
The test
3,002 pictures, set aside before any training
The fault finder is a small model that gives each panel picture one score. It learned from 13,994 labelled pictures of a public set of 20,000. Another 3,004 were used to set it up. The last 3,002 were locked away and opened once, at the end.
With one fixed pass mark, on those 3,002 pictures:
92.2
of every 100 faulty panels found. Likely range 90.8 to 93.5
39.4
of every 1,000 normal panels flagged by mistake. Likely range 30.7 to 50.5
7.8
of every 100 faulty panels missed
A likely range is where the true figure would fall 95 times in 100 if the test were run again on new pictures of the same kind.
By kind of fault
It finds patterns well. It finds evenly warm panels badly.
Faulty panels found, of every 100 of that kind
Against simpler ways
Two plain methods, tested on the same pictures
Before trusting a model we tested two methods with no model in them. Each was set up on the tuning pictures and read once on the held back ones.
Faulty panels found, of every 100
Rare faults
The fewer faults a plant has, the more of the flags are false alarms
The same fault finder, with the same pass mark, on plants with different shares of faulty panels. Worked out for 1,000 panels.
| Faulty panels | Faults found | False alarms | Real, in 100 flags | Faulty, in 1,000 not flagged |
|---|---|---|---|---|
| 5 in 1,000 | 4.6 of 5 | 39.2 | 10.5 | 0.4 |
| 10 in 1,000 | 9.2 of 10 | 39 | 19.1 | 0.8 |
| 20 in 1,000 | 18.4 of 20 | 38.6 | 32.3 | 1.6 |
| 50 in 1,000 | 46.1 of 50 | 37.4 | 55.2 | 4.2 |
Naming the type
One fault type of five passed its test
The app may name a type only if the name is right at least 85 times in 100 and it catches at least 80 of every 100 of its kind.
| Fault type | Name is right, in 100 | Catches, in 100 | Named by the app |
|---|---|---|---|
| Cracked | 83.1 | 76.1 | No |
| Bypass diode active | 99.6 | 92.8 | Yes |
| Hot cell or hot spot | 84 | 76.5 | No |
| Something blocking the panel | 81.6 | 71.3 | No |
| Whole panel not working | 90.3 | 51.6 | No |
Every other suspected fault reads "type not clear", and the technician chooses the type.
Another camera
The test
3,675 pictures of one array, made by other people with other equipment
A model can look good on pictures like its training ones and fail on a new camera. So we tested on a second public set and trained on none of it. It holds 840 faulty panels: 409 that show a pattern and 431 that are evenly warm.
Faulty panels found, of every 100, at 100 false alarms in every 1,000 normal panels
This missed our own mark. Before the test we wrote down that fewer than 70 of every 100 faults found would mean stop and think again. It found 51.8. So panel pictures are offered as a review tool for pattern faults, and evenly warm panels are left to full photos.
The pass mark
A fixed pass mark did not travel
These pictures were saved in another way than the training pictures, and the model scored nearly all of them high. A pass mark set for 50 false alarms in every 1,000 normal panels gave 781 here. Nearly four normal panels in five were flagged.
So the app does not carry a pass mark from one set of pictures to the next. It sets its levels from each batch.
The review rule
The aim
50 false alarms in every 1,000 normal panels, whatever the camera
The rule looks at how the scores of one batch are spread, finds where the ordinary panels sit, and puts its two levels a fixed distance above them. It was chosen from 13 candidates on tuning pictures and then fixed.
Normal panels sent as suspected faults, in every 1,000
Pictures like its trainingAnother camera and site
The numbers behind this chart
| Faulty in 100 | Like training | Other camera | Pattern faults found, like training | Pattern faults found, other camera |
|---|---|---|---|---|
| 0 | 87.2 | 79.7 | none in batch | none in batch |
| 2 | 85.5 | 74.5 | 96.7 | 97 |
| 5 | 75.3 | 78.4 | 96.5 | 97.7 |
| 10 | 63.1 | 73.3 | 96.4 | 97.6 |
| 20 | 39.7 | 69 | 94.9 | 97.6 |
| 30 | 24.1 | 66.5 | 93.4 | 97.5 |
Batch size
Below 200 panels the rule is a guess, so the app does not use it
Normal panels sent as suspected faults, in every 1,000
The numbers behind this chart
| Panels in the batch | On average | Nine batches in ten |
|---|---|---|
| 25 | 136.7 | 0 to 440 |
| 50 | 93.9 | 0 to 298 |
| 100 | 74.2 | 0 to 202.2 |
| 200 | 60.6 | 10.2 to 140 |
| 500 | 52.7 | 18.5 to 101.7 |
| 1,000 | 49.1 | 22.5 to 84 |
| 2,000 | 48.8 | 27.6 to 73.8 |
When it fails
If most of a batch is faulty, the rule cannot find the ordinary panels
The rule needs ordinary panels to measure against. When too many panels stand high it refuses to set a level and lists every panel for review. But from about 40 faulty panels in every 100 upwards it often does not refuse, and then it leaves faults in the pile that is not flagged. The app shows a warning when a batch scores high overall.
| Faulty panels, in 100 | It refused, in 100 batches | Warning shown, in 100 batches | Faults left not flagged, in 100 |
|---|---|---|---|
| 2 | 0 | 0 | 6.1 |
| 10 | 0 | 0 | 6.7 |
| 20 | 0 | 0 | 7.6 |
| 30 | 0.4 | 0 | 8.9 |
| 40 | 32.4 | 5.2 | 13.5 |
| 50 | 57.5 | 48.6 | 20.5 |
| 60 | 58.1 | 99.7 | 28.1 |
| 70 | 40.3 | 100 | 40.9 |
| 80 | 0.1 | 100 | 64.6 |
Faults left not flagged are counted in the batches where a level was set.
Short of time
If a person can only look at the top of the list
The review list is sorted, most suspicious first. This is how many faults sit in the first part of it, on a plant where 2 panels in every 100 are faulty.
| Panels looked at, in every 100 | Pattern faults found, in 100 | Evenly warm found, in 100 | ||
|---|---|---|---|---|
| Like training | Other camera | Like training | Other camera | |
| The first 3 | 91.5 | 92.4 | 47.6 | 2.1 |
| The first 5 | 94.6 | 97.3 | 60.3 | 4.4 |
| The first 10 | 97 | 98 | 73 | 7.4 |
| The first 15 | 97.6 | 98.8 | 82.5 | 9.7 |
| The first 20 | 98.3 | 99.3 | 84.9 | 12.8 |
The audit sample
Some panels the app did not flag are sent to a person anyway
About 3 of every 100 panels that are not flagged go into the review list as an audit sample. If the reviewer finds faults among them, the pile that was not flagged cannot be trusted. One inspection is too small to be sure, so the count builds up across inspections.
Chance, in 100, of noticing that 10 in every 1,000 panels not flagged are faulty
Where it was wrong
Twelve faults it missed. Twelve normal panels it flagged.
These are real pictures from the held back test, with what the dataset says each one is. They are the worst misses, not a fair sample.
Real drone photos
What the app said
1,485 panels on 8 photos of one roof
On full photos the app compares each panel with its neighbours. Here it marked 162 evenly warm panels, 150 of them in 10 groups, and 51 panels with a hot part. The fault finder alone had called 15 of those 162 a suspected fault.
A hot part, or a suspected faultEvenly warmNeeds review
Not measured. Nobody has marked which panels on these photos are faulty. So these are the app's marks, not a score. We cannot say how many of them a technician would agree with.
The roof, photo by photoNot measured
Plainly
What these tests do not tell you
- How well the neighbour check agrees with a technician. No marked full photos exist for it.
- How the app does on plants and cameras in Nigeria. Every test picture comes from public sets made elsewhere.
- Temperatures. The app works from the camera's own readings or from brightness, and prints no degrees.
- How bad a fault is. The app flags a panel and may name a type. It does not rank faults.
- Energy or money. We have not measured savings, so we promise none.
- Full photos from cameras other than the one model tested, the DJI Zenmuse H20T.
How we test
Four rules we hold ourselves to
- The pictures for a final test are locked away before any training starts.
- What counts as a pass, and what would make us stop, is written down before those pictures are opened.
- A final test is read once, and the reading is logged. No second try.
- Every claim about how well the app works is shown beside how often it is wrong.