Results

How often PanelProof is right, and how often it is wrong.

Every number here comes from pictures the app had never seen. Each test was fixed in writing before its pictures were opened, and each was read once.

A plant of 1,000 panels

Say 20 of your 1,000 panels are faulty. This is what the app does.

It sorts every batch into three piles: suspected faults, panels that need review, and panels it did not flag. A person looks at the first two piles. Here are the piles, panel by panel.

3,002 pictures held back from the public set the app learned from.

246

panels go to a person: 103 as suspected faults and 144 more to review.

19.4 of 20

faulty panels are among them. 7.9 of every 100 panels sent are really faulty. The rest are false alarms.

0.6

faulty panel stays among the 754 that are not flagged. That is 0.8 in every 1,000 of them.

A fault with a patternAn evenly warm panelSent as a suspected faultSent for review

Averages over many batches of 500 panels with 2 faulty panels in every 100, scaled to 1,000 panels and rounded to whole panels in the picture. Where each panel sits is for show.

Read it with care. Most panels the app sends to a person are not faulty. That is the price of missing few. On the other camera it misses nearly every evenly warm panel, because a picture of one panel alone cannot show one. Full photos are checked for that in another way.

The fault finder

The test

3,002 pictures, set aside before any training

The fault finder is a small model that gives each panel picture one score. It learned from 13,994 labelled pictures of a public set of 20,000. Another 3,004 were used to set it up. The last 3,002 were locked away and opened once, at the end.

With one fixed pass mark, on those 3,002 pictures:

92.2

of every 100 faulty panels found. Likely range 90.8 to 93.5

39.4

of every 1,000 normal panels flagged by mistake. Likely range 30.7 to 50.5

7.8

of every 100 faulty panels missed

A likely range is where the true figure would fall 95 times in 100 if the test were run again on new pictures of the same kind.

By kind of fault

It finds patterns well. It finds evenly warm panels badly.

Cracked100of 100, on 142 pictures
Bypass diode active96.4of 100, on 251 pictures
Hot cell or hot spot96.2of 100, on 550 pictures
Something blocking the panel91of 100, on 435 pictures
Whole panel not working61.9of 100, on 126 pictures

Faulty panels found, of every 100 of that kind

A panel that is not working at all is evenly warm from edge to edge. In a picture of that panel alone there is no pattern to see.

Against simpler ways

Two plain methods, tested on the same pictures

Before trusting a model we tested two methods with no model in them. Each was set up on the tuning pictures and read once on the held back ones.

A brightness rule52.355 false alarms in 1,000
A forest of 500 decision trees64.649 false alarms in 1,000
The fault finder92.239 false alarms in 1,000

Faulty panels found, of every 100

Rare faults

The fewer faults a plant has, the more of the flags are false alarms

The same fault finder, with the same pass mark, on plants with different shares of faulty panels. Worked out for 1,000 panels.

Faulty panelsFaults foundFalse alarmsReal, in 100 flagsFaulty, in 1,000 not flagged
5 in 1,0004.6 of 539.210.50.4
10 in 1,0009.2 of 103919.10.8
20 in 1,00018.4 of 2038.632.31.6
50 in 1,00046.1 of 5037.455.24.2

Naming the type

One fault type of five passed its test

The app may name a type only if the name is right at least 85 times in 100 and it catches at least 80 of every 100 of its kind.

Fault typeName is right, in 100Catches, in 100Named by the app
Cracked83.176.1No
Bypass diode active99.692.8Yes
Hot cell or hot spot8476.5No
Something blocking the panel81.671.3No
Whole panel not working90.351.6No

Every other suspected fault reads "type not clear", and the technician chooses the type.

Another camera

The test

3,675 pictures of one array, made by other people with other equipment

A model can look good on pictures like its training ones and fail on a new camera. So we tested on a second public set and trained on none of it. It holds 840 faulty panels: 409 that show a pattern and 431 that are evenly warm.

Faults that show a pattern98.3range 96.5 to 99.2
Evenly warm panels7.7range 5.5 to 10.6
All faults together51.8range 48.4 to 55.1

Faulty panels found, of every 100, at 100 false alarms in every 1,000 normal panels

This missed our own mark. Before the test we wrote down that fewer than 70 of every 100 faults found would mean stop and think again. It found 51.8. So panel pictures are offered as a review tool for pattern faults, and evenly warm panels are left to full photos.

The pass mark

A fixed pass mark did not travel

These pictures were saved in another way than the training pictures, and the model scored nearly all of them high. A pass mark set for 50 false alarms in every 1,000 normal panels gave 781 here. Nearly four normal panels in five were flagged.

So the app does not carry a pass mark from one set of pictures to the next. It sets its levels from each batch.

The review rule

The aim

50 false alarms in every 1,000 normal panels, whatever the camera

The rule looks at how the scores of one batch are spread, finds where the ordinary panels sit, and puts its two levels a fixed distance above them. It was chosen from 13 candidates on tuning pictures and then fixed.

Normal panels sent as suspected faults, in every 1,000

0255075100025102030Faulty panels in every 100 of the batchThe aim: 50Like trainingOther camera

Pictures like its trainingAnother camera and site

It runs above its aim on most batches and below it when many panels are faulty. On the other camera a fixed pass mark gave 781. The rule is far steadier than that. It is not exact.
The numbers behind this chart
Faulty in 100Like trainingOther cameraPattern faults found, like trainingPattern faults found, other camera
087.279.7none in batchnone in batch
285.574.596.797
575.378.496.597.7
1063.173.396.497.6
2039.76994.997.6
3024.166.593.497.5

Batch size

Below 200 panels the rule is a guess, so the app does not use it

Normal panels sent as suspected faults, in every 1,000

No levels are set here0100200300400The aim: 5025501002005001,0002,000Panels in the batch
Each point is the average over many batches of tuning pictures. Each line shows where nine batches in ten fell. A small batch is listed whole, most suspicious first.
The numbers behind this chart
Panels in the batchOn averageNine batches in ten
25136.70 to 440
5093.90 to 298
10074.20 to 202.2
20060.610.2 to 140
50052.718.5 to 101.7
1,00049.122.5 to 84
2,00048.827.6 to 73.8

When it fails

If most of a batch is faulty, the rule cannot find the ordinary panels

The rule needs ordinary panels to measure against. When too many panels stand high it refuses to set a level and lists every panel for review. But from about 40 faulty panels in every 100 upwards it often does not refuse, and then it leaves faults in the pile that is not flagged. The app shows a warning when a batch scores high overall.

Faulty panels, in 100It refused, in 100 batchesWarning shown, in 100 batchesFaults left not flagged, in 100
2006.1
10006.7
20007.6
300.408.9
4032.45.213.5
5057.548.620.5
6058.199.728.1
7040.310040.9
800.110064.6

Faults left not flagged are counted in the batches where a level was set.

Short of time

If a person can only look at the top of the list

The review list is sorted, most suspicious first. This is how many faults sit in the first part of it, on a plant where 2 panels in every 100 are faulty.

Panels looked at, in every 100Pattern faults found, in 100Evenly warm found, in 100
Like trainingOther cameraLike trainingOther camera
The first 391.592.447.62.1
The first 594.697.360.34.4
The first 109798737.4
The first 1597.698.882.59.7
The first 2098.399.384.912.8

The audit sample

Some panels the app did not flag are sent to a person anyway

About 3 of every 100 panels that are not flagged go into the review list as an audit sample. If the reviewer finds faults among them, the pile that was not flagged cannot be trusted. One inspection is too small to be sure, so the count builds up across inspections.

500 audited panels38.4alarm at 6 faults found
1,000 audited panels54.3alarm at 10 faults found
2,000 audited panels84.5alarm at 16 faults found
3,000 audited panels92alarm at 23 faults found
5,000 audited panels99.3alarm at 34 faults found
10,000 audited panels100alarm at 63 faults found

Chance, in 100, of noticing that 10 in every 1,000 panels not flagged are faulty

Where it was wrong

Twelve faults it missed. Twelve normal panels it flagged.

These are real pictures from the held back test, with what the dataset says each one is. They are the worst misses, not a fair sample.

Several hot cellsWhole panel offSeveral hot spotsWhole panel offShadeBypass diodeBypass diodeWhole panel offSeveral hot spotsSeveral hot cellsWhole panel offWhole panel off
Faulty panels that scored lowest. Several show almost nothing in the picture.
NormalNormalNormalNormalNormalNormalNormalNormalNormalNormalNormalNormal
Normal panels that scored highest. Some look as if they do have a warm part.

Real drone photos

What the app said

1,485 panels on 8 photos of one roof

On full photos the app compares each panel with its neighbours. Here it marked 162 evenly warm panels, 150 of them in 10 groups, and 51 panels with a hot part. The fault finder alone had called 15 of those 162 a suspected fault.

A hot part, or a suspected faultEvenly warmNeeds review

Not measured. Nobody has marked which panels on these photos are faulty. So these are the app's marks, not a score. We cannot say how many of them a technician would agree with.

The roof, photo by photo

Not measured

Plainly

What these tests do not tell you

  • How well the neighbour check agrees with a technician. No marked full photos exist for it.
  • How the app does on plants and cameras in Nigeria. Every test picture comes from public sets made elsewhere.
  • Temperatures. The app works from the camera's own readings or from brightness, and prints no degrees.
  • How bad a fault is. The app flags a panel and may name a type. It does not rank faults.
  • Energy or money. We have not measured savings, so we promise none.
  • Full photos from cameras other than the one model tested, the DJI Zenmuse H20T.

How we test

Four rules we hold ourselves to

  • The pictures for a final test are locked away before any training starts.
  • What counts as a pass, and what would make us stop, is written down before those pictures are opened.
  • A final test is read once, and the reading is logged. No second try.
  • Every claim about how well the app works is shown beside how often it is wrong.