Case study

The test it could not study for.

3,002 panel pictures were locked away before any training began, and opened once. This is what the fault finder scored on them.

The facts

Pictures
3,002 panel pictures
1,504 faulty, 1,498 normal
24 by 40 pixels each
Found
92.2 of every 100 faulty panels
Likely range 90.8 to 93.5
Flagged by mistake
39.4 of every 1,000 normal panels
Likely range 30.7 to 50.5
Several hot cellsWhole panel offSeveral hot spotsWhole panel offShadeBypass diodeBypass diodeWhole panel offSeveral hot spotsSeveral hot cellsWhole panel offWhole panel off

Twelve faulty panels from the test that scored lowest. The app missed all of them.

The run

Overview

One public set, split three ways

The fault finder learned from a public set of 20,000 labelled panel pictures. Before training, the set was split: 13,994 pictures to learn from, 3,004 to set the model up, and 3,002 held back. The company that published the set is not a PanelProof customer and has no connection with PanelProof.

The question

How good is the score on pictures the model has never seen?

A model can simply remember its training pictures. The only fair test is on pictures it never met. So the held back pictures stayed closed while the model was built, the pass mark was fixed on the tuning pictures, and the test was read once.

What it found

Most faults, with a known rate of false alarms

92.2

of every 100 faulty panels found

39.4

of every 1,000 normal panels flagged by mistake

95

of every 100 faults that show a pattern

61.9

of every 100 evenly warm panels

Cracked100of 100, on 142 pictures
Bypass diode active96.4of 100, on 251 pictures
Hot cell or hot spot96.2of 100, on 550 pictures
Something blocking the panel91of 100, on 435 pictures
Whole panel not working61.9of 100, on 126 pictures

Faulty panels found, of every 100 of that kind

Where it was wrong

The normal panels it flagged

NormalNormalNormalNormalNormalNormalNormalNormalNormalNormalNormalNormal
Twelve normal panels that scored highest. Some look as if they do have a warm part.

What it cannot tell

This test flatters the model

The held back pictures come from the same public set as the training pictures: the same kind of camera files, cut and saved in the same way. A new camera is a harder test. On one, the same model found 98.3 of every 100 pattern faults and 7.7 of every 100 evenly warm panels.

Credit

Whose pictures these are

InfraredSolarModules, published by Raptor Maps, Inc. under the MIT License. Raptor Maps has no connection with PanelProof.

Shown changed: enlarged for the page. Full credits and links.