Case study
The test it could not study for.
3,002 panel pictures were locked away before any training began, and opened once. This is what the fault finder scored on them.
The facts
Twelve faulty panels from the test that scored lowest. The app missed all of them.
The run
Overview
One public set, split three ways
The fault finder learned from a public set of 20,000 labelled panel pictures. Before training, the set was split: 13,994 pictures to learn from, 3,004 to set the model up, and 3,002 held back. The company that published the set is not a PanelProof customer and has no connection with PanelProof.
The question
How good is the score on pictures the model has never seen?
A model can simply remember its training pictures. The only fair test is on pictures it never met. So the held back pictures stayed closed while the model was built, the pass mark was fixed on the tuning pictures, and the test was read once.
What it found
Most faults, with a known rate of false alarms
92.2
of every 100 faulty panels found
39.4
of every 1,000 normal panels flagged by mistake
95
of every 100 faults that show a pattern
61.9
of every 100 evenly warm panels
Faulty panels found, of every 100 of that kind
Where it was wrong
The normal panels it flagged
What it cannot tell
This test flatters the model
The held back pictures come from the same public set as the training pictures: the same kind of camera files, cut and saved in the same way. A new camera is a harder test. On one, the same model found 98.3 of every 100 pattern faults and 7.7 of every 100 evenly warm panels.
Credit
Whose pictures these are
InfraredSolarModules, published by Raptor Maps, Inc. under the MIT License. Raptor Maps has no connection with PanelProof.
Shown changed: enlarged for the page. Full credits and links.
More case studies