Case study
One array in Louisiana, from another camera and another team.
3,675 panel pictures the app had never trained on. It caught nearly every fault that shows a pattern, and missed nearly every panel that is evenly warm.
The facts
Top row: pattern faults the app sent as suspected faults. Bottom row: evenly warm panels it did not flag.
The run
Overview
The test that matters most
This is a public set of pictures, put through the app as it is. The plant, the author and the author's organisation are not PanelProof customers, took no part in this and have no connection with PanelProof.
A model can look good on pictures like the ones it learned from and fail on a new camera. These pictures were made by other people, with other equipment, on another continent. The app trained on none of them.
What we did
Uploaded the pictures, and changed nothing for them
- The picture check found that every picture was saved with black corners round a slightly turned panel. It straightened each one and cut the corners away.
- The fault finder scored every panel.
- The review rule set its levels from this batch and sorted it into three piles.
- We then compared the piles with the author's own labels, which the app never saw. We used three of them: normal, evenly warm, and any other fault.
What the app said
The app's piles against the author's labels
| The author's label | Panels | Suspected fault | Needs review | Not flagged |
|---|---|---|---|---|
| A fault with a pattern | 409 | 400 | 6 | 3 |
| Evenly warm | 431 | 27 | 29 | 375 |
| Normal | 2,835 | 198 | 364 | 2,273 |
Of every 100 panels with that label
Right
406 of 409 pattern faults reached a person
400 went straight to the top pile as suspected faults. 3 were not flagged.
Ten of them, as the app saw them.
Wrong
375 of 431 evenly warm panels were not flagged
A panel that is warm all over has no pattern inside it. In a picture of that panel alone, the app has nothing to see. This is the plainest limit of panel pictures, and the app says so on every inspection of this kind.
Ten of the evenly warm panels it did not flag.
It also sent 562 normal panels to a person, 198 of them as suspected faults. That is 70 in every 1,000, where the rule aims at 50.
Ten normal panels it sent as suspected faults.
What it cannot tell
Limits of this run
- These are grey pictures, each stretched to its own full range by their author. The app worked from brightness, not from camera readings.
- The author's eight fault classes have not been matched to the app's fault types, so this page says nothing about naming the type.
- It is one array from one flight. Another site could give other numbers.
Credit
Whose pictures these are
Constance Schaefer, University of Louisiana at Lafayette. "sUAS Infrared Image Dataset for Photovoltaic Module Fault Classification", Zenodo. Licensed under Creative Commons Attribution 4.0 International.
Shown changed: each panel turned upright, its black corners cut away, and resized to 24 by 40 pixels. Full credits and links.
More case studies