Case study

One array in Louisiana, from another camera and another team.

3,675 panel pictures the app had never trained on. It caught nearly every fault that shows a pattern, and missed nearly every panel that is evenly warm.

The facts

Pictures
3,675 panel pictures
One drone map of one array
Grey pictures, no camera readings
Marked by their author
2,835 normal
409 with a pattern fault
431 evenly warm
Sent to a person
625 as suspected faults
399 for review
80 as an audit sample

Top row: pattern faults the app sent as suspected faults. Bottom row: evenly warm panels it did not flag.

The run

Overview

The test that matters most

This is a public set of pictures, put through the app as it is. The plant, the author and the author's organisation are not PanelProof customers, took no part in this and have no connection with PanelProof.

A model can look good on pictures like the ones it learned from and fail on a new camera. These pictures were made by other people, with other equipment, on another continent. The app trained on none of them.

What we did

Uploaded the pictures, and changed nothing for them

  • The picture check found that every picture was saved with black corners round a slightly turned panel. It straightened each one and cut the corners away.
  • The fault finder scored every panel.
  • The review rule set its levels from this batch and sorted it into three piles.
  • We then compared the piles with the author's own labels, which the app never saw. We used three of them: normal, evenly warm, and any other fault.

What the app said

The app's piles against the author's labels

The author's labelPanelsSuspected faultNeeds reviewNot flagged
A fault with a pattern40940063
Evenly warm4312729375
Normal2,8351983642,273
Pattern faults sent to a person99.3406 of 409
Evenly warm panels sent to a person13.056 of 431
Normal panels sent to a person19.8562 of 2,835

Of every 100 panels with that label

Right

406 of 409 pattern faults reached a person

400 went straight to the top pile as suspected faults. 3 were not flagged.

Ten of them, as the app saw them.

Wrong

375 of 431 evenly warm panels were not flagged

A panel that is warm all over has no pattern inside it. In a picture of that panel alone, the app has nothing to see. This is the plainest limit of panel pictures, and the app says so on every inspection of this kind.

Ten of the evenly warm panels it did not flag.

It also sent 562 normal panels to a person, 198 of them as suspected faults. That is 70 in every 1,000, where the rule aims at 50.

Ten normal panels it sent as suspected faults.

What it cannot tell

Limits of this run

  • These are grey pictures, each stretched to its own full range by their author. The app worked from brightness, not from camera readings.
  • The author's eight fault classes have not been matched to the app's fault types, so this page says nothing about naming the type.
  • It is one array from one flight. Another site could give other numbers.

Credit

Whose pictures these are

Constance Schaefer, University of Louisiana at Lafayette. "sUAS Infrared Image Dataset for Photovoltaic Module Fault Classification", Zenodo. Licensed under Creative Commons Attribution 4.0 International.

Shown changed: each panel turned upright, its black corners cut away, and resized to 24 by 40 pixels. Full credits and links.