True, but even if you could interrogate it and understand the answer, it's effectively an alien intelligence whose decision-making process is totally foreign to our own.
"Why did you interpret the hazard lights as a road sign?"
"The hue of the pixel at X=753, Y=1063 had a .0006 higher correlation to training data associated with a road sign hazard light than that of a vehicle hazard light. Therefore it was classified as a road sign hazard light"
When humans do this it's a lot easier for us to understand the reasoning behind the defective thought process and develop some sort of higher-level organization or process to either make that failure mode less likely to occur or limit its scope. But even then it's extremely hard: "safety regulations are written in blood" as they say.
True, but even if you could interrogate it and understand the answer, it's effectively an alien intelligence whose decision-making process is totally foreign to our own.
"Why did you interpret the hazard lights as a road sign?"
"The hue of the pixel at X=753, Y=1063 had a .0006 higher correlation to training data associated with a road sign hazard light than that of a vehicle hazard light. Therefore it was classified as a road sign hazard light"
When humans do this it's a lot easier for us to understand the reasoning behind the defective thought process and develop some sort of higher-level organization or process to either make that failure mode less likely to occur or limit its scope. But even then it's extremely hard: "safety regulations are written in blood" as they say.