"the car's hazard lights made the system think the parked car was a road sign"
Often that's just a guess by engineers too. One of the problems we had with Watson is a flaw shared by other neural-networks in general: a lack of self-reflection. They are pretty much black box systems that are almost impossible to interrogate. In these settings it's pretty critical to know why something happened or why an expert system answered a question the way it did. You can do an investigation, probe the i/o logs, and try to reproduce the scenario, but you can't just ask "why did you do that?" like you can with a human.
True, but even if you could interrogate it and understand the answer, it's effectively an alien intelligence whose decision-making process is totally foreign to our own.
"Why did you interpret the hazard lights as a road sign?"
"The hue of the pixel at X=753, Y=1063 had a .0006 higher correlation to training data associated with a road sign hazard light than that of a vehicle hazard light. Therefore it was classified as a road sign hazard light"
When humans do this it's a lot easier for us to understand the reasoning behind the defective thought process and develop some sort of higher-level organization or process to either make that failure mode less likely to occur or limit its scope. But even then it's extremely hard: "safety regulations are written in blood" as they say.
Often that's just a guess by engineers too. One of the problems we had with Watson is a flaw shared by other neural-networks in general: a lack of self-reflection. They are pretty much black box systems that are almost impossible to interrogate. In these settings it's pretty critical to know why something happened or why an expert system answered a question the way it did. You can do an investigation, probe the i/o logs, and try to reproduce the scenario, but you can't just ask "why did you do that?" like you can with a human.
True, but even if you could interrogate it and understand the answer, it's effectively an alien intelligence whose decision-making process is totally foreign to our own.
"Why did you interpret the hazard lights as a road sign?"
"The hue of the pixel at X=753, Y=1063 had a .0006 higher correlation to training data associated with a road sign hazard light than that of a vehicle hazard light. Therefore it was classified as a road sign hazard light"
When humans do this it's a lot easier for us to understand the reasoning behind the defective thought process and develop some sort of higher-level organization or process to either make that failure mode less likely to occur or limit its scope. But even then it's extremely hard: "safety regulations are written in blood" as they say.