The article is quite barebones (probably because the military isn’t sharing much info, although I could be wrong), but I’m assuming what happened is that they hooked the decision making up a glorified counting machine, told it to count to the highest number, and then some lunkhead set important things like “kill friendlies” as “-100” instead of “-10,000,000,” or whatever.
If that’s how it happened, it wouldn’t make the AI itself scary, but it sure would make the incompetence of the people running it terrifying.
The article is quite barebones (probably because the military isn’t sharing much info, although I could be wrong), but I’m assuming what happened is that they hooked the decision making up a glorified counting machine, told it to count to the highest number, and then some lunkhead set important things like “kill friendlies” as “-100” instead of “-10,000,000,” or whatever.
If that’s how it happened, it wouldn’t make the AI itself scary, but it sure would make the incompetence of the people running it terrifying.
"We trained it to do something we don't want, then it did something we didn't want!"