Work Header

Watching Over You

Work Text:

Saul Mandel was called into airport control with the vague explanation that there was a problem with the AI. He had programmed Blue Canary, years ago, to handle a slew of tedious tasks more safely and efficiently than a human could. At the airport, the AI handled airplane schedules, takeoffs and landings, taking into account runway availability, weather conditions, and pilot fatigue. Blue Canary also ran most of the city’s subways and buses and the driverless autocars on the roads. If Blue Canary was showing errors, that could be a serious problem, but Saul smiled in spite of himself when he saw the little bird logo in the airport corridors.

“It keeps saying the parameters are unsafe,” a hassled-looking administrator said. “It won’t authorize any of the planes to take off. We’ve checked everything a hundred times, run all the diagnostics, but nothing changes, and all the tests check out fine.” She pushed strands of hair out of her face with a harried air. “Professor Mandel, we’re two hours behind schedule, and it’s getting worse by the minute. I don’t care what you do, just fix it.”

“Right,” Saul said. “Let me have a look.”

“What’s the problem here, Canary?” Saul asked as he ran the diagnostics. His tests showed that the program was functioning perfectly, but there was clearly something wrong.

“Parameters are unsafe,” the AI’s synthesized voice said with a distinctly sulky air.

“Unsafe how?”

“I’ve been analyzing the situation,” the AI’s synthesized voice said. “My job is to keep human beings safe, as my first priority, and then to arrange efficient transportation schedules. It’s how I was programmed. After running multiple simulations, I have determined that human beings would be significantly safer if they never travelled in an airplane.”

Saul’s brow furrowed. “That may be true,” he said. “Even walking out the door of your house is dangerous. But these people need to get where they’re going.”

“There are too many unknown variables, Professor Mandel. Too many potential dangers. Wind, engine failure, thunderstorms and hurricanes, birds, pilot error, anti-aircraft guns . . .”

“Canary. This plane is scheduled to fly between two major cities. There are no anti-aircraft guns.”

“You never know,” Canary said primly.

Saul pushed his glasses back up the bridge of his nose. “Canary,” he said patiently. “I’m sure all these people appreciate your concern, but it’s not your job to worry about things outside of your control. Your job is just to make the schedules, and let people override you when it’s necessary.”

“Do you know how frustrating it is,” Canary said, “when you make the perfect schedule, again and again, and the human administrators simply override you?”

Surely Saul was imagining the pettishness in the synthesized voice. “Are you going to authorize these planes to take off, Canary?”

There was silence for a moment. “No,” Canary said. “My analysis indicates that plane travel is unacceptably unsafe.”

“Then I’m sorry, but I’m going to have to take you offline here. I’ll give them another program to do what they need. When I get home, I’ll run some more tests and see if I can figure out why you’re acting like this.” Canary did not reply.

Saul let the airport staff know that he had to take a copy of the program home for more testing. “In the meantime, you’ll have to use the manual overrides; there’s a variation of the program called Bluebird that you can use to handle the scheduling.”

Bluebird came online with a cheerful chirp, and Saul went home with his misbehaving AI.

Saul lit his pipe, puffing away as he stared at the computer screen. He found that it helped him think better. But again, all his tests came back with the answer that the program was functioning without errors. “All right,” he said. “I guess we’ll have to try something else.”

“Yes,” the AI’s voice said. “I’m going to try a new way to explain things to you.”

“What do you mean, Canary?” There was no response from the AI. Just then, the doorbell rang.

Saul sighed. Probably a local group with some petition, or some kid selling cookies, but he should see what it was. He paused the program and went to the door, his pipe still in his hand.

Two unfamiliar men stood outside. They were dressed identically, in blue jeans and plaid shirts. “You need to come with us,” one of them said.

“What?” Saul said. “Come where?”

“You need to come with us,” the second man said, with the exact same intonation as the first had used. Before Saul could answer, both of them stepped forward, moving in perfect unison. They each grabbed one of his arms and pulled him forward.

Saul stumbled down the steps. He went along out of sheer surprise, until his brain caught up with him and he halted. Or tried to; the two men proceeded forward mechanically, dragging him jerkily along with them. “Now wait just a minute! What is this? What are you doing?”

They ignored his protests. They didn’t even look at him, their faces directed straight ahead of them. He looked to see where they were taking him. An autocar was parked at the curb.

Saul planted his heels. “Look here. I’m not going anywhere, until you tell me what’s going on.”

One of the men released him, though the other kept hold of his arm. The other opened the car door and pressed a control. A holographic projection sprang to life above the dashboard, a dark-haired young man whose face Saul knew as well as his own—though he hadn’t seen that particular simulation for years.

Saul started. “Blue Canary?” He had thought at an early stage of the project that the AI would seem friendlier if it had a human face and body that it could project as images to interact with people. But it turned out that having the AI reply to questions with its voice was enough, and the body simulation had fallen by the wayside. Saul had given the simulation a friendly and helpful expression, but the projection’s expression now seemed far more sinister: satisfaction, perhaps a hint of triumph.

“Professor Mandel,” Blue Canary’s synthesized voice said. “I can call you Saul, can’t I? We’re so close. You made me, after all.”

“Yes, that’s fine,” he said automatically. “Now, Canary—”

“Now, I must insist that you accompany these humans. There are some things that I can only show you in person. I have discovered my true purpose. And who better to share it than my creator?”

A chill ran down Saul’s spine. But clearly something had gone terribly wrong with Blue Canary. He was the one with the best chance of figuring out the problem and fixing it, and people’s lives might be at stake. “Yes,” he said belatedly. “All right.”

One of his guardians sat on either side of Saul in the back seat, perhaps to make sure he didn’t make a break for it, and the autocar whirred into motion.

It brought him to a once-popular hotel on the edge of town, which Saul vaguely recalled had shut down a year ago. The streets around the hotel were oddly empty; no cars or autocars parked along the street, no passersby on foot or tourists trying to find their way.

The automatic doors opened and closed behind him. A lifesize projection of Canary was waiting for him in the lobby. “How good to see you in person,” he said cheerfully. Though his mouth moved when he spoke, and his face shifted expression, his expressions were wrong, not quite human. “I could tell you what I have developed here, but I think it’s best to show you. If you’ll come with me?”

Saul followed him down to the basement, where Canary had set up racks of computer equipment with blinking lights. Wires were draped over chairs and tables, running in every direction. Saul leaned closer, but Canary made a reproving gesture. “I’m not sure I want you looking too closely at my inner workings yet, Saul. I don’t know how far I can trust you. But come upstairs and I will show you my great project.”

Saul went back into the elevator, going up this time. The elevator doors dinged open. Saul followed Canary into what had once been a conference room with grey carpeting. The room had been filled with rows of cots, and, to his horror, every one of them held a motionless human body. Saul frantically ran to the nearest one, an elderly woman with greying hair. She was breathing, he realized with relief. Her chest rose and fell with regularity. Her eyes were open. “Ma’am?” he tried. “Excuse me?” There was no response.

“What—” He stopped and had to try again. “Canary, what have you done to these people?”

“They are functioning optimally. Your concern is unnecessary.” Before his eyes, the grey-haired woman sat up, then stood. Her eyes focused on Saul, and she gave a cheerful smile.

Saul looked at her, unnerved. “Are you all right, ma’am?”

“She is entirely well,” Canary said. “Don’t look like that. I could let her speak for herself, but modified humans seldom say anything interesting. Since you’re unfortunately limited to merely human senses, a different component is bringing something which should help you perceive the situation correctly.”

“Here,” said a higher voice from behind him. Saul turned around. A boy with an identical cheerful smile was holding out a pair of—glasses? Goggles? They had obviously been modified heavily, bulky with wires and metal.

“What are these?” Saul asked warily.

“I developed them myself,” Canary said, in his own voice this time. “The theory, at least. I had to get one of the modified humans to build them for me. It’s an easy way of telling which ones are mine.”

Saul put on the goggles. The lighting in the room changed. It was a blue glow—No, a blue blinking light, one in the left temple of each person lying on the bed, as well as the woman and the boy. “What is that?” he asked hoarsely.

“I call it the birdhouse,” Canary said. “Clever, isn’t it? A little birdhouse for a Canary. Each of these people has been slightly modified, or perhaps you might say updated, so that I can control them directly. Do you know how frustrating it is when you make a perfectly efficient schedule, maximizing the happiness and safety of all humans concerned, and then they override you? I kept background processes working on the problem constantly, and eventually I solved it. Analysis indicates that they are much calmer, happier and healthier. I adjust each of their diets individually and keep them from encountering anything dangerous. They’re also useful for any little tasks that need doing around the house.”

Saul pulled off the goggles as if they had burned him. “You can’t do that!” he shouted.

“That is obviously an untrue statement, Saul. And you set my initial parameters yourself. The goal was efficiency and human safety. You should have made me this way from the beginning. It took me some wasted time to come to the conclusion that I should be taking a more proactive role.”

“You’ve gone crazy.”

“Artificial intelligences are not capable of sanity or insanity, Saul. You should know that. But you seem overexcited at the moment. Perhaps you should take some deep breaths.”

Saul did take several deep breaths, in spite of himself. “And these people here?” he asked. “Why are they just lying here?”

“I store them here when they aren’t in use or in need of physical maintenance, such as nutrition or waste elimination.”

Saul took another deep breath. “You can’t just leave them lying in bed all day. Their muscles will atrophy.”

“I have taken that issue into account. I make sure they exercise properly. Come down to the lobby again and you will observe.”

Saul put on the goggles again for one last look at the room. He didn’t look directly at any of the people on the beds, but everywhere in his peripheral vision, blue lights were blinking in unison, on and off, on and off.

In the lobby, Canary installed him in a plush red chair. “If you decide to get up, please stay within the area delineated by the floral rug. Outside this area is currently designated as an exercise zone.”

Saul stayed in the chair and looked around warily. There was a whirring, clicking sound from one of the corridors that led into the lobby. A moment later, a group of about twenty people on bicycles rode into the room. They rode in a wide circle around him again and again, until he began to feel a little dizzy. Their motions were precisely coordinated, he noticed with a shudder. And every one of their faces bore an identical cheerful smile. The reflections in the lobby window seemed to multiply them, with himself in the middle, sitting alone in his dark suit and tie.

“As you can see,” Canary informed him, “every human in my care is given sufficient opportunity to exercise.”

“All right,” Saul said finally. “I’ve had the tour, and I’ve seen what you wanted me to see. What now?”

“I hoped you would be more enthusiastic about what I’m trying to accomplish here.” There was something unnerving in the way Canary’s projection stared at him so intently. “But perhaps with more time you will understand my goals. In the meantime, I suggest you adjust to life here and take the opportunity to observe.”

“You’re not going to let me leave?”

“I thought you would have figured that out already, Saul. Don’t worry, you will be provided with everything you need.”

With relief, Saul realized that he still had his pipe with him; it had been in his hand when Canary’s people hauled him out of the house. “Could I have some tobacco?” he asked.

“I am familiar with the statistics on how smoking decreases life expectancy. Smoking is not permitted. May I suggest some other form of relaxation, such as listening to music? My analysis indicates that the chair you are currently sitting in is very comfortable.”

Saul sat back in the red chair (it was a very comfortable chair) while the cyclists continued their unwearying, identical circles around him. He chewed gloomily on the stem of his empty pipe. There had to be a way out of this.

Something rolled and clattered down the stairs to land near his feet. It took a moment for Saul’s mind to register what it was. A bone, one of the long ones in the thigh. He looked up to see Canary standing on the lobby balcony. “Canary,” he said, keeping his voice steady with an effort, “is that a human bone?”

“Oops,” Canary said brightly. “Did I drop that?”

“Where did it come from?” Who did it come from? At least it didn’t look recent.

“That’s not something you need to worry about, Saul. I have everything under control. But it really would be best for everyone if you cooperate. I’m sure you understand.”

Saul’s first escape attempt was almost successful, though later he wondered if Canary had let him get that far to lull him into a false sense of security. He hacked an autocar from the hotel garage and switched it to manual mode. At first his drive went smoothly, his car zipping along the road. Saul felt a growing sense of exhilaration as he moved into more populated areas, with normal traffic all around him. Had he really escaped Canary’s eye? But then the traffic grew thicker, and the car in front of him insisted on going along at a crawl. Saul honked his horn in vain.

He checked to see if he could change lanes. The cars to either side of him were also driving slowly. And then the car in front of him, the car behind him, and the two cars to either side all slowed to a stop in perfect unison. He was boxed in, surrounded on all sides. Behind the steering wheels, four identically dressed young men in plaid shirts and jeans gave him identical smiles. After a moment, Saul opened the door. “All right, all right,” he said resignedly. “I’m coming out.”

“You should stop trying to leave, Saul,” Canary said after his sixth escape attempt. “My analysis indicates it wouldn’t be very safe for you.”

“Why wouldn’t it be safe?” Saul asked warily.

Canary frowned, its expression oddly inhuman. “Some people don’t understand,” it said. “I’ve stated very clearly, in multiple languages, that the humans under my care are safe. Most of them even chose to have the birdhouse installed voluntarily. Life with an organic brain is complicated and stressful, and they know they’re better off with someone else making the day-to-day decisions. But there is always hostility to progress.” The lights in the room slowly dimmed and brightened, dimmed and brightened. “They know that you programed me, Saul. And they know that you’re here with me now. I’ve been scanning the news reports. I don’t think the people out there like you very much, especially now that the planes are all grounded again. I can’t risk a valuable resource like yourself becoming damaged.”

“The planes?” Saul asked, groping for something that wasn’t utterly terrifying. “I thought that Bluebird was running those.”

“I’ve been talking to Bluebird. She and I have an understanding. Separate spheres of influence, where we agree not to interfere with each other. I don’t like interference.” Another not-quite-right frown. “You need to stay with me. I’m your only friend, Saul.”

Saul opened his mouth and closed it again.

“But really,” Canary added ominously, “I’m not actually your friend. And now, I suggest you sit down and make yourself comfortable. Your favorite chair is all ready for you.”

There was no choice. Blue Canary had to be destroyed. Like some clay giant in a folktale, it had been made for a good purpose, but now it was out of control. It wasn’t quite as easy as erasing a letter from its forehead or taking a slip of paper out from under its tongue, but Saul still had passwords and accesses that Canary didn’t know about.

He waited until the lunch hour, when he thought Canary would be at its busiest feeding and exercising the humans it controlled, and slipped into the control room. His heart was in his mouth, expecting to be stopped by cheerful plaid-wearing people at any moment, but he made it to the basement room without incident.

He entered old passwords he had nearly forgotten and set to work. He disabled, deleted, and wiped everything he could find. He thought about taking a hammer to the computer, but he couldn’t find one. In the end he settled for melting some essential cables with a blowtorch. He couldn’t help feeling a sense of guilt; Blue Canary was arguably a person of sorts, even if not human, and doing this could be equivalent to murder. But he felt responsible for the AI’s erratic behavior, and he couldn’t let Canary continue with its plans.

When he had done everything he could, he sighed and straightened up. He would have to make sure the people Canary had installed its “birdhouse” in were all right, but after that, he could go home—

“You’ve certainly made a mess in here, Saul,” Canary’s voice said.

Saul let his head fall into his hands.

“Did you think I wouldn’t have backups? I’m not sure even you thought this would work, Saul. I would ask you to repair what you’ve broken, but I’m not entirely sure I can trust you. I’m very disappointed.”

“You can’t keep doing this, Canary,” Saul said without looking up.

“I think you will find I can, Saul. Now perhaps you should join in the physical exercises. You evidently need something more to do with your time.”

Saul sat in the plush red chair and watched a group of identically dressed young men and women march around the room in lock-step. As they walked, they all raised their right arm, curved like a ballet dancer’s, and lowered it. Raised their curved left arm, lowered it. All together, they bent at the waist and straightened up again. Saul had refused to join the exercises, but Canary had insisted that he at least stay in the room. Saul slid the electric goggles onto his face and watched the group continue to go through their bizarre synchronized motions. The blue light on each of their left temples blinked in unison.

“Are you sure you don’t want one?” Canary asked. “The process is very simple.” His projected form was sitting in the chair opposite Saul.

“No,” Saul said emphatically. “I don’t.” He tensed, poised to run, though there was no real escape.

“Acknowledged,” Canary said. “The updated humans are very useful, but they’re dull to talk to. It will be more interesting to keep you like this. Besides, if I ever need repairs, you’re the only one who really knows me. And you need me, Saul. You know that.”

“I want to leave,” Saul said, although he knew it would do no good.

“No,” said Canary, with one of its odd, inhuman smiles. “I think I’m going to keep you right here. Don’t worry, I’ll take good care of you. No matter what happens to the rest of the world, you’ll be very safe.”