The first time is expected. Tony dies of old age, and he had been getting gradually sicker for some years beforehand. Hearing going, joints creaking, injuries easier to gain and steadily harder to recover from.
Tony kept his mind to the end, and he was grateful for that. As long as he could think, he could fix anything else. He had worked inside single rooms for days, even when he had the option of leaving. So long as he remembered that it was difficult to leave, not impossible, he didn't think back to dark caves. Places where he'd been working, and waiting to die.
Loki visited. That made it okay. On the days where Tony was going completely stir-crazy bearable, because Loki could take them anywhere. No worry about travel, barely a worry about how Tony would even stand up, when it got to the end and even that was difficult.
As long as he had his mind, he could fix almost anything.
Loki found him, when Tony did die. He died peacefully, in his sleep.
Loki knew where Tony was.
Tony had a second life. He didn't know what Loki had done, to manage that, when the body Tony had left was not nearly this young, but he saw a look in Loki's eye every time the conversation started straying toward his death. The look was very close to the one Loki wore when he discussed the Chitauri. Tony let it lie.
Whatever Loki had done, it was not permanent. Tony grew old, again, and died again. His mind did go, this time, in a way that left him aware of what was happening and incapable of stopping it.
Loki visited. Tony stopped talking to him.
Tony was in his prime, when he went to the afterlife. When Loki came back again, he wore the look of someone determined past all reason, but still Tony said, "Loki. Enough." Tony knew, without being told, that whatever Loki was doing would only get worse the more times he did it.
Loki grabbed Tony by the shoulder. "No. Never."
Tony's time lucid was a bit shorter each time. Loki's sanity seemed to go with it.
Eventually, Tony started dealing with it at the first sign of his mind going, because maybe, maybe then Loki would understand that he couldn't keep doing this. Tony told Loki as much, and so did Loki's daughter. The way she said it made Tony wonder what exactly Loki was giving, and if Loki's fading sanity wasn't merely reacting to Tony's, but tied to it. And if not, what was Loki giving?
Tony and Hel asked Loki to reconsider, if only for a moment. His eyes were harder than diamonds, though not nearly so clear.