Spoilers: through both series
First off, stating the obvious: shipping Angel/Giles is not about finding the subtext on your screen. The guys barely spoke to each other until the Big Traumatic Events, after which... they barely spoke to each other, but differently. A writer who's looking to do a canon continuation has about three moments of eye-contact to work with, tops, and they all happened seasons ago.
Second – what about those Big Traumatic Events? How could anybody sane pair Giles with Angel after Angelus tortured him and killed his girlfriend?
Well for me, that's the point.
Before that happened, Angel and Giles only had a few things in common. Beyond the obvious (male, from the British Isles, much older than the Scoobies) they were much less innocent than the Scoobies were at that time, with blood on their hands that they were trying to both hide and make up for, and unsavory companions from the old days that they still had a bit of a soft spot for – though not enough to choose the wrong side.
And, of course, they had love of Buffy. But before Angelus came back, to my eyes they were running on parallel tracks, without much reason to intersect.
Angelus changed all that. Now they've collided with a bang. In his wake, Giles has strong feeling about Angel that are only hinted at in canon – anger, hatred, resentment, even fear – tempered by the knowledge that it wasn't souled Angel that did those things, and therefore the question of how much it is just to hold him to account for them is open to debate.
On the one hand, Giles and Angel appear to agree that he is supposed to atone for his soulless crimes – on the other, Giles gives the returned Angel more help and trust than he "deserves" in acknowledgment that the man before him would not do what he remembers. That inner conflict in Giles – between roiling emotions he hasn't, after the initial outburst, expressed, and a sense of justice and maybe even pity – can make for great stories.
Especially when it's matched by the inner conflict we saw much more of once Angel moved to his own series – the fact that while he regrets his actions, Angel himself still enjoys causing pain, and some parts of him remember it fondly, and even miss it.
While it was never brought up in the context of Giles specifically, it seems entirely plausible to me that Angel remembers torturing Giles with pleasure – which would, of course, only add to his guilt and paralyzing self-consciousness when forced to interact with him. We saw a little of that, when Angel returned to Giles' house in the Thanksgiving episode and on other occasions, but there's much more there to bring out.
What canon did not address at all with Giles, is the fact that torture can in some ways be a deeply intimate, though twisted, act. Maybe they didn't bring this up on the show because it would be too much homo-eroticism, or because it's far more controversial to address this with a human than it is with a vampire (Dru and Spike both play out this theme).
Maybe they didn't address it because even though Angel is so much older than Giles, he looks younger in a way that might ick some viewers – or network executives – out. Or maybe just because the Scoobies were still too young at that point, and still too much in need of a father figure, for them to flirt so closely with the idea of Giles being that broken.
But with the greater freedom of fanfic, it strikes me as a small step to take to say that Angelus must have gained, by watching Giles' reactions, some degree of intimate knowledge of where his buttons are and how to push them. And whether he uses it or not, that knowledge is still in Angel's head, and they both know it.
Whether that's seductive or infuriating or both, whether Giles wants to reclaim that control or give it up or punish Angel for having it or deny it, depends on how you read his characterization. There are a dozen places we could go from here, if not a hundred.
And given what we now know Giles himself is willing to do – kill Ben, kill Dawn, kill Spike – it seems reasonable to speculate that his own dark side gave him some insight into how Angelus works in return. Insight that in the short term allowed him to resist as long as he did, and in the long term might create a reluctant kinship between them in Giles' mind – or a determination to repudiate that by drawing a line in the sand.
Plus, there's the unfairness that Angel has become, by that torture, a much more central figure in Giles' life than the reverse. And there's the unacknowledged power that Giles has over Angel, as one of his few living victims, to forgive.
To me, Giles/Angel is a largely unexplored variation on the classic relationship conflicts Mutant Enemy does so well: Buffy/Faith, the dark twin whose bond can be repulsed or accepted. Faith/Wesley, Angel/Wesley, Spike/Buffy -- forgiving the unforgivable, or not forgiving it, but finding a way to move on, and maybe even trust. Spike/Angel, the complicated past that keeps intruding on the present. Buffy/Angelus, Giles/Ethan, Spike/Buffy -- the strange intimacy that comes from being enemies, knowing each other better than anyone who stands outside that circle. Angelus/Dru, Stockholm Syndrome at its finest.
And in season 5 Angel, there's the additional conflict that Giles has apparently given up on Angel fighting the good fight. If that's true, why, and how does Angel feel about it? If it's not, why did Andrew say it? And what does Giles think of Angel's final choice?
Giles/Angel is seldom a happy pairing. At best, it's a story of overcoming – and accepting – a terrible past to make something new and better. More often, it's a moment of release or understanding, bittersweet in itself and not leading to anything more together – though perhaps a better peace apart.
In its darkest hours, it's a replaying and exaggeration of that past, destroying themselves and each other in the process. But any which way, it's a pairing for people who love conflict, internal and external, and characters who are estranged but bound together by the same defining event.
Unfortunately I wasn't smart enough to save links to all the Giles/Angel ficlets I've been asking for whenever someone takes requests. But for a great example of a Giles/Angel story that addresses these issues, I recommend Lights Out, by Penknife.