How can I analyze the three-way relationship between Hamlet, Gertrude, and Claudius?

This is a complex relationship. Claudius sums it up best at the beginning of the play.

Therefore our sometime sister, now our queen,  Th' imperial jointress to this warlike state,  Have we, as 'twere with a defeated joy,  With an auspicious, and a dropping eye,  With mirth in funeral, and with dirge in marriage,  In equal scale weighing delight and dole,  Taken to wife; (I.ii.208-213).

He explains to the reader (or audience) that Gertrude was once...

This is a complex relationship. Claudius sums it up best at the beginning of the play.

Therefore our sometime sister, now our queen, 
Th' imperial jointress to this warlike state, 
Have we, as 'twere with a defeated joy, 
With an auspicious, and a dropping eye, 
With mirth in funeral, and with dirge in marriage, 
In equal scale weighing delight and dole, 
Taken to wife; (I.ii.208-213).

He explains to the reader (or audience) that Gertrude was once his sister-in-law, married to his brother, but is now his wife. This makes Claudius not only Hamlet’s uncle but also his stepfather. Additionally, it creates strain in the mother and son relationship since Hamlet can’t understand how his mother could love his father so dearly yet marry his father’s brother so soon after the funeral.

Though the play begins after King Hamlet is dead, it is this exposition that shows us the complexity of Hamlet’s situation not only in his home at Elsinore but in the kingdom as well. Denmark is at odds with Norway. Young Fortinbras wants to attack and avenge his father, and now Denmark is ripe for the picking with a changeover in the throne. Everything about this set up is precarious, and when we soon after learn that the new king is the one who murdered the old, it’s easy to understand why Hamlet’s relationship with his mother becomes even more strained. Either she was in on the plot to murder King Hamlet or she wasn’t, but either way, she is married to her husband’s killer and places her trust in her husband up until the moment she drinks the poisoned wine. And when Claudius believes Hamlet has uncovered his secrets, he aims to get rid of Hamlet as well, which is what catalyzes the deaths of Gertrude, Laertes, Hamlet, and Claudius himself in the end.

The relationship between Hamlet, Claudius, and Gertrude is volatile from the beginning. It is this set-up that sets the rest of the play in motion until it comes to a tragic end.