What has Macduff done to upset Macbeth?

In Act 3, Scene 4, after Banquo's Ghost exits, Macbeth asks his wife:

How say'st thou, that Macduff denies his personAt our great bidding?

Macduff has failed to attend this coronation banquet. Macbeth tells his wife:

I hear it by the way, but I will send.There's not a one of them but in his houseI keep a servant fee'd. I will tomorrow,And betimes I will, to the weird sisters.More shall they speak; for...

In Act 3, Scene 4, after Banquo's Ghost exits, Macbeth asks his wife:

How say'st thou, that Macduff denies his person
At our great bidding?

Macduff has failed to attend this coronation banquet. Macbeth tells his wife:

I hear it by the way, but I will send.
There's not a one of them but in his house
I keep a servant fee'd. I will tomorrow,
And betimes I will, to the weird sisters.
More shall they speak; for now I am bent to know,
By the worst means, the worst. 

Then in Act 3, Scene 6 we learn through the conversation between Lennox and another Lord that Macduff has fled to England. But Macbeth does not know that yet. He will be greatly upset when he finds out. Macduff was the one who discovered King Duncan's body while Macbeth was with him. Macduff knows that Macbeth is guilty of that murder, and that is why he refused to attend Macbeth's coronation banquet.

In the opening scene of Act 4, Macbeth has a meeting with the three Witches and the apparitions they raise at his behest. The First Apparition tells him:

Macbeth, Macbeth, Macbeth, beware Macduff,
Beware the Thane of Fife. Dismiss me. Enough.

So Macduff has upset Macbeth by coldly refusing to attend his coronation banquet, also because he has been warned by the First Apparition to beware Macduff, and because he will subsequently find out that Macduff has fled to England and thereby is his enemy. Although the First Apparition warns Macbeth to beware Macduff, the Second Apparition tells him that none of woman born shall harm Macbeth, and the Third Apparition tells him that he can never be vanquished until

Great Birnam Wood to high Dunsinane Hill
Shall come against him.

This encourages Macbeth to believe that Macduff can do nothing to harm him. But then Lennox informs him that Macduff has fled to England. Macbeth reflects that he should have attacked Macduff before he had a chance to get away. He took time out to visit the Three Witches when he should have dealt with Macduff first. 

Macduff's flight to England upsets Macbeth the most, partly because the Thane of Fife is deserting him and setting an example for others to follow, and also because Macbeth is mad at himself for letting Macduff get away. This anger and frustration are responsible for the dire reprisals he orders. His soldiers make a surprise attack on Macduff's castle and slaughter his entire family, along with everybody else inside. This will eventually result in the death duel between Macduff and Macbeth when the English army led by Malcolm and Macduff invade Scotland.