The audience hearing the Sermon on the Mount were mostly Jews, those who were obligated to make the pilgrimage to Jerusalem three times each year and make offerings there. Jesus explained why it is dangerous for us to have angry thoughts, or say angry words toward others, and then verse 23 begins with "Therefore."

Therefore if thou bring thy gift to the altar, and there rememberest that thy brother hath ought against thee;

Leave there thy gift before the altar, and go thy way; first be reconciled to thy brother, and then come and offer thy gift.

The LORD wants us to keep short accounts with each other.

Ephesians 4:26-27 Be ye angry, and sin not: let not the sun go down upon your wrath: Neither give place to the devil.

The LORD sees worship and offerings as matters of the heart, not of duty.

If I've caused someone to be angry with me, it is my duty to do whatever I can to free that person from that anger. Their anger at me could be hurting their relationship with God. If I decide it's more important to go to church than be reconciled to my brother, then I've given the devil something to work with; he'll use it against both of us, and eventually the whole church.

I can't say this with any certainty, but because of the context of this passage, and bearing in mind those three mandatory offerings each year, I think the brother who had been wronged was also on his way to the altar in Jerusalem. It is the responsibility of the offender to seek out the offended, and through the healing process of reconciliation, the offended could therefore worship without bitterness, resentment, or anger in his heart.

It's important for us to keep a close relationship with Jesus, but it's hypocrisy if we allow another to continue to be angry because of us. This begs the question; "What if I can't be reconciled to that person?" It's true that the separation by long distances, not knowing where they live, or the fact that they're just not approachable, or even if they've died can prevent reconciliation. I can't speak to that, except to say that the LORD sees both hearts, and I think He makes some allowances for these things. I believe there is forgiveness for repentant hearts. In fact, I know He makes allowances, because in verse 23, He puts that little word "if" in there, and to those of us who can't remember stuff so good, this is a bit of good news! If you bring your gift to the altar, and there remember that someone is angry with you, then it's your responsibility to make it right.

In the Bible, as in life, it's always good to get a second witness, so I'll close with this passage:

And the LORD spake unto Moses, saying,

If a soul sin, and commit a trespass against the LORD, and lie unto his neighbour in that which was delivered him to keep, or in fellowship, or in a thing taken away by violence, or hath deceived his neighbour;

Or have found that which was lost, and lieth concerning it, and sweareth falsely; in any of all these that a man doeth, sinning therein:

Then it shall be, because he hath sinned, and is guilty, that he shall restore that which he took violently away, or the thing which he hath deceitfully gotten, or that which was delivered him to keep, or the lost thing which he found,

Or all that about which he hath sworn falsely; he shall even restore it in the principal, and shall add the fifth part more thereto, and give it unto him to whom it appertaineth, in the day of his trespass offering.

And he shall bring his trespass offering unto the LORD, a ram without blemish out of the flock, with thy estimation, for a trespass offering, unto the priest:

And the priest shall make an atonement for him before the LORD: and it shall be forgiven him for any thing of all that he hath done in trespassing therein.