Your frame of reference is wrong. The action of the liturgy is neither "toward the people" nor "away from the people" because the people are not the focus, God is. The Mass is, essentially, the supreme act of praise and worship of God. It is not, essentially, the gathering of the people of God. Thus, the liturgical action is directed to God. The priest, and all the people, traditionally faced God. This great drawing, courtesy of Christopher's Apologies, sums this up perfectly
But the Mass was instituted by Christ at the Last Supper where He and the apostles would have faced each other across the table. Shouldn't we do the same at Mass?
Seating arrangements were quite different in the ancient world, with the guests seated all together on the same side of the table leaving the other side open for the servers to take and bring the dishes. In all likelyhood Jesus and the apostles were all facing the same direction, just like the traditional arrangement of the Mass.
What is the significance of facing east?
The entire assembled people, including the priest, face east for a few ver significant reasons. First, the sun rises in the east. Since antiquity, Christians have seen the rising sun as a symbol for the Risen Son, Jesus Christ. Facing toward the east, and remember Mass traditionally was said in the morning, means facing the rising sun, the Rising Son. Second, again since antiquity, the Church's tradition tells us Christ will come, at the end of time, from the east, thus we face the direction of the parousia, of the second coming. Thus, ad orientem worship is not dissimilar to Muslims facing Mecca, in that all Catholics turn and face the same direction to worship God.
What of Churches that are not constructed in such a way that the altar faces the east?
Ideally a Catholic church ought to be properly oriented so that the congregation, including the priest, can face east together, but this isn't always the case. This isn't a new problem, either. Nor is it one the Church doesn't have a solution for. When a congregation cannot face directional east, they may face "liturgical east" - that is they can all face the crucifix together.
Are there any negative side-effects when the priest celebrates versus populum?
There can be. Turning the priest around makes it seem like he is the center of the rite. That puts pressure on him to let his personality carry the weight of the Mass. The Mass quickly can become a form of religious entertainment complete with jokes and applause from the "audience." Each priest then impacts the Mass in an undo way. I've heard Mass in the old rite from several different priests and notice no difference expect when he turns around and preaches the sermon. In the new rite, each proest's personality dramatically effect the Mass. This isn't all attributable to the versus populum posture, but such a stance leads to a mentality that tends toward a corrupt understanding of the Mass.
Wouldn't changing the direction the priest faces confuse the people?
Not if it is done correctly and with proper catechesis. A great example of how not to institute this kind of change can be found by looking back at how liturgical changes were implemented following the Council. Changing the direction of the priest with little or no explanation at all caused a great amount of confusion that still hangs over many parishes. An example to emulate is the implementation of the new translation of the Mass. If the priest takes the time to explain why something familiar is changing, Catholics are bright enough to understand.
Hasn't versus populum worship become traditional? It has been half a century!
In a Church that has existed for 20 centuries, a mere 50 years is a fleeting moment. Think of it like this, some Catholics today can still remember what worship was like before the innovations.
Didn't the Second Vatican Council call for this? How can you go against the Council?
This is a common myth. The sixteen documents produced by the Second Vatican Council mention turning the priest around not at all.
Still, isn't it more ecumenical for us to do the same?
The best approach to ecumenism is for both Protestants and Catholics to be honest about what separates us. Papering over the deep differences between what Protestants and Catholics are doing on Sunday mornings by making the Catholic and Protestant worship look similar will not, and indeed has not, ended the sad divisions within Christendom.
This still sounds very pre-Vatican Two. Have any recent popes supported ad orientem worship?
Cardinal Joseph Ratzinger, the future Pope Benedict XVI, wrote, in The Spirit of the Liturgy, "The common turning towards the east was not a "celebration towards the wall"; it did not mean that the priest "had his back to the people": the priest himself was not regarded as so important. For just as the congregation in the synagogue looked together toward Jerusalem, so in the Christian liturgy the congregation looked "toward The Lord."
But surely Pope Francis would never turn his back on the people!
Does all this mean versus populum worship is bad (or at least worse than ad orientem)?
Not at all. Both are valid ways of celebrating the Mass and both have advantages and disadvantages compared to the other style. The focus here has been on ad orientem because it is the more maligned or misunderstood posture. Whichever your parish uses or you prefer, remember the maxim quoted by Pope St. John XXIII in Ad Petri Cathedram (frequently ascribed to St. Augustine),
In necessariis, unitas, in dubiis libertas, in omnibus caritas
Unity in necessary things; liberty in doubtful things; charity in all things.