An away team is a Federation landing party. It consists of a handful of men, usually some of the ship's senior officers, security personnel, and a few specialists. Larger away teams can be assembled from ship rosters if necessary.
Depiction in Canon
Once the Captain of a starship has deemed an away team necessary, the Captain or ranking officer selects members of an away team, usually including himself.
Away teams generally use the transporter to travel. If using the transporter is not feasible for some reason, away teams can deploy by shuttlecraft. In general, there is no dedicated away team on a Federation starship; away team members are picked at the Captain's whim.
Federation away teams generally arrive armed with the Type-II hand phaser. Away team members may have concealed weapons in the form of the Type-I phaser in their uniform pockets. In addition, away team members often bring tricorders.
If a medical officer is present, the away team may have a medical kit and a medical tricorder. Other specialists are also likely to have specialized equipment -- an engineer, for example, would have repair tools and support items such as portable power units.
In Star Trek III: The Search for Spock, Klingons drew their weapons mid-transport, so they were ready to fire as soon as they materialized. Federation away teams generally do not do this, and only draw their weapons when provoked.
Away teams are generally accompanied by a security officer, either the head of security or redshirts. It is a violation of Starfleet regulations to allow a senior officer to disembark without an armed escort.
If an away team encounters hostility, the general practice is to retreat. This is typically the most intelligent decision, since away team weapons are comparable to those of a modern police officer, and in most cases they would be insufficient against a large group of heavily armed locals. An orbiting starship may maintain a transporter lock on its away team, enabling it to beam the team away at the first hint of trouble. Any away mission involves risk, however, due to the many phenomena that can interfere with transporter function.
Large-scale away teams formed the backbone of Federation ground forces in the Dominion War. Away teams were never intended to hold territory against enemy soldiers, but the Federation's lack of an army forced Starfleet to arm away teams with Type-III phasers and press them into infantry roles. Admiral Leyton, in DS9 "Paradise Lost", attempted to institute martial law on Earth with away teams.
Age of Sail Analogy
In military history, the ranking officer would often accompany a landing party when going ashore. During the Age of Sail, the Captain of a vessel was the law. Moreover, ancient militaries led by example, and leaders often had to demonstrate their heroism to their men. This continued even to the the 20th Century, where Captains would leave the safety of their vessels to negotiate, such as when the Germans in WWII attempted to convince French Captains to surrender their vessels at Toulon.
In TOS, the Captain and his ship were analogies to the Age of Sail. The Captain was the law, and Captain Merrik in "Bread and Circuses" remarked on the special nature of a starship and its crew. Captain Tracy complimented Kirk on the high degree of training of his bridge crew in "The Omega Glory." Most likely Captain Kirk was one of a special breed of commanders, cultivated specifically for dangerous away missions.
Star Trek, and especially more modern Trek, presents problems 21st-century Captains never had to face. Firstly, a Federation starship's mission to "Boldly go where no man has gone before" -- as stated by the Enterprise's dedication plaque -- places Captains in unknown, potentially lethal situations. Federation away teams often land in hostile conditions. New civilizations or natives may not adhere to the same "code of honor" that military forces aspire to, and many have not signed treaties with the Federation.
The Captain in modern Trek possesses command codes that allow complete control of his vessel, even remotely.
The Prime Directive
Modern Trek's Captains have vastly different interpretations of the Prime Directive. In Kirk's era, the prime directive was a guideline, and Kirk often violated it when necessary to accomplish his mission. In TNG and beyond, Captains have a much more rigid view of the Prime Directive, refusing to assist pre-warp civilizations even if their population is suffering immensely.