The machine is an automated warship designed to destroy planets and then consume the debris to refuel itself, presumably converting the mass directly into energy by unknown means. It would also react to starships that approached too close, attacking and disabling them before resuming its primary mission.
According to Spock, the hull of the vessel was composed of "pure neutronium", making it opaque to Federation sensors and essentially invulnerable to Federation weapons. The Constitution-class starships USS Constellation and USS Enterprise both attempted to destroy it with phasers, with no effect.
The machine attacked with a beam weapon that Commodore Decker of the Constellation described as "pure antiproton", although its visible effects were inconsistent with a beam composed entirely of negatively charged antimatter particles. Shots from this beam defeated the shields of the Enterprise with just a few shots, but it did not instantly annihilate the ship even after the shields failed. This same beam was apparently capable of blasting large chunks out of planets.
The machine had a tractor beam which was capable of overcoming the propulsion systems of the Enterprise (although the Enterprise had sustained some damage by that time). The primary purpose of the tractor beam was presumably to bring pieces blasted out of planets into the machine's "maw" for consumption.
The machine generated subspace interference that prevented long-range communications in its vicinity.
The machine was finally destroyed by flying the crippled USS Constellation into its "maw" and detonating the impulse engines. The resulting 98-megaton explosion caused internal damage that destroyed the vessel.
It is interesting that the Enterprise made no attempt to launch photon torpedos into the mouth of the machine after learning that its internal mechanisms could be damaged that way. If torpedos have the multi-megaton yields often claimed by Trekkies, a few torpedos into the machine's mouth should have been sufficient to disable it.
One possible explanation relates to how the machine was able to somehow "deactivate" the antimatter on the USS Constellation. If attacks from the machine similarly affected the USS Enterprise, the torpedo warheads might have been rendered inert. The episode provides no confirmation of this hypothesis, however.