Self-destruct refers to a mechanism by which a vehicle or a guided missile can destroy itself. For a ship or other vehicle, this is done to prevent capture; for a weapon, it is done to abort an attack after the weapon has been launched.
Modern warships typically don't have a built-in mechanism for self-destruction; the crew must manually plant explosive charges in critical areas of the ship and detonate them to scuttle the ship. In WW2, crippled warships were sometimes torpedoed by friendly ships to prevent their capture. A dedicated self-destruct system would actually represent a security flaw, since it would require only a few traitorous crew members to destroy a warship.
Modern weapons may have self-destruct features to prematurely detonate the warhead upon receipt of a special command code.
Self-destruct in Star Trek
Federation starships have the ability to self-destruct via a computer command. In the 23rd century, executing the command required command codes from at least three officers of the ship, including the Captain. In the 24th century, the requirement was reduced to the consent of both the Captain and First Officer, identified only by biometric recognition. The self-destruct system of the USS Voyager can be activated by the Captain without confirmation from another officer.
The self-destruct command includes a countdown timer to allow the crew time to abandon the ship; in the 24th century, the countdown time is variable.
The exact mechanism used to destroy the ship isn't specified, but deliberately dropping the force fields containing the ship's antimatter fuel seems to be a likely method. On the USS Voyager, if the secondary command processors are damaged, the self-destruct system will not function.
Federation hand phasers have a self-destruct feature that will cause the weapon to explode with enough energy to significantly damage a starship if it detonates inside; why they have this feature is a mystery.