The project was to proceed in three stages. Stage one was conducted in a lab. Stage two was conducted in a lifeless underground cavern. Reliant searched for a lifeless planet for stage three while the scientists put the finishing touches on their research and development. Unfortunately, Reliant got its planets mixed up and was hijacked by Khan.
As with most pieces of miracle technology in fiction, Genesis had a horrible downside. Spock speculated that it would destroy any life in the area of its effect. David confirmed that it could be "perverted into a dreadful weapon". Khan learned of the project from a mind-controlled Pavel Chekov and added stealing the device to his objectives in addition to seeking revenge on Admiral Kirk. The devastating potential of the Genesis project became a subject of great controversy in the quadrant, with the Klingons doing everything from launching protests against the project to trying to steal it for themselves in Star Trek III: The Search for Spock.
Revenge and new birthMutara nebula, destroying the Reliant. The device reorganized the nebula gas and dust to form a new planet, named "Genesis", and the system was subsequently quarantined because of the previously mentioned controversy.
The planet did not exist long, however, as David Marcus used proto-matter in the matrix in order to solve unspecified problems with the mechanism. The use of unstable proto-matter in the matrix resulted in an unstable planet which soon self-destructed. David Marcus was killed by the Klingons trying to steal the secrets of the device, and the knowledge behind the technology was lost as a result.
In the versus debate
Trekkies see this device as an ultimate weapon in the Trek universe. In their eyes it could destroy entire fleets of enemy starships. Some have gone so far as to calculate exorbitant energy figures for the collapse of the nebula into a planet. These calculations are completely backwards, as the collapse of matter into a planet is a net-negative energy event.
Exactly how effective is really a moot point considering that the researchers destroyed all of the data from the project, and most of the team was killed by Khan.