We present a paradigm for effective descriptions of quantum magnets. Typically, a magnet has many classical ground states — configurations of spins (as classical vectors) that have the least energy. The set of all such ground states forms an abstract space. At low energies, the physics of the quantum magnet maps to that of a single particle moving in this space. A particularly interesting example is the XY tetrahedral magnet. Here, the ground state space is a 'non-manifold’ as it self-intersects at singularities. These singularities behave like impurity potentials to create bound states. The low energy physics of the magnet is entirely determined by these bound states. We call this phenomenon 'order by singularity’. This leads to a preference for certain classical ground states over others purely due to topology. Unlike order-by-disorder, this effect persists even in the classical limit.