Only gemstones containing iron react to magnets; the best examples are some of the garnet family.
Depending on the combination of garnet type in a specific stone, the iron level may vary. For example, since Malaya garnets contain Pyrope and Spessartite, they don’t react as strongly as Rhodolite (a combination of Pyrope and Almandine). We tested several Spessartites, and only some of them had a strong reaction.
Functionally at a mine, magnets can separate dark red spinels from garnets since pyrope garnets contain a lot of iron.