E3D have a blog post on the subject of hardened nozzles. The common abrasive materials are carbon fibre, wood impregnated filament (or any other particulate fill), and some pigments. The blog describes that as little as 250g of the more abrasive filaments can wreck a brass nozzle.
For wear resistance, different manufacturers will have different options. Stainless steel is a little harder than brass, and hardened steel is a big step forward. If you fancy spending more money, aluminium oxide (i.e. Ruby) is an option.
The downside is printability and cost. Specifically thermal conductivity of brass (or copper) is very good when compared to steel. However, this may not be the limiting factor for your printing unless you're running a very large machine. The various materials may also give different friction/wetting performance which can affect jams.
Depending on the printer and the type of hardened nozzle, it probably makes sense to avoid swapping between hardened and standard nozzles most of the time.
Hardened nozzles cost between 3 and 20 times what you will pay for a branded brass nozzle (and arguably a bag of no-name brass nozzles can be treated as near enough free disposables in comparison).