One thing that I have learned in my extensive youtubing is that sometimes the hot end can soften the previous layers. Though, take the following advice with a grain of salt as the following is just conceptual for me (I'm still waiting for my first printer to arrive to follow this advice).
Tomb of 3D Printed Horrors has a pretty good video on printing D&D miniatures that require high detail at small sizes (particularly at the 2 minute mark). His recommendation is to have 0.2 mm thick lines as thinner lines are more susceptible melting when the hot end is laying down subsequent layers. Thicker layers means more plastic to absorb the heat and less deformity. This is probably why your 0.1 mm lines look better than your 0.05 mm lines.
Another fix is to keep print speeds somewhat normal or slightly slow. A faster print head heats the previous layers less and can counteract the issue. Running at 60 mm/s may be too quick for quality prints, but running at 10 mm/s is too slow and can reheat the lower layers. Keeping an "average" pace paired with thick enough lines could help you get the prints you want. Though, finding what speeds/thickness is up to you. I don't have the experience to suggest any at the moment.