If I'm understanding correctly, your process is using a single stainless scoop to scoop from multiple different non-allergen ingredients? Scoops were super common in my spice/dried veg plant as we established blends. I can see this both ways: relatively harmless if the ingredients are all dry and not super dissimilar, and generally known to be safe from micro issues.
But the auditor had a point that the cross-contamination factor isn't zero. If you were to receive one bad batch of an ingredient, scooped it first, then used that scoop on another bin or case of another ingredient, you've introduced contamination into your raw that could linger through following lots. Depending on how careful your employees are or are not when scooping, I could envision seeing a scoop not being fully emptied from one ingredient before it's thrust into a second ingredient (meaning now your package of raw salt has some sugar into it via carelessness).
You could do your own risk assessment to figure out if you should keep this process going. The fact your program calls for different colored scoops for allergens is an acknowledgement that residue transfer is a potential hazard. Justifying scooping salt and sugar from their raw bulk containers with the same scoop isn't impossible, but it'll forever raise eyebrows with some.