As Emportllc says, the amount of allergen may vary. Your supplier could give you an ingredient that contains 20ppm, 500ppm, 5000ppm...
As long as it is labeled correctly on their end, the liability would fall on you.
I will acknowledge that the theoretical ppm in your finished product is small.
Even at a serving size of 10g, 0.02g @ 9ppm would yield ~ 0.018ppm in final product.
Milk-containing ingredient= 0.02g
Serving size= 10g
The milk-containing ingredient makes up 1/500th of the serving size.
9ppm / 500 = 0.018ppm
But now let's assume the worst-case scenario. You are supplied an ingredient that contains 0.9% lactose.
Since you are using such a small amount in your recipe, I assume a 0.9% difference in composition would be hard to detect.
0.9% = 9000ppm
9000ppm/500 = 18ppm
A variation of less than 1% from your supplier could cause illnesses with your customers.
Regulators would look at your suppliers ingredient and say "They have listed milk listed as an allergen. Why don't you?"
"We don't use very much in our recipe" would not be an appropriate response.