Doug, if you happen to have a heat gun, you can use it to pre-melt some of the HDPE shavings, at least soften it up, then press it into your hole and melt it a little more, smooth it out. Be very careful not to hold the gun too close or for too long. It'll damage your kayak. I think you might find some vids on YouTube showing this technique.
One other thing. When you get light scratches from branches and things, usually on the sides of your kayak, a heat gun will actually close those back up. Again, you have to be careful not to overheat the kayak by moving the heat gun over a large area, in and out.
I wouldn't worry about defections on the bottom. Out of sight and expected.