I picked up one of these Plastic to replace my useless 4 pole stock connector, it was all they had at AutoZone but the more I think about it the more plastic seems flimsy, all the 6 pole connectors I've seen have been metal. What do you guys think should I replace it with a metal one or does it matter?

It's fine. Many of the OEM are plastic.

For what its worth, last time I had one installed, I asked for plastic because the metal was chrome and I wanted black. The guy told me they quit selling them because they tend to crack with time.