Anything you add that increases weight and/or wind resistance so going to slow you down some. I would put power poles at the top of the list. But a larger graph is probably going to have less of an effect and possibly there are some mounting bracket options that will minimize the issue. That being said, I think losing "several" mph just by upsizing your graph is questionable. JMO though.
"There is a grey blur and a green blur. I try to stay on the gray one" Joey Dunlap, Isle of Man rider