How much difference does the hosting provider make to WordPress speed?

My WordPress website is reasonably optimized, but I’m wondering whether changing hosting providers would improve its loading times.

I’m looking at hostpro.com and a couple of alternatives. Their shared hosting is one of the options I’m considering, although I know that performance depends on more than just the provider.

For those who have switched shared hosting companies, did you notice a real improvement in TTFB and page loading times? Or was optimizing caching and plugins a much bigger factor?