There was a study years ago from skilled.co about how speed affect your website. The figures now should have some updates but this gives you an idea how important it is with speed and performance.
- 47% of customers expect a web page to be loaded within 2 seconds.
- 64% of smartphone users expect a mobile site to be loaded within 4 seconds.
- For SEO prospective, 1 second delay means 11% loss of pageviews.
- For conversion, 1 second delay lead to 7% reduction in conversion.
So speed does affect various factors to your website or WordPress website. But nowadays website has plenty of website which require to load various css, js etc so there is a balance between feature and speed/performance. E.g. if a media company has a lengthy web page, they should consider shorten with pagination or using lazy load in order to maximise the speed/performance.
Also, there are some un-necessary theme files, plugin in your WordPress, those should be removed as well as do a regular optimisation check on the database. If there are many js / css issue, you could consider minify plugin to perform those tasks but please keep in mind that not all js / css can be minified so that it is always about testing in a staging site / mirror site before deploying to the live site.
Beside those, there are more ways to improve the performance.
- Image compression
Google offers PageSpeed insight tool which you could have a try on the site to have an idea about improvement direction but do not take the score too serious as you could have a amazon website and they do not receive a good score also due to their website complexity requiring more features which takes more steps to load their website than others.