Design pattern is not a magic ingredient which makes code perform better. It's goal is to make code easier to understand for a programmer (by pushing some effort to organize function calls/data/etc to computer).
So by definition, your code using design pattern is more compact (easier to understand) but makes more calls (unless your original code before applying patterns was really suboptimal and badly written).
Cost of these function calls is negligible, so using patterns is considered beneficial - like using high-level language is considered beneficial, even if programming in low-level language like assembly could made code more effective. And is beneficial for exactly the same reasons: because with patterns, like with high-level language, code is easier (cheaper in terms of time of the programmer) develop and maintain (adapt it to changes in the future).
If you are concerned about the performance before you measured the performance and found the bottleneck, you are wasting time. If and after you found the bottleneck, you may consider how to overcome it. Redesigning it is just one (and more expensive) way to handle it. Usually running tests in parallel, or provisioning servers with better performance, is easier, cheaper, and preferable.
Don't worry about the performance until business side tells you that you have performance problem. Don't trouble troubles before trouble troubles you. Be more concerned about code validity, coverage, flakiness, extensibility and maintainability.
So yes, do use design patterns like page factory, but your goal is NOT to improve the performance of the code, but performance and productivity of the programmer writing (and debugging and maintaining) the code, for a price of using higher-level abstractions in the code.