That is quite possible. It may. If it is increasing as observed it suggests that some part of your system under test (SUT) is approaching overload. A common approach to load testing is to run the test at a slowly increasing number of threads. Expect to see fairly constant response times from the start of the test until reaching some number of threads and after that the response times increase steadily. As the number of threads continue to increase the SUT may become overloaded and then you may get lots of failing requests or possibly the SUT may crash.
Running the test at two loads does not tell you much. Running it at several different loads (including running a test that steadily ramps up the load) allows you to draw a graph of load against response time and thus see easily where the system behaviour changes.