I have tried to open new tab in existing browser but it opens in new browser.
Since you didn't really specify how you're using Selenium this is a general answer.
First of all, make sure it's absolutely necessary to open a new tab. It seems a bit strange as you rarely need to do that.
One way to do it though is to send the keys directly to the browser.
The following should work in C#:
IWebElement body = driver.FindElement(By.TagName("body")); body.SendKeys(Keys.Control + 't');