Time to first byte is almost always a problem on the server side. On a heavily shared host you can expect to get this.

Time to first byte is really the time it takes for the web server to start sending data back to your browser after the server has received your request.

We're having the same problem with our site (also on Dreamhost). Yours seems to be resolved- did you take any further action, or was the problem solved after your site was transferred?

I too am running Concrete5 on dreamhost, and
I have been seeing full page load times of around 9 seconds.

Elsewhere in these forums was a suggestion to look at:
http://defunctlife.com/2010/05/slow-speeds-and-speeding-up-concrete...

For me, this has significantly reduced the full page load times.

Here is the change I made.
The change is in:
concrete/elements/header_required.php

From:
$this->addHeaderItem($html->css('ccm.base.css'), 'CORE');
$this->addHeaderItem($html->javascript('jquery.js'), 'CORE');
$this->addHeaderItem($html->javascript('ccm.base.js', false, true), 'CORE');

To:
if ($u->isRegistered()) {
$this->addHeaderItem($html->css('ccm.base.css'), 'CORE');
$this->addHeaderItem($html->javascript('jquery.js'), 'CORE');
$this->addHeaderItem($html->javascript('ccm.base.js', false, true), 'CORE');
} else {
echo("<script type=\"text/javascript\" src=\"http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js\"></script>");
}

Note that there are other suggestions that remove the 'else' clause.
That may be OK for some, but removing the else (not loading jquery) will break extensions such as the slide show.

I think the idea is not to really remove the else but to not load external jquery and keep loading C5 included jquery instead so you would have: