Large Site Scalability?
Permalink
I discovered concrete5 a few days ago and I am VERY impressed.
A big concern of mine, however, is scalability. The first project I would use c5 for involves 1,500+ individual pages and 3,000+ images. If I were to bring my full site into c5, we're talking 20,000+ pages, 1 million+ images and around 100 user accounts.
I'm worried some of the dashboard features would be overloaded by a site the size of mine. What about site performance once you get this big?
The showcase sites are nice looking, but they all appear pretty small-scale to me. What is the largest site actively using this CMS? Has there been any testing in this area? Thanks.
A big concern of mine, however, is scalability. The first project I would use c5 for involves 1,500+ individual pages and 3,000+ images. If I were to bring my full site into c5, we're talking 20,000+ pages, 1 million+ images and around 100 user accounts.
I'm worried some of the dashboard features would be overloaded by a site the size of mine. What about site performance once you get this big?
The showcase sites are nice looking, but they all appear pretty small-scale to me. What is the largest site actively using this CMS? Has there been any testing in this area? Thanks.
check this thread.http://www.concrete5.org/community/forums/usage/performance_on_larg...
....I searched around a bit before posting but didn't find this particular thread.
We average 50,000 visits per day, but usually don't get hit that hard concurrently. My main concern was the ability of c5 to handle so many objects/files in the system.
We average 50,000 visits per day, but usually don't get hit that hard concurrently. My main concern was the ability of c5 to handle so many objects/files in the system.
That is a great post!! Enterprise level scalability. Theoretically, you shouldn't have a problem with this. Your hosting and database requirements would be though.
I imagine with the right server tweaks and management you shouldn't have a problem but then, I am no C5 expert (yet).
Franz or Andrew would have the best replies unless of course other users have done this at the scale your referencing. As a front end, this shouldn't be an issue. Your layout and design will be standard Web2.0 but that back end .... I will hazard the best thing to say is "If your current site is using PHP and your site is working well now, the main issue will be importing the data to work with the templating/theme and single page functionality. I would think if you have a site working now you are using a dedicated server set. But again, Franz or Andrew would better be able to advise you on the intricacies.
That's my two pence worth.
I imagine with the right server tweaks and management you shouldn't have a problem but then, I am no C5 expert (yet).
Franz or Andrew would have the best replies unless of course other users have done this at the scale your referencing. As a front end, this shouldn't be an issue. Your layout and design will be standard Web2.0 but that back end .... I will hazard the best thing to say is "If your current site is using PHP and your site is working well now, the main issue will be importing the data to work with the templating/theme and single page functionality. I would think if you have a site working now you are using a dedicated server set. But again, Franz or Andrew would better be able to advise you on the intricacies.
That's my two pence worth.
That is a great post!! Enterprise level scalability. Theoretically, you shouldn't have a problem with this. Your hosting and database requirements would be though.
I imagine with the right server tweaks and management you shouldn't have a problem but then, I am no C5 expert (yet).
Franz or Andrew would have the best replies unless of course other users have done this at the scale your referencing. As a front end, this shouldn't be an issue. Your layout and design will be standard Web2.0 but that back end .... I will hazard the best thing to say is "If your current site is using PHP and your site is working well now, the main issue will be importing the data to work with the templating/theme and single page functionality. I would think if you have a site working now you are using a dedicated server set. But again, Franz or Andrew would better be able to advise you on the intricacies.
That's my two pence worth.
I imagine with the right server tweaks and management you shouldn't have a problem but then, I am no C5 expert (yet).
Franz or Andrew would have the best replies unless of course other users have done this at the scale your referencing. As a front end, this shouldn't be an issue. Your layout and design will be standard Web2.0 but that back end .... I will hazard the best thing to say is "If your current site is using PHP and your site is working well now, the main issue will be importing the data to work with the templating/theme and single page functionality. I would think if you have a site working now you are using a dedicated server set. But again, Franz or Andrew would better be able to advise you on the intricacies.
That's my two pence worth.
We have robust servers and run on a university network, so hardware shouldn't a limitation (or is something we could tweak if needed).
We're actually running an IIS/ASP setup currently (with no CMS), but are
undergoing a complete visual redesign along with a move to a LAMP stack in the next 6-8 months. I have the luxury of getting to design this from the ground up.
We're actually running an IIS/ASP setup currently (with no CMS), but are
undergoing a complete visual redesign along with a move to a LAMP stack in the next 6-8 months. I have the luxury of getting to design this from the ground up.
not sure if this is a problem, never had that problem.
But I know that certain file system have a limit of files per directory. It might be a problem to put 1mio files into one directory (which c5 is probably going to do).
Anyone to confirm this?
But I know that certain file system have a limit of files per directory. It might be a problem to put 1mio files into one directory (which c5 is probably going to do).
Anyone to confirm this?
Biggest c5 install I know ofhttp://schoolpulse.com. over a million pages, photogalleries, forums, event calendar - many many pictures.. etc
That site runs on a single server with no load balancing or slave databases.. We did have to spend some time optimizing queries on the home page, but that was all custom work in the first place.
You are quite correct (and astute) in predicting that some bits of the UI in concrete5 don't behave well on huge sites today. Here's what I know.
That graph in the dashboard home - just turn off statistics. Its cute on small sites but only a step away from useless - it also takes for ever to load. We routinely turn it off on any mid sized site and everything loads faster.
The sitemap - very handy for small sites, a total wreck when you have 100's of nodes at a single level. That's all loading in JS so I'm not sure there's much optimization that can happen to make dragging-n-dropping effective when you're looking at a 1000 pages in a single node. The good news, we know this and are already solving it. We've got this problem in our faces because of forums where you have to use it to move threads, so were starting the process of creating different view modes (like an OS would have)... search, drilldown/list.. etc I think you'll see a little infrastructure for this in the upcoming 5.3 release, and more sexiness in a 5.3.1 release.
That's really it.
The limit on the number of files you can have in one directory in unix is huge, and we've already dumped that approach in 5.3 so that's not relevant. The sitemap problem is the only real downer in my experience doing this - and there are ways around that even today.. a small hack to have to make for the glory of in-context editing..
Some other big sites that have been made with earlier versions of concrete cms:
Http://lemonade.com
http://indie911.com
http://lewisandclark200.org (first concrete cms site ever!)
That site runs on a single server with no load balancing or slave databases.. We did have to spend some time optimizing queries on the home page, but that was all custom work in the first place.
You are quite correct (and astute) in predicting that some bits of the UI in concrete5 don't behave well on huge sites today. Here's what I know.
That graph in the dashboard home - just turn off statistics. Its cute on small sites but only a step away from useless - it also takes for ever to load. We routinely turn it off on any mid sized site and everything loads faster.
The sitemap - very handy for small sites, a total wreck when you have 100's of nodes at a single level. That's all loading in JS so I'm not sure there's much optimization that can happen to make dragging-n-dropping effective when you're looking at a 1000 pages in a single node. The good news, we know this and are already solving it. We've got this problem in our faces because of forums where you have to use it to move threads, so were starting the process of creating different view modes (like an OS would have)... search, drilldown/list.. etc I think you'll see a little infrastructure for this in the upcoming 5.3 release, and more sexiness in a 5.3.1 release.
That's really it.
The limit on the number of files you can have in one directory in unix is huge, and we've already dumped that approach in 5.3 so that's not relevant. The sitemap problem is the only real downer in my experience doing this - and there are ways around that even today.. a small hack to have to make for the glory of in-context editing..
Some other big sites that have been made with earlier versions of concrete cms:
Http://lemonade.com
http://indie911.com
http://lewisandclark200.org (first concrete cms site ever!)
Wow, thank you...I really appreciate the honest assessment. This area was the only red flag for me so far.
Dashboard stat graph - no issue there, we already have other analytics in place. Wouldn't be that useful to me anyway.
Sitemap - glad to hear this is getting some attention in the next release. As long as there are ways to get around that, I'd be willing to put up with it for now...I agree it would probably be a small price to pay for all of the other functionality c5 offers.
As I said, I'm very impressed with the product and this community so far. I'm planning on downloading c5 and getting a test site running on our servers soon. Perhaps one day I can add our museum's name to your PR efforts...
Dashboard stat graph - no issue there, we already have other analytics in place. Wouldn't be that useful to me anyway.
Sitemap - glad to hear this is getting some attention in the next release. As long as there are ways to get around that, I'd be willing to put up with it for now...I agree it would probably be a small price to pay for all of the other functionality c5 offers.
As I said, I'm very impressed with the product and this community so far. I'm planning on downloading c5 and getting a test site running on our servers soon. Perhaps one day I can add our museum's name to your PR efforts...
Hi
Other than sitemap, hope you can work on improvements in managing files/images. The ability to create subdirectories, define alternate locations for file repositories, sorting, searching, tagging content and others would be great to see in 5.3 or later releases. Thanks!
Other than sitemap, hope you can work on improvements in managing files/images. The ability to create subdirectories, define alternate locations for file repositories, sorting, searching, tagging content and others would be great to see in 5.3 or later releases. Thanks!
and reasonably well commented on throughout the forums i think... file manager has been rebuilt from the bottom up in 5.3 - slated to release end of month..
Can't wait to see how you implemented that new file manager. Can't wait for the release :)
"The limit on the number of files you can have in one directory in unix is huge, and we've already dumped that approach in 5.3 so that's not relevant."
You are abandoning the Unix approach for version 5.3?
Rick
You are abandoning the Unix approach for version 5.3?
Rick
Heh, we're still very much supporting UNIX =) all this statement means is that, in the past, we had dumped every single file uploaded to a site into the root of the files/ directory. Now, we're breaking that up into a series of subdirectories, for a number of reasons.
1. File path is somewhat obfuscated (although this isn't really done for security reasons.)
2. Files are no longer renamed when they are uploaded. That means no more ugly filenames with a bunch of characters/numbers at the beginning of them. Instead, the characters/numbers make up the path to the file.
3. We didn't like that our large sites really filled up directories with thousands and thousands of files. Now we'll still have lots of directories (and it may be difficult to find files unless you use the file manager to locate them) but they should not result in hundreds of thousands of files in only one directory.
1. File path is somewhat obfuscated (although this isn't really done for security reasons.)
2. Files are no longer renamed when they are uploaded. That means no more ugly filenames with a bunch of characters/numbers at the beginning of them. Instead, the characters/numbers make up the path to the file.
3. We didn't like that our large sites really filled up directories with thousands and thousands of files. Now we'll still have lots of directories (and it may be difficult to find files unless you use the file manager to locate them) but they should not result in hundreds of thousands of files in only one directory.
Thank you for the explaination.
Rick
Rick
It scales fairly well. The biggest problem is as always IO. You might need a master-slave mysql system if you have like 3000 concurrent users, but just having a lot of pages isn't a problem at all...