5.6 to 5.7 migration tool is coming!
Permalink
Hi,
Like many people here I'm looking at Concrete5 evolution and followed the discussions about people having issues with the facture that there is no solution yet to upgrade from 5.6 to 5.7.
As a freelance working with Concrete5 this is not an issue now but will be in a near future.
While looking on github the number of thing that need to be done and that are more urgent than this and I'm not skilled enough to create such add-on. So an idea pop in my mind:
Why not make it happens all-together?
Who would be interested in a full migration script (users, files, pages, core blocks content, ...) ?
Who would be interested and have the full knowledge in developing this add-on?
If many people are interested in this add-on and could give 50$ (for exemple9 to make it happens then it will come real soon! Kind of kickstarter dev experiment..
The idea is to speed-up dev of this with some money and then make this add-on available for the whole community (for free).
First step would be to have enough people
Then define specifications
Then find someone who could make the dev.
Have some testers
..
Would you be in ?
Like many people here I'm looking at Concrete5 evolution and followed the discussions about people having issues with the facture that there is no solution yet to upgrade from 5.6 to 5.7.
As a freelance working with Concrete5 this is not an issue now but will be in a near future.
While looking on github the number of thing that need to be done and that are more urgent than this and I'm not skilled enough to create such add-on. So an idea pop in my mind:
Why not make it happens all-together?
Who would be interested in a full migration script (users, files, pages, core blocks content, ...) ?
Who would be interested and have the full knowledge in developing this add-on?
If many people are interested in this add-on and could give 50$ (for exemple9 to make it happens then it will come real soon! Kind of kickstarter dev experiment..
The idea is to speed-up dev of this with some money and then make this add-on available for the whole community (for free).
First step would be to have enough people
Then define specifications
Then find someone who could make the dev.
Have some testers
..
Would you be in ?
I'm in!
This is interesting.
Sounds like a good idea.
I'm in, would be good to know the rough time frame for completion from the developer. I guess it could be developed as an add-on which we are pledging in advance to buy.
Cheers
Dave
I'm in, would be good to know the rough time frame for completion from the developer. I guess it could be developed as an add-on which we are pledging in advance to buy.
Cheers
Dave
Throw it up on Kickstarter and I'll support it.
This is actually something that's already on our roadmap. We're envisioning a single solution using the existing content import XML format, beefed up and much less prone to errors, as well as an existing 5.6 and 5.7 exporter add-on, and a 5.7 importer add-on.
https://github.com/concrete5/concrete5-5.7.0/issues/1991...
We're interested in feedback.
https://github.com/concrete5/concrete5-5.7.0/issues/1991...
We're interested in feedback.
Andrew,
Thanks for jumping in. I remember hearing that this was on the roadmap, but we all acknowledge (I think) that the core team's efforts have to be directed and prioritized by need and budget.
For example, if a corporate client came along and decided they had to have 5.7 right now, and were willing to pay you for a solid, fully-featured migration tool, then the C5 project would have this tool available sooner rather than later. Right?
Can you see a scenario where those of us with some level of motivation contributed to a Kickstarter-type fund in order to bring this along? (Details to be determined, of course.)
Thanks for jumping in. I remember hearing that this was on the roadmap, but we all acknowledge (I think) that the core team's efforts have to be directed and prioritized by need and budget.
For example, if a corporate client came along and decided they had to have 5.7 right now, and were willing to pay you for a solid, fully-featured migration tool, then the C5 project would have this tool available sooner rather than later. Right?
Can you see a scenario where those of us with some level of motivation contributed to a Kickstarter-type fund in order to bring this along? (Details to be determined, of course.)
Yes and no....
It's absolutely true that large client projects have always driven a lot of the concrete5 development over the years. When there's a need that aligns with a corporate project there's often a great win-win in getting some new core features in....
Some examples:
The sitemap flat view was funded by the Great Nation of Kuwait because they had an extranet with 10,000 pages at a level in it - drag and dropping didn't make any sense.
The advanced permissions in 5.6 came about because Planned Parenthood and Topcon Positioning both had challenges on projects we were working on that needed them - and budgets were such that they could happen.
An earlier version of this spec was written because a client we're working with is starting to offer "prefab" 5.7 sites. Pick a bunch of stuff you like in a wizard and it'll build a site with custom content for you. As we heard about their challenges, and kept our own challenges in mind (5.6 upgrades, easy wordpress imports,) this spec emerged as the ideal solution. They might still pay us to build this, it's floating.
In my experience, convincing a client to pay for changes like this can be a bit of a complicated art that requires you to deliver the exact solution they need for their use case, while also creating the reusable parts we need. First you worry about them, and then you're careful to keep a margin to worry about us (free community version.) Not every client wants to work this way, but we're deeply appreciative of the ones who have.
Sometimes even the best of intentions backfires when we do this. We had an agency contract us to change the file manager for their client at one point and while the changes happened the agency was frustrated we didn't ask them to lead the UI design process and now we don't work together any more even though I'm close friends with one of the principles. The in-explicably present "schedule guest access" item in the block menu of 5.6 is because of awkward "but I want it!!!" requests from a client we had engaged with to flush out timed permissions. I'm not trying to disparage these folks, they both know their businesses well and are wonderful people, just to offer that even when it's a 1-on-1 client relationship this "were going to make some improvements to the core on your dime!" thing isn't always a slam dunk.
Extending this approach to work to be crowd funded instead of angel funded is scary to me for a number of reasons:
1) I'm not convinced we could raise the required funds.
It's no secret that our "onwards and upwards" mentality to 5.7's release has alienated some of our senior community members here. There's really nothing I can do about that at this point beyond working hard at delivering on onwards and upwards. I think sticking our hand out for donations to build something as fundamental as some migration scripts might not be well received by the folks who in the past might have been willing to chip in $50-100 towards the cause. We don't have a formal estimate for the cost of building what Andy has described, but "more than 100 hours" seems reasonable. If you were a company asking me what this would cost it'd be a low 5 figure check. I'm concerned that trying to raise that kind of money on a crowd funding site will go no where and be pretty crappy PR.
2) I'm not convinced we could satisfy 100+ clients who each gave us $100 each on this.
It's gotta be free, we're certainly not going to win any hearts and minds by trying to sell a migration tool set. So the typical model of kickstarter being essentially a pre-order system falls flat. What are we selling then? The promise that these import tools will work perfectly for everyone who contributes? Well that's a disaster waiting to happen. I think we all can read Andy's spec and agree that those would be awesome tools to have, but we all also know that if my wife donated $50 with the expectation that her wordpress blog would be easily imported into concrete5 and retain its complete functionality magically, she'd be sorely disappointed. It seems very likely that if we even could raise something even in the neighborhood of the costs, we'd end up with dozens if not hundreds of folks that would all expect a bit of personal hand holding, training, support. Someone would quite reasonably expect that for the money they're chipping in you're going to have some extra time to give them some personal attention. We don't. There's countless companies that have been torn apart by having ten angel investors instead of one. Same problem, different scale.
3) A year or so ago someone tried to start a kickstarter for making some training videos for concrete5. I don't think anything was contributed at all. My sense is there's an emerging backlash against these crowdfunding campaigns when they feel like you're just paying someone to do their job.
4) We did our own little crowd funding thing years ago to get to SxSw.. We needed 10k to make it. We raised 5k. It was hard. There was a lot of overhead in promoting/running the thing, and in the end I had to cover the rest of the bill anyway. Didn't help that sxsw was a complete waste of time, but regardless that's the only time we've ever "accepted donations" and that's fine with me. I still get emails from people on occasion asking a technical question and mentioning in passing that they have "supported the project" and it takes me a wile to remember they're talking about that campaign from 2011. I know we're far from perfect and we're certainly not rich, but turning concrete5 into NPR isn't something any of us here are excited about.
5) If/when we do engage in a crowdfunding thing, it would be to fund an event. I've been chatting with Katz for a while about doing a "conCon" in Portland around OsCon. I have no idea if 3 people, 30 people, or 300 people would come. For something like that crowdfunding makes a lot more sense to me. There's a clear service you're buying (couple days of training) there's some spiffs we can offer for those who want to "donate" without me feeling like a beggar for years because of it. Yadda. That makes total sense to me so please don't cry hypocrite if you see us doing that in the future. ;)
6) Really what we need, and what I believe we have the most chance of getting donated, is time. There's smart folks doing a lot of great work in github, and while it's sometimes a challenge, we have been really working at helping them help us. We've started doing a video conference on the first and third Wednesdays of the month to talk about what's been accepted and how to get things accepted. We've started using git issues instead of trello for our own 5.7 work. We're taking the time to write up specs like what Andy posted instead of "just doing it.." The folks in Github are doing work because their own projects need it, and/or they want to gain expertise in concrete5 and turn it into something better. That's the type of donation that doesn't come with any caveats which we can certainly use more of. My hope is that Andrew's spec drives some good discussion in github, which then turns into a bunch of tasks that not just Andrew can work on.
Beyond that all I can offer is the awareness that we've taken a strategic shift over the end of last year. Our overhead has been dramatically reduced, and we're trying to build up our hosting and SLA business as a larger revenue stream so we can simply manage the marketplace and hosting and not chase after these big fish any more. As time has gone by, it is harder and harder to find a corporate client whose needs align with where we'd like to see the project go. That, and being both a services shop and product company is challenging at best. It's organic, but there's a plan there we're working towards this year. Passive & recurring revenue of hosting/SLA's let us fund core development without having to be completely beholden to a cycling array of corporate masters.
My apologies for the long reply, I've tried to stop doing this but you caught me with too much coffee on a Friday. ;)
TL:DR - Help us by working in github, save your bucks for a more deserving charity than us. ;)
It's absolutely true that large client projects have always driven a lot of the concrete5 development over the years. When there's a need that aligns with a corporate project there's often a great win-win in getting some new core features in....
Some examples:
The sitemap flat view was funded by the Great Nation of Kuwait because they had an extranet with 10,000 pages at a level in it - drag and dropping didn't make any sense.
The advanced permissions in 5.6 came about because Planned Parenthood and Topcon Positioning both had challenges on projects we were working on that needed them - and budgets were such that they could happen.
An earlier version of this spec was written because a client we're working with is starting to offer "prefab" 5.7 sites. Pick a bunch of stuff you like in a wizard and it'll build a site with custom content for you. As we heard about their challenges, and kept our own challenges in mind (5.6 upgrades, easy wordpress imports,) this spec emerged as the ideal solution. They might still pay us to build this, it's floating.
In my experience, convincing a client to pay for changes like this can be a bit of a complicated art that requires you to deliver the exact solution they need for their use case, while also creating the reusable parts we need. First you worry about them, and then you're careful to keep a margin to worry about us (free community version.) Not every client wants to work this way, but we're deeply appreciative of the ones who have.
Sometimes even the best of intentions backfires when we do this. We had an agency contract us to change the file manager for their client at one point and while the changes happened the agency was frustrated we didn't ask them to lead the UI design process and now we don't work together any more even though I'm close friends with one of the principles. The in-explicably present "schedule guest access" item in the block menu of 5.6 is because of awkward "but I want it!!!" requests from a client we had engaged with to flush out timed permissions. I'm not trying to disparage these folks, they both know their businesses well and are wonderful people, just to offer that even when it's a 1-on-1 client relationship this "were going to make some improvements to the core on your dime!" thing isn't always a slam dunk.
Extending this approach to work to be crowd funded instead of angel funded is scary to me for a number of reasons:
1) I'm not convinced we could raise the required funds.
It's no secret that our "onwards and upwards" mentality to 5.7's release has alienated some of our senior community members here. There's really nothing I can do about that at this point beyond working hard at delivering on onwards and upwards. I think sticking our hand out for donations to build something as fundamental as some migration scripts might not be well received by the folks who in the past might have been willing to chip in $50-100 towards the cause. We don't have a formal estimate for the cost of building what Andy has described, but "more than 100 hours" seems reasonable. If you were a company asking me what this would cost it'd be a low 5 figure check. I'm concerned that trying to raise that kind of money on a crowd funding site will go no where and be pretty crappy PR.
2) I'm not convinced we could satisfy 100+ clients who each gave us $100 each on this.
It's gotta be free, we're certainly not going to win any hearts and minds by trying to sell a migration tool set. So the typical model of kickstarter being essentially a pre-order system falls flat. What are we selling then? The promise that these import tools will work perfectly for everyone who contributes? Well that's a disaster waiting to happen. I think we all can read Andy's spec and agree that those would be awesome tools to have, but we all also know that if my wife donated $50 with the expectation that her wordpress blog would be easily imported into concrete5 and retain its complete functionality magically, she'd be sorely disappointed. It seems very likely that if we even could raise something even in the neighborhood of the costs, we'd end up with dozens if not hundreds of folks that would all expect a bit of personal hand holding, training, support. Someone would quite reasonably expect that for the money they're chipping in you're going to have some extra time to give them some personal attention. We don't. There's countless companies that have been torn apart by having ten angel investors instead of one. Same problem, different scale.
3) A year or so ago someone tried to start a kickstarter for making some training videos for concrete5. I don't think anything was contributed at all. My sense is there's an emerging backlash against these crowdfunding campaigns when they feel like you're just paying someone to do their job.
4) We did our own little crowd funding thing years ago to get to SxSw.. We needed 10k to make it. We raised 5k. It was hard. There was a lot of overhead in promoting/running the thing, and in the end I had to cover the rest of the bill anyway. Didn't help that sxsw was a complete waste of time, but regardless that's the only time we've ever "accepted donations" and that's fine with me. I still get emails from people on occasion asking a technical question and mentioning in passing that they have "supported the project" and it takes me a wile to remember they're talking about that campaign from 2011. I know we're far from perfect and we're certainly not rich, but turning concrete5 into NPR isn't something any of us here are excited about.
5) If/when we do engage in a crowdfunding thing, it would be to fund an event. I've been chatting with Katz for a while about doing a "conCon" in Portland around OsCon. I have no idea if 3 people, 30 people, or 300 people would come. For something like that crowdfunding makes a lot more sense to me. There's a clear service you're buying (couple days of training) there's some spiffs we can offer for those who want to "donate" without me feeling like a beggar for years because of it. Yadda. That makes total sense to me so please don't cry hypocrite if you see us doing that in the future. ;)
6) Really what we need, and what I believe we have the most chance of getting donated, is time. There's smart folks doing a lot of great work in github, and while it's sometimes a challenge, we have been really working at helping them help us. We've started doing a video conference on the first and third Wednesdays of the month to talk about what's been accepted and how to get things accepted. We've started using git issues instead of trello for our own 5.7 work. We're taking the time to write up specs like what Andy posted instead of "just doing it.." The folks in Github are doing work because their own projects need it, and/or they want to gain expertise in concrete5 and turn it into something better. That's the type of donation that doesn't come with any caveats which we can certainly use more of. My hope is that Andrew's spec drives some good discussion in github, which then turns into a bunch of tasks that not just Andrew can work on.
Beyond that all I can offer is the awareness that we've taken a strategic shift over the end of last year. Our overhead has been dramatically reduced, and we're trying to build up our hosting and SLA business as a larger revenue stream so we can simply manage the marketplace and hosting and not chase after these big fish any more. As time has gone by, it is harder and harder to find a corporate client whose needs align with where we'd like to see the project go. That, and being both a services shop and product company is challenging at best. It's organic, but there's a plan there we're working towards this year. Passive & recurring revenue of hosting/SLA's let us fund core development without having to be completely beholden to a cycling array of corporate masters.
My apologies for the long reply, I've tried to stop doing this but you caught me with too much coffee on a Friday. ;)
TL:DR - Help us by working in github, save your bucks for a more deserving charity than us. ;)
Thanks for the detailed answer, Frz. Over-caffeinated or no, I appreciate the time and detail involved.
I think you raise some valid points, and it's certainly your call, so for the sake of discussion (not argument to change a position) let me throw out a couple answers to items you raised.
#2 - Satisfying Everyone
You're right. The best we could hope for is a clearly defined target functionality so that people should be aware of what they are buying. But most won't read the details, and some will read what they want whatever words you use. It's much the same as putting out a free piece of software (where everyone has ideas on what you obviously should have done), except that there's a financial component.
The only question is: Is that a deal breaker? Maybe on its own, no? Maybe in conjunction with other issues it helps tip the scale?
You also asked the question: "What are we selling?" I think the answer is "time". It appears this tool will happen at some point, we just have to wait until it's ready in it's first iteration, then hopefully improved upon. So, there's the time to first version, then the time to really tight version, then time to tight version with more features.
What motivates me to contribute is the time that would be gained by having a polished version available sooner. That saves me on the wait to migrate, and the time required to clean up after the migration. That's my pain point at the moment.
I've been running vBulletin sites since 2002, Drupal around the same time, and just bought three Joomla sites. We also operate a XenForo forum that was migrated from vBulletin. I list these only to say I'm well aware that not everyone can hop major versions like WordPress with a click or two. I'm used to having to wait a few months for the themes and plugins our sites require to upgrade (or not!) to support new versions. I'm used to praying my way through an upgrade process hoping that what comes out the other end is usable.
The only difference is vBulletin (admittedly a paid product, so apples to pears) launches their new versions with an upgrade script. It upgrades the core features and company-developed add-ons, but not the 3rd party stuff (though outdated 3rd party stuff rarely breaks the site, it just may not work/look as expected).
I agree in the end that the migration tool should be free, but I'm willing to contribute toward having a free and nicely functioning tool faster. That's just me, and it's because it suits my needs. Would anyone else contribute? Would it be an effective approach? I don't know ... that's why you get paid the big bucks ;) <jokes>
I think you raise some valid points, and it's certainly your call, so for the sake of discussion (not argument to change a position) let me throw out a couple answers to items you raised.
#2 - Satisfying Everyone
You're right. The best we could hope for is a clearly defined target functionality so that people should be aware of what they are buying. But most won't read the details, and some will read what they want whatever words you use. It's much the same as putting out a free piece of software (where everyone has ideas on what you obviously should have done), except that there's a financial component.
The only question is: Is that a deal breaker? Maybe on its own, no? Maybe in conjunction with other issues it helps tip the scale?
You also asked the question: "What are we selling?" I think the answer is "time". It appears this tool will happen at some point, we just have to wait until it's ready in it's first iteration, then hopefully improved upon. So, there's the time to first version, then the time to really tight version, then time to tight version with more features.
What motivates me to contribute is the time that would be gained by having a polished version available sooner. That saves me on the wait to migrate, and the time required to clean up after the migration. That's my pain point at the moment.
I've been running vBulletin sites since 2002, Drupal around the same time, and just bought three Joomla sites. We also operate a XenForo forum that was migrated from vBulletin. I list these only to say I'm well aware that not everyone can hop major versions like WordPress with a click or two. I'm used to having to wait a few months for the themes and plugins our sites require to upgrade (or not!) to support new versions. I'm used to praying my way through an upgrade process hoping that what comes out the other end is usable.
The only difference is vBulletin (admittedly a paid product, so apples to pears) launches their new versions with an upgrade script. It upgrades the core features and company-developed add-ons, but not the 3rd party stuff (though outdated 3rd party stuff rarely breaks the site, it just may not work/look as expected).
I agree in the end that the migration tool should be free, but I'm willing to contribute toward having a free and nicely functioning tool faster. That's just me, and it's because it suits my needs. Would anyone else contribute? Would it be an effective approach? I don't know ... that's why you get paid the big bucks ;) <jokes>
