Anybody out there?

hi,

guess that's what beta team does. there is some debuging and logging stuff in concrete5.

what tests are you planning to integrate?
cheers

unit tests are not done by a team, they are written by the developers and run without any team..

The beta team doesn't use self running tests.

As far as I know there are no unit tests and I think it's not that easy to do as with most web applications.

Please note that there are some ajax interfaces. Using a tool like selenium might work but that's a lot of effort as well.

But the get a "proper answer", you have to wait until someone of the core team sees this thread.

yeah, I was thinking of unit tests, probably in PHPUnit or SimpleTest. Not the user-level QA testing (although that can also be automated using tools like Selenium).

It's really not that hard to unit-test ajax apps, since ajax calls are just calls to web urls: you test the server-side with fake http requests, and you test the client-side ajax with fake server-side responses. At least, if you're into testing, it's no harder than anything else.

Anyway, the question was (and is) directed at the core team, in hopes of discovering whether there are tests not included as part of the standard sourceforge download.

I used to write libraries for financial statistics. There it is quite easy.

fact(5) = 120 and otherwise, report and error.

With ajax url's you do not only have to check the return value but also the state of the database. Some function modify the database in a way you can't judge by the return value.

In this case you need to have a "test database" which you can reset all the time and a few more things.

If you have a different opinion - tell me how you'd write a unit test for the ajax uploader? I'd like to be sure it creates the correct meta data, thumbnail, sets and permissions..

yeah we've actually started poking around at this a bit, while its going to be impossible to model all the use cases around something as flexible as a CMS, there's still a lot of value in breaking out some basic "does this stuff still work" tests...

we've been experimenting with it, but if anyone has a lot of expertise here and wants to volunteer their help you should surely reach out.

I'd love to help. whom should I email?

about helping with testing :-)

JavaScript's really easy to test because it's so loose. Easiest way to test ajax is to override the XHR object with a test double that expects a certain value, or returns a simple string, then assert accordingly. Google 'mock xmlhttprequest' for a ton of listings, including:

see the section 'mock objects':

http://developer.yahoo.com/yui/3/test/...

also

http://ajaxpatterns.org/wiki/index.php?title=Simulation_Service...

There's a really cool community evolving in the javascript world right now :-)

The YUI guys are always friendly and insightful, and put up a nice talk about test-driven javascript development

http://developer.yahoo.com/yui/yuitest/#video...

This is a very interesting thread and I understand how some folks would want to see some sort of benchmark statistics on how well Concrete 5 runs on various platforms.

Some of the issues I see:
a. Tests run off the Concrete 5 hosting platform may be optimized to run only Concrete 5.
b. Tests run from a shared hosting platform such as GoDaddy will be inaccurate based on their server architecture.
c. Are tests run from a UML standard or which/whose standard.
d. Are we wanting to stress test a DB server or an HTTP server (Apache or IIS) with PHP.

Just a few of the questions that come to mind. I believe these benchmark tests should be run at some point because as Concrete 5 grows in distribution and as the community grows, someone (if it already hasn't happened) will want to have these performance results to base the possibility of increased corporate sustainability.

That is my few thoughts for now.

bill, you got that completely wrong.

You write unit tests to ensure the quality and make sure you don't have any negative sideeffects while adding new features.

A stress test or benchmark is something completely different. If you care about performance, just use "ab" on your linux box to test the speed

...because it's wholly dependent on server-side components. What flavor of linux and which web server (nginx/apache/lighttpd/others) is a big question. How many servers you're pushing load across is another: you might easily have a cluster of apache boxes sitting on top of a cluster of mysql boxes.

Then you have to take caching into account: html caching at the app level (concrete5), php caching at the opcode level (APC/Zend Cache), and the mysql query cache at the database level.

Aside from all this, Steve Souders has written a couple of books based on the observation that ~80-90% of download time has to do with front-end configuration: how many HTTP requests are needed, whether your content is served from a CDN, and about 35 other rules (see YSlow for more). Odds are that front-end optimization and caching generated html are the best places to start for a smaller site.

So, you might pick up a good book on benchmarking and server performance (scalable web sites, maybe), but concrete5 can't really do this for you. Even if someone stress-tested a standard GoDaddy-type crapola account, you can't control how many spammer/porn accounts are also riding on any given oversold GoDaddy server box, so the performance across accounts, and across different days or times of day, can vary pretty substantially.

Any movement on this?

I would really like to see unit tests for concrete5. In fact, it's surprising that anybody would build such a large and complex application and make it available to the public without unit testing it.

One problem I see is that in order to properly unit test an application the code has to be testable. I see some fundamental issues with concrete5 which would affect its testability. For example, the configuration is largely based on constants. This will make testing more difficult because you cannot change the value of a constant once it is created. This means that you cannot run a bunch of tests that use different values for these constants. You will have to have different tests for different constant values and you will have to run these tests separately. By using variables instead of constants you can pass them into class constructors or setters allowing you to test the class with different settings. Constants may still be used for some things which will not change from test to test (such as paths) but for many things constants are not ideal.

Another issue is the use of singletons. Singletons make objects globally available and it encourages code that loads these objects as they are needed from inside other objects. Using dependency injection would have been a better approach as it allows you to inject objects into the classes you want to test. When testing you would want to create a fresh instance of each of these objects for each test rather than using a singleton which will reuse the same instance for each test. This allows artifacts from one test to find its way into another. Dependency injection also allows for mocking objects because instead of injecting a particular object you can instead inject a mock version of the object. This is useful with resources for example because you do not want a test to fail because of a problem with a resource when it is not the resource that you are testing. For example, if you are testing a class that sends an email you want that class to use a mock version of the email object so that the test doesn't fail if the email is not sent. So if you are testing a class that handles a form submission and then sends an email you do not want the test to fail because the email could not be sent. You are testing the form submission, not the email object. The email object should have its own unit tests. If you want to test what happens when the email succeeds you pass in a mock version of the email object which is configured to give a successful response and in you want to test how it handles failure you pass in a mock email object that is configured to fail.

So, though I would love to see unit testing, and I believe that unit testing is crucial for an application like this, it might have to wait until version 6. This is because proper unit testing may require a lot of changes in the application architecture which could require web sites to update their code to upgrade their concrete5 install. These kinds of changes are necessary from time to time but should probably wait until a full point upgrade.

Sparky,

You seem to know a fair amount about unit testing. I've learnt the basics, but run into conceptual problems when it comes to things like mocking and injecting.

I committed some code athttps://github.com/concrete5/concrete5/pull/65/files... . It's definitely only a start, but it does have a helper already setup. Helpers are easy(ish), but I don't know where to start with the Request() stuff or models tied to the db. My thought would be to have a basic database import code that's called in the setup.... but that can't be best practice. I spent some time last night googling how to do unit testing with adodb_active_record, and found surprisingly little.

Would you be able to expand out the framework for the more complicated stuff (requests and models that hit the database)?

Also, coincidentally, these types of contributions have been formalized.http://www.concrete5.org/about/blog/open-source-and-strategy/announ...

James

I'm no expert on unit testing though I do have some experience. Though I don't have time right now to look into this stuff for you I would make a few suggestions. For object mocking look at Mockery:https://github.com/padraic/mockery...

Also, I would recommend not hitting the db for your unit tests. You don't want your tests to fail because of db issues and you probably don't want to have to keep track of primary key values and what not. I would recommend saving that for acceptance testing which should be separate from the unit tests, though you can use phpunit for that as well.

Yes, there is a high interest. We started unit testing in house but we haven't really released them, since we didn't have a great hosted version control system in place when we did them (Well, we used SVN, but it wasn't great.)

Now that we're on github I'd really love to see unit tests take off more for concrete5. sparkymeister makes good points about concrete5 and its testability, but I think there is a lot that we can do even in spite of some of the way the application works. For example, I have 10-12 unit tests just covering block custom templates, header items, etc.. This was using SimpleTest, but I've heard that PHPUnit is the better way to go (these days). Thoughts from anyone on this particular topic in mid-2011? I'd love to commit our unit test code into a new github repository specifically dedicated to unit testing, and give some community members commit and management access to really help out in this regard.

I'm sure SimpleTest is great, but does anybody use it? I went to a session on test driven development at tek-x last month and the presenter was a contributor to SimpleTest so that's what he used. Afterwards everybody was saying "why he didn't use PHPUnit? Nobody uses SimpleTest." So I would suggest tests be done in PHPUnit since that's what most people are familiar with.

At the time, the lack of development on PHPUnit probably caused it to come across as an orphaned project. I also had high hopes for the Web Tester that was part of SimpleTest (although phpQuery appears to be better for that, anyway.)

Additionally, I had plans for integrating this into an actual unit test package that could install into a concrete5 site, and saw that Drupal had done this using SimpleTest, and assumed it could be more easily bundled (As in, it doesn't require a PEAR module, etc...)

Not sure when you were looking at it but PHPUnit has been under active development for quite some time. Based on its commit history on github this must have been before 2009.

Yeah, it's been around forever, but I saw the project and the latest entry in their news was sometime around 2005. Additionally, SimpleTest appeared to be gaining a bit of momentum at the time (and appeared to be more easily packageable.)

It's kind of moot at this point, since the number of tests written is pretty low. We have our pick of whichever platform is best.

I did some research and set up a POC testing environment on my C5 package using PHPUnit (nothing complicated, a bootstrap.php file that duplicates some of the dispatch.php stuff), and I've been pretty happy with it.

James

The company I work with does unit testing on every Zend framework based project that has the budget- and certain projects (i.e. auction websites) have time for unit testing built into the project estimate, given the highly complex logic involved.

In these cases, the goal is to provide a code-based proof of concept that your code works; each test that relies on any sort of state in the database should use fixture files (sql, yaml, whatever you want) to setup a sqlite or mysql database in exactly the state it needs to be for your tests to pass.

It is for this reason that the Zend framework includes two testing- oriented classes- Zend_Test_PHPUnit_Db to help you with setting up the database state for each test, and Zend_Test_PHPUnit_ControllerTestCase for testing request / response cycles. For everything else- testing your models or non-controller code- extend one of the PHPUnit classes.

When we evaluate a third party library or framework, a lot of the senior developers who prefer to use Zend will use the unit tests provided as much, if not more than, the documentation provided, since simply running the unit tests will prove whether or not they're out of date, and shows you exactly how the code is expected to be used.

I can't say that you have to use PHPUnit, though it would seem odd to me to use anything else, given that Zend has classes that use it. However, it's really hard to evaluate something we haven't used before for a client's project when there's no way to test it, so anything would be better than nothing.

That's cool. I didn't know that. Yes, we have gone with phpunit in our official testing repository:

http://github.com/concrete5/concrete5-tests...

I'm a bit late on this but I'd really like to see unit testing taking hold. My one an only bug fix would have come with a unit test if it was there.

Here's a fairly good article that aligns with my unit test experience (from the many years in the java world)http://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-dec... (EDIT: fixed link). I'm reluctant to trust code without test these days, both because because there's no test, but also because unit testing forces you to modularise your code and in my experience this has dramatically my code. I'm by no means a unit test fanatic (I don't particularly enjoy writing them and I'm definitely not a "test first" type of guy), but they are such an important part of large scale system.

I'd also love to see more solid active record style query framework. There's an awful lot of table/column names in strings through out the code (i.e. it's not very DRY) and sooo many switch/case and if/else block of code (i.e. not taking advantage of polymorphism). I've been keen to help out and extend things, but issues like these have really given me cause to hold back. Has anyone tried anything likehttp://www.phpactiverecord.org/,...http://www.propelorm.org/ orhttp://www.doctrine-project.org... ? I noticed there's some dynamic query methods on PageList so at least the concepts aren't foreign.

Is there any plan (short or long term) to address issues like this? These kind of things have kept me somewhat at arms length and keeping one eye open for other CMS options. Trouble is I really like the usability of C5 for my users so I'd like to stay with it... I'm just too scared to touch the code and to rack up a large technical debt for myself!

Thanks and cheers
Andrew

Another mocking framework to check out ishttps://github.com/hafriedlander/phockito... although mockery (as mentioned above) looks like it might be more advanced.

There's also a good article on mock/stubs and various testing terminology at:http://martinfowler.com/articles/mocksArentStubs.html...

Cheers
Andrew

Setting up fully automated unit tests is something I am interested in, especially where a unit can be isolated. I used to do a lot of work with more conventional test harnesses in other languages. But I am having trouble grasping the big picture on how a unit test could be built for, say, the controller of a C5 block.

Phockito looks neat, but to grasp it I need to see a fully fledged example unit test with a couple of test cases and explanation. A dummies guide to applying it within the context of C5. With that bit of basic learning out of the way, I am confident that with previous experience my learning curve would be fast.

I think part of the problem is that the current architecture for C5 is pretty un-testable as it currently stands. This is mainly due to the structure of the code (lots of use of if/else/switch/case instead of strategy/state patterns, lots of reliance on global state, no clear services layers that can be mocked/stubbed, lots of sql in strings etc) This makes it very difficult (near impossible) to write maintainable unit tests. There's probably scope for integration style tests where you prep the database, do things and check the results, but these tend to only test happy paths and are more useful for checking that components that already have unit tests play well together in real life. These however aren't a substitute for proper unit testing.

I'm not really sure what the best approach forward is, there's a lot of code in C5, and it works pretty well (mostly) which can limit the motivation to change the internals. I think unit tests on the current code base will themselves become very complex and become a heavy technical burden and in the worst case will be a hindrance to refactoring and future improvement. I'd love to see C5 improve, but I think it'll will require considerable effort and commitment to writing unit tests from the start since it substantially changes the way you develop your framework (for the better). But this will change things substantially for 3 party developers, which is something I'd love, but other's have significant investment in the current code base (both code and knowledge).

I'm only just getting unit testing up and going in my local environment as I'm playing with some autonav changes but it'll be a while before I've had a chance to figure anything out.

Cheers
Andrew

Whilst unit testing within the core is obviously of long term benefit, my immediate interest is just putting together test harnesses for blocks in packages I am developing. Whether this is truly isolated, or a vertical slice of integration, or both, I don't mind. I would just like a clean and obvious test design pattern in a C5 context I could understand and then follow for my own blocks.

I suspect that other developers are in a similar situation, having a desire to do automated and structured testing of their add-ons, but needing guidance on how to do get started.

That's hard to answer without seeing actual code. Is your package public? I'm happy to have a look at it and give my 2 cents (probably inflated). Typically I break my designs down to common OO patterns and test each constituent part. Testing at the controller/view level might be bit harder since it depends on the C5 environment and views aren't objects.

Cheers
Andrew

I've just been going over some of my code for a package I use (not really production standard) which hasn't got tests and have thought about how I'd do it. In this example I'm testing a dashboard controller but I think the ideas would translate. I think a lot of the problem is that the controllers are both the controller and the model (i.e. C5 injects the db properties into the controller making it harder to test).

My dashboard controller configures global properties for popups using C5 Config store. I created a very simple FormModel class that uses Property classes that know how to read and write from Config (which can be tested independently). The form model then handles reading the state from the controller on post and is used by the view to render the form. From here each part is testable and things start to get a bit DRYer.

NOTE: If I had my time again I'd look at extending/using something like the form model in Zend rather than write my own from scratch.

ALSO: Because I haven't written tests this is going to be a bit fictional but I'll try and step through how I'd test it.

Firstly, here's my basic form model. Each property is an instance of ConfigProperty and is (should be) tested independently. BooleanConfigProperty has special handing for the fact that checkboxes are missing from the request when they're not set. It also handles the issue that Config returns empty strings when things aren't set instead of nulls. But because this is in my property class, I don't need to test this each time I use it.


Then in my controller I have.



To make the above testable I'd probably invent a `Request` interface with common methods like isPost() and get(key) etc. I then have a seam at which I can inject my mocks.


Then I can use a mock Request to check that things are configured as I expect.

The next stage would be to move the validation into the FormModel so the above code would become:



From here I can test most of the interactions in the method with our request mock to ensure our conditionals are working. Plus my form model etc would all be tested separately.



Anyway, that's what I'm thinking.

Cheers

Could you sketch/scan and attach a diagram of how this fits together?

Do you mean some UML of the Controller, RequestDecorator, FormModel and View?

It could be UML, or just any blocks and arrows with a bit of explanation on top, anything to help visualise.

The process is similar for any PHP app. If you are using PHPUnit for unit testing your php app, then you can use this guide:https://www.cloudways.com/blog/getting-started-with-unit-testing-php...