I have been struggling to understand why my index queue isnt clearing a fast as I would like.
I am running a new website based on V8.1.
The site is being populated with about 100 pages per day.
I found that the page index queue had reached over 1 million records, this got me looking as to why. I have managed to clear the queue down using the reset all jobs button on the Automation page.
I am running the indexing as a queueable job on a cron schedule. I have found that every couple of days the queue refills with pages to index and the processes the queue at the rate of 1 page every 5 minutes.I have added in a user config for job_queue_batch and increase the default of to to 50. Which had no effect. I then changed the code so that I set the batch size to 50
// $messages = $q->receive($job->getJobQueueBatchSize());
$messages = $q->receive(50);
I finally reloaded php7.0-fpm incase the php wasnt loading. This still had no effect.
I very much suspect that there is an issue with zendqueue. Having done some research I found mention to an issue called short polling were zendqueue may return 1 or even none from a queue. This was a while ago and may not be the issue here.

As the site grows the page count is only going to increase, the plan was to do a full reindex on a weekly basis and incrementals daily. But it is looking like even at this point with about 900 pages the reindex is taking 3 to 4 days.

Has anyone else encountered this issue and how was it fixed?

Thanks

Martyn
https://www.la-maison-de-cordelia.com...