blog.plee.me About software, technology and random things

7Jun/150

Memory Leaks / Problems with Long-Running Symfony / Doctrine Console Applications

Hi!

I recently built several console applications that have to keep running as daemons as part of a more complex website. As soon as I added Doctrine for its highly comfortable ORM functionality, I noticed a significant increase in the applications' memory usage, which made sense because it would load Doctrine's code in order to use it. The worst part of it, however, was that the processes kept eating up more and more memory the longer they ran and with each Doctrine query they executed. Finally the processes ended up being killed by Linux's OOM Killer due to the high amount of memory that they wanted allocated for them.

Clearing the Doctrine Entity Manager ($entityManager->clear()) and triggering PHP's garbage collection manually did not help at all. So I assumed it had something to do with the data that Doctrine accumulates in the background.

During my research I finally stumbled upon this Stack Overflow question: Memory leaks Symfony2 Doctrine2 / exceed memory limit

Apparently, Doctrine uses an SQL Logger to log each and every query it uses when running in debug mode.

Due to the nature of Symfony console applications starting up in the dev environment by default when launching them from the command line, it also kept enabling the debug mode. That in turn enabled the SQL Logger which gathered more and more data and kept it in memory.

I decided to launch the processes by simply manually disabling the debug flag. Using the dev environment is not a problem for me, so I just did the following:

$ php app/console custom:command --no-debug

You could also set the environment variable SYMFONY_DEBUG to 0 before launching the command without the --no-debug flag, as it also respects its value - refer to app/console's source code.

Alternatively, you could start it in a non-dev environment:

$ php app/console custom:command --env=prod

or

$ php app/console custom:command -e prod

or just set the environment variable SYMFONY_ENV before launching it.

This way, the SQL Logger is not enabled.

My processes have been running at stable memory usage size ever since.

Another way would be to deactivate the SQL Logger in the code by principle, but that would be defeating the purpose of the nice built-in features of enabling/disabling the debugging mode, so I rather chose the first path. It might come in handy sooner or later after all, and reverting those changes just to re-enable the SQL Logger would be an unnecessary pain.

I hope this was helpful to you. It certainly took me some time to get behind it.