Sharing Memory In Gunicorn?

I have a large read-only data structure (a graph loaded in networkx, though this shouldn't be important) that I use in my web service. The webservice is built in Flask and then ser

Solution 1:

It looks like the easiest way to do this is to tell gunicorn to preload your application using the preload_app option. This assumes that you can load the data structure as a module-level variable:

from flask import Flask
from your.application import CustomDataStructure

CUSTOM_DATA_STRUCTURE = CustomDataStructure('/data/lives/here')

# @app.routes, etc.

Alternatively, you could use a memory-mapped file (if you can wrap the shared memory with your custom data structure), gevent with gunicorn to ensure that you're only using one process, or the multi-processing module to spin up your own data-structure server which you connect to using IPC.