Guewen Baconnier
624cc0b743
An Object Storage read is slower than a disk of database access. It might take ~200 to 300ms to retrieve a file content. This is not an issue for attachments such as the pdf files or any attachment that we want to read on demand. But that's too slow for files needed to render a web page. We'll store in the database: * Assets (js, css, ...). As a side effect, the databases will be more portable, as assets are rebuilt frequently, storing them in the Object Storage led the integration server to try to read assets deleted since long ago * Attachments linked to Binary fields named 'image_small', 'image_medium', 'web_icon_data'. Those fields are often used on kanban views that display a lot a images and retrieving them all was then very slow (Odoo does not do async requests). The migration to S3 is no longer called during initialization of the registry: it would be too slow as we would have to define if the attachments must be kept in database or sent to S3 on each new start. It means we have to call `env['ir.attachment'].force_storage()` to run the migration.
Cloud Platform
Camptocamp odoo addons used on our Cloud Platform.
Introduction
On the platform we want to achieve having:
- No data stored on the local filesystem so we can move an instance between hosts and even have several running front-ends
- Metrics read from the logs or sent to Prometheus to monitor the instances
- Logs sent to ElasticSearch-Kibana structured as JSON for better searching
For the storage, we store all the attachments on a object storage such as S3 or a S3 compatible one, and we store the werkzeug sessions on Redis.
Setup
Python dependencies
Libraries that must be added in requirements.txt:
boto==2.42.0
redis==2.10.5
python-json-logger==0.1.5
statsd==3.2.1
Odoo Startup
The --load option of Odoo must contains the following addons:
attachment_s3session_redislogging_json
Example:
--load=web,web_kanban,attachment_s3,session_redis,logging_json
Server Environment
The server environments in server_environment_files must be at least:
prodintegrationtestdev
The exact naming is important, because the cloud_platform addon rely on these keys to know and check the running environment.
Attachments in the Object Storage
- prod: stored RW in the object storage
AWS_HOST: depends of the platformAWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<client>-odoo-prod- integration:
AWS_HOST: depends of the platformAWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<client>-odoo-integration- test: attachments are stored in database
Besides, the attachment location should be set to s3 (but this is
automatically done by the install methods of the cloud_platform module.
ir.config_parameterir_attachment.location:s3
Sessions in Redis
- prod:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-prod- integration:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-integration- test:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-testODOO_SESSION_REDIS_EXPIRATION:86400(1 day)
JSON Logging
At least on production and integration, activate:
ODOO_LOGGING_JSON: 1- Add
logging_jsonin theserver_wide_modulesoption in the configuration file
Metrics (Statsd/Prometheus for Grafana)
Should be active at least on the production server
ODOO_STATSD: 1STATSD_CUSTOMER:<client>STATSD_ENVIRONMENT: set if you want to send metrics for a special environment which does not match with theserver_environmentSTATSD_HOST: depends of the platformSTATSD_PORT: depends of the platform
Automatic Configuration
Calling ctx.env['cloud.platform'].install_exoscale() in an
anthem song will configure some parameters such as the
ir_attachment.location and migrate the existing attachments to the
object storage.
Startup checks
At loading of the database, the addon will check if the environment variables for Redis and the object storage are set as expected for the loaded environment. It will refuse to start if anything is badly configured.
The checks can be bypassed with the environment variable
ODOO_CLOUD_PLATFORM_UNSAFE set to 1.