Guewen Baconnier
5d2fdeafb1
Assume the following situation: * We have installed addons base, sale and attachment_s3 (hence base_attachment_object_storage as dependency) * All attachments are in S3 already * We run an upgrade of the 'base' addon, 'sale' is upgraded before attachment_s3 in the order of loading. * Sale updates the icon of the Sale menu * As attachment_s3 is not loaded yet, the attachment is created in the filestore Now if we don't persist the filestore or use different servers, we'll lose the images of the menus (or any attachment loaded by the install/upgrade of an addon). The implemented solution is to move the attachments from the filestore to the object storage at the loading of the module. However, this operation can take time and it shouldn't be run by 2 processes at the same time, so we want to detect if the module is loaded during a normal odoo startup or when some addons have been upgraded. There is nothing anymore at this point which allow us to know that modules just have been upgraded except... in the caller frame (load_modules). We have to rely on the inpect module and get the caller frame, which is not recommended, but seems the only way, besides, it's not called often and if _register_hook was called from another place, it would have no effect (unless the other place has a variable 'update_module' too).
Cloud Platform
Camptocamp odoo addons used on our Cloud Platform.
Introduction
On the platform we want to achieve having:
- No data stored on the local filesystem so we can move an instance between hosts and even have several running front-ends
- Metrics read from the logs or sent to Prometheus to monitor the instances
- Logs sent to ElasticSearch-Kibana structured as JSON for better searching
For the storage, we store all the attachments on a object storage such as S3 or Swift, and we store the werkzeug sessions on Redis.
Two providers are available for the Cloud Platform, Exoscale based in Switzerland and OVH in France.
The main difference between the two is the Object Store they use :
- Exoscale uses S3
- OVH uses Swift
Setup
Python dependencies
Libraries that must be added in requirements.txt:
redis==2.10.5
python-json-logger==0.1.5
statsd==3.2.1
# For S3 object storage (Exoscale, AWS)
boto==2.42.0
# For Swift object storage (OVH)
python-swiftclient==3.4.0
python-keystoneclient==3.13.0
Odoo Startup
The --load option of Odoo must contains the following addons:
attachment_s3orattachment_swiftdepending of the provider used.session_redislogging_json
Example:
--load=web,web_kanban,attachment_s3,session_redis,logging_json
--load=web,web_kanban,attachment_swift,session_redis,logging_json
Server Environment
The server environments in server_environment_files must be at least:
prodintegrationtestdev
The exact naming is important, because the cloud_platform addon rely on these keys to know and check the running environment.
Attachments in the Object Storage S3
- prod: stored RW in the object storage
AWS_HOST: depends of the platformAWS_REGION: depends of the platformAWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<client>-odoo-prod- integration:
AWS_HOST: depends of the platformAWS_REGION: depends of the platformAWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<client>-odoo-integration- test: attachments are stored in database
Besides, the attachment location should be set to s3 (but this is
automatically done by the install methods of the cloud_platform module.
ir.config_parameterir_attachment.location:s3
Attachments in the Object Storage Swift
- prod: stored RW in the object storage
SWIFT_AUTH_URL: depends of the platformSWIFT_ACCOUNT: depends of the platformSWIFT_PASSWORD: depends of the platformSWIFT_WRITE_CONTAINER:<client>-odoo-prod- integration:
SWIFT_AUTH_URL: depends of the platformSWIFT_ACCOUNT: depends of the platformSWIFT_PASSWORD: depends of the platformSWIFT_WRITE_CONTAINER:<client>-odoo-integration- test: attachments are stored in database
Besides, the attachment location should be set to swift (but this is
automatically done by the install methods of the cloud_platform module.
ir.config_parameterir_attachment.location:swift
Sessions in Redis
- prod:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-prod- integration:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-integration- test:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<client>-odoo-testODOO_SESSION_REDIS_EXPIRATION:86400(1 day)
JSON Logging
At least on production and integration, activate:
ODOO_LOGGING_JSON: 1- Add
logging_jsonin theserver_wide_modulesoption in the configuration file
Metrics (Statsd/Prometheus for Grafana)
Should be active at least on the production server
ODOO_STATSD: 1STATSD_CUSTOMER:<client>STATSD_ENVIRONMENT: set if you want to send metrics for a special environment which does not match with theserver_environmentSTATSD_HOST: depends of the platformSTATSD_PORT: depends of the platform
Automatic Configuration
An automatic configuration can be executed from an anthem song to configure
some parameters such as the ir_attachment.location and migrate the existing
attachments to the object storage.
It can be called like this:
ctx.env['cloud.platform'].install(cloud_platform_kind)
Replacing cloud_platform_kind with 'exoscale' or 'ovh'
Or using one of the direct shortcuts:
ctx.env['cloud.platform'].install_exoscale()ctx.env['cloud.platform'].install_ovh()
Startup checks
At loading of the database, the addon will check if the environment variables for Redis and the object storage are set as expected for the loaded environment. It will refuse to start if anything is badly configured.
The checks can be bypassed with the environment variable
ODOO_CLOUD_PLATFORM_UNSAFE set to 1.