mirror of https://github.com/camptocamp/odoo-cloud-platform.git synced 2026-06-24 02:08:36 +00:00

T

Guewen Baconnier ddb7656abd Fix attachments stored in FS instead of object storage

Assume the following situation:

* We have installed addons base, sale and attachment_s3 (hence
base_attachment_object_storage as dependency)
* All attachments are in S3 already
* We run an upgrade of the 'base' addon, 'sale' is upgraded before
attachment_s3 in the order of loading.
* Sale updates the icon of the Sale menu
* As attachment_s3 is not loaded yet, the attachment is created in the
filestore

Now if we don't persist the filestore or use different servers, we'll
lose the images of the menus (or any attachment loaded by the
install/upgrade of an addon).

The implemented solution is to move the attachments from the filestore
to the object storage at the loading of the module. However, this
operation can take time and it shouldn't be run by 2 processes at the
same time, so we want to detect if the module is loaded during a normal odoo
startup or when some addons have been upgraded. There is nothing anymore
at this point which allow us to know that modules just have been
upgraded except... in the caller frame (load_modules). We have to rely
on the inpect module and get the caller frame, which is not recommended,
but seems the only way, besides, it's not called often and if
_register_hook was called from another place, it would have no effect
(unless the other place has a variable 'update_module' too).

2018-06-13 17:15:45 +02:00

attachment_s3

Fix attachments stored in FS instead of object storage

2018-06-13 17:15:45 +02:00

cloud_platform

Allow to use a Redis Sentinel

2017-05-16 09:07:30 +02:00

delivery_carrier_label_s3

Fix calls on multi instead of record

2017-04-26 15:58:39 +02:00

logging_json

Make pylint happy

2016-11-03 16:58:36 +01:00

monitoring_log_requests

Make pylint happy

2016-11-03 16:58:36 +01:00

monitoring_statsd

Set statsd customer

2016-11-04 13:39:50 +01:00

monitoring_status

Expire sessions generated by health checks quickly

2018-01-04 09:27:50 +01:00

session_redis

Merge pull request #25 from guewen/monitoring-short-living-sessions

2018-01-11 08:20:39 +01:00

.gitignore

Add base files

2016-08-31 15:31:44 +02:00

.travis.yml

Fix server_environment_files error

2016-11-03 17:07:47 +01:00

LICENSE

Add base files

2016-08-31 15:31:44 +02:00

oca_dependencies.txt

[ADD] OCA dependencies

2017-02-15 13:58:44 +01:00

README.md

Document --load option

2016-11-10 16:14:19 +01:00

requirements.txt

Add monitoring_statsd in cloud_platform

2016-11-04 10:30:10 +01:00

README.md

Cloud Platform

Camptocamp odoo addons used on our Cloud Platform.

Introduction

On the platform we want to achieve having:

No data stored on the local filesystem so we can move an instance between hosts and even have several running front-ends
Metrics read from the logs or sent to Prometheus to monitor the instances
Logs sent to ElasticSearch-Kibana structured as JSON for better searching

For the storage, we store all the attachments on a object storage such as S3 or a S3 compatible one, and we store the werkzeug sessions on Redis.

Setup

Python dependencies

Libraries that must be added in requirements.txt:

boto==2.42.0
redis==2.10.5
python-json-logger==0.1.5
statsd==3.2.1

Odoo Startup

The --load option of Odoo must contains the following addons:

attachment_s3
session_redis
logging_json

Example:

--load=web,web_kanban,attachment_s3,session_redis,logging_json

Server Environment

The server environments in server_environment_files must be at least:

prod
integration
test
dev

The exact naming is important, because the cloud_platform addon rely on these keys to know and check the running environment.

Attachments in the Object Storage

prod: stored RW in the object storage
AWS_HOST: depends of the platform
AWS_ACCESS_KEY_ID: depends of the platform
AWS_SECRET_ACCESS_KEY: depends of the platform
AWS_BUCKETNAME: <client>-odoo-prod
integration:
AWS_HOST: depends of the platform
AWS_ACCESS_KEY_ID: depends of the platform
AWS_SECRET_ACCESS_KEY: depends of the platform
AWS_BUCKETNAME: <client>-odoo-integration
test: attachments are stored in database

Besides, the attachment location should be set to s3 (but this is automatically done by the install methods of the cloud_platform module.

ir.config_parameter ir_attachment.location: s3

Sessions in Redis

prod:
ODOO_SESSION_REDIS: 1
ODOO_SESSION_REDIS_HOST: depends of the platform
ODOO_SESSION_REDIS_PASSWORD: depends of the platform
ODOO_SESSION_REDIS_PREFIX: <client>-odoo-prod
integration:
ODOO_SESSION_REDIS: 1
ODOO_SESSION_REDIS_HOST: depends of the platform
ODOO_SESSION_REDIS_PASSWORD: depends of the platform
ODOO_SESSION_REDIS_PREFIX: <client>-odoo-integration
test:
ODOO_SESSION_REDIS: 1
ODOO_SESSION_REDIS_HOST: depends of the platform
ODOO_SESSION_REDIS_PASSWORD: depends of the platform
ODOO_SESSION_REDIS_PREFIX: <client>-odoo-test
ODOO_SESSION_REDIS_EXPIRATION: 86400 (1 day)

JSON Logging

At least on production and integration, activate:

ODOO_LOGGING_JSON: 1
Add logging_json in the server_wide_modules option in the configuration file

Metrics (Statsd/Prometheus for Grafana)

Should be active at least on the production server

ODOO_STATSD: 1
STATSD_CUSTOMER: <client>
STATSD_ENVIRONMENT: set if you want to send metrics for a special environment which does not match with the server_environment
STATSD_HOST: depends of the platform
STATSD_PORT: depends of the platform

Automatic Configuration

Calling ctx.env['cloud.platform'].install_exoscale() in an anthem song will configure some parameters such as the ir_attachment.location and migrate the existing attachments to the object storage.

Startup checks

At loading of the database, the addon will check if the environment variables for Redis and the object storage are set as expected for the loaded environment. It will refuse to start if anything is badly configured.

The checks can be bypassed with the environment variable ODOO_CLOUD_PLATFORM_UNSAFE set to 1.