Guewen Baconnier
e3e3899143
The initial issue that triggered this rework is that the forced storage in
database was working only on writes, and was never applied on attachment
creations.
This feature is used to store small files that need to be read in a fast way in
database rather than in the object storage. Reading a file from the object
storage can take 150-200ms, which is fine for downloading a PDF file or a single
image, but not if you need 40 thumbnails.
Down the path to make a correction, I found that:
* the logic to force storage was called in `_inverse_datas`, which is not called
during a create
* odoo implemented a new method `_get_datas_related_values`, which is a model
method that receive only the data and the mimetype, and return the attachment
values and write the file to the correct place
The `_get_datas_related_values` is where we want to plug this special storage,
as it is called for create and write, and already handle the values and
conditional write. But using this method, we have less information than before
about the attachment, so let's review the different criterias we had before:
* res_model: we were using it to always store attachments related to
'ir.ui.view' in db, because assets are related to this model. However, we
don't really need to check this: we should store any javascript and css
documents in database.
* exclude res_model: we could have an exclusion list, to tell that for instance,
for mail.message, we should never store any image in db. We don't have this
information anymore, but I think it was never used and added "in case of".
Because the default configuration is "mail.mail" and "mail.message" and I
couldn't find any attachment with such res_model in any of our biggest
databases. So this is removed.
* mimetype and data (size) are the last criteria and we still have them
The new system is only based on mimetype and data size and I think it's actually
more versatile. Previously, we could set a global size and include mimetypes,
but we couldn't say "I want to store all images below 50KB and all files of type
X below 10KB". Now, we have a single system parameter with a dict configuration
(`ir_attachment.storage.force.database`) defaulting to:
{"image/": 51200, "application/javascript": 0, "text/css": 0}
Assets have a limit of zero, which means they will all be stored in the database
whatever their size is.
Overall, this is a great simplification of the module too, as the method
`_get_datas_related_values` integrates it better in the base calls of IrAttachment.
Note for upgrade:
I doubt we customized the previous system parameters which are now obsolete, but
if yes, the configuration may need to be moved to `ir_attachment.storage.force.database`.
For the record, the params were:
* mimetypes.list.storedb (default: image)
* file.maxsize.storedb (default: 51200)
* excluded.models.storedb (mail.message,mail.mail), no equivalent now
The method IrAttachment.force_storage_to_db_for_special_fields() should be called
through a migration script on existing databases to move the attachments back into
the database.
Odoo Cloud Addons
Camptocamp odoo addons used on our Cloud Platform.
Introduction
On the platform we want to achieve having:
- No data stored on the local filesystem so we can move an instance between hosts and even have several running front-ends
- Metrics read from the logs or sent to Prometheus to monitor the instances
- Logs sent to ElasticSearch-Kibana structured as JSON for better searching
For the storage, we store all the attachments on an object storage such as S3 or Swift, and we store the werkzeug sessions on Redis.
Setup
Python dependencies
Libraries that must be added in requirements.txt:
redis==2.10.5
python-json-logger==0.1.5
statsd==3.2.1
# For S3 object storage (Exoscale, AWS)
boto==2.42.0
# For Swift object storage (Openstack, OVH)
python-swiftclient==3.4.0
python-keystoneclient==3.13.0
Odoo Startup
The --load option of Odoo must contains the following addons:
session_redislogging_json
Example:
--load=web,web_kanban,session_redis,logging_json
Server Environment
The addon cloud_platform is an addon that we use for 2 things:
- validate that we setup the required environment variables depending on the running environment
- install and configure the cloud addons
For this purpose, we use the server_environment with the following envs:
prodintegrationtestdev
The exact naming is important, because the cloud_platform addon rely on these keys to know and check the running environment.
Attachments in the Object Storage S3
- prod: stored RW in the object storage
AWS_HOST: depends of the platformAWS_REGION: region's endpointAWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<project>-odoo-prod- integration:
AWS_ACCESS_KEY_ID: depends of the platformAWS_SECRET_ACCESS_KEY: depends of the platformAWS_BUCKETNAME:<project>-odoo-integration- test: attachments are stored in database
Besides, the attachment location should be set to s3 (this is
automatically done by the install methods of the cloud_platform module).
ir.config_parameterir_attachment.location:s3
Structure of bucket name is checked against environment.
It is possible to by-pass this behavior by using the following environment variable:
AWS_BUCKETNAME_UNSTRUCTURED.
Attachments in the Object Storage Swift
- prod: stored RW in the object storage
SWIFT_AUTH_URL: depends of the platformSWIFT_ACCOUNT: depends of the platformSWIFT_PASSWORD: depends of the platformSWIFT_WRITE_CONTAINER:<project>-odoo-prod- integration:
SWIFT_AUTH_URL: depends of the platformSWIFT_ACCOUNT: depends of the platformSWIFT_PASSWORD: depends of the platformSWIFT_WRITE_CONTAINER:<project>-odoo-integration- test: attachments are stored in database
Besides, the attachment location should be set to swift (this is
automatically done by the install methods of the cloud_platform module).
ir.config_parameterir_attachment.location:swift
Structure of container name is checked against environment.
It is possible to by-pass this behavior by using the following environment variable:
SWIFT_WRITE_CONTAINER_UNSTRUCTURED.
Sessions in Redis
- prod:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<project>-odoo-prod- integration:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<project>-odoo-integration- test:
ODOO_SESSION_REDIS: 1ODOO_SESSION_REDIS_HOST: depends of the platformODOO_SESSION_REDIS_PASSWORD: depends of the platformODOO_SESSION_REDIS_PREFIX:<project>-odoo-testODOO_SESSION_REDIS_EXPIRATION:86400(1 day)
JSON Logging
At least on production and integration, activate:
ODOO_LOGGING_JSON: 1- Add
logging_jsonin theserver_wide_modulesoption in the configuration file
Metrics (Statsd/Prometheus for Grafana)
Should be active at least on the production server
ODOO_STATSD: 1STATSD_CUSTOMER:<project>STATSD_ENVIRONMENT: set if you want to send metrics for a special environment which does not match with theserver_environmentSTATSD_HOST: depends of the platformSTATSD_PORT: depends of the platform
Automatic Configuration
An automatic configuration can be executed from an anthem song to configure
some parameters such as the ir_attachment.location and migrate the existing
attachments to the object storage.
From anthem, it can be called like this:
ctx.env['cloud.platform'].install(cloud_platform_kind)
Replacing cloud_platform_kind with 'exoscale' or 'ovh'
Or using one of the direct shortcuts:
ctx.env['cloud.platform'].install_exoscale()ctx.env['cloud.platform'].install_ovh()
Startup checks
At loading of the database, the addon will check if the environment variables for Redis and the object storage are set as expected for the loaded environment. It will refuse to start if anything is badly configured.
The checks can be bypassed with the environment variable
ODOO_CLOUD_PLATFORM_UNSAFE set to 1.