OVH's Swift applies a rate limit on the authentication.
attachment_swift authenticates again each time it has to read/write an
attachment. When running upgrades on upgrades of files or installing a
new DB, at some point, we get rejected with HTTP 429.
This commit introduces a shared storage for Swift Session. All
connections will reuses the same authentication token created the first
time a connection needs a Session.
Note: needs python-swiftclient>=3.7.0 to have
https://github.com/openstack/python-swiftclient/commit/1971ef880ff225379d4a91f00f89f323a1605eeb
Assume the following situation:
* We have installed addons base, sale and attachment_s3 (hence
base_attachment_object_storage as dependency)
* All attachments are in S3 already
* We run an upgrade of the 'base' addon, 'sale' is upgraded before
attachment_s3 in the order of loading.
* Sale updates the icon of the Sale menu
* As attachment_s3 is not loaded yet, the attachment is created in the
filestore
Now if we don't persist the filestore or use different servers, we'll
lose the images of the menus (or any attachment loaded by the
install/upgrade of an addon).
The implemented solution is to move the attachments from the filestore
to the object storage at the loading of the module. However, this
operation can take time and it shouldn't be run by 2 processes at the
same time, so we want to detect if the module is loaded during a normal odoo
startup or when some addons have been upgraded. There is nothing anymore
at this point which allow us to know that modules just have been
upgraded except... in the caller frame (load_modules). We have to rely
on the inpect module and get the caller frame, which is not recommended,
but seems the only way, besides, it's not called often and if
_register_hook was called from another place, it would have no effect
(unless the other place has a variable 'update_module' too).
When moving attachments from the filestore to an object storage, the
filesystem files will be deleted only after the commit, so if the
transaction is rollbacked, we still have the local files for another
try.
The default expiration of sessions is 7 days. With healthchecks run
every few seconds, we quickly have millions of anonymous sessions in
Redis. Allow to define a custom expiration for some sessions and set a
very short one for the monitoring requests.
The previous error message let think that you should set AWS_BUCKETNAME,
although you should set it only if you are trying to write in this
repository.
The 'base' Kanban view for res.partner in 9.0 loads the large 'image'
field in the 'search_read' but display the 'small_image'. The js code
also has to issue js calls to get the small images as they have not been
prefetched. This is noticeably slow to load when the large images are
read from S3 which is slower than a local filesystem/database.
This fix loads the 'small_image' instead of the large one.
This bug has been fixed in Odoo 10.0.
An Object Storage read is slower than a disk of database access.
It might take ~200 to 300ms to retrieve a file content.
This is not an issue for attachments such as the pdf files or any
attachment that we want to read on demand. But that's too slow for
files needed to render a web page.
We'll store in the database:
* Assets (js, css, ...). As a side effect, the databases will be more
portable, as assets are rebuilt frequently, storing them in the Object
Storage led the integration server to try to read assets deleted since
long ago
* Attachments linked to Binary fields named 'image_small',
'image_medium', 'web_icon_data'. Those fields are often used on kanban
views that display a lot a images and retrieving them all was then
very slow (Odoo does not do async requests).
The migration to S3 is no longer called during initialization of the
registry: it would be too slow as we would have to define if the
attachments must be kept in database or sent to S3 on each new start. It
means we have to call `env['ir.attachment'].force_storage()` to run the
migration.
Because it provokes serialization errors during the installation or
update of addons. Do not commit as we might commit unwanted things...
Later, we might want to add a specific, more elaborate, migration
process.
Guewen BaconnierandSimone Orsi
Timon TschanzandGitHub
jcouxandGitHub
Vincent RenavilleandGitHub