Store files that need fast access in the database

An Object Storage read is slower than a disk of database access. It might take ~200 to 300ms to retrieve a file content. This is not an issue for attachments such as the pdf files or any attachment that we want to read on demand. But that's too slow for files needed to render a web page. We'll store in the database: * Assets (js, css, ...). As a side effect, the databases will be more portable, as assets are rebuilt frequently, storing them in the Object Storage led the integration server to try to read assets deleted since long ago * Attachments linked to Binary fields named 'image_small', 'image_medium', 'web_icon_data'. Those fields are often used on kanban views that display a lot a images and retrieving them all was then very slow (Odoo does not do async requests). The migration to S3 is no longer called during initialization of the registry: it would be too slow as we would have to define if the attachments must be kept in database or sent to S3 on each new start. It means we have to call `env['ir.attachment'].force_storage()` to run the migration.
2026-06-24 08:47:40 +00:00 · 2017-01-16 17:12:52 +01:00
parent 022f9003e8
commit 624cc0b743
4 changed files with 145 additions and 16 deletions
@@ -0,0 +1,62 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016 Camptocamp SA
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html)
+
+import logging
+import os
+
+from contextlib import closing
+
+import openerp
+
+_logger = logging.getLogger(__name__)
+
+
+def migrate(cr, version):
+    if not version:
+        return
+    cr.execute("""
+        SELECT value FROM ir_config_parameter
+        WHERE key = 'ir_attachment.location'
+    """)
+    row = cr.fetchone()
+    bucket = os.environ.get('AWS_BUCKETNAME')
+
+    if row[0] == 's3' and bucket:
+        uid = openerp.SUPERUSER_ID
+        registry = openerp.modules.registry.RegistryManager.get(cr.dbname)
+        new_cr = registry.cursor()
+        with closing(new_cr):
+            with openerp.api.Environment.manage():
+                env = openerp.api.Environment(new_cr, uid, {})
+                store_local = env['ir.attachment'].search(
+                    [('store_fname', '=like', 's3://%'),
+                     '|', ('res_model', '=', 'ir.ui.view'),
+                          ('res_field', 'in', ['image_small',
+                                               'image_medium',
+                                               'web_icon_data'])
+                     ],
+                )
+
+                _logger.info(
+                    'Moving %d attachments from S3 to DB for fast access',
+                    len(store_local)
+                )
+                for attachment_id in store_local.ids:
+                    # force re-storing the document, will move
+                    # it from the object storage to the database
+
+                    # This is a trick to avoid having the 'datas' function
+                    # fields computed for every attachment on each
+                    # iteration of the loop.  The former issue being that
+                    # it reads the content of the file of ALL the
+                    # attachments on each loop.
+                    try:
+                        env.clear()
+                        attachment = env['ir.attachment'].browse(attachment_id)
+                        _logger.info('Moving attachment %s (id: %s)',
+                                     attachment.name, attachment.id)
+                        attachment.write({'datas': attachment.datas})
+                        new_cr.commit()
+                    except:
+                        new_cr.rollback()