Optimizar la consulta de MySQL en las tablas JOINed con GROUP BY y ORDER BY sin usar consultas anidadas

Question

Esto me parece un poco de una pregunta SQL para principiantes, pero aquí va. Esto es lo que intento hacer:

• unir tres tablas, productos, etiquetas y una tabla de enlaces.
• agregado de las etiquetas en un solo campo delimitado por comas (de ahí el GROUP_CONCAT y el GRUPO POR)
• límite de los resultados (a 30)
• tener los resultados en el orden de la fecha 'creado'
• evitar el uso de subconsultas siempre que sea posible, ya que son particularmente desagradable al código utilizando un marco Active Record

que he descrito las tablas implicadas en la parte inferior de este post, pero aquí es la consulta que estoy realizando

SELECT p.*, GROUP_CONCAT(pt.name) 
    FROM products p 
LEFT JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
LEFT JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
GROUP BY p.id 
ORDER BY p.created 
    LIMIT 30; 

Hay aproximadamente 280,000 productos, 130 etiquetas, 524,000 registros de enlace y ANALIZADO las tablas. El problema es que está tardando más de 80 segundos en ejecutarse (con hardware decente), lo que me parece mal.

Aquí está el resultado de explicar:

id select_type table type possible_keys     key        key_len ref     rows Extra 
1 SIMPLE   p  index NULL        created       4   NULL     30 "Using temporary" 
1 SIMPLE   pt4p  ref  idx_product_tags_for_products idx_product_tags_for_products 3   s.id     1  "Using index" 
1 SIMPLE   pt  eq_ref PRIMARY       PRIMARY       4   pt4p.product_tag_id 1  

Creo que está haciendo las cosas en el orden equivocado, es decir, ordenando los resultados después de la unión, usando una tabla temporal grande y, a continuación, limitando. El plan de consulta en la cabeza sería algo como esto:

• ordenar la tabla de productos a través del 'creado' clave
• paso a través de cada fila, IZQUIERDA unirse contra las otras mesas hasta el límite del 30 ha sido alcanzado.

Esto suena simple, pero no parece funcionar así - ¿me falta algo?

---

CREATE TABLE `products` (
    `id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT, 
    `title` varchar(255) COLLATE utf8_unicode_ci NOT NULL, 
    `rating` float NOT NULL, 
    `created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', 
    `last_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, 
    `active` tinyint(1) NOT NULL, 
    PRIMARY KEY (`id`), 
    KEY `created` (`created`), 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci 

CREATE TABLE `product_tags_for_products` (
    `id` bigint(20) NOT NULL AUTO_INCREMENT, 
    `product_id` mediumint(8) unsigned NOT NULL, 
    `product_tag_id` int(10) unsigned NOT NULL, 
    PRIMARY KEY (`id`), 
    UNIQUE KEY `idx_product_tags_for_products` (`product_id`,`product_tag_id`), 
    KEY `product_tag_id` (`product_tag_id`), 
    CONSTRAINT `product_tags_for_products_ibfk_1` FOREIGN KEY (`product_id`) REFERENCES `products` (`id`), 
    CONSTRAINT `product_tags_for_products_ibfk_2` FOREIGN KEY (`product_tag_id`) REFERENCES `product_tags` (`id`) 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci 


CREATE TABLE `product_tags` (
    `id` int(10) unsigned NOT NULL AUTO_INCREMENT, 
    `name` varchar(100) COLLATE utf8_unicode_ci NOT NULL, 
    PRIMARY KEY (`id`), 
    UNIQUE KEY `name` (`name`) 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci 

---

Actualizado con información de perfiles a petición de Salman R:

Status, 
    Duration,CPU_user,CPU_system,Context_voluntary,Context_involuntary,Block_ops_in,Block_ops_out,Messages_sent,Messages_received,Page_faults_major,Page_faults_minor,Swaps,Source_function,Source_file,Source_line 
starting,    
    0.000124,0.000106,0.000015,0,0,0,0,0,0,0,0,0,NULL,NULL,NULL 
"Opening tables",  
    0.000022,0.000020,0.000003,0,0,0,0,0,0,0,0,0,"unknown function",sql_base.cc,4519 
"System lock", 
    0.000007,0.000004,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",lock.cc,258 
"Table lock", 
    0.000011,0.000009,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",lock.cc,269 
init,   
    0.000055,0.000054,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,2524 
optimizing,  
    0.000008,0.000006,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,833 
statistics,  
    0.000116,0.000051,0.000066,0,0,0,0,0,0,0,1,0,"unknown function",sql_select.cc,1024 
preparing,  
    0.000027,0.000023,0.000003,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,1046 
"Creating tmp table", 
    0.000054,0.000053,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,1546 
"Sorting for group", 
    0.000018,0.000015,0.000003,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,1596 
executing,  
    0.000004,0.000002,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,1780 
"Copying to tmp table", 
    0.061716,0.049455,0.013560,0,18,0,0,0,0,0,3680,0,"unknown function",sql_select.cc,1927 
"converting HEAP to MyISAM", 
    0.046731,0.006371,0.017543,3,5,0,3,0,0,0,32,0,"unknown function",sql_select.cc,10980 
"Copying to tmp table on disk", 
10.700166,3.038211,1.191086,538,1230,1,31,0,0,0,65,0,"unknown function",sql_select.cc,11045 
"Sorting result", 
    0.777887,0.155327,0.618896,2,137,0,1,0,0,0,634,0,"unknown function",sql_select.cc,2201 
"Sending data", 
    0.000336,0.000159,0.000178,0,0,0,0,0,0,0,1,0,"unknown function",sql_select.cc,2334 
end, 
    0.000005,0.000003,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,2570 
"removing tmp table", 
    0.106382,0.000058,0.080105,4,9,0,11,0,0,0,0,0,"unknown function",sql_select.cc,10912 
end, 
    0.000015,0.000007,0.000007,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,10937 
"query end", 
    0.000004,0.000002,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_parse.cc,5083 
"freeing items", 
    0.000012,0.000012,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_parse.cc,6107 
"removing tmp table", 
    0.000010,0.000009,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_select.cc,10912 
"freeing items", 
    0.000084,0.000022,0.000057,0,1,0,0,1,0,0,0,0,"unknown function",sql_select.cc,10937 
"logging slow query", 
    0.000004,0.000001,0.000001,0,0,0,0,0,0,0,0,0,"unknown function",sql_parse.cc,1723 
"logging slow query", 
    0.000049,0.000031,0.000018,0,0,0,0,0,0,0,0,0,"unknown function",sql_parse.cc,1733 
"cleaning up", 
    0.000007,0.000005,0.000002,0,0,0,0,0,0,0,0,0,"unknown function",sql_parse.cc,1691 

Las tablas son:

Productos = 84.1MiB (hay campos adicionales en la tabla de productos que omití para mayor claridad) Etiquetas = 32KiB Linking table = 46.6MiB

Answer 1

yo intentaría limitar el número de productos a 30 primera y luego unirse con sólo 30 productos

SELECT p.*, GROUP_CONCAT(pt.name) as tags 
    FROM (SELECT p30.* FROM products p30 ORDER BY p30.created LIMIT 30) p 
LEFT JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
LEFT JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
GROUP BY p.id 
ORDER BY p.created 

Sé que dijo no subconsultas pero no explicó por qué, y No veo otra forma de resolver tu problema.

Tenga en cuenta que puede eliminar la subselección poniendo que en una vista:

CREATE VIEW v_last30products AS 
    SELECT p30.* FROM products p30 ORDER BY p30.created LIMIT 30; 

A continuación, la consulta se simplifica a:

SELECT p.*, GROUP_CONCAT(pt.name) as tags 
    FROM v_last30products p 
LEFT JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
LEFT JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
GROUP BY p.id 
ORDER BY p.created 

Otro tema, su mesa n-to-nproduct_tags_for_products

No tiene sentido, lo reestructuraría así:

CREATE TABLE `product_tags_for_products` ( 
    `product_id` mediumint(8) unsigned NOT NULL,  
    `product_tag_id` int(10) unsigned NOT NULL,  
    PRIMARY KEY (`product_id`,`product_tag_id`),  
    CONSTRAINT `product_tags_for_products_ibfk_1` FOREIGN KEY (`product_id`) REFERENCES `products` (`id`),  
    CONSTRAINT `product_tags_for_products_ibfk_2` FOREIGN KEY (`product_tag_id`) REFERENCES `product_tags` (`id`)  
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci 

Esto debería hacer que la consulta sea más rápida por:
- acortando la tecla utilizada (En InnoDB, la PK siempre se incluye en las teclas secundarias);
- Permitiéndole usar PK que debería ser más rápido que usar una clave secundaria;

Más velocidad emite
Si reemplaza el select * con sólo los campos que necesite select p.title, p.rating, ... FROM que también acelerará un poco las cosas.

Answer 2

Ah - Veo que ninguna de las teclas que GROUP BY activa son BTREE, por defecto las teclas PRIMARY son hashes. Ayuda a agrupar cuando hay un índice de ordenamiento ... de lo contrario tiene que escanear ...

Lo que quiero decir es que creo que ayudaría significativamente si agregara un índice basado en BTREE para p.id y p .creado. En ese caso, creo que el motor evitará tener que escanear/ordenar todas esas teclas para ejecutar group by order by.

Answer 3

En cuanto a la filtración en las etiquetas (que usted ha mencionado en los comentarios sobre Johan's answer), si el obvias

SELECT p.*, GROUP_CONCAT(pt.name) AS tags 
FROM products p 
    JOIN product_tags_for_products pt4p2 ON (pt4p2.product_id = p.id) 
    JOIN product_tags pt2 ON (pt2.id = pt4p2.product_tag_id) 
    LEFT JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
    LEFT JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
WHERE pt2.name IN ('some', 'tags', 'here') 
GROUP BY p.id 
ORDER BY p.created LIMIT 30 

no corre lo suficientemente rápido, siempre se puede intentar esto:

CREATE TEMPORARY TABLE products30 
    SELECT p.* 
    FROM products p 
    JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
    JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
    WHERE pt.name IN ('some', 'tags', 'here') 
    GROUP BY p.id 
    ORDER BY p.created LIMIT 30 

SELECT p.*, GROUP_CONCAT(pt.name) AS tags 
FROM products30 p 
    LEFT JOIN product_tags_for_products pt4p ON (pt4p.product_id = p.id) 
    LEFT JOIN product_tags pt ON (pt.id = pt4p.product_tag_id) 
GROUP BY p.id 
ORDER BY p.created 

(Utilicé una tabla temporal porque dijiste "no subconsultas"; no sé si son más fáciles de usar en un marco de registro activo, pero al menos es otra forma de hacerlo).

---

Ps. Una idea realmente fuera de lo común sobre su problema original: ¿sería importante si cambió la cláusula GROUP BY p.id al GROUP BY p.created, p.id? Probablemente no, pero al menos lo intentaré.