Consultas complejas de MySQL que todavía usan filesort aunque existen índices

Question

Tengo una tabla de Joomla con miles de filas de contenido (aproximadamente 3 millones). Tengo problemas para reescribir las consultas de la base de datos para que sean lo más rápidas posible al consultar las tablas.

Aquí es mi consulta completa:

SELECT cc.title AS category, a.id, a.title, a.alias, a.title_alias, a.introtext, a.fulltext, a.sectionid, a.state, a.catid, a.created, a.created_by, a.created_by_alias, a.modified, a.modified_by, a.checked_out, a.checked_out_time, a.publish_up, a.publish_down, a.attribs, a.hits, a.images, a.urls, a.ordering, a.metakey, a.metadesc, a.access, CASE WHEN CHAR_LENGTH(a.alias) THEN CONCAT_WS(":", a.id, a.alias) ELSE a.id END AS slug, CASE WHEN CHAR_LENGTH(cc.alias) THEN CONCAT_WS(":", cc.id, cc.alias) ELSE cc.id END AS catslug, CHAR_LENGTH(a.`fulltext`) AS readmore, u.name AS author, u.usertype, g.name AS groups, u.email AS author_email 
FROM j15_content AS a 
LEFT JOIN j15_categories AS cc 
ON a.catid = cc.id 
LEFT JOIN j15_users AS u 
ON u.id = a.created_by 
LEFT JOIN j15_groups AS g 
ON a.access = g.id 
WHERE 1 
AND a.access <= 0 
AND a.catid = 108 
AND a.state = 1 
AND (publish_up = '0000-00-00 00:00:00' OR publish_up <= '2012-02-08 00:16:26') 
AND (publish_down = '0000-00-00 00:00:00' OR publish_down >= '2012-02-08 00:16:26') 
ORDER BY a.title, a.created DESC 
LIMIT 0, 10 

Aquí está la salida de un explico:

+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-----------------------------+ 
| id | select_type | table | type | possible_keys           | key  | key_len | ref      | rows | Extra      | 
+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-----------------------------+ 
| 1 | SIMPLE  | a  | ref | idx_access,idx_state,idx_catid,idx_access_state_catid | idx_catid | 4  | const      | 3108187 | Using where; Using filesort | 
| 1 | SIMPLE  | cc | const | PRIMARY            | PRIMARY | 4  | const      |  1 |        | 
| 1 | SIMPLE  | u  | eq_ref | PRIMARY            | PRIMARY | 4  | database.a.created_by  |  1 |        | 
| 1 | SIMPLE  | g  | eq_ref | PRIMARY            | PRIMARY | 1  | database.a.access   |  1 |        | 
+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-----------------------------+ 

Y para mostrar lo que existen índices, SHOW INDEX DE j15_content:

+-------------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 
| Table  | Non_unique | Key_name    | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | 
+-------------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 
| j15_content |   0 | PRIMARY    |   1 | id   | A   |  3228356 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_section   |   1 | sectionid | A   |   2 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_access    |   1 | access  | A   |   1 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_checkout   |   1 | checked_out | A   |   2 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_state    |   1 | state  | A   |   2 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_catid    |   1 | catid  | A   |   6 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_createdby   |   1 | created_by | A   |   1 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | title     |   1 | title  | A   |  201772 |  4 | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_access_state_catid |   1 | access  | A   |   1 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_access_state_catid |   2 | state  | A   |   2 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_access_state_catid |   3 | catid  | A   |   7 |  NULL | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_title_created  |   1 | title  | A   |  3228356 |  8 | NULL |  | BTREE  |   | 
| j15_content |   1 | idx_title_created  |   2 | created  | A   |  3228356 |  NULL | NULL |  | BTREE  |   | 
+-------------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 

Como puede ver, hay algunos datos tomados de la base de datos. Ahora lo he probado simplificando la consulta de que el problema real reside en la cláusula ORDER BY. Sin necesidad de ordenar los resultados, la consulta es muy sensible, aquí es una explicación:

+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-------------+ 
| id | select_type | table | type | possible_keys           | key  | key_len | ref      | rows | Extra  | 
+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-------------+ 
| 1 | SIMPLE  | a  | ref | idx_access,idx_state,idx_catid,idx_access_state_catid | idx_catid | 4  | const      | 3108187 | Using where | 
| 1 | SIMPLE  | cc | const | PRIMARY            | PRIMARY | 4  | const      |  1 |    | 
| 1 | SIMPLE  | u  | eq_ref | PRIMARY            | PRIMARY | 4  | database.a.created_by  |  1 |    | 
| 1 | SIMPLE  | g  | eq_ref | PRIMARY            | PRIMARY | 1  | database.a.access   |  1 |    | 
+----+-------------+-------+--------+-------------------------------------------------------+-----------+---------+---------------------------+---------+-------------+ 

Como se puede ver que es el filesort fatal que está matando el servidor. Con tantas filas, estoy haciendo todo lo posible para optimizar todo a través de los índices, pero algo no está bien con esto. Cualquier contribución será muy apreciada.

intentado usar el índice de Fuerza fue en vano:

explain  SELECT cc.title AS category, a.id, a.title, a.alias, a.title_alias, a.introtext, a.fulltext, a.sectionid, a.state, a.catid, a.created, a.created_by, a.created_by_alias, a.modified, a.modified_by, a.checked_out, a.checked_out_time, a.publish_up, a.publish_down, a.attribs, a.hits, a.images, a.urls, a.ordering, a.metakey, a.metadesc, a.access, CASE WHEN CHAR_LENGTH(a.alias) THEN CONCAT_WS(":", a.id, a.alias) ELSE a.id END AS slug, CASE WHEN CHAR_LENGTH(cc.alias) THEN CONCAT_WS(":", cc.id, cc.alias) ELSE cc.id END AS catslug, CHAR_LENGTH(a.`fulltext`) AS readmore, u.name AS author, u.usertype, g.name AS groups, u.email AS author_email 
    ->  FROM bak_content AS a 
    ->  FORCE INDEX (idx_title_created) 
    ->  LEFT JOIN bak_categories AS cc 
    ->  ON a.catid = cc.id 
    ->  LEFT JOIN bak_users AS u 
    ->  ON u.id = a.created_by 
    ->  LEFT JOIN bak_groups AS g 
    ->  ON a.access = g.id 
    ->  WHERE 1 
    ->  AND a.access <= 0 
    ->  AND a.catid = 108 
    ->  AND a.state = 1 
    ->  AND (publish_up = '0000-00-00 00:00:00' OR publish_up <= '2012-02-08 
    ->  AND (publish_down = '0000-00-00 00:00:00' OR publish_down >= '2012-0 
    ->  ORDER BY a.title, a.created DESC 
    ->  LIMIT 0, 10; 

Produce:

+----+-------------+-------+--------+---------------+---------+---------+------- 
| id | select_type | table | type | possible_keys | key  | key_len | ref 
+----+-------------+-------+--------+---------------+---------+---------+------- 
| 1 | SIMPLE  | a  | ALL | NULL   | NULL | NULL | NULL 
| 1 | SIMPLE  | cc | const | PRIMARY  | PRIMARY | 4  | const 
| 1 | SIMPLE  | u  | eq_ref | PRIMARY  | PRIMARY | 4  | database 
| 1 | SIMPLE  | g  | eq_ref | PRIMARY  | PRIMARY | 1  | database 
+----+-------------+-------+--------+---------------+---------+---------+------- 

Answer 1

veces MySQL tiene problemas para encontrar el índice adecuado. Puede resolver esto haciendo alusión al índice apropiado.

sintaxis Consejo: http://dev.mysql.com/doc/refman/4.1/en/index-hints.html

Asegúrese de que tiene el índice derecho y ajustar su rendimiento mediante la experimentación.

¡Salud!

Answer 2

Se puede tratar esta variación:

SELECT cc.title AS category, ... 
FROM 
    (SELECT * 
     FROM j15_content AS a 
       USE INDEX (title)    --- with and without the hint 
     WHERE 1 
     AND a.access <= 0 
     AND a.catid = 108 
     AND a.state = 1 
     AND (publish_up = '0000-00-00 00:00:00' 
      OR publish_up <= '2012-02-08 00:16:26') 
     AND (publish_down = '0000-00-00 00:00:00' 
      OR publish_down >= '2012-02-08 00:16:26') 
     ORDER BY a.title, a.created DESC 
     LIMIT 0, 10 
    ) AS a 
    LEFT JOIN j15_categories AS cc 
    ON a.catid = cc.id 
    LEFT JOIN j15_users AS u 
    ON u.id = a.created_by 
    LEFT JOIN j15_groups AS g 
    ON a.access = g.id 

un índice en (catid, state, title) sería aún mejor, creo.

Answer 3

Tal vez tratando esto podría ayudar:

CREATE INDEX idx_catid_title_created ON j15_content (catid,title(8),created); 
DROP INDEX idx_catid ON j15_content; 

Answer 4

Aparentemente, este no puede ser resuelto razonablemente utilizando un índice, pistas o reestructuración de la propia consulta.

La razón por la que esto es lento es el hecho de que requiere un archivador de filas de 2M que en realidad lleva mucho tiempo. Si hace un acercamiento en el orden, se especifica como ORDER BY a.title, a.created DESC. El problema es la combinación de clasificar en más de 1 columna y tener una parte DESC. Mysql no admite índices descendentes (la palabra clave DESC es compatible con el CREATE INDEX statement, pero solo para uso futuro).

La solución sugerida es crear una columna adicional 'reverse_created' que se rellene automáticamente de tal forma que su consulta pueda usar ORDER BY a.title, a.reverse_created. Entonces lo completa con max_time - created_time.Luego, cree un índice en esa combinación y (si es necesario) especifique ese índice como una sugerencia.

Hay un par de muy buenos artículos de blog sobre este tema que explican esta mucho mejor y con ejemplos:

• http://www.mysqlperformanceblog.com/2006/05/09/descending-indexing-and-loose-index-scan/
• http://www.mysqlperformanceblog.com/2007/02/16/using-index-for-order-by-vs-restricting-number-of-rows/

-Update- Usted debe ser capaz para hacer una prueba rápida al eliminar la parte "DESC" del pedido en su consulta. Los resultados serán funcionalmente incorrectos, pero debería usar el índice existente que tiene (o de lo contrario la fuerza debería funcionar).

Answer 5

Ha intentado aumentando estos valores tmp_table_size y max_heap_table_size:

Hay una breve explicación here y también enlaces a los detalles de cada uno de ellos.

Espero que esto ayude!

Answer 6

espero que esto es sintácticamente correcta

SELECT 
    cc.title AS category, 
    a.id, a.title, a.alias, a.title_alias, 
    a.introtext, a.fulltext, a.sectionid, 
    a.state, a.catid, a.created, a.created_by, 
    a.created_by_alias, a.modified, a.modified_by, 
    a.checked_out, a.checked_out_time, 
    a.publish_up, a.publish_down, a.attribs, 
    a.hits, a.images, a.urls, a.ordering, a.metakey, 
    a.metadesc, a.access, 
    CASE WHEN CHAR_LENGTH(a.alias) THEN CONCAT_WS(":", a.id, a.alias) ELSE a.id END AS slug, 
    CASE WHEN CHAR_LENGTH(cc.alias) THEN CONCAT_WS(":", cc.id, cc.alias) ELSE cc.id END AS catslug, CHAR_LENGTH(a.`fulltext`) AS readmore, 
    u.name AS author, u.usertype, g.name AS groups, u.email AS author_email 
FROM 
(
    SELECT aa.* 
    FROM 
    (
     SELECT id FROM 
     FROM j15_content 
     WHERE catid=108 AND state=1 
     AND a.access <= 0 
     AND (publish_up = '0000-00-00 00:00:00' OR publish_up <= '2012-02-08 00:16:26') 
     AND (publish_down = '0000-00-00 00:00:00' OR publish_down >= '2012-02-08 00:16:26') 
     ORDER BY title,created DESC 
     LIMIT 0,10 
    ) needed_keys 
    LEFT JOIN j15_content aa USING (id) 
) a 
LEFT JOIN j15_categories AS cc ON a.catid = cc.id 
LEFT JOIN j15_users AS u ON a.created_by = u.id 
LEFT JOIN j15_groups AS g ON a.access = g.id; 

Tendrá un índice de apoyo para needed_keys subconsulta

ALTER TABLE j15_content ADD INDEX subquery_ndx (catid,state,access,title,created); 

darle una oportunidad !!!