No se puede manejar DeadlineExceededError durante el uso de UrlFetch

Question

Tengo esta clase de utilidad básica recupera (posiblemente) URL acortadas en paralelo y devuelve un diccionario que tiene URL finales. Utiliza la funcionalidad wait_any que se describió en este blog post.

class UrlFetcher(object): 

    @classmethod 
    def fetch_urls(cls,url_list): 
    rpcs = [] 
    for url in url_list: 
     rpc = urlfetch.create_rpc(deadline=5.0) 
     urlfetch.make_fetch_call(rpc, url,method = urlfetch.HEAD) 
     rpcs.append(rpc) 

    result = {} 
    while len(rpcs) > 0: 
     rpc = apiproxy_stub_map.UserRPC.wait_any(rpcs) 
     rpcs.remove(rpc) 
     request_url = rpc.request.url() 
     try: 
     final_url = rpc.get_result().final_url 
     except AttributeError: 
     final_url = request_url 
     except DeadlineExceededError: 
     logging.error('Handling DeadlineExceededError for url: %s' %request_url) 
     final_url = None 
     except (DownloadError,InvalidURLError): 
     final_url = None   
     except UnicodeDecodeError: #Funky url with very evil characters 
     final_url = unicode(rpc.get_result().final_url,'utf-8') 

     result[request_url] = final_url 

    logging.info('Returning results: %s' %result) 
    return result 

Incluso si trato de manejar DeadlineExceededError, los registros de la aplicación muestran lo contrario.

2011-04-20 17:06:17.755 
UrlFetchWorker started 
E 2011-04-20 17:06:22.769 
The API call urlfetch.Fetch() took too long to respond and was cancelled. 
Traceback (most recent call last): 
    File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__ 
    handler.post(*groups) 
    File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/handlers/taskworker.py", line 80, in post 
    result_dict = UrlFetcher.fetch_urls(fetch_targets) 
    File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/utils/rpc.py", line 98, in fetch_urls 
    final_url = rpc.get_result().final_url 
    File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result 
    return self.__get_result_hook(self) 
    File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 345, in _get_fetch_result 
    rpc.check_success() 
    File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 558, in check_success 
    self.__rpc.CheckSuccess() 
    File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess 
    raise self.exception 
DeadlineExceededError: The API call urlfetch.Fetch() took too long to respond and was cancelled. 
W 2011-04-20 17:06:22.858 
Found 1 RPC request(s) without matching response (presumably due to timeouts or other errors) 

¿Qué es lo que falta aquí? ¿Hay alguna otra manera de manejar DeadlineExceededError?

¿O hay diferentes tipos de DeadlineExceededErrors y estoy importando el incorrecto? Estoy usando: from google.appengine.runtime import DeadlineExceededError

Answer 1

por la documentación en línea para google.appengine.runtime.DeadlineExceededError:

Exception raised when the request reaches its overall time limit.

Not to be confused with runtime.apiproxy_errors.DeadlineExceededError. That one is raised when individual API calls take too long.

Ésta es una buena demostración de por qué debería utilizar las importaciones calificados (from google.appengine import runtime, después hacer referencia a runtime.DeadlineExceededError), también!

Answer 2

Como has adivinado y otros señalaron, que desea una diferente DeadlineExceededError:

De https://developers.google.com/appengine/articles/deadlineexceedederrors, de junio de 2012:

Currently, there are several errors named DeadlineExceededError for the Python runtime:>

google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests;

google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option;

google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

Catching google.appengine.api.urlfetch_errors.DeadlineExceededError parece hacer el truco para mí. También digno de mención que (al menos en el servidor de aplicaciones dev 1.7.1), el urlfetch_errors.DeadlineExceededError es una subclase de DownloadError, lo cual tiene sentido:

class DeadlineExceededError(DownloadError): 
    """Raised when we could not fetch the URL because the deadline was exceeded. 

    This can occur with either the client-supplied 'deadline' or the system 
    default, if the client does not supply a 'deadline' parameter. 
    """