¿Se puede calcular el algoritmo SHA-1 en una secuencia? Con poca huella de memoria?

Question

Estoy buscando una forma de calcular sumas de comprobación SHA-1 de archivos muy grandes sin tener que cargarlos completamente en la memoria a la vez.

No conozco los detalles de la implementación de SHA-1 y, por lo tanto, me gustaría saber si es posible hacerlo.

Si conoces el analizador XML de SAX, entonces lo que busco sería algo similar: calcular la suma de comprobación SHA-1 solo cargando siempre una pequeña parte en la memoria a la vez.

Todos los ejemplos que encontré, al menos en Java, siempre dependen de cargar completamente el archivo/matriz de bytes/cadena en la memoria.

Si incluso conoce implementaciones (en cualquier idioma), ¡por favor hágamelo saber!

Answer 1

Sí, es totalmente posible. SHA-1 es un algoritmo de bloque, por lo que opera en un bloque a la vez. El tamaño exacto del bloque varía con el tamaño del hash que está produciendo, pero siempre es bastante pequeño (por ejemplo, 20 - 50 bytes más o menos). Eso, por supuesto, supone que quiere incluir SHA-1 256, 384 y/o 512 (también conocido como SHA-256, SHA-384 y SHA-512). Si solo está incluyendo la versión original de 160 bits, el tamaño del bloque siempre será de 20 bytes (160 bits).

Edit: oops: releyando la especificación, los tamaños de bloque son 512 bits para SHA-1, SHA-224, SHA-256 y 1024 bits para SHA-384 y SHA-512. Gracias Charles!

Edit2: Casi me olvido de la parte en la que busca el código fuente, no solo consejos. Aquí hay un código. En primer lugar un encabezado:

// Sha.h: 
#ifndef SHA_1_H_INCLUDED 
#define SHA_1_H_INCLUDED 

// This is a relatively straightforward implementation of SHA-1. It makes no particular 
// attempt at optimization, instead aiming toward easy verification against the standard. 
// To that end, many of the variable names are identical to those used in FIPS 180-2 and 
// FIPS 180-3. 
// 
// The code should be fairly portable, within a few limitations: 
// 1. It requires that 'char' have 8 bits. In theory this is avoidable, but I don't think 
// it's worth the bother. 
// 2. It only deals with inputs in (8-bit) bytes. In theory, SHA-1 can deal with a number of 
// bits that's not a multiple of 8, but I've never needed it. Since the padding always results 
// in a byte-sized stream, the only parts that would need changing would be reading and padding 
// the input. The main hashing portion would be unaffected. 
// 
// Compiles cleanly with: 
// MS VC++ 9.0SP1 (x86 or x64): -W4 -Za 
// gc++ 3.4: -ansi -pedantic -Wall 
// comeau 4.3.3: --vc71 
// Appears to work corectly in all cases. 
// You can't use maximum warnings with Comeau though -- this code itself doesn't give problems 
// (that I know of) but Microsoft's headers give it *major* heartburn. 
// 
// 
// Written by Jerry Coffin, February 2008 
// 
// You can use this software any way you want to, with following limitations 
// (shamelessly stolen from the Boost software license): 
// 
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
// FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT 
// SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE 
// FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, 
// ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 
// DEALINGS IN THE SOFTWARE. 
// 
// If you put this to real use, I'd be happy to hear about it. If you find a bug, 
// I'd be interested in hearing about that too. There's even a pretty good chance 
// that I'll try to fix it, though I certainly can't guarantee that. 
// 
#include <algorithm> 
#include <vector> 
#include <string> 
#include <assert.h> 
#include <iostream> 
#include <sstream> 
#include <iomanip> 

#if defined(_MSC_VER) && _MSC_VER < 1600 
typedef unsigned int uint32_t; 
typedef unsigned __int64 uint64_t; 
#else 
#include <stdint.h> 
#endif 

namespace crypto { 
namespace { 
    struct ternary_operator { 
     virtual uint32_t operator()(uint32_t x, uint32_t y, uint32_t z) = 0; 
    }; 
} 

class sha1 { 
    static const size_t hash_size = 5; 
    static const size_t min_pad = 64; 
    static const size_t block_bits = 512; 
    static const size_t block_bytes = block_bits/8; 
    static const size_t block_words = block_bytes/4; 

    std::vector<uint32_t> K; 
    std::vector<uint32_t> H; 
    std::vector<uint32_t> W; 
    std::vector<ternary_operator *> fs; 
    uint32_t a, b, c, d, e, T; 
    static const size_t block_size = 16; 
    static const size_t bytes_per_word = 4; 
    size_t total_size; 

    // hash a 512-bit block of input. 
    // 
    void hash_block(std::vector<uint32_t> const &block); 

    // Pad the input to a multiple of 512 bits, and add the length 
    // in binary to the end. 
    static std::string pad(std::string const &input, size_t size); 

    // Turn 64 bytes into a block of 16 uint32_t's. 
    std::vector<uint32_t> make_block(std::string const &in); 

public: 

    // Construct a SHA-1 object. More expensive that typical 
    // ctor, but not expected to be copied a lot or anything 
    // like that, so it should be fairly harmless. 
    sha1(); 

    // The two ways to provide input for hashing: as a stream or a string. 
    // Either way, you get the result as a vector<uint32_t>. It's a fairly 
    // small vector, so even if your compiler doesn't do return-value 
    // optimization, the time taken for copying it isn't like to be 
    // significant. 
    // 
    std::vector<uint32_t> operator()(std::istream &in); 
    std::vector<uint32_t> operator()(std::string const &input); 

    friend std::ostream &operator<<(std::ostream &os, sha1 const &s); 
}; 
} 

#endif 

Y la aplicación:

// Sha1.cpp: 
#include "sha.h" 
// Please see comments in sha.h for licensing information, etc. 
// 

// Many people don't like the names I usually use for namespaces, so I've kept this one 
// short and simple. 
// 
namespace crypto { 
namespace { 
uint32_t ROTL(uint32_t const &value, unsigned bits) { 
    uint32_t mask = (1 << bits) - 1; 
    return value << bits | (value >> (32-bits))&mask; 
} 

struct f1 : ternary_operator { 
    uint32_t operator()(uint32_t x, uint32_t y, uint32_t z) { 
     return (x & y)^(~x&z); 
    } 
}; 

struct f2 : ternary_operator { 
    uint32_t operator()(uint32_t x, uint32_t y, uint32_t z) { 
     return x^y^z; 
    } 
}; 

struct f3 : ternary_operator { 
    uint32_t operator()(uint32_t x, uint32_t y, uint32_t z) { 
     return (x&y)^(x&z)^(y&z); 
    } 
}; 

uint32_t word(int a, int b, int c, int d) { 
    a &= 0xff; 
    b &= 0xff; 
    c &= 0xff; 
    d &= 0xff; 
    int val = a << 24 | b << 16 | c << 8 | d; 
    return val; 
} 
} 

// hash a 512-bit block of input. 
// 
void sha1::hash_block(std::vector<uint32_t> const &block) { 
    assert(block.size() == block_words); 

    int t; 
    std::copy(block.begin(), block.end(), W.begin()); 
    for (t=16; t<80; t++) { 
     W[t] = ROTL(W[t-3]^W[t-8]^W[t-14]^W[t-16], 1); 
    } 

    a = H[0]; b = H[1]; c = H[2]; d = H[3]; e = H[4]; 

    for (t=0; t<80; t++) { 
     T = ROTL(a, 5) + (*fs[t])(b, c, d) + e + K[t] + W[t]; 
     e = d; 
     d = c; 
     c = ROTL(b, 30); 
     b = a; 
     a = T; 
    } 
    H[0] += a; H[1] += b; H[2] += c; H[3] += d; H[4] += e; 
} 

// Pad the input to a multiple of 512 bits, and put the length 
// in binary at the end. 
std::string sha1::pad(std::string const &input, size_t size) { 
    size_t length = size * 8 + 1; 
    size_t remainder = length % block_bits; 
    size_t pad_len = block_bits-remainder; 

    if (pad_len < min_pad) 
     pad_len += block_bits; 
    ++pad_len; 

    pad_len &= ~7; 
    std::string padding(pad_len/8, '\0'); 

    for (size_t i=0; i<sizeof(padding.size()); i++) 
     padding[padding.size()-i-1] = (length-1) >> (i*8) & 0xff; 
    padding[0] |= (unsigned char)0x80; 

    std::string ret(input+padding); 
    return ret; 
} 

// Turn 64 bytes into a block of 16 uint32_t's. 
std::vector<uint32_t> sha1::make_block(std::string const &in) { 
    assert(in.size() >= block_bytes); 

    std::vector<uint32_t> ret(block_words); 

    for (size_t i=0; i<block_words; i++) { 
     size_t s = i*4; 
     ret[i] = word(in[s], in[s+1], in[s+2], in[s+3]); 
    } 
    return ret; 
} 


// Construct a SHA-1 object. More expensive that typical 
// ctor, but not expected to be copied a lot or anything 
// like that, so it should be fairly harmless. 
sha1::sha1() : K(80), H(5), W(80), fs(80), total_size(0) { 
    static const uint32_t H0[] = { 
     0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476, 0xc3d2e1f0 
    }; 
    static const uint32_t Ks[] = { 
     0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6 
    }; 

    std::copy(H0, H0+hash_size, H.begin()); 

    std::fill_n(K.begin()+00, 20, Ks[0]); 
    std::fill_n(K.begin()+20, 20, Ks[1]); 
    std::fill_n(K.begin()+40, 20, Ks[2]); 
    std::fill_n(K.begin()+60, 20, Ks[3]); 

    static f1 sf1; 
    static f2 sf2; 
    static f3 sf3; 

    std::fill_n(fs.begin()+00, 20, &sf1); 
    std::fill_n(fs.begin()+20, 20, &sf2); 
    std::fill_n(fs.begin()+40, 20, &sf3); 
    std::fill_n(fs.begin()+60, 20, &sf2); 
} 

// The two ways to provide input for hashing: as a stream or a string. 
// Either way, you get the result as a vector<uint32_t>. It's a fairly 
// small vector, so even if your compiler doesn't do return-value 
// optimization, the time taken for copying it isn't like to be 
// significant. 
// 
std::vector<uint32_t> sha1::operator()(std::string const &input) { 
    std::string temp(pad(input, total_size + input.size())); 
    std::vector<uint32_t> block(block_size); 

    size_t num = temp.size()/block_bytes; 

    for (unsigned block_num=0; block_num<num; block_num++) { 
     size_t s; 
     for (size_t i=0; i<block_size; i++) { 
      s = block_num*block_bytes+i*4; 
      block[i] = word(temp[s], temp[s+1], temp[s+2], temp[s+3]); 
     } 
     hash_block(block); 
    } 
    return H; 
} 

std::vector<uint32_t> sha1::operator()(std::istream &in) { 
    char raw_block[65]; 

    while (in.read(raw_block, block_bytes)) { 
     total_size += block_bytes; 
     std::string b(raw_block, in.gcount()); 
     hash_block(make_block(b)); 
    } 
    std::string x(raw_block, in.gcount()); 
    return operator()(x); 
} 

std::ostream &operator<<(std::ostream &os, sha1 const &s) { 
    // Display a SHA-1 result in hex. 
    for (size_t i=0; i<(s.H).size(); i++) 
     os << std::fixed << std::setprecision(8) << std::hex << std::setfill('0') << (s.H)[i] << " "; 
    return os << std::dec << std::setfill(' ') << "\n"; 
} 

} 

#ifdef TEST 
#include <iostream> 
#include <iomanip> 
#include <string> 
#include <sstream> 

// A minimal test harness to check that it's working correctly. Strictly black-box 
// testing, with no attempt at things like coverage analysis. Nonetheless, I believe 
// it should cover most of the code -- the core hashing code all gets used for every 
// possible value. The padding code should be tested fairly thoroughly as well -- the 
// first test is a fairly simple case, and the second the more complex one (where the 
// padding requires adding another block). 
class tester { 
    bool verify(uint32_t *test_val, std::vector<uint32_t> const &hash, std::ostream &os) { 
     // Verify that a result matches a test value and report result. 
     for (size_t i=0; i<hash.size(); i++) 
      if (hash[i] != test_val[i]) { 
       os << "Mismatch. Expected: " << test_val[i] << ", but found: " << hash[i] << "\n"; 
       return false; 
      } 
      os << "Message digest Verified.\n\n"; 
      return true; 
    } 

public: 

    bool operator()(uint32_t *test_val, std::string const &input) { 
     std::cout << "Testing hashing from string:\n\"" << input << "\"\n"; 
     crypto::sha1 hasher1; 
     std::vector<uint32_t> hash = hasher1(input); 
     std::cout << "Message digest is:\n\t" << hasher1; 
     bool verified = verify(test_val, hash, std::cerr); 

     crypto::sha1 hasher2; 
     std::cout << "Testing hashing from Stream:\n"; 
     std::istringstream buf(input); 
     hash = hasher2(buf); 
     std::cout << "Message digest is:\n\t" << hasher2; 

     return verified & verify(test_val, hash, std::cerr); 
    } 
}; 

int main() { 
    // These test values and results come directly from the FIPS pub. 
    // 
    char const *input1 = "abc"; 
    char const *input2 = "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"; 
    uint32_t result1[] = {0xA9993E36, 0x4706816A, 0xBA3E2571, 0x7850C26C, 0x9CD0D89D}; 
    uint32_t result2[] = {0x84983E44, 0x1C3BD26E, 0xBAAE4AA1, 0xF95129E5, 0xE54670F1 }; 
    bool correct = tester()(result1, input1); 
    correct &= tester()(result2, input2); 
    if (correct) 
     std::cerr << "All Tests passed!\n"; 
    else 
     std::cerr << "Test Failed!\n"; 
} 
#elif defined(MAIN) 

#include <sstream> 
#include <fstream> 
#include <iostream> 

int main(int argc, char **argv) { 
    if (argc < 2) { 
     std::cerr << "Usage: sha1 [filename]\n"; 
     return EXIT_FAILURE; 
    } 
    for (int i=1; i<argc; i++) { 
     crypto::sha1 hash; 
     std::ifstream in(argv[i], std::ios_base::binary); 
     if (in.good()) { 
      hash(in); 
      std::cout << "SHA-1(" << argv[i] << ") = " << hash << "\n"; 
     } 
    } 
    return 0; 
} 
#endif 

Answer 2

Sí, se puede usar para transmitir hash porque es iterativo: va 512 bits en cada iteración, y obtiene un nuevo bloque de 512 bits que puede usar para el próximo.

Aquí puede encontrar el pseudocódigo: link. Debería ser bastante fácil de implementar en Java. ¡Solo necesitas hacer algo de relleno cuando encuentres el último bloque y algunas operaciones a nivel de bit!

ADVERTENCIA: la única cosa es que se necesitan por lo general enteros sin signo mientras que Java ofrece acaba de firmar uno, usted debe hacer algunos trucos para evitar problemas ..

Answer 3

Sí, lo es. Solo necesita leer bloques de 512 bits (64 bytes) a la vez para calcular un hash SHA-1.

Debe realizar un seguimiento de la longitud del flujo y realizar el relleno correcto en los últimos uno o dos bloques, pero sí, es perfectamente posible.

He escrito una implementación de este tipo en C++ antes, pero me temo que no soy libre de distribuirlo.

Answer 4

SHA-1 procesa los datos en bloques, por lo que puede procesar su archivo en la secuencia. Estoy bastante seguro de que la biblioteca crypto castillo hinchable implementa SHA-1 a un nivel lo suficientemente bajo como para que pueda transmitir sus datos. Lo he hecho de forma muy similar usando otras claves de bloque con la biblioteca de crypto castillo hinchable.

docs

http://www.bouncycastle.org/

Answer 5

El Java dicen utilizar una clase MessageDigest para calcular SHA-1 en cualquier dato de tamaño arbitrario.

Answer 6

Puede hacer exactamente esto, de forma transparente y casi sin esfuerzo, utilizando las clases DigestInputStream o DigestOutputStream. O puede usar MessageDigest manualmente y es casi igual de fácil.