Skip to content

shake128 support for big_catalogs #26

Closed
Closed
shake128 support for big_catalogs#26

Description

@rptaylor
rptaylor
opened on Jun 7, 2018

Hi @DrDaveD
I was very interested to try the big_catalogs utility.
However:

$ big_catalogs  http://cvmfs-s1-arbutus.computecanada.ca:8000/cvmfs/soft.computecanada.ca/
Traceback (most recent call last):
  File "/usr/bin/big_catalogs", line 24, in <module>
    repo = cvmfs.open_repository(repo_identifier)
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 200, in open_repository
    repo = Repository.from_source(repository_path, cache_dir)
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 41, in from_source
    return cls(Repository.__make_fetcher(source, cache_dir))
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 33, in __init__
    self._read_manifest()
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 64, in _read_manifest
    self.manifest = Manifest(manifest_file)
  File "/usr/lib/python2.7/site-packages/cvmfs/manifest.py", line 25, in __init__
    RootFile.__init__(self, manifest_file)
  File "/usr/lib/python2.7/site-packages/cvmfs/root_file.py", line 55, in __init__
    self._read_signature(file_object)
  File "/usr/lib/python2.7/site-packages/cvmfs/root_file.py", line 89, in _read_signature
    raise IncompleteRootFileSignature("Signature checksum malformed")
cvmfs._exceptions.IncompleteRootFileSignature: Signature checksum malformed

Does it not support shake-128 hashes?

Thanks.

Activity

  1. DrDaveD commented on Jun 8, 2018

    @DrDaveD
    DrDaveD
    on Jun 8, 2018
    Contributor

    Yep, looks like it predates shake128. Would you be willing to do a PR? I'm as blind as you are on this, I haven't looked at the code before.

  2. rptaylor commented on Jun 8, 2018

    @rptaylor
    rptaylor
    on Jun 8, 2018
    Author

    I took a look and got as far as needing to calculate the hash, but the shake128 hash isn't available in Python until version 3.6:
    https://docs.python.org/3.6/library/hashlib.html

  3. rptaylor
    changed the title [-]big_catalogs exception: signature checksum malformed[/-] [+]shake128 support for big_catalogs[/+] on Jun 8, 2018

  4. rptaylor commented on Jun 8, 2018

    @rptaylor
    rptaylor
    on Jun 8, 2018
    Author

    Here is one bit of code with some of the changes that would be needed, in /usr/lib/python2.7/site-packages/cvmfs/root_file.py

        def _read_signature(self, file_object):
        """ Reads the signature's checksum and the binary signature string """
        file_object.seek(0)
        message_digest = self._hash_over_content(file_object)

        self.signature_checksum = file_object.readline().rstrip()
        if len(self.signature_checksum) == 40:
            pass # sha-1
        elif len(self.signature_checksum) == 49 and self.signature_checksum[40:49] == "-shake128":
            pass # shake-128
        else:
            raise IncompleteRootFileSignature("Signature checksum malformed")

    Then there would need to be a function similar to _hash_over_content which calculates the shake128 hash. Maybe that will be helpful if someone comes along to work on it.

  5. DrDaveD commented on Jun 11, 2018

    @DrDaveD
    DrDaveD
    on Jun 11, 2018
    Contributor

    One could always import a custom function. Alternatively, python3 is available in EPEL for all EL* variants. yum install /usr/bin/python3.

  6. DrDaveD commented on Jul 14, 2026

    @DrDaveD
    DrDaveD
    on Jul 14, 2026
    Contributor

    Should be fixed by #45

  7. DrDaveD
    closed this as completedon Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions