CollecTor Filesystem Protocol

CollecTor Filesystem Protocol.

class bushel.collector.filesystem.CollecTorIndexCompression(extension: str, decompress: Callable[[bytes], bytes])[source]

Enumeration of supported compression types for CollecTor indexes.

Name

Description

UNCOMPRESSED

Uncompressed

BZ2

bzip2

XZ

xz

GZ

gzip

Variables

extension (str) – Filename extension with leading dot (“.”).

class bushel.collector.filesystem.CollectorOutBridgeDescsMarker[source]

Enumeration of marker names under the “bridge-descriptors” directory as specified in §5.2 of [collector-protocol].

Name

Description

EXTRA_INFO

Bridge extra-info descriptors (§5.2.1)

SERVER_DESCRIPTOR

Bridge server descriptors (§5.2.1)

STATUS

Bridge statuses (§5.2.2)

class bushel.collector.filesystem.CollectorOutRelayDescsMarker[source]

Enumeration of marker names under the “relay-descriptors” directory as specified in §5.3 of [collector-protocol].

Name

Description

CONSENSUS

Network status consensuses (§5.3.2)

EXTRA_INFO

Relay extra-info descriptors (§5.3.2)

SERVER_DESCRIPTOR

Relay server descriptors (§5.3.2)

VOTE

Network status votes (§5.3.2)

class bushel.collector.filesystem.CollectorOutSubdirectory[source]

Enumeration of subdirectory names under the “out” directory as specified in §5.0 of [collector-protocol].

Name

Description

BRIDGE_DESCRIPTORS

Bridge descriptors (§5.2)

EXIT_LISTS

Exit lists (§5.1)

RELAY_DESCRIPTORS

Relay descriptors (§5.3)

TORPERF

Torperf and Onionperf (§5.1)

WEBSTATS

Web server access logs (§5.4)

class bushel.collector.filesystem.CollectorRecentSubdirectory[source]

Enumeration of subdirectory names under the “recent” directory as specified in §4.0 of [collector-protocol].

Name

Description

BRIDGE_DESCRIPTORS

Bridge descriptors (§4.2)

EXIT_LISTS

Exit lists (§4.1.1)

RELAY_DESCRIPTORS

Relay descriptors (§4.3)

TORPERF

Torperf and Onionperf (§4.1.2)

WEBSTATS

Web server access logs (§4.4)

bushel.collector.filesystem.collector_422_filename(valid_after: datetime.datetime, fingerprint: str) → str[source]

Create a filename for a bridge status according to §4.2.2 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge
>>> collector_422_filename(valid_after, fingerprint)
'20181119-150000-BA44A889E64B93FAA2B114E02C2A279A8555C533'
Parameters
  • valid_after (datetime) – The valid-after time.

  • fingerprint (str) – The fingerprint of the bridge authority.

Returns

Filename as a str.

bushel.collector.filesystem.collector_431_filename(valid_after: datetime.datetime) → str[source]

Create a filename for a network status consensus according to §4.3.1 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_431_filename(valid_after)
'2018-11-19-15-00-00-consensus'
Parameters

valid_after (datetime) – The valid-after time.

Returns

Filename as a str.

bushel.collector.filesystem.collector_433_filename(valid_after: datetime.datetime, v3ident: str, digest: str) → str[source]

Create a filename for a network status vote according to §4.3.3 of the [collector-protocol].

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> v3ident = "D586D18309DED4CD6D57C18FDB97EFA96D330566"  # moria1
>>> digest = "663B503182575D242B9D8A67334365FF8ECB53BB"
>>> collector_433_filename(valid_after, v3ident, digest)  # doctest: +ELLIPSIS
'2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'

Paths in the Collector File Structure Protocol using this filename expect upper-case hex-encoded SHA-1 digests.

>>> v3ident = "d586d18309ded4cd6d57c18fdb97efa96d330566"  # Lower case gets corrected
>>> digest = "663b503182575d242b9d8a67334365ff8ecb53bb"  # Lower case gets corrected
>>> collector_433_filename(valid_after, v3ident, digest)  # doctest: +ELLIPSIS
'2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'
Parameters
  • valid_after (datetime) – The valid-after time.

  • v3ident (str) – The v3ident of the directory authority.

  • digest (str) – The digest of the vote.

Returns

Filename as a str.

bushel.collector.filesystem.collector_434_filename(valid_after: datetime.datetime) → str[source]

Create a filename for a microdesc-flavoured network status consensus according to §4.3.4 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_434_filename(valid_after)
'2018-11-19-15-00-00-consensus-microdesc'
Parameters

valid_after (datetime) – The valid-after time.

Returns

Filename as a str.

bushel.collector.filesystem.collector_521_path(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], published: datetime.datetime, digest: str) → str[source]

Create a path according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:

>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS
>>> marker = CollectorOutRelayDescsMarker.SERVER_DESCRIPTOR
>>> published = datetime.datetime(2018, 11, 19, 9, 17, 56)
>>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389"
>>> collector_521_path(subdirectory, marker, published, digest)  # doctest: +ELLIPSIS
'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'

Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.

>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected
>>> collector_521_path(subdirectory, marker, published, digest)  # doctest: +ELLIPSIS
'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'
Parameters
Returns

Path for the descriptor as a str.

bushel.collector.filesystem.collector_521_substructure(published: datetime.datetime, digest: str) → str[source]

Create a path substructure according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:

>>> published = datetime.datetime(2018, 11, 19, 9, 17, 56)
>>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389"
>>> collector_521_substructure(published, digest)
'2018/11/a/9'

Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.

>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected
>>> collector_521_substructure(published, digest)
'2018/11/a/9'
Parameters
  • published (datetime) – The published time.

  • digest (str) – The hex-encoded SHA-1 digest for the descriptor. The case will automatically be fixed to lower-case.

Returns

Path substructure as a str.

bushel.collector.filesystem.collector_522_path(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], valid_after: datetime.datetime, filename: str) → str[source]

Create a path according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses (both ns- and microdesc- flavors) and votes. For a bridge status for example:

>>> subdirectory = CollectorOutSubdirectory.BRIDGE_DESCRIPTORS
>>> marker = CollectorOutBridgeDescsMarker.STATUSES
>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge
>>> filename = collector_422_filename(valid_after, fingerprint)
>>> collector_522_path(subdirectory, marker, valid_after, filename)  # doctest: +ELLIPSIS
'bridge-descriptors/statuses/2018/11/19/20181119-150000-BA44...533'

Or alternatively for a network-status consensus:

>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS
>>> marker = CollectorOutRelayDescsMarker.CONSENSUS
>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> filename = collector_431_filename(valid_after)
>>> collector_522_path(subdirectory, marker, valid_after, filename)
'relay-descriptors/consensus/2018/11/19/2018-11-19-15-00-00-consensus'
Parameters
Returns

Path for the descriptor as a str.

bushel.collector.filesystem.collector_522_substructure(valid_after: datetime.datetime) → str[source]

Create a path substructure according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses and votes. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_522_substructure(valid_after)
'2018/11/19'
Parameters

valid_after (datetime) – The valid-after time.

Returns

Path substructure as a str.

bushel.collector.filesystem.collector_533_substructure(valid_after: datetime.datetime) → str[source]

Create a substructure according to §5.3.3 of the [collector-protocol]. This is used for microdesc-flavored consensuses and microdescriptors. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_533_substructure(valid_after)
'2018/11'
bushel.collector.filesystem.collector_534_consensus_path(valid_after)[source]

Create a path according to §5.3.4 of the [collector-protocol] for a microdesc-flavored consensus. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_534_consensus_path(valid_after)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/consensus-microdesc/19/2018-11-1...sc'
bushel.collector.filesystem.collector_534_microdescriptor_path(valid_after: datetime.datetime, digest: str) → str[source]

Create a path according to §5.3.4 of the [collector-protocol] for a microdescriptor. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> digest = "00d91cf96321fbd536dd07e297a5e1b7e6961ddd10facdd719716e351453168f"
>>> collector_534_microdescriptor_path(valid_after, digest)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'

This path in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-256 digests.

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> digest = "00D91CF96321FBD536DD07E297A5E1B7E6961DDD10FACDD719716E351453168F"
>>> collector_534_microdescriptor_path(valid_after, digest)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'
bushel.collector.filesystem.collector_index_path(compression: bushel.collector.filesystem.CollecTorIndexCompression) → str[source]

Create a path to the CollecTor index file, using the specified compression algorithm.

Parameters

compression (CollecTorIndexCompression) – Compression algorithm to use.