CollecTor Filesystem Protocol¶
CollecTor Filesystem Protocol.
-
class
bushel.collector.filesystem.
CollecTorIndexCompression
(extension: str, decompress: Callable[[bytes], bytes])[source]¶ Enumeration of supported compression types for CollecTor indexes.
Name
Description
UNCOMPRESSED
Uncompressed
BZ2
bzip2
XZ
xz
GZ
gzip
- Variables
extension (str) – Filename extension with leading dot (“.”).
-
class
bushel.collector.filesystem.
CollectorOutBridgeDescsMarker
[source]¶ Enumeration of marker names under the “bridge-descriptors” directory as specified in §5.2 of [collector-protocol].
Name
Description
EXTRA_INFO
Bridge extra-info descriptors (§5.2.1)
SERVER_DESCRIPTOR
Bridge server descriptors (§5.2.1)
STATUS
Bridge statuses (§5.2.2)
-
class
bushel.collector.filesystem.
CollectorOutRelayDescsMarker
[source]¶ Enumeration of marker names under the “relay-descriptors” directory as specified in §5.3 of [collector-protocol].
Name
Description
CONSENSUS
Network status consensuses (§5.3.2)
EXTRA_INFO
Relay extra-info descriptors (§5.3.2)
SERVER_DESCRIPTOR
Relay server descriptors (§5.3.2)
VOTE
Network status votes (§5.3.2)
-
class
bushel.collector.filesystem.
CollectorOutSubdirectory
[source]¶ Enumeration of subdirectory names under the “out” directory as specified in §5.0 of [collector-protocol].
Name
Description
BRIDGE_DESCRIPTORS
Bridge descriptors (§5.2)
EXIT_LISTS
Exit lists (§5.1)
RELAY_DESCRIPTORS
Relay descriptors (§5.3)
TORPERF
Torperf and Onionperf (§5.1)
WEBSTATS
Web server access logs (§5.4)
-
class
bushel.collector.filesystem.
CollectorRecentSubdirectory
[source]¶ Enumeration of subdirectory names under the “recent” directory as specified in §4.0 of [collector-protocol].
Name
Description
BRIDGE_DESCRIPTORS
Bridge descriptors (§4.2)
EXIT_LISTS
Exit lists (§4.1.1)
RELAY_DESCRIPTORS
Relay descriptors (§4.3)
TORPERF
Torperf and Onionperf (§4.1.2)
WEBSTATS
Web server access logs (§4.4)
-
bushel.collector.filesystem.
collector_422_filename
(valid_after: datetime.datetime, fingerprint: str) → str[source]¶ Create a filename for a bridge status according to §4.2.2 of the [collector-protocol]. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge >>> collector_422_filename(valid_after, fingerprint) '20181119-150000-BA44A889E64B93FAA2B114E02C2A279A8555C533'
-
bushel.collector.filesystem.
collector_431_filename
(valid_after: datetime.datetime) → str[source]¶ Create a filename for a network status consensus according to §4.3.1 of the [collector-protocol]. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> collector_431_filename(valid_after) '2018-11-19-15-00-00-consensus'
-
bushel.collector.filesystem.
collector_433_filename
(valid_after: datetime.datetime, v3ident: str, digest: str) → str[source]¶ Create a filename for a network status vote according to §4.3.3 of the [collector-protocol].
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> v3ident = "D586D18309DED4CD6D57C18FDB97EFA96D330566" # moria1 >>> digest = "663B503182575D242B9D8A67334365FF8ECB53BB" >>> collector_433_filename(valid_after, v3ident, digest) # doctest: +ELLIPSIS '2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'
Paths in the Collector File Structure Protocol using this filename expect upper-case hex-encoded SHA-1 digests.
>>> v3ident = "d586d18309ded4cd6d57c18fdb97efa96d330566" # Lower case gets corrected >>> digest = "663b503182575d242b9d8a67334365ff8ecb53bb" # Lower case gets corrected >>> collector_433_filename(valid_after, v3ident, digest) # doctest: +ELLIPSIS '2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'
-
bushel.collector.filesystem.
collector_434_filename
(valid_after: datetime.datetime) → str[source]¶ Create a filename for a microdesc-flavoured network status consensus according to §4.3.4 of the [collector-protocol]. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> collector_434_filename(valid_after) '2018-11-19-15-00-00-consensus-microdesc'
-
bushel.collector.filesystem.
collector_521_path
(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], published: datetime.datetime, digest: str) → str[source]¶ Create a path according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:
>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS >>> marker = CollectorOutRelayDescsMarker.SERVER_DESCRIPTOR >>> published = datetime.datetime(2018, 11, 19, 9, 17, 56) >>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389" >>> collector_521_path(subdirectory, marker, published, digest) # doctest: +ELLIPSIS 'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'
Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.
>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected >>> collector_521_path(subdirectory, marker, published, digest) # doctest: +ELLIPSIS 'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'
- Parameters
subdirectory (str) – The subdirectory under the “out” directory to use. Standard values can be found in
CollectorOutSubdirectory
.marker (str) – The marker under the subdirectory to use. Standard values can be found in
CollectorOutRelayDescsMarker
andCollectorOutBridgeDescsMarker
.published (datetime) – The published time.
digest (str) – The hex-encoded SHA-1 digest for the descriptor. The case will automatically be fixed to lower-case.
- Returns
Path for the descriptor as a
str
.
-
bushel.collector.filesystem.
collector_521_substructure
(published: datetime.datetime, digest: str) → str[source]¶ Create a path substructure according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:
>>> published = datetime.datetime(2018, 11, 19, 9, 17, 56) >>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389" >>> collector_521_substructure(published, digest) '2018/11/a/9'
Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.
>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected >>> collector_521_substructure(published, digest) '2018/11/a/9'
-
bushel.collector.filesystem.
collector_522_path
(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], valid_after: datetime.datetime, filename: str) → str[source]¶ Create a path according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses (both ns- and microdesc- flavors) and votes. For a bridge status for example:
>>> subdirectory = CollectorOutSubdirectory.BRIDGE_DESCRIPTORS >>> marker = CollectorOutBridgeDescsMarker.STATUSES >>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge >>> filename = collector_422_filename(valid_after, fingerprint) >>> collector_522_path(subdirectory, marker, valid_after, filename) # doctest: +ELLIPSIS 'bridge-descriptors/statuses/2018/11/19/20181119-150000-BA44...533'
Or alternatively for a network-status consensus:
>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS >>> marker = CollectorOutRelayDescsMarker.CONSENSUS >>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> filename = collector_431_filename(valid_after) >>> collector_522_path(subdirectory, marker, valid_after, filename) 'relay-descriptors/consensus/2018/11/19/2018-11-19-15-00-00-consensus'
- Parameters
subdirectory (str) – The subdirectory under the “out” directory to use. Standard values can be found in
CollectorOutSubdirectory
.marker (str) – The marker under the subdirectory to use. Standard values can be found in
CollectorOutRelayDescsMarker
andCollectorOutBridgeDescsMarker
.valid_after (datetime) – The valid_after time.
filename (str) – The filename to use as a
str
, typically created withcollector_422_filename()
for bridge statuses,collector_431_filename()
for network-status consensuses, orcollector_433_filename()
for network-status votes.
- Returns
Path for the descriptor as a
str
.
-
bushel.collector.filesystem.
collector_522_substructure
(valid_after: datetime.datetime) → str[source]¶ Create a path substructure according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses and votes. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> collector_522_substructure(valid_after) '2018/11/19'
-
bushel.collector.filesystem.
collector_533_substructure
(valid_after: datetime.datetime) → str[source]¶ Create a substructure according to §5.3.3 of the [collector-protocol]. This is used for microdesc-flavored consensuses and microdescriptors. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> collector_533_substructure(valid_after) '2018/11'
-
bushel.collector.filesystem.
collector_534_consensus_path
(valid_after)[source]¶ Create a path according to §5.3.4 of the [collector-protocol] for a microdesc-flavored consensus. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> collector_534_consensus_path(valid_after) # doctest: +ELLIPSIS 'relay-descriptors/microdesc/2018/11/consensus-microdesc/19/2018-11-1...sc'
-
bushel.collector.filesystem.
collector_534_microdescriptor_path
(valid_after: datetime.datetime, digest: str) → str[source]¶ Create a path according to §5.3.4 of the [collector-protocol] for a microdescriptor. For example:
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> digest = "00d91cf96321fbd536dd07e297a5e1b7e6961ddd10facdd719716e351453168f" >>> collector_534_microdescriptor_path(valid_after, digest) # doctest: +ELLIPSIS 'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'
This path in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-256 digests.
>>> valid_after = datetime.datetime(2018, 11, 19, 15) >>> digest = "00D91CF96321FBD536DD07E297A5E1B7E6961DDD10FACDD719716E351453168F" >>> collector_534_microdescriptor_path(valid_after, digest) # doctest: +ELLIPSIS 'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'
-
bushel.collector.filesystem.
collector_index_path
(compression: bushel.collector.filesystem.CollecTorIndexCompression) → str[source]¶ Create a path to the CollecTor index file, using the specified compression algorithm.
- Parameters
compression (CollecTorIndexCompression) – Compression algorithm to use.