Re: [pragmatique] GNUnet

Delete this message

Reply to this message
Author: Catonano
Date:  
To: pragmatique
Subject: Re: [pragmatique] GNUnet
a couple more

2017-05-21 8:59 GMT+02:00 Catonano <catonano@???>:

> one more
>
>
>

From f2a4ed8ae682129886e9215771dc80eae8d96c91 Mon Sep 17 00:00:00 2001
From: Adriano Peluso <catonano@???>
Date: Sun, 21 May 2017 08:47:57 +0200
Subject: [PATCH 4/5] some more warnings eliminated from developer.texi

---
developer.texi | 290 +++++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 263 insertions(+), 27 deletions(-)

diff --git a/developer.texi b/developer.texi
index 13a1eba..c3aae19 100644
--- a/developer.texi
+++ b/developer.texi
@@ -42,9 +42,79 @@ these restrictions on your account. We're sorry for this inconvenience;
however, few people would want to read this site if 99% of it was
advertisements for bogus websites.

+
+
@c ***************************************************************************
-@node DevelperIntroduction
-@chapter DeveloperIntroduction
+
+
+
+
+
+
+
+@menu
+* Developer Introduction::
+* Code overview::
+* System Architecture::
+* Subsystem stability::
+* Naming conventions and coding style guide::
+* Build-system::
+* Developing extensions for GNUnet using the gnunet-ext template::
+* Writing testcases::
+* GNUnet's TESTING library::
+* API::
+* Finer control over peer stop::
+* Helper functions::
+* Testing with multiple processes::
+* Performance regression analysis with Gauger::
+* GNUnet's TESTBED Subsystem::
+* libgnunetutil::
+* Logging::
+* Interprocess communication API (IPC)::
+* Cryptography API::
+* Message Queue API::
+* Service API::
+* Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps::
+* The CONTAINER_MDLL API::
+* The Automatic Restart Manager (ARM)::
+* Basic functionality::
+* Key configuration options::
+* Availability2::
+* Reliability::
+* GNUnet's TRANSPORT Subsystem::
+* Address validation protocol::
+* NAT library::
+* Distance-Vector plugin::
+* SMTP plugin::
+* Bluetooth plugin::
+* WLAN plugin::
+* The ATS Subsystem::
+* GNUnet's CORE Subsystem::
+* Limitations::
+* When is a peer "connected"?::
+* libgnunetcore::
+* The CORE Client-Service Protocol::
+* The CORE Peer-to-Peer Protocol::
+* GNUnet's CADET subsystem::
+* libgnunetcadet::
+* GNUnet's NSE subsystem::
+* GNUnet's HOSTLIST subsystem::
+* GNUnet's IDENTITY subsystem::
+* GNUnet's NAMESTORE Subsystem::
+* GNUnet's PEERINFO subsystem::
+* GNUnet's PEERSTORE subsystem::
+* GNUnet's STATISTICS subsystem::
+* GNUnet's Distributed Hash Table (DHT)::
+* The GNU Name System (GNS)::
+* The GNS Namecache::
+* The REVOCATION Subsystem::
+* Dissemination::
+* GNUnet's File-sharing (FS) Subsystem::
+* GNUnet's REGEX Subsystem::
+@end menu
+
+@node Developer Introduction
+@section Developer Introduction

This developer handbook is intended as first introduction to GNUnet for new
developers that want to extend the GNUnet framework. After the introduction,
@@ -100,9 +170,15 @@ Automatically generated, current reports on the test suite are here.
Current reports on test coverage are here.
@end itemize

+
+
@c ***************************************************************************
+@menu
+* Project overview::
+@end menu
+
@node Project overview
-@section Project overview
+@subsection Project overview

The GNUnet project consists at this point of several sub-projects. This section
is supposed to give an initial overview about the various sub-projects. Note
@@ -477,11 +553,31 @@ This subsystem does not have an API/IPC-protocol/P2P-protocol

Here you can find some rules to help you write code for GNUnet.

+
+
@c ***************************************************************************
+@menu
+* Naming conventions::
+* Coding style::
+@end menu
+
@node Naming conventions
@subsection Naming conventions

+
@c ***************************************************************************
+@menu
+* include files::
+* binaries::
+* logging::
+* configuration::
+* exported symbols::
+* private (library-internal) symbols (including structs and macros)::
+* testcases::
+* performance tests::
+* src/ directories::
+@end menu
+
@node include files
@subsubsection include files

@@ -808,7 +904,7 @@ and run @code{ldconfig} or your add it to the environmental variable

@c ***************************************************************************
@node Writing testcases
-@chapter Writing testcases
+@section Writing testcases

Ideally, any non-trivial GNUnet code should be covered by automated testcases.
Testcases should reside in the same place as the code that is being tested. The
@@ -1151,7 +1247,18 @@ use this path to start helper binaries both locally and remotely.
Testbed API can accessed by including "gnunet_testbed_service.h" file and
linking with -lgnunettestbed.

+
+
@c ***************************************************************************
+@menu
+* Supported Topologies::
+* Hosts file format::
+* Topology file format::
+* Testbed Barriers::
+* Automatic large-scale deployment of GNUnet in the PlanetLab testbed::
+* TESTBED Caveats::
+@end menu
+
@node Supported Topologies
@subsection Supported Topologies

@@ -1328,7 +1435,12 @@ uninitialised barrier results in failure.
@code{GNUNET_TESTBED_barrier_wait_cancel()} cancels the notification registered
by @code{GNUNET_TESTBED_barrier_wait()}.

+
@c ***************************************************************************
+@menu
+* Implementation::
+@end menu
+
@node Implementation
@subsubsection Implementation

@@ -1409,7 +1521,15 @@ Please also check @uref{https://gnunet.org/installation-fedora8-svn} and@
@uref{https://gnunet.org/installation-fedora12-svn} to find detailled
instructions how to install GNUnet on a PlanetLab node.

+
@c ***************************************************************************
+@menu
+* PlanetLab Automation for Fedora8 nodes::
+* Install buildslave on PlanetLab nodes running fedora core 8::
+* Setup a new PlanetLab testbed using GPLMT::
+* Why do i get an ssh error when using the regex profiler?::
+@end menu
+
@node PlanetLab Automation for Fedora8 nodes
@subsubsection PlanetLab Automation for Fedora8 nodes

@@ -1517,7 +1637,13 @@ login, then you should be good to go.
This section documents a few caveats when using the GNUnet testbed
subsystem.

+
@c ***************************************************************************
+@menu
+* CORE must be started::
+* ATS must want the connections::
+@end menu
+
@node CORE must be started
@subsubsection CORE must be started

@@ -1548,8 +1674,8 @@ largely arises for dense overlay topologies, especially if you try to create
cliques with more than 20 peers.

@c ***************************************************************************
-@chapter libgnunetutil
@node libgnunetutil
+@section libgnunetutil

libgnunetutil is the fundamental library that all GNUnet code builds upon.
Ideally, this library should contain most of the platform dependent code
@@ -1747,7 +1873,13 @@ At the moment GNUnet will stop processing a log definition when it encounters
an error in definition formatting or an error in regular expression syntax, and
will not report the failure in any way.

+
@c ***************************************************************************
+@menu
+* Examples::
+* Log files::
+@end menu
+
@node Examples
@subsection Examples

@@ -1792,6 +1924,7 @@ definition, as opposed to earlier examples, which use the shell).@ Another
limitation, on Windows, GNUNET_FORCE_LOGFILE @strong{MUST} be set in order to
GNUNET_FORCE_LOG to work.

+
@c ***************************************************************************
@node Log files
@subsection Log files
@@ -1829,7 +1962,12 @@ kept for 4 years and the logs from the first year would be deleted once year 5
begins. If you do not use any date-related string format codes, logs would
never be automatically deleted by GNUnet.

+
@c ***************************************************************************
+@menu
+* Updated behavior of GNUNET_log::
+@end menu
+
@node Updated behavior of GNUNET_log
@subsubsection Updated behavior of GNUNET_log

@@ -1910,7 +2048,15 @@ service and client.@ (Here, a client uses the @code{struct
AddressLookupMessage} as a request to ask the server to return the address of
any other peer connecting to the service.)

+
@c ***************************************************************************
+@menu
+* Define new message types::
+* Define message struct::
+* Connection between client and server::
+* Server Setting::
+@end menu
+
@node Define new message types
@subsection Define new message types

@@ -1952,7 +2098,16 @@ both ensure correct alignment when sending structs over the network
For typical communication, the connection should be created first, in other
words, a connection between the client and the service should be
established.
+
+
@c ***************************************************************************
+@menu
+* Client setting::
+* Establish connection::
+* Initialize request message::
+* Send request and receive response::
+@end menu
+
@node Client setting
@subsubsection Client setting
@c %**end of header
@@ -2019,7 +2174,17 @@ message from the service.
@node Server Setting
@subsection Server Setting

+
@c ***************************************************************************
+@menu
+* Startup service::
+* Add new handles for specified messages::
+* Process request message::
+* Response to client::
+* Notification of clients::
+* Conversion between Network Byte Order (Big Endian) and Host Byte Order::
+@end menu
+
@node Startup service
@subsubsection Startup service

@@ -2457,7 +2622,16 @@ sometimes responsible for a large share of GNUnet's overall memory consumption
API quirks (and their implications for applications) that were recently
introduced to minimize the footprint of the hash map.

+
@c ***************************************************************************
+@menu
+* Analysis::
+* Solution::
+* Migration::
+* Conclusion::
+* Availability::
+@end menu
+
@node Analysis
@subsection Analysis
@c %**end of header
@@ -2641,7 +2815,7 @@ accessing the "next_XX" and/or "prev_XX" members.

@c ***************************************************************************
@node The Automatic Restart Manager (ARM)
-@chapter The Automatic Restart Manager (ARM)
+@section The Automatic Restart Manager (ARM)
@c %**end of header

GNUnet's Automated Restart Manager (ARM) is the GNUnet service responsible for
@@ -2830,7 +3004,7 @@ trying to restart a problematic service.

@c ***************************************************************************
@node GNUnet's TRANSPORT Subsystem
-@chapter GNUnet's TRANSPORT Subsystem
+@section GNUnet's TRANSPORT Subsystem
@c %**end of header

This chapter documents how the GNUnet transport subsystem works. The GNUnet
@@ -3056,6 +3230,15 @@ performance results presented are quite old and maybe outdated at this point.
@item Is there any additional documentation?
@end itemize

+
+@menu
+* Why use SMTP for a peer-to-peer transport?::
+* How does it work?::
+* How do I configure my peer?::
+* How do I test if it works?::
+* How fast is it?::
+@end menu
+
@node Why use SMTP for a peer-to-peer transport?
@subsection Why use SMTP for a peer-to-peer transport?
@c %**end of header
@@ -3238,6 +3421,17 @@ have any questions or problems just post them here or ask on the IRC channel.
@item How can I test it?
@end itemize

+
+
+@menu
+* What do I need to use the Bluetooth plugin transport?::
+* How does it work2?::
+* What possible errors should I be aware of?::
+* How do I configure my peer2?::
+* How can I test it?::
+* The implementation of the Bluetooth transport plugin::
+@end menu
+
@node What do I need to use the Bluetooth plugin transport?
@subsection What do I need to use the Bluetooth plugin transport?
@c %**end of header
@@ -3421,6 +3615,17 @@ platforms.
@item Pending Features
@end itemize

+
+
+@menu
+* Linux functionality::
+* THE INITIALIZATION::
+* THE LOOP::
+* Details about the broadcast implementation::
+* Windows functionality::
+* Pending features::
+@end menu
+
@node Linux functionality
@subsubsection Linux functionality
@c %**end of header
@@ -3635,7 +3840,7 @@ This section documents how the wlan transport plugin works. Parts which are not
implemented yet or could be better implemented are described at the end.

@node The ATS Subsystem
-@chapter The ATS Subsystem
+@section The ATS Subsystem
@c %**end of header

ATS stands for "automatic transport selection", and the function of ATS in
@@ -3656,7 +3861,7 @@ strategies which differ significantly in their performance and maturity, and it
is still unclear if any particular plugin is generally superior.

@node GNUnet's CORE Subsystem
-@chapter GNUnet's CORE Subsystem
+@section GNUnet's CORE Subsystem
@c %**end of header

The CORE subsystem in GNUnet is responsible for securing link-layer
@@ -3813,6 +4018,13 @@ re-established, the applications will be receive matching connect events.
This section describes the protocol between an application using the CORE
service (the client) and the CORE service process itself.

+
+@menu
+* Setup2::
+* Notifications::
+* Sending::
+@end menu
+
@node Setup2
@subsection Setup2
@c %**end of header
@@ -3882,6 +4094,14 @@ for each request).
@section The CORE Peer-to-Peer Protocol
@c %**end of header

+
+@menu
+* Creating the EphemeralKeyMessage::
+* Establishing a connection::
+* Encryption and Decryption::
+* Type maps::
+@end menu
+
@node Creating the EphemeralKeyMessage
@subsection Creating the EphemeralKeyMessage
@c %**end of header
@@ -3999,7 +4219,7 @@ the correct hash of the type map) is not received, the sender will retransmit
the type map (with exponential back-off).

@node GNUnet's CADET subsystem
-@chapter GNUnet's CADET subsystem
+@section GNUnet's CADET subsystem

The CADET subsystem in GNUnet is responsible for secure end-to-end
communications between nodes in the GNUnet overlay network. CADET builds on the
@@ -4036,8 +4256,9 @@ the sender to send more traffic than the receiver or the network are able to
process.
@end itemize

-@section libgnunetcadet
@node libgnunetcadet
+@section libgnunetcadet
+

The CADET API (defined in gnunet_cadet_service.h) is the messaging API used by
P2P applications built using GNUnet. It provides applications the ability to
@@ -4112,8 +4333,9 @@ Finally, when an application no longer wants to use CADET, it should call
@code{GNUNET_CADET_disconnect}, but first all channels and pending
transmissions must be closed (otherwise CADET will complain).

-@chapter GNUnet's NSE subsystem
@node GNUnet's NSE subsystem
+@section GNUnet's NSE subsystem
+

NSE stands for Network Size Estimation. The NSE subsystem provides other
subsystems and users with a rough estimate of the number of peers currently
@@ -4126,8 +4348,9 @@ of [2/3 estimate, 3/2 estimate]. We will now give an overview of the algorithm
used to calcualte the estimate; all of the details can be found in this
technical report.

-@subsection Motivation
@node Motivation
+@subsection Motivation
+

Some subsytems, like DHT, need to know the size of the GNUnet network to
optimize some parameters of their own protocol. The decentralized nature of
@@ -4386,8 +4609,9 @@ Finally, when it comes to send the stored message for the current round to the
neighbors there is a random delay added for each neighbor, to avoid traffic
spikes and minimize cross-messages.

-@chapter GNUnet's HOSTLIST subsystem
@node GNUnet's HOSTLIST subsystem
+@section GNUnet's HOSTLIST subsystem
+
@c %**end of header

Peers in the GNUnet overlay network need address information so that they can
@@ -4638,8 +4862,9 @@ For more information on how to configure the HOSTLIST subsystem see the
installation handbook:@ Configuring the hostlist to bootstrap@ Configuring your
peer to provide a hostlist

-@chapter GNUnet's IDENTITY subsystem
@node GNUnet's IDENTITY subsystem
+@section GNUnet's IDENTITY subsystem
+
@c %**end of header

Identities of "users" in GNUnet are called egos. Egos can be used as pseudonyms
@@ -4797,8 +5022,9 @@ message, which includes the name of the service function. The identity service
will respond to a GET_DEFAULT request with a SET_DEFAULT message containing the
respective information, or with a RESULT_CODE to indicate an error.

-@chapter GNUnet's NAMESTORE Subsystem
@node GNUnet's NAMESTORE Subsystem
+@section GNUnet's NAMESTORE Subsystem
+
@c %**end of header

The NAMESTORE subsystem provides persistent storage for local GNS zone
@@ -4910,8 +5136,9 @@ zone, the label and the records and their number.
To stop monitoring, the client call @code{GNUNET_NAMESTORE_zone_monitor_stop}
and passes the handle obtained from the function to start the monitoring.

-@chapter GNUnet's PEERINFO subsystem
@node GNUnet's PEERINFO subsystem
+@section GNUnet's PEERINFO subsystem
+
@c %**end of header

The PEERINFO subsystem is used to store verified (validated) information about
@@ -5102,8 +5329,9 @@ will notify you about every change and the callback function will be called to
notify you about changes. The function returns a handle to cancel notifications
with @code{GNUNET_PEERINFO_notify_cancel}.

-@chapter GNUnet's PEERSTORE subsystem
@node GNUnet's PEERSTORE subsystem
+@section GNUnet's PEERSTORE subsystem
+
@c %**end of header

GNUnet's PEERSTORE subsystem offers persistent per-peer storage for other
@@ -5197,7 +5425,7 @@ the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will delay the
disconnection until all pending STORE requests are sent to the PEERSTORE
service, otherwise, the pending STORE requests will be destroyed as well.

-@chapter GNUnet's SET Subsystem
+@section GNUnet's SET Subsystem
@node GNUnet's SET Subsystem
@c %**end of header

@@ -5490,8 +5718,9 @@ All Bloom filter operations use a salt to mingle keys before hasing them into
buckets, such that future iterations have a fresh chance of succeeding if they
failed due to collisions before.

-@chapter GNUnet's STATISTICS subsystem
@node GNUnet's STATISTICS subsystem
+@section GNUnet's STATISTICS subsystem
+
@c %**end of header

In GNUnet, the STATISTICS subsystem offers a central place for all subsystems
@@ -5663,8 +5892,9 @@ notifications through messages of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH_VALUE} whenever the statistic
parameter's value is changed.

-@chapter GNUnet's Distributed Hash Table (DHT)
@node GNUnet's Distributed Hash Table (DHT)
+@section GNUnet's Distributed Hash Table (DHT)
+
@c %**end of header

GNUnet includes a generic distributed hash table that can be used by developers
@@ -6027,8 +6257,9 @@ filter accordingly, to ensure that the same result is never forwarded more than
once. The DHT service may also cache forwarded results locally if the
"CACHE_RESULTS" option is set to "YES" in the configuration.

-@chapter The GNU Name System (GNS)
@node The GNU Name System (GNS)
+@section The GNU Name System (GNS)
+
@c %**end of header

The GNU Name System (GNS) is a decentralized database that enables users to
@@ -6337,8 +6568,9 @@ use alternative means of resolving names (such as sending queries to a DNS
server directly by themselves). This includes some of well known utilities,
like "ping" and "nslookup".

-@chapter The GNS Namecache
@node The GNS Namecache
+@section The GNS Namecache
+
@c %**end of header

The NAMECACHE subsystem is responsible for caching (encrypted) resolution
@@ -6447,8 +6679,8 @@ can done either by simply adding new blocks and selecting for the most recent
expiration time during lookup, or by checking which block is more recent during
the store operation.

-@chapter The REVOCATION Subsystem
@node The REVOCATION Subsystem
+@section The REVOCATION Subsystem
@c %**end of header

The REVOCATION subsystem is responsible for key revocation of Egos. If a user
@@ -6457,8 +6689,9 @@ REVOCATION system to inform all of the other users that this private key is no
longer valid. The subsystem thus includes ways to query for the validity of
keys and to propagate revocation messages.

-@section Dissemination
@node Dissemination
+@section Dissemination
+
@c %**end of header

When a revocation is performed, the revocation is first of all disseminated by
@@ -6514,6 +6747,7 @@ been revoked. The given callback will be invoked with the result of the check.
The query can be cancelled using @code{GNUNET_REVOCATION_query_cancel} on the
return value.

+
@subsection Preparing revocations
@node Preparing revocations
@c %**end of header
@@ -6600,8 +6834,9 @@ peers at any time; however, well-behaved peers should only initiate this
operation once after establishing a connection to a peer with a larger hashed
peer identity.

-@chapter GNUnet's File-sharing (FS) Subsystem
@node GNUnet's File-sharing (FS) Subsystem
+@section GNUnet's File-sharing (FS) Subsystem
+
@c %**end of header

This chapter describes the details of how the file-sharing service works. As
@@ -6752,8 +6987,9 @@ this directory structure is flat and does not mirror the structure of the
publishing operation. Note that unindex operations cannot have associated child
operations.

-@chapter GNUnet's REGEX Subsystem
@node GNUnet's REGEX Subsystem
+@section GNUnet's REGEX Subsystem
+
@c %**end of header

Using the REGEX subsystem, you can discover peers that offer a particular
--
2.13.0

From 7608a4818223eccc734b70672ea33399967bf1aa Mon Sep 17 00:00:00 2001
From: Adriano Peluso <catonano@???>
Date: Sun, 21 May 2017 21:17:27 +0200
Subject: [PATCH 5/5] almost all warnings eliminated. Only a few left

---
developer.texi | 640 +++++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 505 insertions(+), 135 deletions(-)

diff --git a/developer.texi b/developer.texi
index c3aae19..07c4234 100644
--- a/developer.texi
+++ b/developer.texi
@@ -52,6 +52,7 @@ advertisements for bogus websites.



+
@menu
* Developer Introduction::
* Code overview::
@@ -98,19 +99,75 @@ advertisements for bogus websites.
* GNUnet's CADET subsystem::
* libgnunetcadet::
* GNUnet's NSE subsystem::
+* libgnunetnse::
+* The NSE Client-Service Protocol::
+* The NSE Peer-to-Peer Protocol::
* GNUnet's HOSTLIST subsystem::
+* HELLOs::
+* Overview for the HOSTLIST subsystem::
+* Interacting with the HOSTLIST daemon::
+* Hostlist security address validation::
+* The HOSTLIST daemon::
+* The HOSTLIST server::
+* The HOSTLIST client::
+* Usage::
* GNUnet's IDENTITY subsystem::
+* libgnunetidentity::
+* The IDENTITY Client-Service Protocol::
* GNUnet's NAMESTORE Subsystem::
+* libgnunetnamestore::
* GNUnet's PEERINFO subsystem::
+* Features2::
+* Limitations3::
+* DeveloperPeer Information::
+* Startup::
+* Managing Information::
+* Obtaining Information::
+* The PEERINFO Client-Service Protocol::
+* libgnunetpeerinfo::
* GNUnet's PEERSTORE subsystem::
+* Functionality::
+* Architecture::
+* libgnunetpeerstore::
+* GNUnet's SET Subsystem::
+* Local Sets::
+* Set Modifications::
+* Set Operations::
+* Result Elements::
+* libgnunetset::
+* The SET Client-Service Protocol::
+* The SET-Intersection Peer-to-Peer Protocol::
+* The SET-Union Peer-to-Peer Protocol::
* GNUnet's STATISTICS subsystem::
+* libgnunetstatistics::
+* The STATISTICS Client-Service Protocol::
* GNUnet's Distributed Hash Table (DHT)::
+* Block library and plugins::
+* libgnunetdht::
+* The DHT Client-Service Protocol::
+* The DHT Peer-to-Peer Protocol::
* The GNU Name System (GNS)::
+* libgnunetgns::
+* libgnunetgnsrecord::
+* GNS plugins::
+* The GNS Client-Service Protocol::
+* Hijacking the DNS-Traffic using gnunet-service-dns::
+* Serving DNS lookups via GNS on W32::
* The GNS Namecache::
+* libgnunetnamecache::
+* The NAMECACHE Client-Service Protocol::
+* The NAMECACHE Plugin API::
* The REVOCATION Subsystem::
* Dissemination::
+* Revocation Message Design Requirements::
+* libgnunetrevocation::
+* The REVOCATION Client-Service Protocol::
+* The REVOCATION Peer-to-Peer Protocol::
* GNUnet's File-sharing (FS) Subsystem::
+* Encoding for Censorship-Resistant Sharing (ECRS)::
+* File-sharing persistence directory structure::
* GNUnet's REGEX Subsystem::
+* How to run the regex profiler::
@end menu

@node Developer Introduction
@@ -4348,6 +4405,13 @@ of [2/3 estimate, 3/2 estimate]. We will now give an overview of the algorithm
used to calcualte the estimate; all of the details can be found in this
technical report.

+
+
+@menu
+* Motivation::
+* Principle::
+@end menu
+
@node Motivation
@subsection Motivation

@@ -4361,8 +4425,15 @@ protocols may allow any malicious peer to manipulate the final result or to
take advantage of the system to perform DoS (Denial of Service) attacks against
the network. GNUnet's NSE protocol avoids these drawbacks.

-@subsubsection Security
+
+
+@menu
+* Security::
+@end menu
+
@node Security
+@subsubsection Security
+

The NSE subsystem is designed to be resilient against these attacks. It uses
@uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs of work} to
@@ -4373,8 +4444,9 @@ calculated periodically and out-of-time traffic is either ignored or stored for
later retransmission by benign peers. In particular, peers cannot trigger
global network communication at will.

-@subsection Principle
@node Principle
+@subsection Principle
+

The algorithm calculates the estimate by finding the globally closest peer ID
to a random, time-based value.
@@ -4382,8 +4454,20 @@ to a random, time-based value.
The idea is that the closer the ID is to the random value, the more "densely
packed" the ID space is, and therefore, more peers are in the network.

-@subsubsection Example
+
+
+@menu
+* Example::
+* Algorithm::
+* Target value::
+* Timing::
+* Controlled Flooding::
+* Calculating the estimate::
+@end menu
+
@node Example
+@subsubsection Example
+

Suppose all peers have IDs between 0 and 100 (our ID space), and the random
value is 42. If the closest peer has the ID 70 we can imagine that the average
@@ -4394,15 +4478,17 @@ them. Naturally, we could have been rather unlucky, and there is only one peer
and happens to have the ID 44. Thus, the current estimate is calculated as the
average over multiple rounds, and not just a single sample.

-@subsubsection Algorithm
@node Algorithm
+@subsubsection Algorithm
+

Given that example, one can imagine that the job of the subsystem is to
efficiently communicate the ID of the closest peer to the target value to all
the other peers, who will calculate the estimate from it.

-@subsubsection Target value
@node Target value
+@subsubsection Target value
+
@c %**end of header

The target value itself is generated by hashing the current time, rounded down
@@ -4411,8 +4497,8 @@ to an agreed value. If the rounding amount is 1h (default) and the time is
rouning amount (in this example would be every hour). Every repetition is
called a round.

-@subsubsection Timing
@node Timing
+@subsubsection Timing
@c %**end of header

The NSE subsystem has some timing control to avoid everybody broadcasting its
@@ -4426,8 +4512,9 @@ the middle of the round. If its bigger it will be earlier and if its smaler
(the most likely case) it will be later. This ensures that the peers closests
to the target value start broadcasting their ID the first.

-@subsubsection Controlled Flooding
@node Controlled Flooding
+@subsubsection Controlled Flooding
+
@c %**end of header

When a peer receives a value, first it verifies that it is closer than the
@@ -4443,8 +4530,9 @@ values, since a better value can come before the broadcast time, rendering the
previous one obsolete and saving the traffic that would have been used to
broadcast it to the neighbors.

-@subsubsection Calculating the estimate
@node Calculating the estimate
+@subsubsection Calculating the estimate
+
@c %**end of header

Once the closest ID has been spread across the network each peer gets the exact
@@ -4459,8 +4547,9 @@ size could be half of the estimate or twice as much). Note that the actual
network size is calculated in powers of two of the raw input, thus one bit of
uncertainty means a factor of two in the size estimate.

-@section libgnunetnse
@node libgnunetnse
+@section libgnunetnse
+
@c %**end of header

The NSE subsystem has the simplest API of all services, with only two calls:
@@ -4476,8 +4565,16 @@ the first. The default round time is set to 1 hour.
The disconnect call disconnects from the NSE subsystem and the callback is no
longer called with new estimates.

-@subsection Results
+
+
+@menu
+* Results::
+* Examples2::
+@end menu
+
@node Results
+@subsection Results
+
@c %**end of header

The callback provides two values: the average and the
@@ -4512,8 +4609,9 @@ standard deviation value, not only the average (in particular, if the standard
veriation is very high, the average maybe meaningless: the network size is
changing rapidly).

-@subsection Examples2
@node Examples2
+@subsection Examples2
+
@c %**end of header

Let's close with a couple examples.
@@ -4538,8 +4636,9 @@ To put this in perspective, if someone remembers the LHC Higgs boson results,
were announced with "5 sigma" and "6 sigma" certainties. In this case a 5 sigma
minimum would be 2 million and a 6 sigma minimum, 1.8 million.

-@section The NSE Client-Service Protocol
@node The NSE Client-Service Protocol
+@section The NSE Client-Service Protocol
+
@c %**end of header

As with the API, the client-service protocol is very simple, only has 2
@@ -4557,8 +4656,9 @@ respective round.
When the @code{GNUNET_NSE_disconnect} API call is executed, the client simply
disconnects from the service, with no message involved.

-@section The NSE Peer-to-Peer Protocol
@node The NSE Peer-to-Peer Protocol
+@section The NSE Peer-to-Peer Protocol
+
@c %**end of header

The NSE subsystem only has one message in the P2P protocol, the
@@ -4636,8 +4736,9 @@ outdated set of HELLO messages from the distribution. In this case, getting new
peers to connect to the network requires either manual effort or the use of a
HOSTLIST to obtain HELLOs.

-@section HELLOs
@node HELLOs
+@section HELLOs
+
@c %**end of header

The basic information peers require to connect to other peers are contained in
@@ -4646,8 +4747,9 @@ identity of the peer (based on the cryptographic public key) a HELLO message
may contain address information that specifies ways to contact a peer. By
obtaining HELLO messages, a peer can learn how to contact other peers.

-@section Overview for the HOSTLIST subsystem
@node Overview for the HOSTLIST subsystem
+@section Overview for the HOSTLIST subsystem
+
@c %**end of header

The HOSTLIST subsystem provides a way to distribute and obtain contact
@@ -4663,8 +4765,15 @@ hostlist format is a binary blob containing a sequence of HELLO messages. Note
that any HTTP server can theoretically serve a hostlist, the build-in hostlist
server makes it simply convenient to offer this service.

-@subsection Features
+
+@menu
+* Features::
+* Limitations2::
+@end menu
+
@node Features
+@subsection Features
+
@c %**end of header

The HOSTLIST daemon can:
@@ -4679,8 +4788,9 @@ gossip
@item automatically learn about hostlist servers from the gossip of other peers
@end itemize

-@subsection Limitations2
@node Limitations2
+@subsection Limitations2
+
@c %**end of header

The HOSTLIST daemon does not:
@@ -4690,8 +4800,9 @@ The HOSTLIST daemon does not:
@item verify the address information in the HELLO messages
@end itemize

-@section Interacting with the HOSTLIST daemon
@node Interacting with the HOSTLIST daemon
+@section Interacting with the HOSTLIST daemon
+
@c %**end of header

The HOSTLIST subsystem is currently implemented as a daemon, so there is no
@@ -4714,8 +4825,9 @@ shutdown if changes to this value are to have any effect on the daemon (as
HOSTLIST does not monitor STATISTICS for changes to the download
frequency).

-@section Hostlist security: address validation
-@node Hostlist security: address validation
+@node Hostlist security address validation
+@section Hostlist security address validation
+
@c %**end of header

Since information obtained from other parties cannot be trusted without
@@ -4732,8 +4844,9 @@ HOSTLIST servers specified in the configuration, downloads the (unvalidated)
list of HELLO messages and forwards these information to the TRANSPORT server
to validate the addresses.

-@section The HOSTLIST daemon
@node The HOSTLIST daemon
+@section The HOSTLIST daemon
+
@c %**end of header

The hostlist daemon is the main component of the HOSTLIST subsystem. It is
@@ -4759,8 +4872,9 @@ events.
To clean up on shutdown, the daemon has a cleaning task, shutting down all
subsystems and disconnecting from CORE.

-@section The HOSTLIST server
@node The HOSTLIST server
+@section The HOSTLIST server
+
@c %**end of header

The server provides a way for other peers to obtain HELLOs. Basically it is a
@@ -4768,8 +4882,15 @@ small web server other peers can connect to and download a list of HELLOs using
standard HTTP; it may also advertise the URL of the hostlist to other peers
connecting on CORE level.

-@subsection The HTTP Server
+
+@menu
+* The HTTP Server::
+* Advertising the URL::
+@end menu
+
@node The HTTP Server
+@subsection The HTTP Server
+
@c %**end of header

During startup, the server starts a web server listening on the port specified
@@ -4791,8 +4912,9 @@ as a HTTP response and the the server will terminate the connection with the
result code HTTP 200 OK. The connection will be closed immediately if no
hostlist is available.

-@subsection Advertising the URL
@node Advertising the URL
+@subsection Advertising the URL
+
@c %**end of header

The server also advertises the URL to download the hostlist to other peers if
@@ -4800,8 +4922,9 @@ hostlist advertisement is enabled. When a new peer connects and has hostlist
learning enabled, the server sends a GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT
message to this peer using the CORE service.

-@section The HOSTLIST client
@node The HOSTLIST client
+@section The HOSTLIST client
+
@c %**end of header

The client provides the functionality to download the list of HELLOs from a set
@@ -4813,8 +4936,15 @@ validation.
The client supports two modes of operation: download of HELLOs (bootstrapping)
and learning of URLs.

-@subsection Bootstrapping
+
+@menu
+* Bootstrapping::
+* Learning::
+@end menu
+
@node Bootstrapping
+@subsection Bootstrapping
+
@c %**end of header

For bootstrapping, it schedules a task to download the hostlist from the set of
@@ -4835,8 +4965,9 @@ full HELLO was downloaded, the HOSTLIST client offers this HELLO message to the
TRANSPORT service for validation. When the download is finished or failed,
statistical information about the quality of this URL is updated.

-@subsection Learning
@node Learning
+@subsection Learning
+
@c %**end of header

The client also manages hostlist advertisements from other peers. The HOSTLIST
@@ -4851,8 +4982,9 @@ through successful downloads and number of HELLOs e.g.) is discarded. During
shutdown the list of URLs is saved to a file for persistance and loaded on
startup. URLs from the configuration file are never discarded.

-@section Usage
@node Usage
+@section Usage
+
@c %**end of header

To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES section
@@ -4906,12 +5038,22 @@ fixed and known to everyone. Thus, anyone can perform actions as anonymous.
This can be useful as with this trick, code does not have to contain a special
case to distinguish between anonymous and pseudonymous egos.

-@section libgnunetidentity
@node libgnunetidentity
+@section libgnunetidentity
@c %**end of header

-@subsection Connecting to the service
+
+@menu
+* Connecting to the service::
+* Operations on Egos::
+* The anonymous Ego::
+* Convenience API to lookup a single ego::
+* Associating egos with service functions::
+@end menu
+
@node Connecting to the service
+@subsection Connecting to the service
+
@c %**end of header

First, typical clients connect to the identity service using
@@ -4943,8 +5085,9 @@ The ego handle passed to the callback remains valid until the callback is
invoked with a name of NULL, so it is safe to store a reference to the ego's
handle.

-@subsection Operations on Egos
@node Operations on Egos
+@subsection Operations on Egos
+
@c %**end of header

Given an ego handle, the main operations are to get its associated private key
@@ -4964,8 +5107,9 @@ respective continuations would be called. It is not guaranteed that the
operation will not be completed anyway, only the continuation will no longer be
called.

-@subsection The anonymous Ego
@node The anonymous Ego
+@subsection The anonymous Ego
+
@c %**end of header

A special way to obtain an ego handle is to call
@@ -4976,8 +5120,9 @@ signatures (for example, to avoid a special path in the code). The anonymous
ego is always valid and accessing it does not require a connection to the
identity service.

-@subsection Convenience API to lookup a single ego
@node Convenience API to lookup a single ego
+@subsection Convenience API to lookup a single ego
+

As applications commonly simply have to lookup a single ego, there is a
convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to
@@ -4987,16 +5132,18 @@ will only be valid during that callback. The operation can be cancelled via
@code{GNUNET_IDENTITY_ego_lookup_cancel} (cancellation is only legal before the
callback is invoked).

-@subsection Associating egos with service functions
@node Associating egos with service functions
+@subsection Associating egos with service functions
+

The @code{GNUNET_IDENTITY_set} function is used to associate a particular ego
with a service function. The name used by the service and the ego are given as
arguments. Afterwards, the service can use its name to lookup the associated
ego using @code{GNUNET_IDENTITY_get}.

-@section The IDENTITY Client-Service Protocol
@node The IDENTITY Client-Service Protocol
+@section The IDENTITY Client-Service Protocol
+
@c %**end of header

A client connecting to the identity service first sends a message with type
@@ -5050,8 +5197,9 @@ a specific or all zones and to monitor zones for changes. NAMESTORE
functionality can be accessed using the NAMESTORE api or the NAMESTORE command
line tool.

-@section libgnunetnamestore
@node libgnunetnamestore
+@section libgnunetnamestore
+
@c %**end of header

To interact with NAMESTORE clients first connect to the NAMESTORE service using
@@ -5067,8 +5215,16 @@ private keys can be obtained from the IDENTITY subsytem. Here @emph{egos@emph{
can be used to refer to zones or the default ego assigned to the GNS subsystem
can be used to obtained the master zone's private key.}}

-@subsection Editing Zone Information
+
+@menu
+* Editing Zone Information::
+* Iterating Zone Information::
+* Monitoring Zone Information::
+@end menu
+
@node Editing Zone Information
+@subsection Editing Zone Information
+
@c %**end of header

NAMESTORE provides functions to lookup records stored under a label in a zone
@@ -5102,8 +5258,9 @@ and records stored in a zone. Here the client uses the
zone, the nickname as string plus a the callback with the result of the
operation.

-@subsection Iterating Zone Information
@node Iterating Zone Information
+@subsection Iterating Zone Information
+
@c %**end of header

A client can iterate over all information in a zone or all zones managed by
@@ -5119,8 +5276,9 @@ NAMESTORE calls the callback for every result and expects the client to call@
NAMESTORE reached the last item it will call the callback with a NULL value to
indicate.

-@subsection Monitoring Zone Information
@node Monitoring Zone Information
+@subsection Monitoring Zone Information
+
@c %**end of header

Clients can also monitor zones to be notified about changes. Here the clients
@@ -5156,8 +5314,9 @@ subsystems tend to need to store per-peer information in persistent way. To not
duplicate this functionality we plan to provide a PEERSTORE service providing
this functionality

-@section Features2
@node Features2
+@section Features2
+
@c %**end of header

@itemize @bullet
@@ -5167,15 +5326,17 @@ this functionality
@item Differentiation between public and friend-only HELLO
@end itemize

-@section Limitations3
@node Limitations3
+@section Limitations3
+

@itemize @bullet
@item Does not perform HELLO validation
@end itemize

-@section DeveloperPeer Information
@node DeveloperPeer Information
+@section DeveloperPeer Information
+
@c %**end of header

The PEERINFO subsystem stores these information in the form of HELLO messages
@@ -5202,8 +5363,9 @@ purposes. Clients are for example the HOSTLIST component providing these
information to other peers in form of a hostlist or the TRANSPORT subsystem
using these information to maintain connections to other peers.

-@section Startup
@node Startup
+@section Startup
+
@c %**end of header

During startup the PEERINFO services loads persistent HELLOs from disk. First
@@ -5216,8 +5378,9 @@ the distribution. The use of these HELLOs can be prevented by setting the
@code{USE_INCLUDED_HELLOS} in the @code{PEERINFO} configuration section to
@code{NO}. Files containing invalid information are removed.

-@section Managing Information
@node Managing Information
+@section Managing Information
+
@c %**end of header

The PEERINFO services stores information about known PEERS and a single HELLO
@@ -5235,8 +5398,9 @@ it finds. Expired TRANSPORT addresses are removed from the HELLO and if the
HELLO does not contain any valid addresses, it is discarded and removed from
disk.

-@section Obtaining Information
@node Obtaining Information
+@section Obtaining Information
+
@c %**end of header

When a client requests information from PEERINFO, PEERINFO performs a lookup
@@ -5250,8 +5414,9 @@ list of clients interested in this notifications. Such a notification occurs if
a HELLO for a peer was updated (due to a merge for example) or a new peer was
added.

-@section The PEERINFO Client-Service Protocol
@node The PEERINFO Client-Service Protocol
+@section The PEERINFO Client-Service Protocol
+
@c %**end of header

To connect and disconnect to and from the PEERINFO Service PEERINFO utilizes
@@ -5279,16 +5444,25 @@ message is @code{struct GNUNET_MessageHeader} with type
@code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO}. If the client receives this message,
he can proceed with the next request if any is pending

-@section libgnunetpeerinfo
@node libgnunetpeerinfo
+@section libgnunetpeerinfo
+
@c %**end of header

The PEERINFO API consists mainly of three different functionalities:
maintaining a connection to the service, adding new information and retrieving
information form the PEERINFO service.

-@subsection Connecting to the Service
+
+@menu
+* Connecting to the Service::
+* Adding Information::
+* Obtaining Information2::
+@end menu
+
@node Connecting to the Service
+@subsection Connecting to the Service
+
@c %**end of header

To connect to the PEERINFO service the function @code{GNUNET_PEERINFO_connect}
@@ -5296,8 +5470,9 @@ is used, taking a configuration handle as an argument, and to disconnect from
PEERINFO the function @code{GNUNET_PEERINFO_disconnect}, taking the PEERINFO
handle returned from the connect function has to be called.

-@subsection Adding Information
@node Adding Information
+@subsection Adding Information
+
@c %**end of header

@code{GNUNET_PEERINFO_add_peer} adds a new peer to the PEERINFO subsystem
@@ -5309,8 +5484,9 @@ operation allowing to cancel the operation with the respective cancel function
you can iterate over all information stored with PEERINFO or you can tell
PEERINFO to notify if new peer information are available.

-@subsection Obtaining Information2
@node Obtaining Information2
+@subsection Obtaining Information2
+
@c %**end of header

To iterate over information in PEERINFO you use @code{GNUNET_PEERINFO_iterate}.
@@ -5347,8 +5523,9 @@ the following fields:
@item expiry: record expiry date.
@end itemize

-@section Functionality
@node Functionality
+@section Functionality
+
@c %**end of header

Subsystems can store any type of value under a (subsystem, peerid, key)
@@ -5371,8 +5548,9 @@ Subsystems can also request to be notified about any new values stored under a
(subsystem, peerid, key) combination by sending a "watch" request to
PEERSTORE.

-@section Architecture
@node Architecture
+@section Architecture
+
@c %**end of header

PEERSTORE implements the following components:
@@ -5385,8 +5563,9 @@ issue commands to the PEERSTORE service.
"sqlite" plugin is implemented.
@end itemize

-@section libgnunetpeerstore
@node libgnunetpeerstore
+@section libgnunetpeerstore
+
@c %**end of header

libgnunetpeerstore is the library containing the PEERSTORE API. Subsystems
@@ -5425,8 +5604,9 @@ the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will delay the
disconnection until all pending STORE requests are sent to the PEERSTORE
service, otherwise, the pending STORE requests will be destroyed as well.

-@section GNUnet's SET Subsystem
@node GNUnet's SET Subsystem
+@section GNUnet's SET Subsystem
+
@c %**end of header

The SET service implements efficient set operations between two peers over a
@@ -5435,8 +5615,9 @@ operations. Elements of a set consist of an @emph{element type} and arbitrary
binary @emph{data}. The size of an element's data is limited to around 62
KB.

-@section Local Sets
@node Local Sets
+@section Local Sets
+
@c %**end of header

Sets created by a local client can be modified and reused for multiple
@@ -5447,8 +5628,9 @@ of a set is determined upon its creation. If a the elements of a set are needed
for an operation of a different type, all of the set's element must be copied
to a new set of appropriate type.

-@section Set Modifications
@node Set Modifications
+@section Set Modifications
+
@c %**end of header

Even when set operations are active, one can add to and remove elements from a
@@ -5458,8 +5640,9 @@ sees a snapshot of the set from the time the operation was started. This
mechanism is @emph{not} implemented by copying the whole set, but by attaching
@emph{generation information} to each element and operation.

-@section Set Operations
@node Set Operations
+@section Set Operations
+
@c %**end of header

Set operations can be started in two ways: Either by accepting an operation
@@ -5473,8 +5656,9 @@ application id. Once notified of an incoming set request, the client can
accept the set request (providing a local set for the operation) or reject
it.

-@section Result Elements
@node Result Elements
+@section Result Elements
+
@c %**end of header

The SET service has three @emph{result modes} that determine how an operation's
@@ -5495,12 +5679,22 @@ elements. This can be useful if only the remove peer is actually interested in
the result of the set operation.
@end itemize

-@section libgnunetset
@node libgnunetset
+@section libgnunetset
+
@c %**end of header

-@subsection Sets
+@menu
+* Sets::
+* Listeners::
+* Operations::
+* Supplying a Set::
+* The Result Callback::
+@end menu
+
@node Sets
+@subsection Sets
+
@c %**end of header

New sets are created with @code{GNUNET_SET_create}. Both the local peer's
@@ -5513,8 +5707,9 @@ functions dealing with sets. This return value must always be checked.
Elements are added and removed with @code{GNUNET_SET_add_element} and
@code{GNUNET_SET_remove_element}.

-@subsection Listeners
@node Listeners
+@subsection Listeners
+
@c %**end of header

Listeners are created with @code{GNUNET_SET_listen}. Each time time a remote
@@ -5524,8 +5719,9 @@ synchronously call either @code{GNUNET_SET_accept} or @code{GNUNET_SET_reject}.
Note that the operation will not be started until the client calls
@code{GNUNET_SET_commit} (see Section "Supplying a Set").

-@subsection Operations
@node Operations
+@subsection Operations
+
@c %**end of header

Operations to be initiated by the local peer are created with
@@ -5533,8 +5729,9 @@ Operations to be initiated by the local peer are created with
the client calls @code{GNUNET_SET_commit} (see Section "Supplying a
Set").

-@subsection Supplying a Set
@node Supplying a Set
+@subsection Supplying a Set
+
@c %**end of header

To create symmetry between the two ways of starting a set operation (accepting
@@ -5546,8 +5743,9 @@ The client must call @code{GNUNET_SET_commit} to specify a set to use for an
operation. @code{GNUNET_SET_commit} may only be called once per set
operation.

-@subsection The Result Callback
@node The Result Callback
+@subsection The Result Callback
+
@c %**end of header

Clients must specify both a result mode and a result callback with
@@ -5558,12 +5756,22 @@ result mode. The callback needs to know which result mode it is used in, as the
arguments do not indicate if an element is part of the full result set, or if
it is in the difference between the original set and the final set.

-@section The SET Client-Service Protocol
@node The SET Client-Service Protocol
+@section The SET Client-Service Protocol
+
@c %**end of header

-@subsection Creating Sets
+@menu
+* Creating Sets::
+* Listeners2::
+* Initiating Operations::
+* Modifying Sets::
+* Iterating Sets::
+@end menu
+
@node Creating Sets
+@subsection Creating Sets
+
@c %**end of header

For each set of a client, there exists a client connection to the service. Sets
@@ -5571,8 +5779,9 @@ are created by sending the @code{GNUNET_SERVICE_SET_CREATE} message over a new
client connection. Multiple operations for one set are multiplexed over one
client connection, using a request id supplied by the client.

-@subsection Listeners2
@node Listeners2
+@subsection Listeners2
+
@c %**end of header

Each listener also requires a seperate client connection. By sending the
@@ -5583,29 +5792,39 @@ client connection. In contrast, when accepting an incoming request, a a
@code{GNUNET_SERVICE_SET_ACCEPT} message must be sent over the@ set that is
supplied for the set operation.

-@subsection Initiating Operations
@node Initiating Operations
+@subsection Initiating Operations
+
@c %**end of header

Operations with remote peers are initiated by sending a
@code{GNUNET_SERVICE_SET_EVALUATE} message to the service. The@ client
connection that this message is sent by determines the set to use.

-@subsection Modifying Sets
@node Modifying Sets
+@section Modifying Sets
+
@c %**end of header

Sets are modified with the @code{GNUNET_SERVICE_SET_ADD} and
@code{GNUNET_SERVICE_SET_REMOVE} messages.

+
+@menu
+* Results and Operation Status::
+* Iterating Sets::
+@end menu
+
@node Results and Operation Status
+@subsection Results and Operation Status
@c %**end of header

The service notifies the client of result elements and success/failure of a set
operation with the @code{GNUNET_SERVICE_SET_RESULT} message.

-@subsection Iterating Sets
@node Iterating Sets
+@subsection Iterating Sets
+
@c %**end of header

All elements of a set can be requested by sending
@@ -5615,8 +5834,9 @@ with @code{GNUNET_SERVICE_SET_ITER_DONE}. After each received element, the
client@ must send @code{GNUNET_SERVICE_SET_ITER_ACK}. Note that only one set
iteration may be active for a set at any given time.

-@section The SET-Intersection Peet-to-Peer Protocol
@node The SET-Intersection Peer-to-Peer Protocol
+@section The SET-Intersection Peet-to-Peer Protocol
+
@c %**end of header

The intersection protocol operates over CADET and starts with a
@@ -5639,8 +5859,15 @@ Bloom filter exchange, unless the set size is indicated to be zero, in which
case the intersection is considered finished after just the initial
handshake.

-@subsection The Bloom filter exchange
+
+@menu
+* The Bloom filter exchange::
+* Salt::
+@end menu
+
@node The Bloom filter exchange
+@subsection The Bloom filter exchange
+
@c %**end of header

In this phase, each peer transmits a Bloom filter over the remaining keys of
@@ -5662,8 +5889,9 @@ a@ GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_DONE back to indicate that the
latest set is the final result. Otherwise, the receiver starts another Bloom
fitler exchange, except this time as the sender.

-@subsection Salt
@node Salt
+@subsection Salt
+
@c %**end of header

Bloomfilter operations are probablistic: With some non-zero probability the
@@ -5679,8 +5907,9 @@ The iterations terminate once both peers have established that they have sets
of the same size, and where the XOR over all keys computes the same 512-bit
value (leaving a failure probability of 2-511).

-@section The SET-Union Peer-to-Peer Protocol
@node The SET-Union Peer-to-Peer Protocol
+@section The SET-Union Peer-to-Peer Protocol
+
@c %**end of header

The SET union protocol is based on Eppstein's efficient set reconciliation
@@ -5761,8 +5990,9 @@ before terminating itself. This is to prevent the loss of data during peer
shutdown --- delaying the STATISTICS service shutdown helps other services to
store important data to STATISTICS during shutdown.

-@section libgnunetstatistics
@node libgnunetstatistics
+@section libgnunetstatistics
+
@c %**end of header

@strong{libgnunetstatistics} is the library containing the API for the
@@ -5782,8 +6012,16 @@ configuration all calls to @code{GNUNET_STATISTICS_create()} return @code{NULL}
as the STATISTICS subsystem is unavailable and no other functions from the API
can be used.

-@subsection Statistics retrieval
+
+@menu
+* Statistics retrieval::
+* Setting statistics and updating them::
+* Watches::
+@end menu
+
@node Statistics retrieval
+@subsection Statistics retrieval
+
@c %**end of header

Once a connection to the statistics service is obtained, information about any
@@ -5803,8 +6041,9 @@ the function @code{GNUNET_STATISTICS_get_cancel()}. This is helpful when
retrieving statistics takes too long and especially when we want to shutdown
and cleanup everything.

-@subsection Setting statistics and updating them
@node Setting statistics and updating them
+@subsection Setting statistics and updating them
+
@c %**end of header

So far we have seen how to retrieve statistics, here we will learn how we can
@@ -5828,8 +6067,9 @@ the client performs requests at a rate that is faster than the available IPC
with the STATISTICS service. Thus, the client does not have to worry about
sending requests too quickly.

-@subsection Watches
@node Watches
+@subsection Watches
+
@c %**end of header

As interesting feature of STATISTICS lies in serving notifications whenever a
@@ -5845,12 +6085,20 @@ A registered watch will keep notifying any value changes until
@code{GNUNET_STATISTICS_watch_cancel()} is called with the same parameters that
are used for registering the watch.

-@section The STATISTICS Client-Service Protocol
@node The STATISTICS Client-Service Protocol
+@section The STATISTICS Client-Service Protocol
@c %**end of header

-@subsection Statistics retrieval2
+
+@menu
+* Statistics retrieval2::
+* Setting and updating statistics::
+* Watching for updates::
+@end menu
+
@node Statistics retrieval2
+@subsection Statistics retrieval2
+
@c %**end of header

To retrieve statistics, the client transmits a message of type
@@ -5861,8 +6109,9 @@ statistics parameters that match the client request for the client. The end of
information retrieved is signaled by the service by sending a message of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_END}.

-@subsection Setting and updating statistics
@node Setting and updating statistics
+@subsection Setting and updating statistics
+
@c %**end of header

The subsystem name, parameter name, its value and the persistence flag are
@@ -5882,8 +6131,9 @@ relative update by sending a message of type
(@code{GNUNET_STATISTICS_SETFLAG_RELATIVE}) signifying that the value in the
message should be treated as an update value.

-@subsection Watching for updates
@node Watching for updates
+@subsection Watching for updates
+
@c %**end of header

The function registers the watch at the service by sending a message of type
@@ -5928,12 +6178,22 @@ significant delay. Your application logic must be written to tolerate this
(naturally, some loss of performance or quality of service is expected in this
case).

-@section Block library and plugins
@node Block library and plugins
+@section Block library and plugins
+
@c %**end of header

-@subsection What is a Block?
+@menu
+* What is a Block?::
+* The API of libgnunetblock::
+* Queries::
+* Sample Code::
+* Conclusion2::
+@end menu
+
@node What is a Block?
+@subsection What is a Block?
+
@c %**end of header

Blocks are small (< 63k) pieces of data stored under a key (struct
@@ -5946,8 +6206,9 @@ for the maintenance of the DHT are both stored using blocks). The block
subsystem provides a few common functions that must be available for any type
of block.

-@subsection The API of libgnunetblock
@node The API of libgnunetblock
+@subsection The API of libgnunetblock
+
@c %**end of header

The block library requires for each (family of) block type(s) a block plugin
@@ -5974,8 +6235,8 @@ replies (by adding the current response to the Bloom filter and rejecting it if
it is encountered again). If a plugin fails to do this, responses may loop in
the network.

-@subsection Queries
@node Queries
+@subsection Queries
@c %**end of header

The query format for any block in GNUnet consists of four main components.
@@ -5998,8 +6259,9 @@ Depending on the results from the plugin, the DHT will then discard the
(invalid) query, forward the query, discard the (invalid) reply, cache the
(valid) reply, and/or forward the (valid and non-duplicate) reply.

-@subsection Sample Code
@node Sample Code
+@subsection Sample Code
+
@c %**end of header

The source code in @strong{plugin_block_test.c} is a good starting point for
@@ -6007,8 +6269,9 @@ new block plugins --- it does the minimal work by implementing a plugin that
performs no validation at all. The respective @strong{Makefile.am} shows how to
build and install a block plugin.

-@subsection Conclusion2
@node Conclusion2
+@subsection Conclusion2
+
@c %**end of header

In conclusion, GNUnet subsystems that want to use the DHT need to define a
@@ -6018,8 +6281,9 @@ and any reply as matching any query. This type is also used for the DHT command
line tools. However, it should NOT be used for normal applications due to the
lack of error checking that results from this primitive implementation.

-@section libgnunetdht
@node libgnunetdht
+@section libgnunetdht
+
@c %**end of header

The DHT API itself is pretty simple and offers the usual GET and PUT functions
@@ -6027,8 +6291,17 @@ that work as expected. The specified block type refers to the block library
which allows the DHT to run application-specific logic for data stored in the
network.

-@subsection GET
+
+@menu
+* GET::
+* PUT::
+* MONITOR::
+* DHT Routing Options::
+@end menu
+
@node GET
+@subsection GET
+
@c %**end of header

When using GET, the main consideration for developers (other than the block
@@ -6051,8 +6324,9 @@ more). This way, the DHT will filter the respective blocks using the block
library in the network, which may result in a significant reduction in
bandwidth consumption.

-@subsection PUT
@node PUT
+@subsection PUT
+
@c %**end of header

In contrast to GET operations, developers @strong{must} manually re-run PUT
@@ -6069,8 +6343,9 @@ rule of thumb is that there should be at least a dozen PUT operations within
the content lifetime. Content in the DHT typically expires after one day, so
DHT PUT operations should be repeated at least every 1-2 hours.

-@subsection MONITOR
@node MONITOR
+@subsection MONITOR
+
@c %**end of header

The DHT API also allows applications to monitor messages crossing the local
@@ -6088,8 +6363,9 @@ workers have no good way to guess the keys under which work would be stored.
Naturally, additional protocols might be needed to ensure that the desired
number of workers will process the distributed workload.

-@subsection DHT Routing Options
@node DHT Routing Options
+@subsection DHT Routing Options
+
@c %**end of header

There are two important options for GET and PUT requests:
@@ -6117,12 +6393,20 @@ the DHT's peer discovery mechanism and should not be used by applications.
the future offer performance improvements for clique topologies.
@end table

-@section The DHT Client-Service Protocol
@node The DHT Client-Service Protocol
+@section The DHT Client-Service Protocol
+
@c %**end of header

-@subsection PUTting data into the DHT
+@menu
+* PUTting data into the DHT::
+* GETting data from the DHT::
+* Monitoring the DHT::
+@end menu
+
@node PUTting data into the DHT
+@subsection PUTting data into the DHT
+
@c %**end of header

To store (PUT) data into the DHT, the client sends a@ @code{struct
@@ -6140,8 +6424,9 @@ the same key-value pair was already stored in the DHT. However, changing this
would also require additional state and messages in the P2P
interaction.

-@subsection GETting data from the DHT
@node GETting data from the DHT
+@subsection GETting data from the DHT
+
@c %**end of header

To retrieve (GET) data from the DHT, the client sends a@ @code{struct
@@ -6176,8 +6461,9 @@ is more common as this allows a client to run many concurrent GET operations
over the same connection with the DHT service --- and to stop them
individually.

-@subsection Monitoring the DHT
@node Monitoring the DHT
+@subsection Monitoring the DHT
+
@c %**end of header

To begin monitoring, the client sends a @code{struct
@@ -6191,12 +6477,20 @@ GNUNET_DHT_MonitorPutMessage} for PUT events and@ @code{struct
GNUNET_DHT_MonitorGetRespMessage} for RESULTs. Each of these messages contains
all of the information about the event.

-@section The DHT Peer-to-Peer Protocol
@node The DHT Peer-to-Peer Protocol
+@section The DHT Peer-to-Peer Protocol
@c %**end of header

-@subsection Routing GETs or PUTs
+
+@menu
+* Routing GETs or PUTs::
+* PUTting data into the DHT2::
+* GETting data from the DHT2::
+@end menu
+
@node Routing GETs or PUTs
+@subsection Routing GETs or PUTs
+
@c %**end of header

When routing GETs or PUTs, the DHT service selects a suitable subset of
@@ -6208,8 +6502,9 @@ estimate, the selection of the peers maybe randomized or by proximity to the
key. Furthermore, requests include a set of peers that a request has already
traversed; those peers are also excluded from the selection.

-@subsection PUTting data into the DHT2
@node PUTting data into the DHT2
+@subsection PUTting data into the DHT2
+
@c %**end of header

To PUT data into the DHT, the service sends a @code{struct PeerPutMessage} of
@@ -6226,8 +6521,9 @@ previous hop; however, the path should not include the identity of the previous
hop and the receiver should append the identity of the sender to the path, not
its own identity (this is done to reduce bandwidth).

-@subsection GETting data from the DHT2
@node GETting data from the DHT2
+@subsection GETting data from the DHT2
+
@c %**end of header

A peer can search the DHT by sending @code{struct PeerGetMessage}s of type
@@ -6295,8 +6591,9 @@ users, the NAMESTORE to store information specific to the local users, and the
DHT to exchange data between users. A plugin API is used to enable applications
to define new GNS record types.

-@section libgnunetgns
@node libgnunetgns
+@section libgnunetgns
+
@c %**end of header

The GNS API itself is extremely simple. Clients first connec to the GNS service
@@ -6305,8 +6602,17 @@ using @code{GNUNET_GNS_connect}. They can then perform lookups using
@code{GNUNET_GNS_lookup_cancel}. Once finished, clients disconnect using
@code{GNUNET_GNS_disconnect}.

-@subsection Looking up records
+
+@menu
+* Looking up records::
+* Accessing the records::
+* Creating records::
+* Future work::
+@end menu
+
@node Looking up records
+@subsection Looking up records
+
@c %**end of header

@code{GNUNET_GNS_lookup} takes a number of arguments:
@@ -6344,8 +6650,9 @@ longer be cancelled.
@item proc_cls The closure for proc.
@end table

-@subsection Accessing the records
@node Accessing the records
+@subsection Accessing the records
+
@c %**end of header

The @code{libgnunetgnsrecord} library provides an API to manipulate the GNS
@@ -6357,8 +6664,9 @@ applications.
For DNS records, the @code{libgnunetdnsparser} library provides functions for
parsing (and serializing) common types of DNS records.

-@subsection Creating records
@node Creating records
+@subsection Creating records
+
@c %**end of header

Creating GNS records is typically done by building the respective record
@@ -6367,16 +6675,18 @@ information (possibly with the help of @code{libgnunetgnsrecord} and
publish the information. The GNS API is not involved in this
operation.

-@subsection Future work
@node Future work
+@subsection Future work
+
@c %**end of header

In the future, we want to expand @code{libgnunetgns} to allow applications to
observe shortening operations performed during GNS resolution, for example so
that users can receive visual feedback when this happens.

-@section libgnunetgnsrecord
@node libgnunetgnsrecord
+@section libgnunetgnsrecord
+
@c %**end of header

The @code{libgnunetgnsrecord} library is used to manipulate GNS records (in
@@ -6393,8 +6703,15 @@ We will now discuss the four commonly used functions of the API.@
uses plugins to perform the operation. GNUnet includes plugins to support
common DNS record types as well as standard GNS record types.

-@subsection Value handling
+
+@menu
+* Value handling::
+* Type handling::
+@end menu
+
@node Value handling
+@subsection Value handling
+
@c %**end of header

@code{GNUNET_GNSRECORD_value_to_string} can be used to convert the (binary)
@@ -6406,8 +6723,9 @@ available plugin.
readable string to the respective (binary) representation of a GNS record
value.

-@subsection Type handling
@node Type handling
+@subsection Type handling
+
@c %**end of header

@code{GNUNET_GNSRECORD_typename_to_number} can be used to obtain the numeric
@@ -6422,8 +6740,9 @@ time.}
associated with a given numeric value. For example, given the type number 1,
the function will return the typename "A".

-@section GNS plugins
@node GNS plugins
+@section GNS plugins
+
@c %**end of header

Adding a new GNS record type typically involves writing (or extending) a
@@ -6445,8 +6764,9 @@ The central functions of the block APIs (plugin and main library) are the same
four functions for converting between values and strings, and typenames and
numbers documented in the previous section.

-@section The GNS Client-Service Protocol
@node The GNS Client-Service Protocol
+@section The GNS Client-Service Protocol
+
@c %**end of header

The GNS client-service protocol consists of two simple messages, the
@@ -6462,8 +6782,9 @@ The response includes the number of records and the records themselves in the
format created by @code{GNUNET_GNSRECORD_records_serialize}. They can thus be
deserialized using @code{GNUNET_GNSRECORD_records_deserialize}.

-@section Hijacking the DNS-Traffic using gnunet-service-dns
@node Hijacking the DNS-Traffic using gnunet-service-dns
+@section Hijacking the DNS-Traffic using gnunet-service-dns
+
@c %**end of header

This section documents how the gnunet-service-dns (and the gnunet-helper-dns)
@@ -6488,8 +6809,14 @@ application using the DNS service will be sent to the original recipient. The
answer to the query will always be sent back through the virtual interface with
the original nameserver as source address.

-@subsection Network Setup Details
+
+@menu
+* Network Setup Details::
+@end menu
+
@node Network Setup Details
+@subsection Network Setup Details
+
@c %**end of header

The DNS interceptor adds the following rules to the Linux kernel:
@@ -6504,8 +6831,9 @@ beforehand (@code{$LOCALPORT}) will be routed normally. Line 2 marks every
other packet to a DNS-Server with mark 3 (chosen arbitrarily). The third line
adds a routing policy based on this mark 3 via the routing table.

-@section Serving DNS lookups via GNS on W32
@node Serving DNS lookups via GNS on W32
+@section Serving DNS lookups via GNS on W32
+
@c %**end of header

This section documents how the libw32nsp (and gnunet-gns-helper-service-w32) do
@@ -6596,8 +6924,9 @@ always small enough (a few MB) to fit on the drive.

The NAMECACHE supports the use of different database backends via a plugin API.

-@section libgnunetnamecache
@node libgnunetnamecache
+@section libgnunetnamecache
+
@c %**end of header

The NAMECACHE API consists of five simple functions. First, there is
@@ -6616,8 +6945,9 @@ the NAMECACHE.
The maximum size of a block that can be stored in the NAMECACHE is
@code{GNUNET_NAMECACHE_MAX_VALUE_SIZE}, which is defined to be 63 kB.

-@section The NAMECACHE Client-Service Protocol
@node The NAMECACHE Client-Service Protocol
+@section The NAMECACHE Client-Service Protocol
+
@c %**end of header

All messages in the NAMECACHE IPC protocol start with the @code{struct
@@ -6626,8 +6956,15 @@ standard message header. The request ID is used to match requests with the
respective responses from the NAMECACHE, as they are allowed to happen
out-of-order.

-@subsection Lookup
+
+@menu
+* Lookup::
+* Store::
+@end menu
+
@node Lookup
+@subsection Lookup
+
@c %**end of header

The @code{struct LookupBlockMessage} is used to lookup a block stored in the
@@ -6637,8 +6974,9 @@ sets the expiration time in the response to zero. Otherwise, the response is
expected to contain the expiration time, the ECDSA signature, the derived key
and the (variable-size) encrypted data of the block.

-@subsection Store
@node Store
+@subsection Store
+
@c %**end of header

The @code{struct BlockCacheMessage} is used to cache a block in the NAMECACHE.
@@ -6647,16 +6985,23 @@ service responds with a @code{struct BlockCacheResponseMessage} which contains
the result of the operation (success or failure). In the future, we might want
to make it possible to provide an error message as well.

-@section The NAMECACHE Plugin API
@node The NAMECACHE Plugin API
+@section The NAMECACHE Plugin API
@c %**end of header

The NAMECACHE plugin API consists of two functions, @code{cache_block} to store
a block in the database, and @code{lookup_block} to lookup a block in the
database.

-@subsection Lookup2
+
+@menu
+* Lookup2::
+* Store2::
+@end menu
+
@node Lookup2
+@subsection Lookup2
+
@c %**end of header

The @code{lookup_block} function is expected to return at most one block to the
@@ -6664,8 +7009,9 @@ iterator, and return @code{GNUNET_NO} if there were no non-expired results. If
there are multiple non-expired results in the cache, the lookup is supposed to
return the result with the largest expiration time.

-@subsection Store2
@node Store2
+@subsection Store2
+
@c %**end of header

The @code{cache_block} function is expected to try to store the block in the
@@ -6711,8 +7057,9 @@ messages whenever two peers (that both support REVOCATION dissemination)
connect. The SET service is used to perform this operation
efficiently.

-@section Revocation Message: Design Requirements
-@node Revocation Message: Design Requirements
+@node Revocation Message Design Requirements
+@section Revocation Message Design Requirements
+
@c %**end of header

However, flooding is also quite costly, creating O(|E|) messages on a network
@@ -6731,15 +7078,24 @@ revoked. Thus, they can only be created while the private key is in the
possession of the respective user. This is another reason to create a
revocation message ahead of time and store it in a secure location.

-@section libgnunetrevocation
@node libgnunetrevocation
+@section libgnunetrevocation
+
@c %**end of header

The REVOCATION API consists of two parts, to query and to issue
revocations.

-@subsection Querying for revoked keys
+
+@menu
+* Querying for revoked keys::
+* Preparing revocations::
+* Issuing revocations::
+@end menu
+
@node Querying for revoked keys
+@subsection Querying for revoked keys
+
@c %**end of header

@code{GNUNET_REVOCATION_query} is used to check if a given ECDSA public key has
@@ -6747,9 +7103,9 @@ been revoked. The given callback will be invoked with the result of the check.
The query can be cancelled using @code{GNUNET_REVOCATION_query_cancel} on the
return value.

-
-@subsection Preparing revocations
@node Preparing revocations
+@subsection Preparing revocations
+
@c %**end of header

It is often desirable to create a revocation record ahead-of-time and store it
@@ -6778,8 +7134,9 @@ the future) is to be revoked and returns the signature. The signature can again
be saved to disk for later use, which will then allow performing a revocation
even without access to the private key.

-@subsection Issuing revocations
@node Issuing revocations
+@subsection Issuing revocations
+

Given a ECDSA public key, the signature from @code{GNUNET_REVOCATION_sign} and
the proof-of-work, @code{GNUNET_REVOCATION_revoke} can be used to perform the
@@ -6788,8 +7145,9 @@ operation. @code{GNUNET_REVOCATION_revoke_cancel} can be used to stop the
library from calling the continuation; however, in that case it is undefined
whether or not the revocation operation will be executed.

-@section The REVOCATION Client-Service Protocol
@node The REVOCATION Client-Service Protocol
+@section The REVOCATION Client-Service Protocol
+

The REVOCATION protocol consists of four simple messages.

@@ -6806,8 +7164,9 @@ and the proof-of-work, The service responds with a
@code{RevokeMessage} was invalid (i.e. proof of work incorrect), or otherwise
indicates that the revocation has been processed successfully.

-@section The REVOCATION Peer-to-Peer Protocol
@node The REVOCATION Peer-to-Peer Protocol
+@section The REVOCATION Peer-to-Peer Protocol
+
@c %**end of header

Revocation uses two disjoint ways to spread revocation information among peers.
@@ -6862,8 +7221,9 @@ in the FS service.

NOTE: The documentation in this chapter is quite incomplete.

-@section Encoding for Censorship-Resistant Sharing (ECRS)
@node Encoding for Censorship-Resistant Sharing (ECRS)
+@section Encoding for Censorship-Resistant Sharing (ECRS)
+
@c %**end of header

When GNUnet shares files, it uses a content encoding that is called ECRS, the
@@ -6877,8 +7237,15 @@ extensions are not in the paper is that we felt that they were obvious or
trivial extensions to the original scheme and thus did not warrant space in
the research report.

-@subsection Namespace Advertisements
+
+@menu
+* Namespace Advertisements::
+* KSBlocks::
+@end menu
+
@node Namespace Advertisements
+@subsection Namespace Advertisements
+
@c %**end of header

An @code{SBlock} with identifier ′all zeros′ is a signed
@@ -6889,8 +7256,9 @@ namespace. The URI should always be empty. The @code{SBlock} is signed with
the content provder′s RSA private key (just like any other SBlock). Peers
can search for @code{SBlock}s in order to find out more about a namespace.

-@subsection KSBlocks
@node KSBlocks
+@subsection KSBlocks
+
@c %**end of header

GNUnet implements @code{KSBlocks} which are @code{KBlocks} that, instead of
@@ -6916,8 +7284,9 @@ Collections are also advertised using @code{KSBlock}s.
@item https://gnunet.org/sites/default/files/ecrs.pdf
@end table

-@section File-sharing persistence directory structure
@node File-sharing persistence directory structure
+@section File-sharing persistence directory structure
+
@c %**end of header

This section documents how the file-sharing library implements persistence of
@@ -7002,8 +7371,9 @@ For the technical details, we have "Max's defense talk and Max's Master's
thesis. An additional publication is under preparation and available to team
members (in Git).

-@section How to run the regex profiler
@node How to run the regex profiler
+@section How to run the regex profiler
+
@c %**end of header

The gnunet-regex-profiler can be used to profile the usage of mesh/regex for a
--
2.13.0