Fetcher Node

Fetcher Nodes are vital components of the Agentic Data Coordination Service (ADCS) Node, designed to intelligently retrieve, process, and aggregate data from diverse external sources. By leveraging a structured data adapter framework, these nodes ensure that requestors have access to standardized data structures.

In the ADCS system, Fetcher Nodes serve as data provision points for oracles, ensuring they are supplied with high-quality, reliable information necessary for efficiently fulfilling consumer requests.

In addition, to further ensure data integrity and bolster network security, we are committed to open-sourcing our Fetcher Nodes. This initiative empowers anyone to run a node, fostering a collaborative environment where the community actively participates in maintaining the integrity of our data ecosystem.

Each Fetcher Node consists of three primary components:

• Fetcher: This component is responsible for intelligently fetching data from designated sources based on predefined configurations. It ensures efficient retrieval of information while adhering to the specifications set in the data adapter.

• Data Adapter: The idea behind our adapter framework is to ensure that users can request various types of data from diverse sources while maintaining compatibility with standardized formats. It acts as the critical interface that standardizes the interaction between external data sources and Fetcher within the ADCS framework.

  Each data adapter is identified by a unique adapterHash that ensures compatibility with the aggregator. This hash guarantees the data retrieved by Fetcher is correctly structured and compatible for aggregation, maintaining data integrity throughout the process.

• Aggregator: Aggregates the data retrieved from the fetchers and provides the final, aggregated data. Each node stores both a local aggregate and a global aggregate.

The Aggregator consolidates this data according to the unique specifications of each Adapter, ensuring accuracy and relevance. When data is fetched from a fetcher, it is processed using different aggregation methods, such as median or majority voting, depending on the type of data. This aggregated information is then stored in the local aggregate. Subsequently, all local aggregates are synchronized with each other to become global aggregates. When the consensus is reached, the global aggregate will be the final data for storage.

We use the Raft consensus mechanism to ensure that all nodes in the network agree on the global aggregate data, maintaining consistency and reliability while effectively managing data replication and fault tolerance.

Steps Raft Takes to Achieve Consensus for a Global Aggregate:

1. Leader Election: Initially, a leader must be elected among the nodes. This node will be responsible for managing the log replication process and ensuring that all nodes (followers) are in sync.

2. Log Entry Creation: Once a leader is elected, it proposes the global aggregate data to be stored. This is done by creating a new log entry that contains the aggregated data and then sends this log entry to all follower nodes

3. Log Replication: Each follower node receives the proposed log entry from the leader and appends it to its own log. Followers send an acknowledgment back to the leader to confirm they received and stored the log entry.

4. Commitment of the Log Entry: The leader waits for acknowledgments from a majority of the nodes (more than half of the total nodes) to confirm that the log entry has been replicated. Once the majority acknowledgment is received, the leader considers the log entry committed. It updates the state machine and finalizes the global aggregate.