Skip to main content

Basic moments

Common​

  • ID: MID-RID .
  • MID - milliseconds part of ID, generated (extracted from doc) by ingestor before sending to store.
  • RID - random part of ID, generated by ingestor before sending to store.
  • docParam - link of ID and block position, position of doc in block.
  • Only active fraction has meta file. It is used for restoring index in memory and in process of sealing fraction it is used to form index file.

Bulk​

Bulk in Ingestor​

  • Max amount of inflight queries - IngestorMaxInflightBulks.
  • Each query uses bulk object from pool, which size is IngestorMaxInflightBulks.
  • Bulk object consists from doc provider, which contains own map for speed up lz4 compression.
  • DocProvider parses input data from query and form docs and metas (parseBulk).
  • DocProvider compresses all doc in one block. The same is done for metas.
  • Send data to hot store, then to cold stores. Store replicaSet is selected by bulkTryCount.Inc() % len(replicaSets).
  • Data is sent to each replica in replicaSet.
  • If at least one replica returned an error, the next replicaSet will be selected.

Bulk in Store​

  • Check number of active requests - inflightBulks.
  • Choose active fraction.
  • Send data to WriteWorkers and IndexWorkers.
  • In write worker wait for 16 requests for writing.
  • In write worker write docs and metas to separate files for all write requests.
  • Do fsync.
  • Update stats (DocsOnDisk, MetaOnDisk).
  • For index worker set position of block of docs in file.
  • Waiting the end of the work for only writeWorker (WriteTask). In case of reloading(stopping) store, index of docs will be restored (ReplayBlocks).
  • In index worker parse metas, form index and store link of block position and ID ( docParam) (appendWorker).

Fetch​

Fetch in Ingestor​

  • Check id in hot. In case of error or lack of data in hot, fetch data from cold stores.
  • Cancel other simultaneous requests in success case.

Fetch in Store​

  • In case of single id and hot store, compare ID.MID and OldestMID on store. If Id.MID is older, returns error.
  • Form FetchTask with pointers to fraction and id/ids.
  • In fetchWorker try to fetch doc by each ids.
  • In fetchOne extracts MID from ID and selects consistently suitable fraction, check fraction.IsIntersects.
  • If fraction (active or sealed) is suitable, find docParam by ID, decode block position and position of doc in block from docParam.
  • Read block, decompress it, returns doc's content and don't start check next fraction.
  • if the document was found, forms disk.DocBlock with ID, length and uncompressed data. Otherwise, form block with zero length.

TODO:

Fraction​

Sealing​