* add free check (#1315)

* chore: bump version to 1.6.1 (#1317)

* fix legacy deal verified status (#1324)

* fix: update go-unixfsnode enough to make sure unixfs-preload is available (#1323)

* release v1.6.2-rc1 (#1328)

* use full path (#1330)

* fix bug (#1332)

* use forks of graphsync, go-data-transfer and go-fil-markets (#1333)

* refactor: use forks of graphsync, go-data-transfer and go-fil-markets

* refactor: convert from data transfer v1 to v2 voucher type

* fix: index provider validation voucher type

* fix: pass index provider engine link system through to graphsync's transport configurer

* feat: use tagged version of boost-gfm

* fix: retrieval client imports

* feat: tagged version of lotus

* feat: require go 1.19

* lint: fix lint errors

* fix: itests

* fix: cbor-gen, docsgen

* fix: update CI lint version

* fix: lint

* fix: docgen

* fix: go mod tidy

* fix: protocol proxy TestOutboundForwarding

* fix: docsgen

* fix: update filecoin-ffi submodule

* fix: prometheus duplicate register panic

* fix: cleanup imports

* fix: legs voucher processing

* chore: release v1.6.2-rc2 (#1340)

* release v1.6.2-rc2

* fix test

* fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (#1350)

* fix: flaky TestDealCompletionOnProcessResumption (#1351)

* fix: occasional panic on shutdown (#1353)

* feat: query UI (#1352)

* log insert

* fix display error

* refactor code

* shorten status strings

* remove comment

* apply suggestion

* feat: add download block link to inspect page (#1312)

* fix(devnet): update golang and lotus default versions (#1354)

* fix(devnet): bump golang to 1.19

* chore(devnet): bump lotus default version

* chore(devnet): remove unused stable env

* booster-http: implement IPFS HTTP gateway (#1225)

* feat: implement http api gateway

* feat: use go-libipfs lib (instead of copying to extern)

* feat: bump booster-bitswap info minor version

* feat: http gateway metrics

* fix: TestHttpInfo

* feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc)

* fix: correct link for download root block (#1355)

* feat: option to cleanup data for offline deals after add piece (#1341)

* chore: add support for multiple node.js versions in makefile (#1356)

* chore: release v.1.7.0-rc1 (#1357)

* release v.1.7.0-rc1

* fix version

* fix: dagstore initialize-all parameter (#1363)

* fix: show verifying commp state for offline deals (#1364)

* fix: boost run missing staging-area dir (#1368)

* merge(wip): main to lid

TODO: remoteblockstore needs to handle nil metrics

* fix: flaky TestNewHttpServer (#1372)

* feat: group agent version by binary name (#1369)

* fix: wrap stats in nil checks for now

we should probably revisit how stats are handled now that we have all 3 transports being tracked

* test(fix): incorrect test urls

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* feat: add local index directory

Internally this is still refered to as the piece directory

Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* refactor: merge from main into lid branch (#1339)

* merge: main to lid (#1370)

* add free check (#1315)

* chore: bump version to 1.6.1 (#1317)

* fix legacy deal verified status (#1324)

* fix: update go-unixfsnode enough to make sure unixfs-preload is available (#1323)

* release v1.6.2-rc1 (#1328)

* use full path (#1330)

* fix bug (#1332)

* use forks of graphsync, go-data-transfer and go-fil-markets (#1333)

* refactor: use forks of graphsync, go-data-transfer and go-fil-markets

* refactor: convert from data transfer v1 to v2 voucher type

* fix: index provider validation voucher type

* fix: pass index provider engine link system through to graphsync's transport configurer

* feat: use tagged version of boost-gfm

* fix: retrieval client imports

* feat: tagged version of lotus

* feat: require go 1.19

* lint: fix lint errors

* fix: itests

* fix: cbor-gen, docsgen

* fix: update CI lint version

* fix: lint

* fix: docgen

* fix: go mod tidy

* fix: protocol proxy TestOutboundForwarding

* fix: docsgen

* fix: update filecoin-ffi submodule

* fix: prometheus duplicate register panic

* fix: cleanup imports

* fix: legs voucher processing

* chore: release v1.6.2-rc2 (#1340)

* release v1.6.2-rc2

* fix test

* fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (#1350)

* fix: flaky TestDealCompletionOnProcessResumption (#1351)

* fix: occasional panic on shutdown (#1353)

* feat: query UI (#1352)

* log insert

* fix display error

* refactor code

* shorten status strings

* remove comment

* apply suggestion

* feat: add download block link to inspect page (#1312)

* fix(devnet): update golang and lotus default versions (#1354)

* fix(devnet): bump golang to 1.19

* chore(devnet): bump lotus default version

* chore(devnet): remove unused stable env

* booster-http: implement IPFS HTTP gateway (#1225)

* feat: implement http api gateway

* feat: use go-libipfs lib (instead of copying to extern)

* feat: bump booster-bitswap info minor version

* feat: http gateway metrics

* fix: TestHttpInfo

* feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc)

* fix: correct link for download root block (#1355)

* feat: option to cleanup data for offline deals after add piece (#1341)

* chore: add support for multiple node.js versions in makefile (#1356)

* chore: release v.1.7.0-rc1 (#1357)

* release v.1.7.0-rc1

* fix version

* fix: dagstore initialize-all parameter (#1363)

* fix: show verifying commp state for offline deals (#1364)

* fix: boost run missing staging-area dir (#1368)

* merge(wip): main to lid

TODO: remoteblockstore needs to handle nil metrics

* fix: flaky TestNewHttpServer (#1372)

* feat: group agent version by binary name (#1369)

* fix: wrap stats in nil checks for now

we should probably revisit how stats are handled now that we have all 3 transports being tracked

* test(fix): incorrect test urls

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* fix: make devnet work for lid (#1375)

* feat: support full addr config in boostd-data

* chore: fix linting for boostd-data

* feat: use addr instead of port for lid

chore: update devnet to work with lid setup

* chore: resolve feedback on lint changes

* feat: fail deal if start epoch passed (#1319)

* fail deal if start epoch passed

* add suggestion

* test: add deal expiry on startup test

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* fix: makefile

* fix: db migration ordering

* fix: correct rootcid formatting

* fix: prevent accidental removal of valid sector index announcements

fix: add cache tests and dont announce cache state
fix: add unique index to sector state db
fix: sealed and unsealed sector state conflict
fix: ensure index provider wrapper starts after db migration has completed

* chore: go mod tidy

* fix: download block (#1440)

* LID yugabyte db impl (#1391)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* script to migrate from couchbase to yugabyte (#1445)

* feat: script to migrate from couchbase to yugabyte

* fix: reduce batch size for yugabyte inserts

* Change service GetIndex / AddIndex to return channel instead of array (#1444)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* wip: service GetIndex returns channel of records instead of array

* feat: return channel from AddIndex and GetIndex

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* local index directory: recover tool (#1410)

* initial disaster recovery tool for LID

* wip

* do not block on individual error

* instantiate lid

* report

* catch signal

* fixup

* comment out sector already in progress

* fixup

* start containers with init: true

* record that we dont have an unsealed copy

* match deals with boost sqlite db and piece store

* fixup

* fixup

* use logger

* fixup

* disable stacktrace

* fixup

* extract piece store away from disaster recovery struct

* add more sanity checks

* compare IsUnsealed vs storage find

* improve safeIsUnseal

* fixup

* better logs

* expand repodir

* calc properly next offset

* fixup

* add sector id to logs

* incr offset

* break after finding expired deal

* more logs

* fewer logs

* better logs

* better error

* refactor

* refactor minerApi

* better logs

* add time around add index

* pd.Start

* LID benchmarking tool (#1276)

* feat: LID benchmarking tool

* fix: bench thread safety

* refactor: structured logging

* refactor: postgres bulk insert

* lid bench: Add foundationdb impl

* lid fdb: Fix Tx sizing, parallel chunk puts

* lid fdb: More efficient sample generation

* feat: array of piece count / blocks per piece (#1314)

* lid bench: print add rate

* lid bench: Add retry to postgres put (#1316)

* lid bench: Make cassandra put much more robust (#1318)

* instrumentation for bench tool (#1337)

* instrument postgres

* more instrumentation

* check for err getoffsetsize

* emit metrics every 10sec

* ignore errors

* add postgres-drop

* use directly tables

* fix: go mod tidy

* use INSERT INTO instead of tmp tables

* try to catch sig

* remove transaction commit

* fixup

* add postgres-init

* fixuop

* split create and init

* fixup

* remove if not exist

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: batch insert queries for postgres

* feat: add flag to insert into postgres using tmp table

* refactor: merge changes from nonsense/lid-bench

* refactor: just use one database (dont create bench database)

* refactor: remove unused params

* refactor: command structure

* fix: cassandra - dont use batch insert for PayloadToPieces

* fix: create tables CQL

* fix: increase payload to pieces insert parallelism

* fix: use simple replication strategy

* feat: use yugabyte cassandra driver

* fix: remove bench binary

* update metrics endpoint

* fix random generated piece cid

* fixup

* fix: cassandra bitswap benchmark

* remove foundationdb

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* fix: failing tests due to bad merge

* fix: flaky TestMultipleDealsConcurrent

* more logs

* piece doctor and sector state manager refactor (#1463)

* fix timer.Reset and improve logs

* revert randomization

* piece doc: handle errors

* adjust piece check

* refactor unsealsectormanager

* refactor piece doctor

* add random ports

* ignore tests

* add version to boostd-data

* fix ctx in Start

* fix: add reader mock to fix tests

* fix: pass new piece directory to provider on test restart

* fix synchronisation

* note that panics are not propagated in tests

* carv1 panics piece directory

* print panics

* fix: use reader that supports Seek in piece reader mock

* fix: reset mock car reader on each invocation

* fix: TestOfflineDealDataCleanup

* add check for nil cancel func

* bump min check period for LevelDB to 5 minutes

* check if sector state mgr is initialised

* debug line for unflagging

* commenting out TestMultipleDealsConcurrent -- flaky test -- works locally

* add SectorStateUpdates pubsub

* add close for pubsub

* add mock sectorstatemgr

* add wrapper tests

* fixup

* cleanup

* cleanup

* better names

* t.Skip for test

* remove TODO above println for panic

* add unit tests for refreshState

* rename tests

* more cases

* more tests

* update description

* better comment

* better names and comments

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Merge from main to lid branch (#1483)

* fix statx output string (#1451)

* fix: flaky TestMultipleDealsConcurrent (#1458)

* Add option to serve index provider ads over http (#1452)

* feat: option to serve index provider ads over http

* fix: config naming, hostname parsing

* fix: update docsgen

* fix: log announce address

* feat: add config for indexer direct announce urls

* refactor: always announce over pubsub

* fix: docsgen

* test: add test case for empty announce address hostname

* Add `boostd index announce-latest` command (#1456)

* feat: boostd index announce-latest

* feat: add announce-latest-http command

* fix: default direct announce url

* feat: update to index-provider v0.11.2

* Signal to index provider to skip announcements (#1457)

* fix: signal to index provider to skip announcements

* fix: ensure multihash lister skip error is of type ipld.ErrNotExists

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* release v1.7.3-rc2 (#1460)

* fix: improve stalled retrieval cancellation (#1449)

* refactor stalled retrieval cancel

* add ctx with timeout

* implement suggestions

* update err wrapping

* fix: set short cancel timeout for unpaid retrievals only

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: enable listen address for booster-http (#1461)

* enable listen address

* modify tests

* fix nil ptr (#1470)

* fix: incorrect check when import offline deal data using proposal CID (#1473)

* fix incorrect early check

* update error msg

* fix(server): properly cancel graphsync requests (#1475)

* set UI default listen address to localhost (#1476)

* feat: display msg params in the mpool UI (#1471)

* show msg params

* fix: mpool nil pointer

* fix width

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Reset read deadline after reading deal proposal message (#1479)

* fix: reset read deadline after reading deal proposal message

* fix: increase client request deadline

* feat: Show elapsed epoch and PSD wait epochs in UI (#1480)

* show epochs

* fix devnet UI, use BlockdDelaySecs

* fix lint err

* Update gql/resolver.go

Co-authored-by: dirkmc <dirkmdev@gmail.com>

---------

Co-authored-by: dirkmc <dirkmdev@gmail.com>

* release v1.7.3-rc3 (#1481)

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>

* update local index directory ui (#1477)

* feat: update local index directory ui

* comment out wrench as docker doesnt build

* rearrange menu

* refactor: remove sectors list

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: surface indexing errors (#1490)

* feat: log panic (instead of just printing to stdout) (#1491)

* split flagged pieces into unsealed/sealed tables (#1493)

* refactor: remove couchbase tests (#1496)

* refactor: remove piece directory couchbase tests (#1497)

* GraphQL resolvers for LID (#1494)

* wip

* rename

* sectorUnsealedCopies and SectorProvingState

* fix: piece directory tests (#1498)

* log line for only sealed sectors

* more logs

* feat: flagged pieces (#1501)

* check that sector has deals for unsealed sectors (#1502)

* check that sector has deals for unsealed sectors

* simplify

* rename heading

* piece doctor to ignore expired/slashed deals (#1503)

* ignore expired/slashed deals

* fix mocks

* add timer for checkPiece

* move ChainHead away from checkPiece

* add nil check for fullnodeApi

* add debug line

* fix pagination

* LID landing page: add stats around Flagged and non-Flagged pieces (#1508)

* wip

* fixup

* add debug line

* fixup

* feat: split flagged pieces page into flagged / flagged because unsealed (#1509)

* fix: display of no flagged pieces (#1511)

* disable dummy panels - block stats; deal data (#1510)

* fix unsealed field in flagged piece (#1515)

* update ffi

* fix main merge issue

* fix go mod

* Add info boxes on LID UI page (#1516)

* feat: add info boxes on LID UI page

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: replace migrate couchbase command with migrate yugavbyte (#1518)

* remove redundant makefile (#1519)

* remove redundant makefile

* add migrate-lid to Makefile

* update gitignore

* move booster-bitswap and booster-http to make and make install

* fix: inspect page - dont try to fetch root cid (#1525)

* feat: add send epoch, time, elapsed epoch and elapsed time for each message in mpool to UI (#1523)

* add message epoch/time details

* implement suggestion

* use moment lib

* fix alerting bug

* update polling interval

* add logs

* fix devnet: use ws instead of http to connect to boostd-data

* feat: make legacy deals optional (#1524)

* make legacy deals optional

* fix gen

* modify itests, create new

* handle legacy stream explicitly

* separate out the protocols

* fix lint error

* enable itest in CI

* fix ci

* apply suggestions

* fix error after conflict resolution

* refactor: simplify legacy deal response code

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* refactor: remove couchbase implementation (#1535)

* Update lotus and boxo versions (#1466) (#1537)

* Update to use packages in go-libipni

* feat: update lotus version

* update boxo (#1492)

* feat: update boxo

* refactor: depend on repo:Jorropo/lotus branch:boxo2

* chore: temporarily update go-fil-markets with replace directive

* feat: switch itests framework ExtractFileFromCAR to use non-global IPLD registry

* feat: switch booster-bitswap client fetch to use the go-ipld-prime globals via go-ipld-legacy

* go fmt

* chore: update dependencies and migrate to boxo

* fix: update boost-gfm

* fix: stop itests framework from prematurely setting listenaddrs via go-libp2p defaults that conflict with lotus

* fix: docs gen

* chore(deps): update deps for boxo v0.10.0

* chore(deps): update boost-gfm

* fix(booster-http): update for boxo v0.10.0

* chore(deps): update to remove kubo dependency

* fix(gen): update docs gen

* feat: update boost-gfm to v1.26.6

* chore(deps): update lotus to master

---------




---------

Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: hannahhoward <hannah@hannahhoward.net>

* feat: update boost-gfm to v1.26.7 (#1538)

* fix: piece doctor tests (#1540)

* refactor: build indexes for legacy deals (#1539)

* feat: http index announcements (#1418)

* feat(indexprovider): announce http transport

refactor: isolate extended provider logic

feat: announce http indexes

refactor(indexprovider): use metadata.Default

fix(wrapper): fix compile error

* fix http ep signing bug

* update comment

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* feat: check unseal status of piece through both apis (#1548)

* fix: metrics and Grafana (#1546)

* fix grafna, metrics

* remove dagstore from name

* fix: add missing PieceDeal (PieceCid) index (#1551)

* fix: iterate all deals to index piece (#1549)

* fix: iterate all deals to index piece

* add test, use multierror

* add and update comments

* refactor: separate yugabyte / leveldb tests for easier local testing (#1553)

* feat: refactor mpool page in UI (#1530)

* modify GQL

* fix count type

* fix locks

* fix js

* migrate config to v5 (#1560)

* migrate config to v5

* change default version

* chore: release v2.0.0-rc1 (#1561)

* Upgrade to index-provider v0.13.4 (#1559)

Upgrade to the latest index-provider library.

* feat: add IPNI itest (#1563)

* ipni itest

* refactor test

* add to circleCI

* add indexer topic

* Print protocol IDs exposed by f.Boost

* generate topic name dynamically

---------

Co-authored-by: Masih H. Derkani <m@derkani.org>

* IPNI UX (#1562)

* feat: IPNI UX

* Update react/src/Ipni.js

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* feat: server side config

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* add peerID to multiaddresses returned by boost provider retrieval-transports query (#1568)

* add peerID to multiaddress

* fix error typos

* undo http-libp2p on vanilla http

* fix: rename indexer config to index provider config (#1573)

* fix retrieve output path (#1574)

* update README.md (#1576)

* update description (#1580)

* tempo: increase max trace size (#1579)

* use go-block-format

---------

Co-authored-by: Jacob Heun <jacob.heun@gmail.com>
Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Jacob Heun <jacobheun@gmail.com>
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>

* add free check (#1315)

* chore: bump version to 1.6.1 (#1317)

* fix legacy deal verified status (#1324)

* fix: update go-unixfsnode enough to make sure unixfs-preload is available (#1323)

* release v1.6.2-rc1 (#1328)

* use full path (#1330)

* fix bug (#1332)

* use forks of graphsync, go-data-transfer and go-fil-markets (#1333)

* refactor: use forks of graphsync, go-data-transfer and go-fil-markets

* refactor: convert from data transfer v1 to v2 voucher type

* fix: index provider validation voucher type

* fix: pass index provider engine link system through to graphsync's transport configurer

* feat: use tagged version of boost-gfm

* fix: retrieval client imports

* feat: tagged version of lotus

* feat: require go 1.19

* lint: fix lint errors

* fix: itests

* fix: cbor-gen, docsgen

* fix: update CI lint version

* fix: lint

* fix: docgen

* fix: go mod tidy

* fix: protocol proxy TestOutboundForwarding

* fix: docsgen

* fix: update filecoin-ffi submodule

* fix: prometheus duplicate register panic

* fix: cleanup imports

* fix: legs voucher processing

* chore: release v1.6.2-rc2 (#1340)

* release v1.6.2-rc2

* fix test

* fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (#1350)

* fix: flaky TestDealCompletionOnProcessResumption (#1351)

* fix: occasional panic on shutdown (#1353)

* feat: query UI (#1352)

* log insert

* fix display error

* refactor code

* shorten status strings

* remove comment

* apply suggestion

* feat: add download block link to inspect page (#1312)

* fix(devnet): update golang and lotus default versions (#1354)

* fix(devnet): bump golang to 1.19

* chore(devnet): bump lotus default version

* chore(devnet): remove unused stable env

* booster-http: implement IPFS HTTP gateway (#1225)

* feat: implement http api gateway

* feat: use go-libipfs lib (instead of copying to extern)

* feat: bump booster-bitswap info minor version

* feat: http gateway metrics

* fix: TestHttpInfo

* feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc)

* fix: correct link for download root block (#1355)

* feat: option to cleanup data for offline deals after add piece (#1341)

* chore: add support for multiple node.js versions in makefile (#1356)

* chore: release v.1.7.0-rc1 (#1357)

* release v.1.7.0-rc1

* fix version

* fix: dagstore initialize-all parameter (#1363)

* fix: show verifying commp state for offline deals (#1364)

* fix: boost run missing staging-area dir (#1368)

* merge(wip): main to lid

TODO: remoteblockstore needs to handle nil metrics

* fix: flaky TestNewHttpServer (#1372)

* feat: group agent version by binary name (#1369)

* fix: wrap stats in nil checks for now

we should probably revisit how stats are handled now that we have all 3 transports being tracked

* test(fix): incorrect test urls

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* feat: add local index directory

Internally this is still refered to as the piece directory

Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* refactor: merge from main into lid branch (#1339)

* merge: main to lid (#1370)

* add free check (#1315)

* chore: bump version to 1.6.1 (#1317)

* fix legacy deal verified status (#1324)

* fix: update go-unixfsnode enough to make sure unixfs-preload is available (#1323)

* release v1.6.2-rc1 (#1328)

* use full path (#1330)

* fix bug (#1332)

* use forks of graphsync, go-data-transfer and go-fil-markets (#1333)

* refactor: use forks of graphsync, go-data-transfer and go-fil-markets

* refactor: convert from data transfer v1 to v2 voucher type

* fix: index provider validation voucher type

* fix: pass index provider engine link system through to graphsync's transport configurer

* feat: use tagged version of boost-gfm

* fix: retrieval client imports

* feat: tagged version of lotus

* feat: require go 1.19

* lint: fix lint errors

* fix: itests

* fix: cbor-gen, docsgen

* fix: update CI lint version

* fix: lint

* fix: docgen

* fix: go mod tidy

* fix: protocol proxy TestOutboundForwarding

* fix: docsgen

* fix: update filecoin-ffi submodule

* fix: prometheus duplicate register panic

* fix: cleanup imports

* fix: legs voucher processing

* chore: release v1.6.2-rc2 (#1340)

* release v1.6.2-rc2

* fix test

* fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (#1350)

* fix: flaky TestDealCompletionOnProcessResumption (#1351)

* fix: occasional panic on shutdown (#1353)

* feat: query UI (#1352)

* log insert

* fix display error

* refactor code

* shorten status strings

* remove comment

* apply suggestion

* feat: add download block link to inspect page (#1312)

* fix(devnet): update golang and lotus default versions (#1354)

* fix(devnet): bump golang to 1.19

* chore(devnet): bump lotus default version

* chore(devnet): remove unused stable env

* booster-http: implement IPFS HTTP gateway (#1225)

* feat: implement http api gateway

* feat: use go-libipfs lib (instead of copying to extern)

* feat: bump booster-bitswap info minor version

* feat: http gateway metrics

* fix: TestHttpInfo

* feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc)

* fix: correct link for download root block (#1355)

* feat: option to cleanup data for offline deals after add piece (#1341)

* chore: add support for multiple node.js versions in makefile (#1356)

* chore: release v.1.7.0-rc1 (#1357)

* release v.1.7.0-rc1

* fix version

* fix: dagstore initialize-all parameter (#1363)

* fix: show verifying commp state for offline deals (#1364)

* fix: boost run missing staging-area dir (#1368)

* merge(wip): main to lid

TODO: remoteblockstore needs to handle nil metrics

* fix: flaky TestNewHttpServer (#1372)

* feat: group agent version by binary name (#1369)

* fix: wrap stats in nil checks for now

we should probably revisit how stats are handled now that we have all 3 transports being tracked

* test(fix): incorrect test urls

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* fix: make devnet work for lid (#1375)

* feat: support full addr config in boostd-data

* chore: fix linting for boostd-data

* feat: use addr instead of port for lid

chore: update devnet to work with lid setup

* chore: resolve feedback on lint changes

* feat: fail deal if start epoch passed (#1319)

* fail deal if start epoch passed

* add suggestion

* test: add deal expiry on startup test

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* fix: makefile

* fix: db migration ordering

* fix: correct rootcid formatting

* fix: prevent accidental removal of valid sector index announcements

fix: add cache tests and dont announce cache state
fix: add unique index to sector state db
fix: sealed and unsealed sector state conflict
fix: ensure index provider wrapper starts after db migration has completed

* chore: go mod tidy

* fix: download block (#1440)

* LID yugabyte db impl (#1391)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* script to migrate from couchbase to yugabyte (#1445)

* feat: script to migrate from couchbase to yugabyte

* fix: reduce batch size for yugabyte inserts

* Change service GetIndex / AddIndex to return channel instead of array (#1444)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* wip: service GetIndex returns channel of records instead of array

* feat: return channel from AddIndex and GetIndex

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* local index directory: recover tool (#1410)

* initial disaster recovery tool for LID

* wip

* do not block on individual error

* instantiate lid

* report

* catch signal

* fixup

* comment out sector already in progress

* fixup

* start containers with init: true

* record that we dont have an unsealed copy

* match deals with boost sqlite db and piece store

* fixup

* fixup

* use logger

* fixup

* disable stacktrace

* fixup

* extract piece store away from disaster recovery struct

* add more sanity checks

* compare IsUnsealed vs storage find

* improve safeIsUnseal

* fixup

* better logs

* expand repodir

* calc properly next offset

* fixup

* add sector id to logs

* incr offset

* break after finding expired deal

* more logs

* fewer logs

* better logs

* better error

* refactor

* refactor minerApi

* better logs

* add time around add index

* pd.Start

* LID benchmarking tool (#1276)

* feat: LID benchmarking tool

* fix: bench thread safety

* refactor: structured logging

* refactor: postgres bulk insert

* lid bench: Add foundationdb impl

* lid fdb: Fix Tx sizing, parallel chunk puts

* lid fdb: More efficient sample generation

* feat: array of piece count / blocks per piece (#1314)

* lid bench: print add rate

* lid bench: Add retry to postgres put (#1316)

* lid bench: Make cassandra put much more robust (#1318)

* instrumentation for bench tool (#1337)

* instrument postgres

* more instrumentation

* check for err getoffsetsize

* emit metrics every 10sec

* ignore errors

* add postgres-drop

* use directly tables

* fix: go mod tidy

* use INSERT INTO instead of tmp tables

* try to catch sig

* remove transaction commit

* fixup

* add postgres-init

* fixuop

* split create and init

* fixup

* remove if not exist

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: batch insert queries for postgres

* feat: add flag to insert into postgres using tmp table

* refactor: merge changes from nonsense/lid-bench

* refactor: just use one database (dont create bench database)

* refactor: remove unused params

* refactor: command structure

* fix: cassandra - dont use batch insert for PayloadToPieces

* fix: create tables CQL

* fix: increase payload to pieces insert parallelism

* fix: use simple replication strategy

* feat: use yugabyte cassandra driver

* fix: remove bench binary

* update metrics endpoint

* fix random generated piece cid

* fixup

* fix: cassandra bitswap benchmark

* remove foundationdb

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* fix: failing tests due to bad merge

* fix: flaky TestMultipleDealsConcurrent

* more logs

* piece doctor and sector state manager refactor (#1463)

* fix timer.Reset and improve logs

* revert randomization

* piece doc: handle errors

* adjust piece check

* refactor unsealsectormanager

* refactor piece doctor

* add random ports

* ignore tests

* add version to boostd-data

* fix ctx in Start

* fix: add reader mock to fix tests

* fix: pass new piece directory to provider on test restart

* fix synchronisation

* note that panics are not propagated in tests

* carv1 panics piece directory

* print panics

* fix: use reader that supports Seek in piece reader mock

* fix: reset mock car reader on each invocation

* fix: TestOfflineDealDataCleanup

* add check for nil cancel func

* bump min check period for LevelDB to 5 minutes

* check if sector state mgr is initialised

* debug line for unflagging

* commenting out TestMultipleDealsConcurrent -- flaky test -- works locally

* add SectorStateUpdates pubsub

* add close for pubsub

* add mock sectorstatemgr

* add wrapper tests

* fixup

* cleanup

* cleanup

* better names

* t.Skip for test

* remove TODO above println for panic

* add unit tests for refreshState

* rename tests

* more cases

* more tests

* update description

* better comment

* better names and comments

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Merge from main to lid branch (#1483)

* fix statx output string (#1451)

* fix: flaky TestMultipleDealsConcurrent (#1458)

* Add option to serve index provider ads over http (#1452)

* feat: option to serve index provider ads over http

* fix: config naming, hostname parsing

* fix: update docsgen

* fix: log announce address

* feat: add config for indexer direct announce urls

* refactor: always announce over pubsub

* fix: docsgen

* test: add test case for empty announce address hostname

* Add `boostd index announce-latest` command (#1456)

* feat: boostd index announce-latest

* feat: add announce-latest-http command

* fix: default direct announce url

* feat: update to index-provider v0.11.2

* Signal to index provider to skip announcements (#1457)

* fix: signal to index provider to skip announcements

* fix: ensure multihash lister skip error is of type ipld.ErrNotExists

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* release v1.7.3-rc2 (#1460)

* fix: improve stalled retrieval cancellation (#1449)

* refactor stalled retrieval cancel

* add ctx with timeout

* implement suggestions

* update err wrapping

* fix: set short cancel timeout for unpaid retrievals only

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: enable listen address for booster-http (#1461)

* enable listen address

* modify tests

* fix nil ptr (#1470)

* fix: incorrect check when import offline deal data using proposal CID (#1473)

* fix incorrect early check

* update error msg

* fix(server): properly cancel graphsync requests (#1475)

* set UI default listen address to localhost (#1476)

* feat: display msg params in the mpool UI (#1471)

* show msg params

* fix: mpool nil pointer

* fix width

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Reset read deadline after reading deal proposal message (#1479)

* fix: reset read deadline after reading deal proposal message

* fix: increase client request deadline

* feat: Show elapsed epoch and PSD wait epochs in UI (#1480)

* show epochs

* fix devnet UI, use BlockdDelaySecs

* fix lint err

* Update gql/resolver.go

Co-authored-by: dirkmc <dirkmdev@gmail.com>

---------

Co-authored-by: dirkmc <dirkmdev@gmail.com>

* release v1.7.3-rc3 (#1481)

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>

* update local index directory ui (#1477)

* feat: update local index directory ui

* comment out wrench as docker doesnt build

* rearrange menu

* refactor: remove sectors list

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: surface indexing errors (#1490)

* feat: log panic (instead of just printing to stdout) (#1491)

* split flagged pieces into unsealed/sealed tables (#1493)

* refactor: remove couchbase tests (#1496)

* refactor: remove piece directory couchbase tests (#1497)

* GraphQL resolvers for LID (#1494)

* wip

* rename

* sectorUnsealedCopies and SectorProvingState

* fix: piece directory tests (#1498)

* log line for only sealed sectors

* more logs

* feat: flagged pieces (#1501)

* check that sector has deals for unsealed sectors (#1502)

* check that sector has deals for unsealed sectors

* simplify

* rename heading

* piece doctor to ignore expired/slashed deals (#1503)

* ignore expired/slashed deals

* fix mocks

* add timer for checkPiece

* move ChainHead away from checkPiece

* add nil check for fullnodeApi

* add debug line

* fix pagination

* LID landing page: add stats around Flagged and non-Flagged pieces (#1508)

* wip

* fixup

* add debug line

* fixup

* feat: split flagged pieces page into flagged / flagged because unsealed (#1509)

* fix: display of no flagged pieces (#1511)

* disable dummy panels - block stats; deal data (#1510)

* fix unsealed field in flagged piece (#1515)

* update ffi

* fix main merge issue

* fix go mod

* Add info boxes on LID UI page (#1516)

* feat: add info boxes on LID UI page

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: replace migrate couchbase command with migrate yugavbyte (#1518)

* remove redundant makefile (#1519)

* remove redundant makefile

* add migrate-lid to Makefile

* update gitignore

* move booster-bitswap and booster-http to make and make install

* fix: inspect page - dont try to fetch root cid (#1525)

* feat: add send epoch, time, elapsed epoch and elapsed time for each message in mpool to UI (#1523)

* add message epoch/time details

* implement suggestion

* use moment lib

* fix alerting bug

* update polling interval

* add logs

* fix devnet: use ws instead of http to connect to boostd-data

* feat: make legacy deals optional (#1524)

* make legacy deals optional

* fix gen

* modify itests, create new

* handle legacy stream explicitly

* separate out the protocols

* fix lint error

* enable itest in CI

* fix ci

* apply suggestions

* fix error after conflict resolution

* refactor: simplify legacy deal response code

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* refactor: remove couchbase implementation (#1535)

* Update lotus and boxo versions (#1466) (#1537)

* Update to use packages in go-libipni

* feat: update lotus version

* update boxo (#1492)

* feat: update boxo

* refactor: depend on repo:Jorropo/lotus branch:boxo2

* chore: temporarily update go-fil-markets with replace directive

* feat: switch itests framework ExtractFileFromCAR to use non-global IPLD registry

* feat: switch booster-bitswap client fetch to use the go-ipld-prime globals via go-ipld-legacy

* go fmt

* chore: update dependencies and migrate to boxo

* fix: update boost-gfm

* fix: stop itests framework from prematurely setting listenaddrs via go-libp2p defaults that conflict with lotus

* fix: docs gen

* chore(deps): update deps for boxo v0.10.0

* chore(deps): update boost-gfm

* fix(booster-http): update for boxo v0.10.0

* chore(deps): update to remove kubo dependency

* fix(gen): update docs gen

* feat: update boost-gfm to v1.26.6

* chore(deps): update lotus to master

---------




---------

Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: hannahhoward <hannah@hannahhoward.net>

* feat: update boost-gfm to v1.26.7 (#1538)

* fix: piece doctor tests (#1540)

* refactor: build indexes for legacy deals (#1539)

* feat: http index announcements (#1418)

* feat(indexprovider): announce http transport

refactor: isolate extended provider logic

feat: announce http indexes

refactor(indexprovider): use metadata.Default

fix(wrapper): fix compile error

* fix http ep signing bug

* update comment

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* feat: check unseal status of piece through both apis (#1548)

* fix: metrics and Grafana (#1546)

* fix grafna, metrics

* remove dagstore from name

* fix: add missing PieceDeal (PieceCid) index (#1551)

* fix: iterate all deals to index piece (#1549)

* fix: iterate all deals to index piece

* add test, use multierror

* add and update comments

* refactor: separate yugabyte / leveldb tests for easier local testing (#1553)

* feat: refactor mpool page in UI (#1530)

* modify GQL

* fix count type

* fix locks

* fix js

* migrate config to v5 (#1560)

* migrate config to v5

* change default version

* chore: release v2.0.0-rc1 (#1561)

* Upgrade to index-provider v0.13.4 (#1559)

Upgrade to the latest index-provider library.

* feat: add IPNI itest (#1563)

* ipni itest

* refactor test

* add to circleCI

* add indexer topic

* Print protocol IDs exposed by f.Boost

* generate topic name dynamically

---------

Co-authored-by: Masih H. Derkani <m@derkani.org>

* IPNI UX (#1562)

* feat: IPNI UX

* Update react/src/Ipni.js

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* feat: server side config

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* feat: use shared piece reader for blockstore get

* fix: booster-bitswap client - extract blocks from identity cids

* go mod tidy

* fixup

* remove refs and use ttlcache

* ignore err on cache

* add lock around ttl cache

* use a new readerCtx

* add expire again

* fix logger

* check for refs=0 when closing individual shared reader

* use cancel instead of direct close

* add expired field

* downgrade warnw to debugw

---------

Co-authored-by: Jacob Heun <jacob.heun@gmail.com>
Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Jacob Heun <jacobheun@gmail.com>
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>

* feat: add local index directory

Internally this is still refered to as the piece directory

Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* refactor: merge from main into lid branch (#1339)

* merge: main to lid (#1370)

* add free check (#1315)

* chore: bump version to 1.6.1 (#1317)

* fix legacy deal verified status (#1324)

* fix: update go-unixfsnode enough to make sure unixfs-preload is available (#1323)

* release v1.6.2-rc1 (#1328)

* use full path (#1330)

* fix bug (#1332)

* use forks of graphsync, go-data-transfer and go-fil-markets (#1333)

* refactor: use forks of graphsync, go-data-transfer and go-fil-markets

* refactor: convert from data transfer v1 to v2 voucher type

* fix: index provider validation voucher type

* fix: pass index provider engine link system through to graphsync's transport configurer

* feat: use tagged version of boost-gfm

* fix: retrieval client imports

* feat: tagged version of lotus

* feat: require go 1.19

* lint: fix lint errors

* fix: itests

* fix: cbor-gen, docsgen

* fix: update CI lint version

* fix: lint

* fix: docgen

* fix: go mod tidy

* fix: protocol proxy TestOutboundForwarding

* fix: docsgen

* fix: update filecoin-ffi submodule

* fix: prometheus duplicate register panic

* fix: cleanup imports

* fix: legs voucher processing

* chore: release v1.6.2-rc2 (#1340)

* release v1.6.2-rc2

* fix test

* fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (#1350)

* fix: flaky TestDealCompletionOnProcessResumption (#1351)

* fix: occasional panic on shutdown (#1353)

* feat: query UI (#1352)

* log insert

* fix display error

* refactor code

* shorten status strings

* remove comment

* apply suggestion

* feat: add download block link to inspect page (#1312)

* fix(devnet): update golang and lotus default versions (#1354)

* fix(devnet): bump golang to 1.19

* chore(devnet): bump lotus default version

* chore(devnet): remove unused stable env

* booster-http: implement IPFS HTTP gateway (#1225)

* feat: implement http api gateway

* feat: use go-libipfs lib (instead of copying to extern)

* feat: bump booster-bitswap info minor version

* feat: http gateway metrics

* fix: TestHttpInfo

* feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc)

* fix: correct link for download root block (#1355)

* feat: option to cleanup data for offline deals after add piece (#1341)

* chore: add support for multiple node.js versions in makefile (#1356)

* chore: release v.1.7.0-rc1 (#1357)

* release v.1.7.0-rc1

* fix version

* fix: dagstore initialize-all parameter (#1363)

* fix: show verifying commp state for offline deals (#1364)

* fix: boost run missing staging-area dir (#1368)

* merge(wip): main to lid

TODO: remoteblockstore needs to handle nil metrics

* fix: flaky TestNewHttpServer (#1372)

* feat: group agent version by binary name (#1369)

* fix: wrap stats in nil checks for now

we should probably revisit how stats are handled now that we have all 3 transports being tracked

* test(fix): incorrect test urls

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* fix: make devnet work for lid (#1375)

* feat: support full addr config in boostd-data

* chore: fix linting for boostd-data

* feat: use addr instead of port for lid

chore: update devnet to work with lid setup

* chore: resolve feedback on lint changes

* feat: fail deal if start epoch passed (#1319)

* fail deal if start epoch passed

* add suggestion

* test: add deal expiry on startup test

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* fix: makefile

* fix: db migration ordering

* fix: correct rootcid formatting

* fix: prevent accidental removal of valid sector index announcements

fix: add cache tests and dont announce cache state
fix: add unique index to sector state db
fix: sealed and unsealed sector state conflict
fix: ensure index provider wrapper starts after db migration has completed

* chore: go mod tidy

* fix: download block (#1440)

* LID yugabyte db impl (#1391)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* script to migrate from couchbase to yugabyte (#1445)

* feat: script to migrate from couchbase to yugabyte

* fix: reduce batch size for yugabyte inserts

* Change service GetIndex / AddIndex to return channel instead of array (#1444)

* feat: yugabyte db impl

* feat: run yugabyte tests against a dockerized yugabyte

* fix: use out own yugabyte docker image

* fix: use yugabyte 2.17.2.0 docker image

* feat: piece doctor yugabyte impl

* fix: go mod tidy

* refactor: remove SetCarSize as its not longer being used

* refactor: remove functionality to mark index as errored (not being used)

* feat: implement delete commands

* refactor: consolidate test params

* feat: add lid yugabyte config

* fix: port map yugabyte postgres to standard port

* Fix yugabyte CI (#1433)

* fix: yugabyte tests in CI

* docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost

* add tag

* test lid

* make gen

* fixup

* move couchbase settings under build tag

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* wip: service GetIndex returns channel of records instead of array

* feat: return channel from AddIndex and GetIndex

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* local index directory: recover tool (#1410)

* initial disaster recovery tool for LID

* wip

* do not block on individual error

* instantiate lid

* report

* catch signal

* fixup

* comment out sector already in progress

* fixup

* start containers with init: true

* record that we dont have an unsealed copy

* match deals with boost sqlite db and piece store

* fixup

* fixup

* use logger

* fixup

* disable stacktrace

* fixup

* extract piece store away from disaster recovery struct

* add more sanity checks

* compare IsUnsealed vs storage find

* improve safeIsUnseal

* fixup

* better logs

* expand repodir

* calc properly next offset

* fixup

* add sector id to logs

* incr offset

* break after finding expired deal

* more logs

* fewer logs

* better logs

* better error

* refactor

* refactor minerApi

* better logs

* add time around add index

* pd.Start

* LID benchmarking tool (#1276)

* feat: LID benchmarking tool

* fix: bench thread safety

* refactor: structured logging

* refactor: postgres bulk insert

* lid bench: Add foundationdb impl

* lid fdb: Fix Tx sizing, parallel chunk puts

* lid fdb: More efficient sample generation

* feat: array of piece count / blocks per piece (#1314)

* lid bench: print add rate

* lid bench: Add retry to postgres put (#1316)

* lid bench: Make cassandra put much more robust (#1318)

* instrumentation for bench tool (#1337)

* instrument postgres

* more instrumentation

* check for err getoffsetsize

* emit metrics every 10sec

* ignore errors

* add postgres-drop

* use directly tables

* fix: go mod tidy

* use INSERT INTO instead of tmp tables

* try to catch sig

* remove transaction commit

* fixup

* add postgres-init

* fixuop

* split create and init

* fixup

* remove if not exist

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: batch insert queries for postgres

* feat: add flag to insert into postgres using tmp table

* refactor: merge changes from nonsense/lid-bench

* refactor: just use one database (dont create bench database)

* refactor: remove unused params

* refactor: command structure

* fix: cassandra - dont use batch insert for PayloadToPieces

* fix: create tables CQL

* fix: increase payload to pieces insert parallelism

* fix: use simple replication strategy

* feat: use yugabyte cassandra driver

* fix: remove bench binary

* update metrics endpoint

* fix random generated piece cid

* fixup

* fix: cassandra bitswap benchmark

* remove foundationdb

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* fix: failing tests due to bad merge

* fix: flaky TestMultipleDealsConcurrent

* more logs

* piece doctor and sector state manager refactor (#1463)

* fix timer.Reset and improve logs

* revert randomization

* piece doc: handle errors

* adjust piece check

* refactor unsealsectormanager

* refactor piece doctor

* add random ports

* ignore tests

* add version to boostd-data

* fix ctx in Start

* fix: add reader mock to fix tests

* fix: pass new piece directory to provider on test restart

* fix synchronisation

* note that panics are not propagated in tests

* carv1 panics piece directory

* print panics

* fix: use reader that supports Seek in piece reader mock

* fix: reset mock car reader on each invocation

* fix: TestOfflineDealDataCleanup

* add check for nil cancel func

* bump min check period for LevelDB to 5 minutes

* check if sector state mgr is initialised

* debug line for unflagging

* commenting out TestMultipleDealsConcurrent -- flaky test -- works locally

* add SectorStateUpdates pubsub

* add close for pubsub

* add mock sectorstatemgr

* add wrapper tests

* fixup

* cleanup

* cleanup

* better names

* t.Skip for test

* remove TODO above println for panic

* add unit tests for refreshState

* rename tests

* more cases

* more tests

* update description

* better comment

* better names and comments

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Merge from main to lid branch (#1483)

* fix statx output string (#1451)

* fix: flaky TestMultipleDealsConcurrent (#1458)

* Add option to serve index provider ads over http (#1452)

* feat: option to serve index provider ads over http

* fix: config naming, hostname parsing

* fix: update docsgen

* fix: log announce address

* feat: add config for indexer direct announce urls

* refactor: always announce over pubsub

* fix: docsgen

* test: add test case for empty announce address hostname

* Add `boostd index announce-latest` command (#1456)

* feat: boostd index announce-latest

* feat: add announce-latest-http command

* fix: default direct announce url

* feat: update to index-provider v0.11.2

* Signal to index provider to skip announcements (#1457)

* fix: signal to index provider to skip announcements

* fix: ensure multihash lister skip error is of type ipld.ErrNotExists

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* release v1.7.3-rc2 (#1460)

* fix: improve stalled retrieval cancellation (#1449)

* refactor stalled retrieval cancel

* add ctx with timeout

* implement suggestions

* update err wrapping

* fix: set short cancel timeout for unpaid retrievals only

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: enable listen address for booster-http (#1461)

* enable listen address

* modify tests

* fix nil ptr (#1470)

* fix: incorrect check when import offline deal data using proposal CID (#1473)

* fix incorrect early check

* update error msg

* fix(server): properly cancel graphsync requests (#1475)

* set UI default listen address to localhost (#1476)

* feat: display msg params in the mpool UI (#1471)

* show msg params

* fix: mpool nil pointer

* fix width

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* Reset read deadline after reading deal proposal message (#1479)

* fix: reset read deadline after reading deal proposal message

* fix: increase client request deadline

* feat: Show elapsed epoch and PSD wait epochs in UI (#1480)

* show epochs

* fix devnet UI, use BlockdDelaySecs

* fix lint err

* Update gql/resolver.go

Co-authored-by: dirkmc <dirkmdev@gmail.com>

---------

Co-authored-by: dirkmc <dirkmdev@gmail.com>

* release v1.7.3-rc3 (#1481)

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>

* update local index directory ui (#1477)

* feat: update local index directory ui

* comment out wrench as docker doesnt build

* rearrange menu

* refactor: remove sectors list

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: surface indexing errors (#1490)

* feat: log panic (instead of just printing to stdout) (#1491)

* split flagged pieces into unsealed/sealed tables (#1493)

* refactor: remove couchbase tests (#1496)

* refactor: remove piece directory couchbase tests (#1497)

* GraphQL resolvers for LID (#1494)

* wip

* rename

* sectorUnsealedCopies and SectorProvingState

* fix: piece directory tests (#1498)

* log line for only sealed sectors

* more logs

* feat: flagged pieces (#1501)

* check that sector has deals for unsealed sectors (#1502)

* check that sector has deals for unsealed sectors

* simplify

* rename heading

* piece doctor to ignore expired/slashed deals (#1503)

* ignore expired/slashed deals

* fix mocks

* add timer for checkPiece

* move ChainHead away from checkPiece

* add nil check for fullnodeApi

* add debug line

* fix pagination

* LID landing page: add stats around Flagged and non-Flagged pieces (#1508)

* wip

* fixup

* add debug line

* fixup

* feat: split flagged pieces page into flagged / flagged because unsealed (#1509)

* fix: display of no flagged pieces (#1511)

* disable dummy panels - block stats; deal data (#1510)

* fix unsealed field in flagged piece (#1515)

* update ffi

* fix main merge issue

* fix go mod

* Add info boxes on LID UI page (#1516)

* feat: add info boxes on LID UI page

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* Update react/src/LID.js

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

---------

Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>

* feat: replace migrate couchbase command with migrate yugavbyte (#1518)

* remove redundant makefile (#1519)

* remove redundant makefile

* add migrate-lid to Makefile

* update gitignore

* move booster-bitswap and booster-http to make and make install

* fix: inspect page - dont try to fetch root cid (#1525)

* feat: add send epoch, time, elapsed epoch and elapsed time for each message in mpool to UI (#1523)

* add message epoch/time details

* implement suggestion

* use moment lib

* fix alerting bug

* update polling interval

* add logs

* fix devnet: use ws instead of http to connect to boostd-data

* feat: make legacy deals optional (#1524)

* make legacy deals optional

* fix gen

* modify itests, create new

* handle legacy stream explicitly

* separate out the protocols

* fix lint error

* enable itest in CI

* fix ci

* apply suggestions

* fix error after conflict resolution

* refactor: simplify legacy deal response code

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* refactor: remove couchbase implementation (#1535)

* Update lotus and boxo versions (#1466) (#1537)

* Update to use packages in go-libipni

* feat: update lotus version

* update boxo (#1492)

* feat: update boxo

* refactor: depend on repo:Jorropo/lotus branch:boxo2

* chore: temporarily update go-fil-markets with replace directive

* feat: switch itests framework ExtractFileFromCAR to use non-global IPLD registry

* feat: switch booster-bitswap client fetch to use the go-ipld-prime globals via go-ipld-legacy

* go fmt

* chore: update dependencies and migrate to boxo

* fix: update boost-gfm

* fix: stop itests framework from prematurely setting listenaddrs via go-libp2p defaults that conflict with lotus

* fix: docs gen

* chore(deps): update deps for boxo v0.10.0

* chore(deps): update boost-gfm

* fix(booster-http): update for boxo v0.10.0

* chore(deps): update to remove kubo dependency

* fix(gen): update docs gen

* feat: update boost-gfm to v1.26.6

* chore(deps): update lotus to master

---------




---------

Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: hannahhoward <hannah@hannahhoward.net>

* feat: update boost-gfm to v1.26.7 (#1538)

* fix: piece doctor tests (#1540)

* refactor: build indexes for legacy deals (#1539)

* feat: http index announcements (#1418)

* feat(indexprovider): announce http transport

refactor: isolate extended provider logic

feat: announce http indexes

refactor(indexprovider): use metadata.Default

fix(wrapper): fix compile error

* fix http ep signing bug

* update comment

---------

Co-authored-by: LexLuthr <lexluthr@protocol.ai>

* feat: check unseal status of piece through both apis (#1548)

* fix: metrics and Grafana (#1546)

* fix grafna, metrics

* remove dagstore from name

* fix: add missing PieceDeal (PieceCid) index (#1551)

* fix: iterate all deals to index piece (#1549)

* fix: iterate all deals to index piece

* add test, use multierror

* add and update comments

* refactor: separate yugabyte / leveldb tests for easier local testing (#1553)

* Parse indexed deals for data segment index

this makes boost aware of the format in
https://github.com/filecoin-project/go-data-segment/
for new deals.

this is the same logic as in filecoin-project/lotus#10674

* first pass at segment fixture

* fix compile issues in test

* test passes

* additional seek on fallback

* code review

* remaining test fix

* refactor: simplify data segment index test

* update to tagged go-data-segment

* feat: refactor mpool page in UI (#1530)

* modify GQL

* fix count type

* fix locks

* fix js

* migrate config to v5 (#1560)

* migrate config to v5

* change default version

* chore: release v2.0.0-rc1 (#1561)

* Upgrade to index-provider v0.13.4 (#1559)

Upgrade to the latest index-provider library.

* itest for data segment

* enable test in circleCI

* feat: add IPNI itest (#1563)

* ipni itest

* refactor test

* add to circleCI

* add indexer topic

* Print protocol IDs exposed by f.Boost

* generate topic name dynamically

---------

Co-authored-by: Masih H. Derkani <m@derkani.org>

* fix file comparison

* Add test for second segment

* IPNI UX (#1562)

* feat: IPNI UX

* Update react/src/Ipni.js

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* feat: server side config

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>

* fix: data segment offset calculation

* fix itest to use carv1

* change itest to use 1.5 mb file

* use fixtures

* use fixture.dat generated by Will

* Remove carv1/carv2 specific semantics from commP calculations

The commp should be directly over the bytes of the deal, it shouldn't care about car format.

* fail on empty files

* fix another instance where deals are assumed to be car files

* And fix a test assertion per expectation

* Update recorded offsets when indexing a carv2 file

* update to play nicely between absolute index and car blockstore

* better logging on read errors

* additional logging

* check if offset not properly accounted

* try other direction

* remove debugging

* replay on get errors

* lint

* see if this identifies the unexpected behavior in skipNext

* offbyone

* print buffer of object

* get slice around target area

* fix: use fixed car BlockReader#SkipNext SourceOffset (#1659)

* fix: use fixed car BlockReader#SkipNext SourceOffset

Ref: ipld/go-car#491

* chore: regenerate cbor-gen types w/ cbor-gen update

* fix compilation errors

* feat: add corshandler to IPFS gateway (#1589)

* add corshandler to IPFS gateway

* use cors lib

* go mod tidy

* refactor test, remove extra cors

* fix a couple files that failed on the merge to master

* fix: CARv2 deal and retrieval support fixes

* fix: make test work

* feat: show both sealing and index / announce status, if there's an error announcing (#1699)

* require data segmetn index to have a valid car at each segment

* force only car deals in commp

* use expected ffi

---------

Co-authored-by: Jacob Heun <jacob.heun@gmail.com>
Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>
Co-authored-by: Jacob Heun <jacobheun@gmail.com>
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>