Remove /entries/txIds endpoint and entry_seqno_index indexer

[Copilot]

The entry_seqno_index (SeqnosForValue_Bucketed<EntryTable>) maintained an ever-growing in-memory index to power /entries/txIds, trading memory and first-response latency on new nodes for skipping deserialization of governance transactions—a premature optimization given governance is rarely a significant portion of ledger volume.

C++

• Remove EntrySeqnoIndexingStrategy typedef, entry_seqno_index member, and its initialization/installation
• Remove the /entries/txIds GET endpoint and all associated logic
• Remove GetEntriesTransactionIds from call_types.h
• Remove indexing namespace constants (SEQNOS_PER_BUCKET, MAX_BUCKETS) and IndexingInProgressRetryLater error from constants.h
• Drop <ccf/indexing/strategies/seqnos_by_key_bucketed.h> include and now-unused get_tx_status helper

Python / CLI

• Remove Client.enumerate_statements() and its unused imports (Iterable, urlencode)
• Delete retrieve_signed_claims.py CLI command (scitt retrieve) which depended on enumerate_statements
• Remove retrieve_signed_claims from cli/main.py

Tests

• Remove test_enumerate_statements from test/test_historical.py

Original prompt

---

This section details on the original issue you should resolve

<issue_title>review the need for entry_seqno_index</issue_title>
<issue_description>The current code maintains a entry_seqno_index with a ccf::indexing::strategies::SeqnosForValue_Bucketed<EntryTable> that's essentially a list of the seqnos at which there is a write to the entry table (i.e. CTS business transactions as opposed to CCF internal/governance).

This is exposed by the /entries/txIds endpoint, presumably to facilitate scanning through all the CTS ledger entries using the API, as opposed to the ledger files.

This may be a premature and ultimately harmful optimisation, because it trades off an ever-growing in-memory index for being able to skip deserialising what should amount to a fairly small amount of transactions normally (governance is rarely the major part of a ledger). Aside from the memory cost, this will also greatly increase first-response latency on new nodes, which currently won't respond usefully before they can build the index up to that point. An index-less historical query will come back much faster in that situation.

My sense is that it would be best to:

1. Remove the index
2. Convert /entries/txIds to a regular historical query, if it's needed at all
3. Establish one or more benchmarks that we think /entries/txIds should meet, and decide what, if any optimisation is needed</issue_description>

<agent_instructions>Remove /entries/txIds API as suggested, along with its own indexer.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes review the need for entry_seqno_index #237

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[achamayou]

This is a positive change, workloads that want to audit substantial portions of the ledger are better served by exposing static ledger files for download. It makes more sense to cache and serve those at scale than to do with inside the TS TEE.

As pointed out in the PR description, governance transactions are only a small portion of a typical ledger, and so skipping them is not a major imposition on an auditing entity.

[ivarprudnikov]

Unfortunately this existing api is tied to the existing rest api spec we have https://github.com/microsoft/azure-rest-api-specs_2024-03-01/blob/main/specification/confidentialledger/Microsoft.CodeTransparency/entries.tsp#L59

Need to create a new simpler version and deprecate these methods.

[achamayou]

It may be possible to create an intermediate version without the index, and which just uses the regular historical query flow, but I am not sure if it's worth doing.