sherlockbeard 6 hours ago

any measured commit latency overhead from the vector index on a streaming job, once it's built? post benchmarkss build time + query latency but not the write path stuff.

  • cloud8bits 2 hours ago

    We had reported index build time, but not continuous maintenance. Good feedback, we will include it a future blog.

    To answer this now:

    a) Assuming the vectors are generated, its one additional set of sharded writes to object storage. updates/deletes are logged, then read side merged, during index lookup.

    b) This is same as how secondary indexing works, and it adds about 10-15% more to writer pipeline. Most expensive part still remains producing columnar files for the actual data.

    Hope that helps.

rjshmhndr 4 hours ago

vector search with < 100msecs on a lake is unheard of - pretty impressive

jauntywundrkind 8 hours ago

Pretty good write up, talking to the problem, surveying what and how other players do things?

Went looking to see if there was going to be anything open (mostly it's product pitch), found:

> Iceberg support builds on the Hudi-backed indexing work in Apache XTable (Incubating). We developed the design in the open with engineers from Uber and Walmart to support their planet-scale tables. The open RFC documents the storage layout, query path and consistency model.

> The design is settled and ships in Lakegres today; the Hudi implementation is being upstreamed and XTable is in release-candidate testing ahead of OSS releases that other engines can integrate.

With two links inline, https://github.com/apache/incubator-xtable/pull/935 https://github.com/apache/hudi/blob/master/rfc/rfc-109/rfc-1...

  • gardnr 8 hours ago

    The problem doesn’t really seem to be solved though. Most attempts at decentralized index storage will be limited by what this project calls nprobes. The authors propose a cheap search first approach but it’s unclear how it scales.

tokYoComPleX 8 hours ago

This is a solidly built feature with great performance eclipsing other competing solutions on the market.

  • CBLT 8 hours ago

    Since you created an account just to compliment this product, it would probably be best to explain your relationship to the product in question. Even a simple "not affiliated, just a happy user" would be great.