ES time based indexing


#1

Hello,

Any interest in supporting time based indexing in ES? Would be happy to supply the PR


#2

Could you please elaborate a bit more on the use case?


#3

So, we have a lot of data, and are currently retiring it using time based indexing. It seems it would make sense to have the old graph data retired in the same manner. I think all that may be necessary for this is to take separate read and write indicies, with the read indicies supporting wildcards.


#4

If I understood correctly, you want each node/quad to have a timestamp and ignore data below a specific timestamp.

This is pretty easy to achieve. All you need is to build a query (without a quadstore), get a shape from it, run a custom optimizer that will replace all Quads query shapes with Intersect(Quads, nosql.Shape{...}), where the nosql.Shape contains a filter for the timestamp field. But the problem is that something will need to populate these timestamps. This would require changing NoSQL meta-backend a bit. I will gladly assit you with the a PR.

However, I’m not sure that we will be able to merge it upstream. The reason behind it is that your feature request is in fact not about time indexes, but more about adding X as a metadata on nodes/quads, where X might be timestamp, weight, subgraph, etc. If we want to bring timestamps, we should consider adding other custom metadata as well.

But, please do not be discouraged by this note. Feel free to modify the code and open a PR for the review. Cayley was meant to be modular to fit custom use cases, and this is exactly what you want to achieve. We will assist you with an implementation, and we are planning to add custom metadata as a part of v0.8-0.9. Your use case will give us a valuable feedback to design a new API properly.