At some point in the race of scaling a search application, as query load (queries per second) and corpus size (number of documents) increase, we need to distribute things. Distribution means in practical terms sharding the index, and we need to solve both distributing indexing as well as distributed search.
For years I’ve been continuously fighting with backups. I have not been particularly good or consistent at it. We’ve been okay with Time Machine and Carbon Copy Cloner, but the recent addition of a digital SLR to our gadget collection has meant running out of space on our shared home drive.
On a theoretical world, given the ability for a processor to run an infinite amount of threads, we could prove the following statements (no attribution purposely given): (1) If you do more CPU than I/O, use threads; (2) If you do more I/O than CPU, use more threads. Which would allow us to conclude with the following corollary: “at full utilization, threads and events have the same theoretical throughput.” Such argument ignores praxis – it is a purely theoretical debate disconnected from the reality of scaling services –.
I want to use a standard and open document format, so that I know I will be able to open my documents in years to come. That’s plain text. I also want to revision control my text files, so I want to use source control system. Which will also allow me to synchronize between computers between computers. I also like to have one standard editor I know I find in any operating system, and that will always work with my documents. I want to make sure that I am not locked-in into any particular cloud or tool or format or …
I am playing with a new query language. I am defining the grammar as EBNF, but I want to show railroad diagrams for those readers that are more graphical and less familiar with BNF. I’ve found limited support for generating syntax diagrams from EBNF.