KServe, AIBrix, and llmaz

As a follower and active contributor for inference platform, I created the llmaz project to provide an unified inference platform for LLMs and also joined the AIBrix community to build the next-gen GenAI infrastructure.

Frankly speaking, I think AIBrix makes a lot of sense to the industry which shows how a modern inference system should be. Definitely will give me some inspirations to design the llmaz in the future.

And in the early discussion with @Jeffwan, the founder of AIBrix, I just found the vision of AIBrix and llmaz is quite similar, both of these projects hope to work smoothly with the inference engines and bridge the gap between local deployment of inference service and large scale distribution deployment. So I think it would be meaningful to have a comparison between these two projects, especially for those who has little knowledge with them or even the inference system.

What I want to emphasize is, I’m not trying to say which one is better, but to show the differences and similarities between them. And usually, open source projects learn from each other, so from the llmaz perspective, this is really beneficial.

Note that, the comparison won’t be too detailed, just a feature matrix, because all these two projects are still at the early stage, we may have a deep dive in the future. And even we tagged Yes for two projects in the comparison matrix, it doesn’t mean they have the same capacities, the implementation details could be really different.

Also, KServe, as one of the most popular inference projects, is also a good comparison target for us. So I will also include it in the comparison matrix. And in the end, I will gather some other inference-related projects for reference as well.

Last but not least, I respect all these projects, and I’m not an expert for all of them, so if you find any mistakes or have any suggestions, please feel free to let me know, and if I missed some fields, let me know. Again, I’m not trying to choose the best, just want to provide an overview for the people who are interested in inference system.

Find the Google slides here.