kweb is a Python application that exposes Kafka API via websocket. The application is based on Tornado asynchronous web framework and kafka-python library.
Help me to improve this project!
kweb is very easy to use. You simply create a websocket connection and you can start issuing command to operate with kafka.
Assuming you already have the source code, in order to start the service you can do:
cd kweb
python main.pyNow you have the service up and running.
In order to create a Kafka consumer you have to connect via websocket protocol to the following url:
ws://localhost:8000/kweb/consumer/ws/v1In order to create a consumer, issue the following command on the websocket:
{
"command_name": "create_consumer",
"parameters": {
"bootstrap_servers": ["<kafka-host-1>:9092", "<kafka-host-2:9092"],
"group_id": "testing_group",
"enable_auto_commit": false
}
}Done. Consumer created. If you want to know which options you can specify, you can check here: Kafka Consumer
After you created a consumer, you should subscribe some topics. You can do that by issuing the following command:
{
"command_name": "subscribe",
"parameters": {
"topics": ["TOPIC1", "OTHER-TOPIC", "ANOTHER-TOPIC"]
}
}More details on this command here: Subscribe Command
If you want to unsubscribe the previously subcribed topics, you can issue the following command:
{
"command_name": "unsubscribe",
"parameters": {}
}More details on this command here: Unsubscribe Command
If you want to seek to end of topic-partition, you can issue the following command:
{
"command_name": "seek_to_end",
"parameters": {
"topic-partitions": [
{
"topic": "<topic-name>",
"partition": 0
},
{
"topic": "<another-assigned-topic>",
"partition": 2
}
]
}
}More details on this command here: Seek to End Command
If you want to seek to beginning of topic-partition, you can issue the following command:
{
"command_name": "seek_to_beginning",
"parameters": {
"topic-partitions": [
{
"topic": "<topic-name>",
"partition": 0
},
{
"topic": "<another-assigned-topic>",
"partition": 2
}
]
}
}More details on this command here: Seek to Beginning Command
If you want to assign a series of topic-partition to your consumer you can issue the following command:
{
"command_name": "assign",
"parameters": {
"topic-partitions": [
{
"topic": "<topic-name>",
"partition": 0
},
{
"topic": "<another-assigned-topic>",
"partition": 2
}
]
}
}More details on this command here: Assign Command
If you want to commit your offset in an asynchronous manner, you can issue the following command:
{
"command_name": "commit_async",
"parameters": {
"topic-offset-metadata": [
{
"key": {
"topic": "<topic-name>",
"partition": 0
},
"value": {
"offset": 1,
"metadata": "your metadata here"
}
}
]
}
}Note that topic-offset-metadata can also be an empty array. In this way all the offset will be committed. Note also
that since this command is asynchronous, a callback will notify you on the same socket when the commit will be
received by the cluster.
More details on this command here: Commit Async Command
If you want to commit your offset you can issue the following command:
{
"command_name": "commit",
"parameters": {
"topic-offset-metadata": [
{
"key": {
"topic": "<topic-name>",
"partition": 0
},
"value": {
"offset": 1,
"metadata": "your metadata here"
}
}
]
}
}Note that topic-offset-metadata can also be an empty array. In this way all the offset will be committed.
More details on this command here: Commit Command
If you want to know the last committed offset for a given topic-partition, you can issue the following command:
{
"command_name": "committed",
"parameters": {
"topic-partition": {
"topic": "<topic-name>",
"partition": 0
},
"metadata": false
}
}If you don't want only the last committed offset as an integer, you can set metadata to true. In this way
more details will be returned.
More details on this command here: Committed
If you want to know the partitions available for a specific topic (assuming you are authorized to do so), you can issue the following command:
{
"command_name": "partitions_for_topic",
"parameters": {
"topic": "<topic-name>"
}
}More details on this command here: Partitions for Topic Command
If you want to know the offset of the next record that will be fetched you can issue the following command:
{
"command_name": "position",
"parameters": {
"topic-partition": {
"topic": "<topic-name>",
"partition": 0
}
}
}More details on this command here: Position Command
If you want to know the last known highwater for a given topic-partition, you can issue the following command:
{
"command_name": "highwater",
"parameters": {
"topic-partition": {
"topic": "<topic-name>",
"partition": 0
}
}
}If you want to know what's an highwater offset, or you want to have more details on this command, go here: Highwater Command
If you want to suspend fetching from a set of topic-partitions, you can issue the following command:
{
"command_name": "pause",
"parameters": {
"topic-partition": [
{
"topic": "<topic-name>",
"partition": 0
}
]
}
}More details on this command here: Pause Command
If you want to resume fetching from a set of previously paused topic-partitions, you can issue the following command:
{
"command_name": "resume",
"parameters": {
"topic-partition": [
{
"topic": "<topic-name>",
"partition": 0
}
]
}
}More details on this command here: Resume Command
If you want to seek on specifically topic-partion, you can issue the following command:
{
"command_name": "seek",
"parameters": {
"topic-partition": {
"topic": "<topic-name>",
"partition": 0
},
"offset": 100
}
}More details on this command here: Seek Command
If you want to get the initial offsets for the specified set of topic-partitions, you can issue the following command:
{
"command_name": "beginning_offsets",
"parameters": {
"topic-partition": [
{
"topic": "<topic-name>",
"partition": 0
}
]
}
}More details on this command here: Beginning Offsets Command
If you want to get the end offsets for the specified set of topic-partitions, you can issue the following command:
{
"command_name": "end_offsets",
"parameters": {
"topic-partition": [
{
"topic": "<topic-name>",
"partition": 0
}
]
}
}More details on this command here: End Offsets Command