GitHub - Infodatamatrix/AIKafkaPipelineDemo: This is a sample kafka driven demo · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Infodatamatrix
AIKafkaPipelineDemo
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>4 Commits<br>4 Commits
api-gateway/app
api-gateway/app
shared
shared
workers
workers
.env
.env
Dockerfile
Dockerfile
README.md
README.md
docker-compose.yml
docker-compose.yml
requirements.txt
requirements.txt
View all files
Repository files navigation
AI Kafka Pipeline Demo
A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain.
Project Structure
ai-kafka-pipeline-demo/<br>├── api-gateway/<br>│ └── app/<br>│ ├── main.py<br>│ ├── routes/submit.py<br>│ ├── services/publisher.py<br>│ └── config.py<br>├── workers/<br>│ ├── extractor/<br>│ ├── summarizer/<br>│ └── notifier/<br>├── shared/<br>│ ├── kafka/<br>│ ├── schemas/<br>│ ├── config/<br>│ └── utils/<br>├── Dockerfile<br>├── docker-compose.yml<br>├── requirements.txt<br>└── .env
Demo Flow
POST /submit to the API gateway
Gateway publishes document.submitted
Extractor consumes and publishes text.extracted
Summarizer consumes and publishes summary.generated
Notifier consumes and logs final completion
Run
docker compose up --build
Test
Open another terminal:
curl -X POST http://localhost:8000/submit \<br>-H "Content-Type: application/json" \<br>-d '{<br>"user_id": "user-1",<br>"content": "Kafka helps decouple AI pipeline stages for scalable processing in production systems."<br>}'
What you should see
API returns Processing started
Extractor logs the incoming event
Summarizer logs the next event
Notifier logs the final pipeline completion
Suggested narration
FastAPI handles intake, not heavy processing.
Kafka turns the request into an event.
Each worker owns one stage.
Shared schemas keep the contracts explicit.
This is the simplest form of a production-style AI pipeline.
I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here: https://youtu.be/c2ijN2KAWXw
https://youtu.be/KjvbABpajUs
About
This is a sample kafka driven demo
Resources
Readme
Uh oh!
There was an error while loading. Please reload this page.
Activity
Stars
stars
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Python<br>98.8%
Dockerfile<br>1.2%
You can’t perform that action at this time.