Agentic Engineering at Zalando: a snapshotWhile the landscape and environment rapidly changes every day, the whole industry is figuring out how to approach Agentic Engineering. With more than 250 engineering teams innovating across our business lines, we're observing value and impact of LLMs in different forms and at different paces. We shared some early wins on using LLMs for product data enrichments, LLM as a judge for improving search quality, relevance assessment in product search, or frontend migrations.<br>Recently, we have been looking back at our progress in the past 2.5 years and we'd like to share a few approaches that worked well for us.<br>LLM proxy for API-based LLM access from day 1<br>We've been a GitHub Copilot user from the early days when it offered autocomplete in the IDE. To complement this offering and provide API-based access to LLMs, our ML platform team deployed in January 2024 a LiteLLM based API proxy with access to models from different providers (now: OpenAI, AWS Bedrock, and Google Vertex). This way, it became easy for our engineers to experiment with different tools and models. The platform team got a single point to measure adoption via: MAU, WAU, model, User-Agent.<br>We like LiteLLM for its extensibility. We use post-call hooks for anonymized cost tracking and pre-call hooks for enforcing client version upgrades by restricting access to the proxy based on the User-Agent header. For self-managed client installations, unfortunately blocking access is the only effective measure. Same goes for retiring models. There is always a long-tail group of users who do not adjust their local configurations and who do not follow new model releases. We also enabled auto-injection of prompt caching checkpoints, which reduced costs for custom agents while their authors still learn about prompt caching. To mitigate stability and memory leak issues of LiteLLM, we enforce restarts after 20k requests using --max_requests_before_restart. This enables us to run the proxy for 2k MAU with just six small (2k CPU, 4 GB) pods. We look forward to the Rust rewrite that's expected to improve performance and stability.<br>Beyond the API: Chat UI and CLI<br>The API offering is complemented with a simple chat UI (fork of a now unmaintained OSS codebase) and a CLI tool (custom-built using pydantic-ai). To our surprise, the Chat UI still has a high adoption rate even though IDE plugins and CLIs are ubiquitous these days and users have many more capable alternatives to choose from. The CLI was incepted in a hackathon (Aug 2024) in times where coding agents did not exist yet. Initially, we used it in maintenance scripts for model access in the terminal. Over time, the repository attracted a small community of maintainers who extended it with additional tools helping us scale adoption of LLMs for coding tasks:<br>generating images with simple file format conversion<br>an interactive mode for multi-turn chats, supported by simple commands to load and save files for context management<br>agent mode with MCP support and automatic Bearer token injection for internally hosted MCP servers<br>http to stdio MCP proxy to make internally built MCP servers easy to access in any other tool<br>built-in MCP server configuration, enabling first-time MCP users to experiment easily without much friction<br>coding agent configuration command which installs safe configurations for claude code, opencode, pi along with custom plugins for model autodiscovery<br>The token injection and MCP proxy helped us to promote safe configuration of MCP servers where no secrets need to be hardcoded in configuration files. It allowed to spread the use of internally-deployed MCP servers without needing to deal with any auth concerns. This is especially important as the user base of the LLM access expands beyond engineering, with varying intuition for security. Deployed MCP servers, hosted by teams, are automatically protected by our default ingress oauth filter.<br>Challenges with LLM-enabled tooling<br>Two classic problems are consistent across different tools. Too often, tools use a generic User-Agent header, making it difficult to identify the tools used as clients of the proxy. For our own LLM applications, such as custom code review agents, we ensure to include a name and originating repository and version as part of the User-Agent header. For other tools, we request changes upstream (or contribute).<br>Second, tools lack support for custom auth commands for auth token generation, supporting only static credentials or defaulting to support subscription offerings. Reliance on environment variables causes user frustration as tokens expire and need to be refreshed manually which involves restarting the applications. To bridge this gap, we have a local proxy that injects auth headers and write plugins for coding agents that handle model access and model discovery along with their parameters. The proxy evolved to include features helping live debug LLM tools of our own making. It ships with a TUI...