Databricks Jobs Monitoring in your custom Databricks App | by Maksim Pachkouski | May, 2026 | Data Engineer ThingsSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Data Engineer Things

Things learned in our data engineering journey and ideas on data and engineering.

Databricks Jobs Monitoring in your custom Databricks App

Maksim Pachkouski

4 min read·<br>22 hours ago

Listen

Share

You're an active Databricks user and have already scheduled several dozen jobs. By default, there are two ways to monitor job failures: set up alerts via email or Teams, or check the Monitoring dashboard in Databricks. But sometimes you forget to set up alerts, and with a large number of jobs, the dashboard becomes inconvenient.<br>To solve this problem, I created a small Databricks app on Streamlit, which works through the API SDK.<br>Press enter or click to view image in full size

Why doesn't the current Databricks dashboard meet our needs? It only displays the last five launches and a large number of pages to navigate:

Press enter or click to view image in full size

I like Databricks, it's like a construction kit. If you understand how it works, you can add a lot of custom solutions to automate processes. We'll use the Databricks App to create a more user-friendly dashboard.<br>In my previous article, I shared a modular template for developing Databricks Apps on Streamlit, which we'll use to build our app. The first version will be fairly simple: the dashboard will display the cumulative status, but in a single, more convenient location. This dashboard is convenient for daily batch uploads. However, it can also be adapted for streaming or DLT pipelines, if desired:

Press enter or click to view image in full size

The app retrieves information via the API SDK. The architecture is fairly simple: one page for timezone and command settings, and a second for displaying jobs.<br>Quite often, employees make mistakes and can set jobs in different time zones, making it difficult to understand when the script actually runs. Therefore, we'll add a filter to consolidate the entire schedule into a single time zone.

Press enter or click to view image in full size

The app has Settings, where you can configure the default display timezone and commands. You can then filter jobs by command:

Press enter or click to view image in full size

The app can be run locally by connecting to Databricks via the CLI, and then it will work for free, since at the moment we only use the API and do not need a cluster.<br>Get Maksim Pachkouski’s stories in your inbox

Join Medium for free to get updates from this writer.

Subscribe

Subscribe

Remember me for faster sign in

When working via the CLI from a local computer, you'll only have access to jobs you have permission to access. When installing the application in Databricks, you need to configure access rights.<br>Installation and launch<br>You can also install it within Databricks itself, but remember to stop it. It will consume an average of $300 per month (Medium cluster: Up to 2 vCPUs, 6 GB memory, 0.5 DBU/hour). If you run it on demand or schedule it, it will cost $10-$100 per month.

Installation inside Databricks<br>Go to Compute -> Apps -> Go to Databricks Apps -> Create App -> Create a custom app .<br>2. Specify a name, for example, jobsadmin , then Next Configuration Git.<br>3. Specify the link non repository https://github.com/protmaks/databricks_app_jobs_admin_streamlit<br>Next, click Next Configure. If you can't install from GitHub, you'll need to either download the repository or enable the setting. More details are in the README.<br>4. In User authorization -> +Add scope, you'll need to select access to jobs (this feature isn't currently available). Then, any user who opens jobs will only see those they have access to.<br>You can also manually open access to jobs for monitoring on the Settings page in the app:<br>Press enter or click to view image in full size

5. Click Create App . The installation will then begin.<br>6. Once the installation is complete, click Deploy. Enter the branch name "main" and click Deploy again.<br>The application is now active and can be opened using the link:

Press enter or click to view image in full size

If you want the application to see all jobs by default, you need to add the Service Principal to the Admin group:<br>Go to Settings -> Identity and access -> Groups.<br>Select the admins group.<br>Click Add members.<br>Search for your App Name (or its Service Principal ID) and add it.

This approach works like Ad-hoc: if you don't open the app, you won't even know there are problems. Therefore, it's still worth configuring error notifications. Perhaps we'll add control over this parameter in the next section. Subscribe to stay up to date. I'd also appreciate your comments and ideas.

Databricks

Databricks Apps

Databricks Jobs

Data Engineering

Data Mesh

Published in Data Engineer Things<br>46K followers<br>·Last published 22 hours ago

Things learned in our data engineering...