Administration: Taking Snowflake’s Git Integration and Schema for a Spin
Written by
Dan Luszcz, Senior Manager
Published
May 14, 2024
Snowflake functionality can be overwhelming. And when you factor in technology partners, marketplace apps, and APIs, the possibilities become seemingly endless. As an experienced Snowflake partner, we understand that customers need help sifting through the possibilities to identify the functionality that will bring them the most value.
Designed to help you digest all that’s possible, our Snowflake Panorama series shines a light on core areas that will ultimately give you a big picture understanding of how Snowflake can help you access and enrich valuable data across the enterprise for innovation and competitive advantage.
After Build 2023, we discussed some exciting announcements from Snowflake around Git integration and declarative schema management. With Git integration now in public preview, we decided to take these new features for a spin to see what the hands-on experience is like.
The new features we are talking about today will enable Snowflake admins to integrate source control and use declarative schema to maintain their Snowflake environment using three key functions:
Git Integration, which essentially lets you to setup a Snowflake stage that’s connected to your Git repository
EXECUTE IMMEDIATE FROM, which lets you run scripts directly from a stage (see above)
CREATE OR ALTER, which is a new DDL command that can perform a schema compare and update the target table to match the new schema in your latest definition
Git Integration
Git integration, which entered public preview in April 2024, enables you connect to a Git repository and interact with it like a Snowflake stage. I found that Snowflake’s implementation is a bit more involved compared to traditional Git alternatives since it integrates through SQL. The bulk of the setup consists of permissions—a song and dance very familiar to anyone who has spent any time in Snowflake.
The first challenge I faced when experimenting with this feature was the requirement to create the Git credentials and Integration in a Snowflake database. While unsurprising, in our case there was not any appropriate candidate database in our environment. The Git integration was going to be used to deploy our Snowflake OS Streamlit app, which is an Application Package and does not use an existing database. Ultimately, rather than use an existing database that had nothing to do with my use case, I decided to create a database and schema specifically for the purposes of the Git integration. Creating a database just for a Git repository rubbed me the wrong way, but it was far preferable to the alternative.
Once the roles, database, schema, secret, and Git Repository were created successfully, I was able to fetch the master branch into Snowflake and list the files that had been pulled into the stage. Again, this is all done in SQL so you can leave your extensive knowledge of Git commands at the door. Developers will not be performing merges in Snowsight; this is simply for pulling files from Git into Snowflake for deployment. Once I had successfully fetched the files from Git into Snowflake, it was relatively easy to copy those files into an Application Package Stage.
In summary, the Git integration meets expectations. It’s not particularly user friendly, but that’s par for the course: Snowflake built an excellent cloud data warehouse and then decided to start adding features on top. That approach comes with pros and cons. For users who spend a lot of time in Snowflake, the mechanics of the new Git integration will feel very natural. For someone who is used to a traditional Git experience, it will take more getting used to.
EXECUTE IMMEDIATE FROM
CREATE OR ALTER
Finally, the star of the show: CREATE OR ALTER. As a reminder, CREATE OR ALTER enables declarative schema management, where developers can evolve table schema in-place over time. Rather than forcing developers to migrate schema by writing ALTER statements, which must be executed sequentially, declarative schema management is as simple as updating the table definition and letting Snowflake determine the necessary ALTER statements to update the target table to match. Declarative schema management enables clean and intuitive source control, collaboration, and automated deployments for your database schema. Unfortunately, this feature is still in Private Preview. Like the Jinja templating, we’ll have to wait for this to be released in Public Preview before we can start playing with it. These two features will be critical to fully unlocking the power of the Git integration feature. Until then, it’s nice to have but ultimately limited in its scope.
Leveraging Snowflake for Modern Dev Ops
In summary, the Git Integration and EXECUTE IMMEDIATE FROM features are an important step on the journey to implementing modern Dev Ops in Snowflake. Combined with the declarative schema management and templating features still in Private Preview, these capabilities will make Snowflake an even more compelling choice for businesses, even those that may not need to leverage it for its traditional cloud data warehouse role. Viewed alongside other upcoming features like Hybrid Tables, an exciting picture is forming where Snowflake can offer cutting-edge solutions for all the data needs of modern data organizations.
Ready to unlock the full potential of data and AI?
Book a free consultation to learn how OneSix can help drive meaningful business outcomes.
