Automating Processes: Running Scripts on Latest Changed Files with GitHub Workflows

In today’s world, where every system is going towards exponential complexity, we want to automate our processes, and same was the case for us. We wanted to update our configs that we were storing in our database in a systematic way. The approach we used was that we took a clone of our configs in database, and we kept those configs in separate files based on their primary ids. The problem was, we wanted to update those configs in our database when we change them in their files. So here comes the solution, we wrote a script to be run on every change to the main branch using GitHub Workflows updating only those configs that were changed. How? Let’s learn.

Overview:

The implemented solution uses GitHub Workflows to automate the execution of scripts based on changes made in commits. Here’s an overview of the process:

1. GitHub Workflows Setup: The process begins with setting up a GitHub Workflow file within the .github/workflows directory of the repository. This file defines the conditions under which scripts should be executed, such as on pushes to specific branches like main.
2. Detecting Changes in Commits: Within the workflow, a step is dedicated to identifying changes between the last commit and the current one. This is achieved using the git diff command, comparing the files in the previous commit (github.event.before) with the files in the current commit (github.sha).
3. Extracting Changed Files: The output of the git diff command provides a list of files that have been modified, added, or deleted between commits. This list of changed files is then captured and processed to ensure it is in a suitable format for further usage.
4. Passing Changed File Names to Script: After obtaining the list of changed files, it is passed as arguments to the command that executes the script. This ensures that the script operates only on the files that have been modified since the last commit, minimizing unnecessary processing and improving efficiency.
5. Executing the Script: Finally, the script is executed with the list of changed file names as arguments. This enables the script to perform its intended actions, such as running tests, generating documentation, or deploying code, specifically targeting the files that have been altered in the latest commit.

Following is the .github/workflow file for accomplishing the above task.

name: Deploymenton: push: branches: - mainjobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 # Don't forget this line, otherwise git wouldn't be able to detect the hashes of the commits - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: "18" - name: Install dependencies run: npm install - name: Identify changed files id: changed-files run: | changed_files=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }}) # Replace newline characters with spaces to avoid data loss. When this variable is transferred  # to the next step, the data after the first new line special character is lost changed_files=${changed_files//$'\n'/ } echo "::set-output name=changed_files::$changed_files" - name: Run script with changed files run: | changed_files="${{ steps.changed-files.outputs.changed_files }}" echo "$changed_files" echo "Executing command: node index.js $changed_files" node index.js $changed_files

The question is, how would we get the names of the changed files in our script file!? Easypeasy:

async function main() {// 'process.argv' is an array containing the command line arguments provided when the Node.js process was invoked.// The '.slice(2)' method is used to extract elements from the 'process.argv' array, starting from index 2, since indexes 0 and 1 contain node and index.js respectively const allModifiedFiles = process.argv.slice(2);// Filter only those files that you want to track const modifiedFiles = allModifiedFiles.filter((file) => file.startsWith("data/") ); modifiedFiles.forEach(async (file) => { // Write code as per your requirements });}main();