SKM IT World

Just another blog about IT


Leave a comment

How to Format a Large Code Base Automatically

If you introduce code formatting rules retroactively, you have to solve the problem how to format existing code base according to the new formatting rules. You could checkout every code repository one by one in your IDE and click on Autoformat the whole project. But this is boring and waste of time. Fortunately, Intellij IDEA has a format CLI tool in its installation. You can locate it in the path <your Intellij IDEA installation>/bin. It’s called format.sh.

In the next section I’d like to show you how you can automate formatting big code base. First, I will show the preparation steps like exporting your code formatting rule setting from the IDE. Then, I will demonstrate how to use the CLI-Tool format.sh. At the end, I will show a small Groovy script that query all repositories (in this case they are Git repositories), formatting the code and push it back to the remote SCM.

Preparations

First at all, we need the code formatting rule setting exported from Intellij IDEA. In your Intellij IDEA follow the next step

  1. Open File -> Settings -Editor-> Code Style
  2. Click on Export…
  3. Choose a name for the XML file (for example, Default.xml) and a location where this file should be saved (for example, /home/foo ).

Then, checkout or clone your SCM repository and remember the location where you checkout/clone it (for example, /home/foo/myrepository).

Format Code Base Via format.sh  CLI Tool

Three parameters are important for format.sh:

  • -s : Set a path to Intellij IDEA code style settings .xml file (in our example: /home/foo/Default.xml).
  • -r : Set that directories should be scanned recursively.
  • path<n> : Set a path to a file or a directory that should be formatted (in our example: /home/foo/myrepository).

> ./format.sh
IntelliJ IDEA 2018.2.4, build IC-182.4505.22 Formatter
Usage: format [-h] [-r|-R] [-s|-settings settingsPath] path1 path2...
-h|-help Show a help message and exit.
-s|-settings A path to Intellij IDEA code style settings .xml file.
-r|-R Scan directories recursively.
-m|-mask A comma-separated list of file masks.
path<n> A path to a file or a directory.

> /format.sh -r -s ~/Default.xml ~/myrepository

It’s possible that the tool cancels scanning because of a java.lang.OutOfMemoryError: Java heap space. Then, you have to increase Java’s maximum memory size (-Xmx) in <your Intellij IDEA installation>/bin/idea64.vmoptions.


> nano idea64.vmoptions
-Xms128m
-Xmx750m // <- here increase the maximum memory size
-XX:ReservedCodeCacheSize=240m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-Djdk.http.auth.tunneling.disabledSchemes=""
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-Dawt.useSystemAAFontSettings=lcd
-Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine

Groovy Script For Formatting Many Repository In a Row

Now, we want to bring everything together. The script should do four things:

  1. Find all repository URLs whose code has to be formatted.
  2. Check out / Clone the repositories.
  3. Format the code in all branches of each repostory.
  4. Commit and push the change to the remote SCM.

I choose Git as SCM in my example. The finding of the repository URLs depends on the Git Management System (like BitBucket, Gitlab, SCM Manager etc.) that you use. But the approach is in all system the same:

  1. Call the RESTful API of your Git Management System.
  2. Parse the JSON object in the response after the URLs.

For example, in BitBucket it’s like that:



@Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1')
import groovyx.net.http.*

import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*

def cloneUrlsForProject() {

    def projectUrl = "https://scm/rest/api/1.0/projects/PROJECT_KEY/repos?limit=1000"
    def token = "BITBUCKET_TOKEN"

    def projects = []
    def cloneUrls = []

    def http = new HTTPBuilder(projectUrl)
    http.request(GET) {
        headers."Accept" = "application/json"
        headers."Authorization" = "Bearer ${token}"

        response.success = { resp -> projects = new JsonSlurper().parseText(resp.entity.content.text)}

        response.failure = { resp ->
            throw new RuntimeException("Error fetching clone urls for '${projectKey}': ${resp.statusLine}")
        }
    }

    projects.values.each { value ->
        def cloneLink = value.links.clone.find { it.name == "ssh" }
        cloneUrls.add(cloneLink.href)
    }

    return cloneUrls
}

Then, we have to clone the repositories and checkout each branch. In each branch, the format.sh has to be called. For the git operation, we use the jgit library and for the format.sh call we use a Groovy feature for process calling. In Groovy it’s possible to define the command as a String and then to call the method execute() on this String like “ls -l”.execute(). So the Groovy script for the last three tasks would be looked like that:

#!/usr/bin/env groovy
@Grab('org.eclipse.jgit:org.eclipse.jgit:5.1.2.201810061102-r')
import jgit.*
import org.eclipse.jgit.api.CreateBranchCommand
import org.eclipse.jgit.api.Git
import org.eclipse.jgit.api.ListBranchCommand
import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider

import java.nio.file.Files


def intellijHome = 'path to your idea home folder'
def codeFormatterSetting = 'path to your exported code formatter setting file'
def allRepositoriesUrls = ["http://scm/repo1","http://scm/repo2"] // for simplifying

allRepositoriesUrls.each { repository ->
    def repositoryName = repository.split('/').flatten().findAll { it != null }.last()
    File localPath = Files.createTempDirectory("${repositoryName}-").toFile()
    println "Clone ${repository} to ${localPath}"
    Git.cloneRepository()
       .setURI(repository)
       .setDirectory(localPath)
       .setNoCheckout(true)
       .setCredentialsProvider(new UsernamePasswordCredentialsProvider("user", "password")) // only needed when clone url is https / http
       .call()
       .withCloseable { git ->
        def remoteBranches = git.branchList().setListMode(ListBranchCommand.ListMode.REMOTE).call()
        def remoteBranchNames = remoteBranches.collect { it.name.replace('refs/remotes/origin/', '') }

        println "Found the following branches: ${remoteBranchNames}"

        remoteBranchNames.each { remoteBranch ->
            println "Checkout branch $remoteBranch"
            git.checkout()
               .setName(remoteBranch)
               .setCreateBranch(true)
               .setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK)
               .setStartPoint("origin/" + remoteBranch)
               .call()

            def formatCommand = "$intellijHome/bin/format.sh -r -s $codeFormatterSetting $localPath"

            println formatCommand.execute().text

            git.add()
               .addFilepattern('.')
               .call()
            git.commit()
               .setAuthor("Automator", "no-reply@yourcompany.com")
               .setMessage('Format code according to IntelliJ setting.')
               .call()

            println "Commit successful!"
        }

        git.push()
           .setCredentialsProvider(new UsernamePasswordCredentialsProvider("user", "password")) // only needed when clone url is https / http
           .setPushAll()
           .call()

        println "Push is done"

    }
}

Do you have another approach? Let me know and write a comment below.

Advertisements


3 Comments

How To Debug Groovy Script From Shell

Groovy is a scripting language, so it is possible to run Groovy code without compiling to Java byte code. The necessary condition is that Groovy is installed on your machine. Then, running a Groovy script in a shell looks like the following line.


~>groovy TestScript.groovy

Now, something is wrong with the script, only on a special environment. So you want to debug your Groovy script from the shell. Fortunately, it works for Groovy just like for Java. You only have to export the Java options for debugging.


~>export JAVA_OPTS="-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=y"

Now, we can debug our script running from the shell with our favorite IDE.


~>groovy TestScript.groovy
Listening for transport dt_socket at address: 4000


1 Comment

Migration Path for Jenkins Subversion Plugin from Version 1.53 to 2.2

Introduction

Jenkins Subversion Plugin changes its credential management. Before version 2.2 you had to set your Subversion credential globally depend on domain name. In version 2.2 it’s changed. The credential management is still central but now you have to configure in every job which credential the job has to use for Subversion authentication .

If you have many jobs, you don’t want to touch every job to change the Subversion configuration. A Groovy script for Jenkins Script Console [1] can help. In the next section I will describe how the migration path looks like.

Migration Path

  1. Update the Subversion Plugin in your Jenkins instance to version 2.2.
  2. Add your Subversion credential to the global credential store:
    1. Go to Jenkins -> Credential -> Global credentials -> Add credential
    2. Add your credential. The scope has to be global.
  3. Go to the installation path of your Jenkins instance and open the file credential.xml.
  4. Search in credential.xml for the element <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl> that contains your above set credential and copy the value of the element <id> .
    Example:

    <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
      <scope>GLOBAL</scope>
      <id>63c3793c-e3fb-49eb-b45b-f6f8e7364876</id>
      <description></description>
      <username>jenkinssvnuser</username>
      HyDUenzpyDkbL9xMoQ0pxdK10l20VkXKEiy4+ZnjL9c=
    </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
    
  5. Go to the Jenkins Script Console and run following Groovy script. Assign your credential id to the variable credentialId before running the script.
    def credentialId = '63c3793c-e3fb-49eb-b45b-f6f8e7364876'
    
    Hudson.instance.items.each { item ->
    
    if(item.scm instanceof hudson.scm.SubversionSCM) {
      println("JOB : "+item.name)
    
      def newLocations = new ArrayList()
    
      item.scm.locations.each {location ->
      newLocations.add(new hudson.scm.SubversionSCM.ModuleLocation(location.remote, credentialId, location.local, location.depthOption,location.ignoreExternalsOption))
    }
    
    def newScm = new hudson.scm.SubversionSCM(newLocations, item.scm.workspaceUpdater,
    item.scm.browser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
    item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog, item.scm.additionalCredentials)
    
    item.scm = newScm
    item.save()
    println("\n=======\n")
    }
    
  6. Finish.

This script is tested with Jenkins version 1.555 and Jenkins Subversion Plugin version 2.2. On Github [2] you can find more Groovy script for Jenkins Script Console.

Links

[1] Jenkins Script Console
[2] More Groovy Script for Jenkins on Github


17 Comments

Groovy Script for Jenkins Script Console to Add or Modify Discard Old Builds Setting

When your Jenkins server runs on for a while, your hard disk usage grows up because the default setting of each job is that every job build and build artifact obtained from this are stored in a build history.  Limiting this build history can save hard disk usage. Each job has a setting option called Discard Old Builds. In this setting you can set how long (in days) old builds keep in the build history (criteria 1) or how many old builds keep at most in the build history (criteria 2).  Jenkins matches first criteria 1 and then criteria 2. Beyond that there exists advanced setting options. This advanced setting options can configured how long and how many builds with build artifacts have to keep in the build history. After that Jenkins deletes the build artifact, but the logs, history, reports, etc for the build will be kept. These will be kept so as long as they match the criteria in the normal setting options.

When you have many jobs, you don’t want to configure every job manually. For that you can modify the Discard Old Builds setting in each job over a Groovy script at one go. Jenkins has a so-called Script Console [1].  In this console you have to put the following Groovy script and every job is modified to discard its old builds.

def daysToKeep = 28
def numToKeep = 10
def artifactDaysToKeep = -1
def artifactNumToKeep = -1

Jenkins.instance.items.each { item ->
  println("=====================")
  println("JOB: " + item.name)
  println("Job type: " + item.getClass())

  if(item.buildDiscarder == null) {
    println("No BuildDiscarder")
    println("Set BuildDiscarder to LogRotator")
  } else {
    println("BuildDiscarder: " + item.buildDiscarder.getClass())
    println("Found setting: " + "days to keep=" + item.buildDiscarder.daysToKeepStr + "; num to keep=" + item.buildDiscarder.numToKeepStr + "; artifact day to keep=" + item.buildDiscarder.artifactDaysToKeepStr + "; artifact num to keep=" + item.buildDiscarder.artifactNumToKeepStr)
    println("Set new setting")
  }

  item.buildDiscarder = new hudson.tasks.LogRotator(daysToKeep,numToKeep, artifactDaysToKeep, artifactNumToKeep)
  item.save()
  println("")

}

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

In my last posts ([2], [3]) I showed two other use cases for the Jenkins Script Console. All Groovy scripts can be found in GitHub [4]

Links

[1] Jenkins Script Console
[2] Post about how to rename Subversion host name in every job
[3] Post about how to add or modify Subversion repository browser in every job
[4] GitHub repository with several Groovy scripts for Jenkins script console


1 Comment

Groovy Script for Jenkins Script Console to Add or Replace Subversion Repository Browser

In my last post I showed how to rename the host name for Subversion in all jobs with one script. In this post I will show you how to set the repository browser in all jobs with one script. This script is based on the script from my last post.

def newRepositoryBrowserRootUrl = new URL("http://root.url.to.your.sventon.instance")
def newRepositoryInstance = "repository-instance-name"
def newRepositoryBrowser = new  hudson.scm.browsers.Sventon2(newRepositoryBrowserRootUrl, newRepositoryInstance)

Hudson.instance.items.each { item ->

    if(item.scm instanceof hudson.scm.SubversionSCM) {
        println("JOB: " + item.name)

        def newScm = new hudson.scm.SubversionSCM(Arrays.asList(item.scm.locations), item.scm.workspaceUpdater,
            newRepositoryBrowser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
            item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog)

        item.scm = newScm
        item.save()

        println("New Repository Browser: " +  item.scm.browser.class)
        println("\n=================\n")

    }
}

As aforementioned the above Groovy script uses Sventon 2.x as Subversion repository browser. However, Jenkins supports more Subversion repository browsers originally like

  • Assembla
  • CollabNetSVN
  • FishEyeSVN
  • SVNWeb
  • Sventon 1.x
  • ViewSVN
  • WebSVN

Jenkins supports other Subversion repository browsers by plugins like

  • Polarion WebClient  for Subversion
  • WebSVN 2.x

If you want to use an another Subversion repository browser, you have to change the first three lines:

def newRepositoryBrowserRootUrl = new URL("http://root.url.to.your.sventon.instance")
def newRepositoryInstance = "repository-instance-name"
def newRepositoryBrowser = new  hudson.scm.browsers.Sventon2(newRepositoryBrowserRootUrl, newRepositoryInstance)

For example, if you want to use SVNWeb as Subversion repository browser, you have to add following lines instead of the aforementioned lines:

def newRepositoryBrowserUrl = new URL("http://root.url.to.your.svn")
def newRepositoryBrowser = new hudson.scm.browsers.SVNWeb(newRepositoryBrowserUrl)

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

Links

  1. Blog Post – Groovy Script for Jenkins Script Console to Rename the Subversion Host Name
  2. Overview about supported Subversion repository browser by Jenkins
  3. Polarion Plugin
  4. WebSVN2 Plugin


4 Comments

Groovy Script for Jenkins Script Console to Rename the Subversion Host Name

The host name of a Subversion instance is renamed and now many Jenkins jobs have to be adjusted. One possibility is to change every job by hand. But this approach is very time-consuming and error-prone. A better possibility is to have a script based approach, that rename the Subversion host name of all jobs in one go. Jenkins has a feature, that helps us thereby. Jenkins has a script console to run Groovy script on Jenkins server [1].

Below you can see a Groovy script  that renames the Subversion host name in all jobs.


def oldHostName = "old.hostname.com"
def newHostName = "new.hostname.com"

Hudson.instance.items.each { item ->

  if(item.scm instanceof hudson.scm.SubversionSCM) {
    println("JOB : "+item.name)

    def newLocations = new ArrayList<hudson.scm.SubversionSCM.ModuleLocation>()

    item.scm.locations.each {location ->

      println("SCM Location Remote : " + location.remote)
      def newRemoteUrl = location.remote.replaceFirst(oldHostName, newHostName)

      newLocations.add(new hudson.scm.SubversionSCM.ModuleLocation(newRemoteUrl, location.local, location.depthOption,location.ignoreExternalsOption))
    }

    def newScm = new hudson.scm.SubversionSCM(newLocations, item.scm.workspaceUpdater,
    item.scm.browser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
    item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog)

    newScm.locations.each { location ->
      println("New SCM Location Remote : " + location.remote)
    }

    item.scm = newScm
    item.save()
    println("\n=======\n")
  }
}

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

Links

[1] Jenkins Script Console