SKM IT World

Just another blog about IT


Leave a comment

Continuous Integration Infrastructure With Windows – Scripting With PowerShell

In one of my current project, I deal with how to run a Continuous Integration (CI) infrastructure on Windows machines. I have had experience in running a CI infrastructure for five years, but it was always on Linux machines. So in the next months I will write some blog posts about my challenge with Windows machines from the perspective of a Linux fan girl :-). This first blog post is about shell scripting on Windows. But bear in mind: This blog post isn’t a tutorial for PowerShell scripting. It only explains striking feature coming from Linux background.

When you run a CI infrastructure, it’s a frequent practice to write little shell scripts to automate repeatable tasks. On a Linux system you would write your scripts in Bash or in a script language like Perl or Phyton. I usually write my script in Bash or in Groovy. I choose Groovy, because I’m a Java Developer and it is possible writing Groovy script in Java-style in the beginning and the second argument for me was, that the administration of Jenkins is easier with Groovy scripts. Jenkins supports a Groovy console for administration tasks and build job’s step also can be automated with Groovy in Jenkins, directly. So I use Groovy for other automated tasks to not use so many script languages at the same time. Now you can say, ok, what’s the problem. It is able to use Groovy on Windows system. The problem is the requirements in my project. It is only allowed to use Java, C# or PowerShell as programming language. But I want to write little scripts, so from this point of view, only PowerShell remains.

Switch from Bash to PowerShell – What is striking

Good news at first, working with PowerShell isn’t as creepy as working with DOS shell. But it’s different compared to Bash shell. In next section, I will report what I notice when I start writing Powershell scripts with a Bash shell background.

First at all, it helps all lot for writing script in PowerShell from the Linux’s point of view when you understand the difference between how Linux and Windows handle their configuration. In Linux, you only have to adjust some text files to change system’s configuration. Here, the configuration is text driven and shells in Linux are optimized to handle text. On the other hand in Windows, you use an API to adjust properties in objects to change system’s configuration. Here, the configuration is API driven. The next important point is that Microsoft provides a large class library, the .NET framework, that has object model of Windows’ system configuration. PowerShell reuses this object model as the base for type system. So scripting in PowerShell feels more like object-oriented programming. Fortunately, we can reuse all functionality of the .NET framework in our PowerShell scripts. So if you’re familiar with C# programming, the start with PowerShell scripting is very easy for you. So writing scripts for PowerShell feels like working with an OOP language.

So let’s look at some code sample for typical situations to see the difference between scripting for Bash and for PowerShell.

Writing Something to the Standard Output Stream

On the Bash side, you have the built-in command echo for that:

echo "Hello World"

For PowerShell, you have a so-called Cmdlet Write-Output:

Write-Output "Hello World"

Now, we want to write the value of a variable to standard output.

Bash

message="Hello World"
echo $message

PowerShell

 $message="Hello World" Write-Output $message 

That was easy, wasn’t it

Parsing Files for a Pattern

In our example, we want to parse only XML files after a specific pattern (in our case “search-pattern”) and count how often this pattern is match in all.

On the Bash side, we use for Linux typically pipeline pattern. First, we use grep for searching the pattern and then we pipe the result of grep to wc to count the matches.

grep -w *.xml -e "search-pattern" | wc -l

On PowerShell, it looks little bit different. First, we have to list all XML files with dir. This result is piped to the Cmdlet ForEach-Object. This Cmdlet gets a script block. In our case, the script block reads the content of a file and pipe it to Select-String Cmdlet. This is responsible for filtering after the given pattern and this filter result is piped to Measure-Object that can calculate the numeric properties of object. In our case, it should only count the matches. At the end, every count has to be added together. The important thing is that the result of every Cmdlet is an object that has properties.

$sum = 0
dir *.xml | ForEach-Object {
$sum += (Get-Content $_ | Select-String -Pattern 'search-pattern' | Measure-Object).Count
}
Write-Output $sum

Conditions

Conditions in Bash and in PowerShell look very similar . The only difference in my opinion is the using of bracket. In PowerShell, conditions look more like in C#.

Bash

if [ true -a true -o false -a 2 -gt 1 -a 1 -eq 1 -a 1 -lt 3 -a !false ];
then
  # do something if the condition is true
else
  # do something if the condition is false
fi

PowerShell

if ( 1 -and ( 1 -or 0) -and (2 -gt 1) -and ( 1 -eq 1) -and (1 -lt 3) -and (-not 0)) {
  # do something if the condition is true
} else {
  # do something if the condition is false
}

Setting System Environment Variable

In both systems, Linux and Window, it is possible to set system environment variable on different context, system-wide, process or user based.

Bash

Setting system environment variable with Bash works like the following line:


export BASH_EXAMPLE="some value"

This variable is active during the current process. If you want that the variable is active system-wide, you have to edit the file /etc/profile or you create executable shell scripts in the directory /etc/profile.d. If the variable should be active only for special user session, you have to edit the file ~/.bash_profile . You can do this programmatically with sed or awk.

Using the variable works like that


echo "$BASH_EXAMPLE"

PowerShell

In the PowerShell, you can call the .NET Framework API for setting the system environment variable.


[Environment]::SetEnvironmentVariable("WindowsExample", "Some value.", "Machine")

If the variable should be active only for special user or process, write User or Process instead of Machine. For process level there also exists another command:

$env:WindowsExample = "Some value."

The .Net Framework has a GetEnviromentVariable method for calling the variable.

[Environment]::GetEnvironmentVariable("WindowsExample", "Machine")
$Env:WindowsExample // shortcut

When setting system environment variable, you can see the big different between the both approach for configuration, text-driven (Linux) and API-driven (Windows).

Tool Support

I was pleasantly surprised when I figured out, that a PowerShell “IDE” exists, called PowerShell Integrated Scripting Environment (ISE). It supports you with code completion and has a complete documentation about the Cmdlets.
powershell-ise

PowerShell Integration in Jenkins

One use case for writing scripts is running them in a build job in Jenkins. Jenkins can’t run PowerShell scripts out of the box. Fortunately, it exists a Jenkins Plugin for that called PowerShell Plugin. After installing this plugin a build step Windows PowerShell is avaible in the job configuration. This step has a command field where you can add your script code directly, or you add the path to your script (see below example). In the second variant it is important to add exit $LASTEXITCODE otherwise the build doesn’t recognize that the PowerShell script failed.

my-first-script.ps1
exit $LASTEXITCODE

Further Information

  1. Manning’s book PowerShell in Action
  2. Jenkins’ PowerShell Plugin
  3. Microsoft TechNet about System Environment Variable.
  4. nixCraft’s blog post Linux: Set Environment Variable


1 Comment

Migration Path for Jenkins Subversion Plugin from Version 1.53 to 2.2

Introduction

Jenkins Subversion Plugin changes its credential management. Before version 2.2 you had to set your Subversion credential globally depend on domain name. In version 2.2 it’s changed. The credential management is still central but now you have to configure in every job which credential the job has to use for Subversion authentication .

If you have many jobs, you don’t want to touch every job to change the Subversion configuration. A Groovy script for Jenkins Script Console [1] can help. In the next section I will describe how the migration path looks like.

Migration Path

  1. Update the Subversion Plugin in your Jenkins instance to version 2.2.
  2. Add your Subversion credential to the global credential store:
    1. Go to Jenkins -> Credential -> Global credentials -> Add credential
    2. Add your credential. The scope has to be global.
  3. Go to the installation path of your Jenkins instance and open the file credential.xml.
  4. Search in credential.xml for the element <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl> that contains your above set credential and copy the value of the element <id> .
    Example:

    <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
      <scope>GLOBAL</scope>
      <id>63c3793c-e3fb-49eb-b45b-f6f8e7364876</id>
      <description></description>
      <username>jenkinssvnuser</username>
      HyDUenzpyDkbL9xMoQ0pxdK10l20VkXKEiy4+ZnjL9c=
    </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
    
  5. Go to the Jenkins Script Console and run following Groovy script. Assign your credential id to the variable credentialId before running the script.
    def credentialId = '63c3793c-e3fb-49eb-b45b-f6f8e7364876'
    
    Hudson.instance.items.each { item ->
    
    if(item.scm instanceof hudson.scm.SubversionSCM) {
      println("JOB : "+item.name)
    
      def newLocations = new ArrayList()
    
      item.scm.locations.each {location ->
      newLocations.add(new hudson.scm.SubversionSCM.ModuleLocation(location.remote, credentialId, location.local, location.depthOption,location.ignoreExternalsOption))
    }
    
    def newScm = new hudson.scm.SubversionSCM(newLocations, item.scm.workspaceUpdater,
    item.scm.browser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
    item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog, item.scm.additionalCredentials)
    
    item.scm = newScm
    item.save()
    println("\n=======\n")
    }
    
  6. Finish.

This script is tested with Jenkins version 1.555 and Jenkins Subversion Plugin version 2.2. On Github [2] you can find more Groovy script for Jenkins Script Console.

Links

[1] Jenkins Script Console
[2] More Groovy Script for Jenkins on Github


15 Comments

Groovy Script for Jenkins Script Console to Add or Modify Discard Old Builds Setting

When your Jenkins server runs on for a while, your hard disk usage grows up because the default setting of each job is that every job build and build artifact obtained from this are stored in a build history.  Limiting this build history can save hard disk usage. Each job has a setting option called Discard Old Builds. In this setting you can set how long (in days) old builds keep in the build history (criteria 1) or how many old builds keep at most in the build history (criteria 2).  Jenkins matches first criteria 1 and then criteria 2. Beyond that there exists advanced setting options. This advanced setting options can configured how long and how many builds with build artifacts have to keep in the build history. After that Jenkins deletes the build artifact, but the logs, history, reports, etc for the build will be kept. These will be kept so as long as they match the criteria in the normal setting options.

When you have many jobs, you don’t want to configure every job manually. For that you can modify the Discard Old Builds setting in each job over a Groovy script at one go. Jenkins has a so-called Script Console [1].  In this console you have to put the following Groovy script and every job is modified to discard its old builds.

def daysToKeep = 28
def numToKeep = 10
def artifactDaysToKeep = -1
def artifactNumToKeep = -1

Jenkins.instance.items.each { item ->
  println("=====================")
  println("JOB: " + item.name)
  println("Job type: " + item.getClass())

  if(item.buildDiscarder == null) {
    println("No BuildDiscarder")
    println("Set BuildDiscarder to LogRotator")
  } else {
    println("BuildDiscarder: " + item.buildDiscarder.getClass())
    println("Found setting: " + "days to keep=" + item.buildDiscarder.daysToKeepStr + "; num to keep=" + item.buildDiscarder.numToKeepStr + "; artifact day to keep=" + item.buildDiscarder.artifactDaysToKeepStr + "; artifact num to keep=" + item.buildDiscarder.artifactNumToKeepStr)
    println("Set new setting")
  }

  item.buildDiscarder = new hudson.tasks.LogRotator(daysToKeep,numToKeep, artifactDaysToKeep, artifactNumToKeep)
  item.save()
  println("")

}

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

In my last posts ([2], [3]) I showed two other use cases for the Jenkins Script Console. All Groovy scripts can be found in GitHub [4]

Links

[1] Jenkins Script Console
[2] Post about how to rename Subversion host name in every job
[3] Post about how to add or modify Subversion repository browser in every job
[4] GitHub repository with several Groovy scripts for Jenkins script console


1 Comment

Groovy Script for Jenkins Script Console to Add or Replace Subversion Repository Browser

In my last post I showed how to rename the host name for Subversion in all jobs with one script. In this post I will show you how to set the repository browser in all jobs with one script. This script is based on the script from my last post.

def newRepositoryBrowserRootUrl = new URL("http://root.url.to.your.sventon.instance")
def newRepositoryInstance = "repository-instance-name"
def newRepositoryBrowser = new  hudson.scm.browsers.Sventon2(newRepositoryBrowserRootUrl, newRepositoryInstance)

Hudson.instance.items.each { item ->

    if(item.scm instanceof hudson.scm.SubversionSCM) {
        println("JOB: " + item.name)

        def newScm = new hudson.scm.SubversionSCM(Arrays.asList(item.scm.locations), item.scm.workspaceUpdater,
            newRepositoryBrowser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
            item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog)

        item.scm = newScm
        item.save()

        println("New Repository Browser: " +  item.scm.browser.class)
        println("\n=================\n")

    }
}

As aforementioned the above Groovy script uses Sventon 2.x as Subversion repository browser. However, Jenkins supports more Subversion repository browsers originally like

  • Assembla
  • CollabNetSVN
  • FishEyeSVN
  • SVNWeb
  • Sventon 1.x
  • ViewSVN
  • WebSVN

Jenkins supports other Subversion repository browsers by plugins like

  • Polarion WebClient  for Subversion
  • WebSVN 2.x

If you want to use an another Subversion repository browser, you have to change the first three lines:

def newRepositoryBrowserRootUrl = new URL("http://root.url.to.your.sventon.instance")
def newRepositoryInstance = "repository-instance-name"
def newRepositoryBrowser = new  hudson.scm.browsers.Sventon2(newRepositoryBrowserRootUrl, newRepositoryInstance)

For example, if you want to use SVNWeb as Subversion repository browser, you have to add following lines instead of the aforementioned lines:

def newRepositoryBrowserUrl = new URL("http://root.url.to.your.svn")
def newRepositoryBrowser = new hudson.scm.browsers.SVNWeb(newRepositoryBrowserUrl)

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

Links

  1. Blog Post – Groovy Script for Jenkins Script Console to Rename the Subversion Host Name
  2. Overview about supported Subversion repository browser by Jenkins
  3. Polarion Plugin
  4. WebSVN2 Plugin


4 Comments

Groovy Script for Jenkins Script Console to Rename the Subversion Host Name

The host name of a Subversion instance is renamed and now many Jenkins jobs have to be adjusted. One possibility is to change every job by hand. But this approach is very time-consuming and error-prone. A better possibility is to have a script based approach, that rename the Subversion host name of all jobs in one go. Jenkins has a feature, that helps us thereby. Jenkins has a script console to run Groovy script on Jenkins server [1].

Below you can see a Groovy script  that renames the Subversion host name in all jobs.


def oldHostName = "old.hostname.com"
def newHostName = "new.hostname.com"

Hudson.instance.items.each { item ->

  if(item.scm instanceof hudson.scm.SubversionSCM) {
    println("JOB : "+item.name)

    def newLocations = new ArrayList<hudson.scm.SubversionSCM.ModuleLocation>()

    item.scm.locations.each {location ->

      println("SCM Location Remote : " + location.remote)
      def newRemoteUrl = location.remote.replaceFirst(oldHostName, newHostName)

      newLocations.add(new hudson.scm.SubversionSCM.ModuleLocation(newRemoteUrl, location.local, location.depthOption,location.ignoreExternalsOption))
    }

    def newScm = new hudson.scm.SubversionSCM(newLocations, item.scm.workspaceUpdater,
    item.scm.browser, item.scm.excludedRegions, item.scm.excludedUsers, item.scm.excludedRevprop, item.scm.excludedCommitMessages,
    item.scm.includedRegions, item.scm.ignoreDirPropChanges, item.scm.filterChangelog)

    newScm.locations.each { location ->
      println("New SCM Location Remote : " + location.remote)
    }

    item.scm = newScm
    item.save()
    println("\n=======\n")
  }
}

This script is tested with Jenkins version 1.534 and Jenkins Subversion Plugin version 1.53.

Links

[1] Jenkins Script Console


Leave a comment

SaxException During a Sucessful Maven Build in Jenkins

During the inspection of  several  sucessful Maven 3 build output, following SaxException catchs my eye:

[INFO] Parsing file:/home/skosmalla/.m2/repository/repository.xml
[Fatal Error] repository.xml:3:1: Premature end of file.
org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:249)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:208)
at org.apache.felix.obrplugin.ObrUpdate.parseFile(ObrUpdate.java:347)
at org.apache.felix.obrplugin.ObrUpdate.parseRepositoryXml(ObrUpdate.java:324)
at org.apache.felix.obrplugin.ObrInstall.execute(ObrInstall.java:140)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:158)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:122)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:74)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I didn’t find a solution with google for this problem. But I found a hint that the repository.xml  is generated by Maven at the  end of  a build.  I noticed that only the Maven builds had this SaxException that were built parallelly (Jenkins had two build processors)  and all Maven builds used the same M2 repository.  So I had the idea that the reason of this exception could be that two almost finished Maven build tried to generate the repository.xml.  I actived in every job the option Use private Maven repository and the SaxException didn’t turn up anymore.