SKM IT World

Just another blog about IT


Leave a comment

Running Ansible on a Windows System

On my last conference talk (it was about Ansible and Docker at DevOpsCon in Berlin), I was asked what is the best way to run Ansible on a Windows system. Ansible itself requires a Linux-based system as the control machine. When I have to develop on a Windows machine, I install a Linux-based virtual machine to run the Ansible’s playbooks inside the virtual machine. I set up the virtual machine with Virtualbox and Vagrant. This tools allow me to share the playbooks easily between host and the virtual machine. so I can develop the playbook on the windows system and the virtual machine can have a headless setup. The next section shows you how to set up this tool chain.

 Tool Chain Setup

 At first, install VirtualBox and Vagrant on your machine. I additionally use Babun, a windows shell based on Cygwin and oh-my-zsh, for a better shell experience on Windows, but this isn’t necessary. Then, go to the directory (let’s called it ansible-workspace), where your Ansible’s playbooks are located. Create there a Vagrant configuration file with the command vagrant init:
ansible-workspace
├── inventories
│   ├── production
│   └── test
├── README.md
├── roles
│   ├── deploy-on-tomcat
│   │   ├── defaults
│   │   │   └── main.yml
│   │   └── tasks
│   │       ├── cleanup-webapp.yml
│   │       ├── deploy-webapp.yml
│   │       ├── main.yml
│   │       ├── start-tomcat.yml
│   │       └── stop-tomcat.yml
│   ├── jdk
│   │   └── tasks
│   │       └── main.yml
│   └── tomcat8
│       ├── defaults
│       │   └── main.yml
│       ├── files
│       │   └── init.d
│       │       └── tomcat
│       ├── tasks
│       │   └── main.yml
│       └── templates
│           └── setenv.sh.j2
├── demo-app-ansible-deploy-1.0-SNAPSHOT.war
├── deploy-demo.yml
├── inventories
│   ├── production
│   └── test
├── roles
│   ├── deploy-on-tomcat
│   │   ├── defaults
│   │   │   └── main.yml
│   │   └── tasks
│   │       ├── cleanup-webapp.yml
│   │       ├── deploy-webapp.yml
│   │       ├── main.yml
│   │       ├── start-tomcat.yml
│   │       └── stop-tomcat.yml
│   ├── jdk
│   │   └── tasks
│   │       └── main.yml
│   └── tomcat8
│       ├── defaults
│       │   └── main.yml
│       ├── files
│       │   └── init.d
│       │       └── tomcat
│       ├── tasks
│       │   └── main.yml
│       └── templates
│           └── setenv.sh.j2
├── setup-app-roles.yml
├── setup-app.yml
└── Vagrantfile

├── setup-app-roles.yml
├── setup-app.yml
└── Vagrantfile


Now, we have to choose a so-called Vagrant Box on Vagrant Cloud. A box is the package format for a Vagrant environment. It depends on the provider and the operation system that you choose to use. In our case, it is a Virtualbox VM image based on a minimal Ubuntu 18.04 system (box name is bento/ubuntu-18.04 ). This box will be configured in our Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-18.04"
end

The next step is to ensure that Ansible will be installed in the box. Thus, we use the shell provisioner of Vagrant. The Vagranfile will be extended by the provisioning code:

Vagrant.configure("2") do |config|
  # ... other Vagrant configuration
  config.vm.provision "shell", inline: <<-SHELL
    sudo apt-get update -y
    sudo apt-get install -y software-properties-common
    sudo apt-add-repository ppa:ansible/ansible
    sudo apt-get update -y
    sudo apt-get install -y ansible
    # ... other Vagrant provision steps
  SHELL
end
end

The last step is to copy the SSH credential into the Vagrant box. Thus, we mark the SSH credential folder of the host system as a Shared folder, so that we can copy them to the SSH config folder inside the box.
Vagrant.configure("2") do |config|
 
  # ... other Vagrant configuration
  config.vm.synced_folder ".", "/vagrant"
  config.vm.synced_folder "path to your ssh config", "/home/vagrant/ssh-host"
  # ... other Vagrant configuration

  config.vm.provision "shell", inline: <<-SHELL
    # ... other Vagrant provision steps
    cp /home/vagrant/ssh-host/* /home/vagrant/.ssh/.
  SHELL
end

On Github’s Gist you can found the whole Vagrantfile.

Workflow

After setting up the tool chain let’s have a look how to work with it. I write my Ansible playbook on the Windows system and run them from the Linux guest system against the remote hosts. For running the Ansible playbooks we have to start the Vagrant box.
> cd ansible-workspace
> vagrant up

When the Vagrant box is ready to use, we can jump into the box with:
 
> vagrant ssh 

You can find the Ansible playbooks inside the box in the folder /vagrant .  In this folder run Ansible:
 
> cd" /vagrant
> ansible-playbook -i inventories/test -u tekkie setup-db.yml

Outlook

Maybe on Windows 10 it’s possible to use Ansible natively, because of the Linux subsystem. But I don’t try it out. Some Docker fans would prefer a container instead of a virtual machine. But remember, before Windows 10 Docker runs on Windows in a virtual machine, so therefore, I don’t see a benefit for using Docker instead of a virtual machine. But of course with Windows 10 native container support a setup with Docker is a good alternative if Ansible doesn’t run on the Linux subsystem.
Do you another idea or approach? Let me know and write a comment.

Links

  1. VirtualBox
  2. Vagrant
  3. Whole Vagrantfile on Github.

 

Advertisements


Leave a comment

Pimp My Git – Git Mergetool

I like to work with git on the command line. But in some cases I prefer UI support. For example, solving merge conflicts is such a case. Git has a command mergetool, which can open a graphical tool to solve merge conflicts. But before you can use this command, you had to configure it. In this blog post I’d like to show you how to configure mergetool and how to use it.

Configuration

First at all, open a shell on Linux. On Windows, open Git Bash. Then choose a graphic tool that should support you solving merge conflicts. git mergetool –tool-help shows a list which tools are supported on your machine


 sparsick@sparsick-ThinkPad-T430s > git mergetool --tool-help
'git mergetool --tool=<tool>' may be set to one of the following:
                araxis
                kdiff3
                meld

The following tools are valid, but not currently available:
                bc
                bc3
                codecompare
                deltawalker
                diffmerge
                diffuse
                ecmerge
                emerge
                gvimdiff
                gvimdiff2
                gvimdiff3
                opendiff
                p4merge
                tkdiff
                tortoisemerge
                vimdiff
                vimdiff2
                vimdiff3
                winmerge
                xxdiff

Some of the tools listed above only work in a windowed
environment. If run in a terminal-only session, they will fail.

This command shows two lists. The first list shows all tools that are supported by git and that are available on your machine (in sample, it is araxis, kdiff3 and meld). The second list shows that are also supported by git, but they aren’t install on your machine.

I use meld as graphical tool. It’s runnable on Windows and Linux. If you haven’t install meld on your machine, then it’s now the right time to do it or choose another tool.

We want to set mergetool globally for all our repositories.


sparsick@sparsick-ThinkPad-T430s > git config --global merge.tool meld
sparsick@sparsick-ThinkPad-T430s > git mergetool
No files need merging

If git mergetool returns more than No files need merging, then the path to your graphic tool isn’t set in your $PATH system variable (The normal case on Windows systems). It’s possible to set the path to the graphical tool directly in git.

sparsick@sparsick-ThinkPad-T430s > git config --global mergetool.meld.path /c/Program\ Files\ \(x86\)/Meld/Meld.exe</pre>

Bear two important things in mind: mergetool is written without a dot between merge and tool and meld is a placeholder for the name of the graphical tool in the above sample. If you use another tool such like vimdiff, then the config key is called mergetool.vimdiff.path .

Now git mergetool is ready to use.

Usage

Now I’d like to demonstrate how to use git mergetool. It is used in when we have merge conflicts during a merge action. Let’s say we want to merge branch branch1 into master and this merge will have some merge conflicts.


sparsick@sparsick-ThinkPad-T430s > git merge branch1

Auto-merging test
CONFLICT (content): Merge conflict in test
Automatic merge failed; fix conflicts and then commit the result.

Now, we want to solve these conflicts with a graphical tool (in the example, it’s meld). git mergetool on the command line open the graphical tool of our choice.

sparsick@sparsick-ThinkPad-T430s > git mergetool

Merging:
test

Normal merge conflict for 'test':
{local}: modified file
{remote}: modified file

After solving the merge conflicts, the change has to commit.

sparsick@sparsick-ThinkPad-T430s > git status

On branch master
All conflicts fixed but you are still merging.
(use "git commit" to conclude merge)

Changes to be committed:

modified:   test

Untracked files:
(use "git add <file>..." to include in what will be committed)

test.orig
sparsick@sparsick-ThinkPad-T430s > git commit

You can see that we have a new untracked file test.orig . This is a backup of the merged file created by mergetool. You can configure that this backup should be removed after a successful merge.

sparsick@sparsick-ThinkPad-T430s > git config --global mergetool.keepBackup false

Further files are created when using git mergetool:

sparsick@sparsick-ThinkPad-T430s > git status

On branch master
Untracked files:
(use "git add ..." to include in what will be committed)

test.BACKUP.7344
test.BASE.7344
test.LOCAL.7344
test.REMOTE.7344

If only these files are currently untracked, then a git clean can help. Otherwise they have to be removed manually.

sparsick@sparsick-ThinkPad-T430s > git clean -f

Removing test.BACKUP.7344
Removing test.BASE.7344
Removing test.LOCAL.7344
Removing test.REMOTE.734

Links

  1. Meld Homepage
  2. git mergetool Documentation


Leave a comment

Automatic Tomcat 8.5 Installation and Configuration as Windows Service

If you want to install Tomcat on Windows system as a service, you’ll get the recommendation to use the 32-/64-Bit Windows Service Installer. If you want to install the Tomcat manually, it’s fine. But you can’t use this installer for an automatic installation and configuration of Tomcat, because the installer is UI-based. The next sections explain how you can install and configure Tomcat on a CMD.

Tomcat Installation

  1. Download from the Apache Tomcat 8.5 download page the Core 64-bit Windows zip (or the 32-Bit zip).
  2. Unzip it (for example to C:\tomcat\)

That’s it. Now we have a ready-to-use Tomcat with default configuration values. But it isn’t install as a service.

Installation and Configuration As Windows Service

  1. Go to the bin folder in the installation folder of Tomcat (in the example  it’s C:\tomcat\apache-tomcat-8.5.11\bin)
  2. Install Tomcat as service named tomcat8 by calling service.bat install <servicename>
    C:\tomcat\apache-tomcat-8.5.11\bin>service.bat install tomcat8
    
  3.  tomcat8.exe //US//<servicename> followed by configuration parameter configures the Tomcat service. For example:
    C:\tomcat\apache-tomcat-8.5.11\bin>tomcat8.exe //US//tomcat8 --Startup=auto --JavaHome="C:\Program Files\Java\jre1.8.0_112" --JvmMs=2048 --JvmMx=4096 ++JvmOptions=-Dkey=value
    
  4. Start the Tomcat service with net start <servicename>
    net start tomcat8
    
  5. You can check on http://localhost:8080 whether Tomcat is installed correctly.

The configuration example (step 3) shows how to configure the JVM (heap space, Java option etc.), where Java is installed and which start type should use for the service. The full list of the possible configuration parameter for the Tomcat service can be found in the Apache Tomcat’s Windows Service documentation.

Now we have everything together for writing a Powershell script that does these steps automatically.

 


Leave a comment

Summary of SoCraTes 2016 Session “Hey dude, where is my tool chain?” – Working on Windows as a Linux User aka Let’s talk about Windows

This year on the conference SoCraTes I hosted a session for the first time. It was about working on a Windows system from the perspective of a Linux user.  A big thank to @ndrssmn, who motivated to host this session.

@yooogan was so nice to summarize the session in the SoCraTes wiki (big thank for that).  But the wiki page is only accessible for SoCraTes participants, so we decided that I republish it on my blog. Enjoy it.


Console/Shell

  • Babun – Based on Cygwin, includes a CLI package manager (pact – like apt, yum, …) and preconfigured oh-my-zsh as shell
  • ConEmu – feature rich console emulator with tabs
  • Console2 (original)/(modified fork) – console emulator, multi tabs, configurable mouse behavior
  • PuTTYssh client (when you don’t have Babun/Cygwin anyway)

File Management

Text Editors

  • Notepad++ – all-purpose editor, syntax highlighting, file monitoring (tail -f)
  • Atom – editor; same settings in all your environments

Diff/Merge

Searching

Viewers

Documentation

  • Zim – Organize notes, saves to plain text
  • Greenshot – Screenshots, including obfuscation / comments, for documentation, connects to JIRA
  • yEd – multi-platform (Win/Linux/MacOS) graph editor, extensible palette, useful pictograms
  • Paint.NET – free image editor
  • GIMP – Swiss army knife for images

System

Disk Usage

  • RidNacs – graphical du
  • WinDirStat – even more colorful graphical du
  • ncdu – CLI, can be installed from Cygwin/Babun

Keyboard

  • WinCompose – a (X.org/X11 like) compose key for Windows – type äöë߀«»←↑↓→¡☺♥… like a boss!
  • SharpKeys – remap keyboard: CapsLockCtrl, ~Esc, etc.
  • AutoHotkey – very sophisticated keyboard macros / automation – full-fledged scripting language

Clipboard

  • PureText – remove formatting from pasted text
  • Ditto – clipboard manager

Pictures (taken from Twitter):

Let's talk about Windows, pt. 1
Let's talk about Windows, pt. 2
Let's talk about Windows, pt. 3

 


Leave a comment

Continuous Integration Infrastructure With Windows – Scripting With PowerShell

In one of my current project, I deal with how to run a Continuous Integration (CI) infrastructure on Windows machines. I have had experience in running a CI infrastructure for five years, but it was always on Linux machines. So in the next months I will write some blog posts about my challenge with Windows machines from the perspective of a Linux fan girl :-). This first blog post is about shell scripting on Windows. But bear in mind: This blog post isn’t a tutorial for PowerShell scripting. It only explains striking feature coming from Linux background.

When you run a CI infrastructure, it’s a frequent practice to write little shell scripts to automate repeatable tasks. On a Linux system you would write your scripts in Bash or in a script language like Perl or Phyton. I usually write my script in Bash or in Groovy. I choose Groovy, because I’m a Java Developer and it is possible writing Groovy script in Java-style in the beginning and the second argument for me was, that the administration of Jenkins is easier with Groovy scripts. Jenkins supports a Groovy console for administration tasks and build job’s step also can be automated with Groovy in Jenkins, directly. So I use Groovy for other automated tasks to not use so many script languages at the same time. Now you can say, ok, what’s the problem. It is able to use Groovy on Windows system. The problem is the requirements in my project. It is only allowed to use Java, C# or PowerShell as programming language. But I want to write little scripts, so from this point of view, only PowerShell remains.

Switch from Bash to PowerShell – What is striking

Good news at first, working with PowerShell isn’t as creepy as working with DOS shell. But it’s different compared to Bash shell. In next section, I will report what I notice when I start writing Powershell scripts with a Bash shell background.

First at all, it helps all lot for writing script in PowerShell from the Linux’s point of view when you understand the difference between how Linux and Windows handle their configuration. In Linux, you only have to adjust some text files to change system’s configuration. Here, the configuration is text driven and shells in Linux are optimized to handle text. On the other hand in Windows, you use an API to adjust properties in objects to change system’s configuration. Here, the configuration is API driven. The next important point is that Microsoft provides a large class library, the .NET framework, that has object model of Windows’ system configuration. PowerShell reuses this object model as the base for type system. So scripting in PowerShell feels more like object-oriented programming. Fortunately, we can reuse all functionality of the .NET framework in our PowerShell scripts. So if you’re familiar with C# programming, the start with PowerShell scripting is very easy for you. So writing scripts for PowerShell feels like working with an OOP language.

So let’s look at some code sample for typical situations to see the difference between scripting for Bash and for PowerShell.

Writing Something to the Standard Output Stream

On the Bash side, you have the built-in command echo for that:

echo "Hello World"

For PowerShell, you have a so-called Cmdlet Write-Output:

Write-Output "Hello World"

Now, we want to write the value of a variable to standard output.

Bash

message="Hello World"
echo $message

PowerShell

 $message="Hello World" Write-Output $message 

That was easy, wasn’t it

Parsing Files for a Pattern

In our example, we want to parse only XML files after a specific pattern (in our case “search-pattern”) and count how often this pattern is match in all.

On the Bash side, we use for Linux typically pipeline pattern. First, we use grep for searching the pattern and then we pipe the result of grep to wc to count the matches.

grep -w *.xml -e "search-pattern" | wc -l

On PowerShell, it looks little bit different. First, we have to list all XML files with dir. This result is piped to the Cmdlet ForEach-Object. This Cmdlet gets a script block. In our case, the script block reads the content of a file and pipe it to Select-String Cmdlet. This is responsible for filtering after the given pattern and this filter result is piped to Measure-Object that can calculate the numeric properties of object. In our case, it should only count the matches. At the end, every count has to be added together. The important thing is that the result of every Cmdlet is an object that has properties.

$sum = 0
dir *.xml | ForEach-Object {
$sum += (Get-Content $_ | Select-String -Pattern 'search-pattern' | Measure-Object).Count
}
Write-Output $sum

Conditions

Conditions in Bash and in PowerShell look very similar . The only difference in my opinion is the using of bracket. In PowerShell, conditions look more like in C#.

Bash

if [ true -a true -o false -a 2 -gt 1 -a 1 -eq 1 -a 1 -lt 3 -a !false ];
then
  # do something if the condition is true
else
  # do something if the condition is false
fi

PowerShell

if ( 1 -and ( 1 -or 0) -and (2 -gt 1) -and ( 1 -eq 1) -and (1 -lt 3) -and (-not 0)) {
  # do something if the condition is true
} else {
  # do something if the condition is false
}

Setting System Environment Variable

In both systems, Linux and Window, it is possible to set system environment variable on different context, system-wide, process or user based.

Bash

Setting system environment variable with Bash works like the following line:


export BASH_EXAMPLE="some value"

This variable is active during the current process. If you want that the variable is active system-wide, you have to edit the file /etc/profile or you create executable shell scripts in the directory /etc/profile.d. If the variable should be active only for special user session, you have to edit the file ~/.bash_profile . You can do this programmatically with sed or awk.

Using the variable works like that


echo "$BASH_EXAMPLE"

PowerShell

In the PowerShell, you can call the .NET Framework API for setting the system environment variable.


[Environment]::SetEnvironmentVariable("WindowsExample", "Some value.", "Machine")

If the variable should be active only for special user or process, write User or Process instead of Machine. For process level there also exists another command:

$env:WindowsExample = "Some value."

The .Net Framework has a GetEnviromentVariable method for calling the variable.

[Environment]::GetEnvironmentVariable("WindowsExample", "Machine")
$Env:WindowsExample // shortcut

When setting system environment variable, you can see the big different between the both approach for configuration, text-driven (Linux) and API-driven (Windows).

Tool Support

I was pleasantly surprised when I figured out, that a PowerShell “IDE” exists, called PowerShell Integrated Scripting Environment (ISE). It supports you with code completion and has a complete documentation about the Cmdlets.
powershell-ise

PowerShell Integration in Jenkins

One use case for writing scripts is running them in a build job in Jenkins. Jenkins can’t run PowerShell scripts out of the box. Fortunately, it exists a Jenkins Plugin for that called PowerShell Plugin. After installing this plugin a build step Windows PowerShell is avaible in the job configuration. This step has a command field where you can add your script code directly, or you add the path to your script (see below example). In the second variant it is important to add exit $LASTEXITCODE otherwise the build doesn’t recognize that the PowerShell script failed.

my-first-script.ps1
exit $LASTEXITCODE

Further Information

  1. Manning’s book PowerShell in Action
  2. Jenkins’ PowerShell Plugin
  3. Microsoft TechNet about System Environment Variable.
  4. nixCraft’s blog post Linux: Set Environment Variable


Leave a comment

Cygwin Embedded In Console2 Under Windows 7

I like to use Cygwin for having a bash shell on a windows machine. In combination with Console2 you have a powerful command-line tool for windows. I’d like to describe how to install Cygwin and Console2 under windows 7 (I think this instruction works on Windows XP, too) and how to embed Cygwin in Console2. I used for Cygwin version 1.7.17-1 32bit and for Console2 version 2.00b148-Beta_32bit.

Cygwin Installation

  1. Download setup.exe.
  2. Double click on setup.exe starts the installation.
  3. Follow the set up instruction. As installation location I use c:\cygwin.
  4. Go to c:\cygwin and run Cygwin.bat for setting up Cygwin user home etc.

Console2 Installation

  1. Download zip file.
  2. Unzip it into installation location.
  3. Start Console2 with a double click on Console2.exe.

How to embed Cygwin in Console2

  1. Open Console2.
  2. Go to Edit -> Settings.
  3. Go to  Tab
  4. In field title insert for example bash. It is the name of the first tab in Console2.
  5. In field Shell you have to insert C:\cygwin\bin\bash.exe –login -i (Don’t forget c:\cygwin is your installation folder).
  6. In field Startup dir you can insert C:\cygwin\home\user.name. This folder is used at every Console2 start.

Links

  1. Cygwin Homepage
  2. Console2 Homepage
  3. Good short feature overview of Cygwin on the Wikipedia page