SKM IT World

Just another blog about IT


Leave a comment

How to Install Serverspec in the Current Version on Ubuntu 14.04 LTS (Trusty)

If you google “serverspec install ubuntu”, you find the information that a package called ruby-serverspec in the standard package repository can be used to install Serverspec on an Ubuntu 14.04 LTS based system. Unfortunately, this package installs an outdated version of Serverspec. The next point is that if you try to install the newest version of Serverspec with gem (that’s the way that it is described on the Serverspec homepage), you will get the following error message:


~> sudo gem install serverspec
ERROR:  Error installing serverspec:
net-ssh requires Ruby version >= 2.0.

 

The problem is, when you install Ruby with sudo apt-get install ruby, the package manager installs Ruby in the version 1.9.1 .

Therefore, the next sections explain how to install Ruby and Serverspec in the newest version on an Ubuntu 14.04 LTS based system. Let’s start with Ruby that is required for Serverspec.

Ruby Installation

The cloud hosting service Brightbox provides Ruby package repositories for several Ubuntu versions and several Ruby version. I chose the repository for Ruby 2.3 packages, so the installation steps are:


~> sudo apt-get install software-properties-common
~> sudo apt-add-repository ppa:brightbox/ruby-ng
~> sudo apt-get update
~> sudo apt-get install ruby2.3
~> ruby --version
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux-gnu]

Serverspec Installation

Now, we can install Serverspec like it is explained on the Serverspec homepage. In my case, I had to install rake separately.

~> sudo gem install serverspec rake

Links

  1. Serverspec Homepage
  2. Brightbox Ruby package repositories for Ubuntu documentation


1 Comment

Installation Cheat Sheet For LivingDoc

We wanted to evaluate the new Confluence plugin LivingDoc as a replacement to Fitnesse in order to execute automated web GUI tests.

  • Confluence 5.7.1 (Non-Cloud version)
  • Inside Confluence HSQL In-Memory DB for evaluation purpose
  • LivingDoc plugin 1.0.0.jar
  • Selenium Webdriver for automated Web Testing of
  • Spring Petclinic running inside a Tomcat 8
  • Java Version SDK 1.7.0.80
  • Maven 3.3.1

The following steps are an extension to the LivingDoc documentation. This documentation is very detailed, but if you struggle around some steps, check the following out. We recommend to use the search function of your browser to find the relevant parts. Additional this is not about Best Practices, but only about getting the setup running fast. Let´s start:

  1. Starting point is the LivingDoc Documentation under https://testit-livingdoc.atlassian.net/wiki > Current Documentation > Getting Started
  2. After following this steps, go to Current Documentation > Confluence plugin
  3. Unfortunately, there is no direct link to the current livingdoc-confluence5-plugin.jar, so go to https://github.com/testIT-LivingDoc/livingdoc-confluence/releases and choose livingdoc-confluence5-plugin-1.0.0.jar  (even there is already a version livingdoc-confluence5-plugin-1.1.0.jar) and download it.
  4. Next is the configuration of the Runner. Please look at the following pic:
    1_2016-03-06 14_20_37-LivingDoc Configuration - Confluence
    We replaced the classpath default value with the path, where you have downloaded the livingdoc-confluence5-plugin-1.0.0.jar on your machine.
  5. Next is Project Management:
    2_2016-03-06 16_37_48-LivingDoc Configuration - Confluence
    There, we choose our above prepared runner. Under classpaths, we copy the path to the jar of our Selenium project inclusive its dependencies. In step 7 below, it will described how this jar is built. This is needed so that the Selenium tests can be executed by LivingDoc.
  6. Next is remote agent: In order to get our scenario running, we got advice from the LivingDoc Developer Team to start a remote agent. You will get the complete remote agent jar under https://github.com/testIT-LivingDoc/livingdoc-core/releases . At that time, we chose livingdoc-remote-agent-1.0.0-complete.jar and downloaded it. Please start it like it is written  in the Livingdoc documentation (Current Documentation > Confluence Plugin > Advanced > Remote Agent).
  7. In order that the Selenium tests are executed by LivingDoc, we need a jar file with all our tests inclusive their dependencies. Therefore, we use Maven Assembly Plugin to build a jar with all dependencies. Below the configuration of the Maven Assembly Plugin (Link to whole POM)
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <executions>
            <execution>
                <goals>
                    <goal>test-jar</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.6</version>
        <executions>
            <execution>
                <id>jar-with-dep</id>
                <goals>
                    <goal>single</goal>
                </goals>
                <phase>package</phase>
                <configuration>
                    <descriptors>
                        <descriptor>src/assembly/src.xml</descriptor>
                    </descriptors>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </execution>
        </executions>
    </plugin>
    

    The descriptor format jar-with-dependencies can be found in the Maven Assembly Plugin site.

Summary

Bringing all together and let the Remote Agent running, we can execute from Confluence our Selenium Test. Now we are ready to rumble.

In the meantime, there is a VirtualBox image with everything inside under LiviningDoc documentation > Showcases. Furthermore, there is a new release 1.1 that supports Confluence 5.9.3. But if you want to install all by yourself, we hope this can save you some time.

4_2016-03-06 15_05_28-PetClinic __ a Spring Framework demonstrationPetClinicOwners - AcceptanceTests

Links

  1. Spring Pet Clinic Project
  2. LivingDoc on GitHub
  3. LivingDoc documentation

 

 

 

 


Leave a comment

My Notes From Conference “Herbstcampus 2015”

In September I visited the conference “Herbstcampus” in Nuremberg. I took notes about some sessions, that I like to share with you. The notes are in German.

SOLIDes Design – Kriterien für objektorientiertes Design by David Tanzer

Solides Design

Wie geht’s? Was geht? Wann hat es Sinn? – Portierung von COBOL-Programmen nach Java by Carsten Siedentop

Cobol

Überzogen – Technische Schulden by Gerrit Beine

Technische Schulden

Follow the Leader – Wie Sie Ihr Team beeinflussen und gestalten können by Sabine Zehnder

Follow the leader0001Follow the leader0002

So sieht’s aus! -Architekturüberblicke: Tipps und Tricks by Stefan Zörner

Architektur0001Architektur0002


1 Comment

How To Debug Groovy Script From Shell

Groovy is a scripting language, so it is possible to run Groovy code without compiling to Java byte code. The necessary condition is that Groovy is installed on your machine. Then, running a Groovy script in a shell looks like the following line.


~>groovy TestScript.groovy

Now, something is wrong with the script, only on a special environment. So you want to debug your Groovy script from the shell. Fortunately, it works for Groovy just like for Java. You only have to export the Java options for debugging.


~>export JAVA_OPTS="-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=y"

Now, we can debug our script running from the shell with our favorite IDE.


~>groovy TestScript.groovy
Listening for transport dt_socket at address: 4000


Leave a comment

Git Resources for Beginner

In this post I’d like to share resources that help me learning and understanding Git.

Links

Do you have some more resources that you can recommend? Let me know it and write a comment.


2 Comments

(Not only) Internet Sources About Writing Software Documentation

I was asked some time ago, which sources I used in order to inform me about writing software documentation. This motivates me to collect all sources, that helps me, in this blog post. Most of the sources are in German.

Links

  • Arc42 Homepage: This site is about the Arc42 template. This template gives you a good framework for your own software documentation. The template is available in English and German. A Spanish version is working in progress.
  • DokChess ( in German): Example for how to write software documentation based on the Arc42 template. This example is used in the book “Softwarearchitekturen dokumentieren und kommunizieren”. The documentation itself describes a chess engine.
  • Arc42 Starschnitt (in German): Again, an example for how to write software documentation based on the Arc42 template. The documentation describes the build tool Gradle. The difference to Dokchess is that this example explains the motivation and goals behind the single chapter of the Arc42 template.
  • Arc42 vs Software Guidebook (in German): Comparison between software documentation after Arc42 template and after Software Guidebook.
  • Kolumne “Architekturen dokumentieren” (in German): Newspaper column about architecture documentation from German Java Magazine.
  • Architektur-Spicker (in German): Cheat Sheet for writing an overview about your software architecture.
  • Agile Modeling: Collection of best practices for effective modeling and documentation in an agile context.

Books

Videos

Do you have some more links, books or videos that you can recommend? Let me know it and write a comment.

 


1 Comment

Commons VFS, SSHJ and JSch in Comparison

Some weeks ago I evaluated some SSH libraries for Java. The main requirements to them are file transferring and file operations on a remote machine. Therefore, it exists a network protocol based on SSH, SSH File Transfer Protocol (or SFTP). So I needed a SSH library that supports SFTP.

A research shows that it exits many SSH libraries for Java. I reduce the number of libraries to three for the comparison. I choose JSch, SSHJ and Apache’s Commons VFS for a deeper look. All of them support SFTP. JSch seems to be the de-facto standard for Java. SSHJ is a newer library. Its goal is to have a clear Java API for SSH. The goal of Commons VFS is to have a clear API for virtual file systems and SFTP is one of the supported protocol. Under the hood it uses JSch for the SFTP protocol. The libraries should cover following requirements:

  • client authentication over password
  • client authentication over public key
  • server authentication
  • upload files from local host over SFTP
  • download files to local host over SFTP
  • file operations on the remote host like move, delete, list all children of a given folder (filtering after type like file or folder) over SFTP
  • execute plain shell commands

Lets have a deeper look how the three libraries cover the requirements.

Client Authentication

All three libraries supports both required authentication methods. SSHJ has the clearest API for authentication (SSHClient.authUserPass(), SSHClient.authUserPublicKey()).


SSHClient sshClient= new SSHClient();
sshClient.connect(host);

// only for public key authentication
sshClient.authPublickey("user", "location to private key file");

// only for password authentication
sshClient.authPassword("user", "password");

In Commons VFS the authentication configuration depends which kind of authentication should be used. For the public key authentication, the private key has to set in the FileSystemOption and the user name is a part of the connection url. For the password authentication, user name and password is a part of the connection url.


StandardFileSystemManager fileSystemManager = new StandardFileSystemManager();
fileSystemManager.init();

// only for public key authentication
SftpFileSystemConfigBuilder sftpConfigBuilder = SftpFileSystemConfigBuilder.getInstance();
FileSystemOptions opts = new FileSystemOptions();
sftpConfigBuilder.setIdentities(opts, new File[]{privateKey.toFile()});
String connectionUrl = String.format("sftp://%s@%s", user, host);

// only for password authentication
String connectionUrl = String.format("sftp://%s:%s@%s", user, password, host);

// Connection set-up
FileObject remoteRootDirectory = fileSystemManager.resolveFile(connectionUrl, connectionOptions);

The authentication configuration in JSch is similar to Commons VFS. It depends which kind of authentication should be used. The private key for the public key authentication has to be configured in the JSch object and the password for the password authentication has to be set in the Session object. For both, the user name is set, when the JSch object gets the Session object.


JSch sshClient = new JSch();

// only for public key authentication
sshClient.addIdentity("location to private key file");

session = sshClient.getSession(user, host);

// only for password authentication
session.setPassword(password);

session.connect();

Server Authentication

All three libraries supports server authentication. In SSHJ the server authentication can be enabled with SSHClient.loadKnownHost. It is possible to  add an own location of known_host file or it is used the default location that depends on the using platform.


SSHClient sshClient = new SSHClient();
sshClient.loadKnownHosts(); // or sshClient.loadKnownHosts(knownHosts.toFile());
sshClient.connect(host);

In Commons VFS the server authentication configuration is also a part of the FileSystemOption like the public key authentication. There, the location of the known_hosts file can be set.


SftpFileSystemConfigBuilder sftpConfigBuilder = SftpFileSystemConfigBuilder.getInstance();
FileSystemOptions opts = new FileSystemOptions();
sftpConfigBuilder.setKnownHosts(opts, new File("location of the known_hosts file"));

In JSch it exists two possibilities to configure the server authentication. One possibility is to use the OpenSSHConfig (see JSch example for OpenSSHConfig). The another possibility is easier. The location of the known_hosts file can be set directly in JSch object.


JSch sshClient = new JSch();
sshClient.setKnownHosts("location of known-hosts file");

Upload/download Files Over SFTP

All three libraries supports uploads and downloads files over SFTP. SSHJ has very clear API for these operations. The SSHClient object creates a SFTPClient object. This object is responsible for the upload (SFTPClient.put) and for the download (SFTPClient.get).


SSHClient sshClient = new SSHClient();
// ... connection

try (SFTPClient sftpClient = sshClient.newSFTPClient()) {
  // download
  sftpClient.get(remotePath, new FileSystemFile(local.toFile()));
  // upload
  sftpClient.put(new FileSystemFile(local.toFile()), remotePath);
}

In Commons VFS the upload and download files is abstracted as operation on a file system. So both are represented by the copyFrom method of a FileObject object. Upload is a copyFrom operation on a RemoteFile  object and download is a copyFrom operation on a LocalFile.


StandardFileSystemManager fileSystemManager = new StandardFileSystemManager();
// ... configuration
remoteRootDirectory = fileSystemManager.resolveFile(connectionUrl, connectionOptions);

LocalFile localFileObject = (LocalFile) fileSystemManager.resolveFile(local.toUri().toString());
FileObject remoteFileObject = remoteRootDirectory.resolveFile(remotePath);
try {
  // download
  localFileObject.copyFrom(remoteFileObject, new AllFileSelector());

  // upload
  remoteFileObject.copyFrom(localFileObject, new AllFileSelector());
} finally {
  localFileObject.close();
  remoteFileObject.close();
}

JSch also supports a SFTPClient. In JSch it is called ChannelSFTP. It has two method for download (ChannelSFTP.get) and upload (ChannelSFTP.put).


// here: creation and configuration of session

ChannelSftp sftpChannel = null;
try {
  sftpChannel = (ChannelSftp) session.openChannel("sftp");
  sftpChannel.connect();

  // download
  InputStream inputStream = sftpChannel.get(remotePath);
  Files.copy(inputStream, localPath);

  // upload
  OutputStream outputStream = sftpChannel.put(remotePath);
  Files.copy(locaPathl, outputStream);
} catch (SftpException | JSchException ex) {
  throw new IOException(ex);
} finally {
  if (sftpChannel != null) {
    sftpChannel.disconnect();
  }
}

Execute Shell Commands

Only Commons VFS doesn’t support executing plain shell commands. In SSHJ it is a two-liner. The SshClient starts a new Session object. This object executes the shell command. It is very intuitive.


// creation and configuration of sshClient

try (Session session = sshClient.startSession()) {
  session.exec("ls");
}

In Jsch the ChannelExec is responsible for executing shell commands over SSH. At first the command is set in the channel and then the channel has to be started. It isn’t so intuitive than in SSHJ.


// here: creation and configuration of session object

ChannelExec execChannel = null;
try {
  execChannel = (ChannelExec) session.openChannel("exec");
  execChannel.connect();
  execChannel.setCommand(command);
  execChannel.start();
} catch (JSchException ex) {
  throw new IOException(ex);
} finally {
  if (execChannel != null) {
    execChannel.disconnect();
  }
}

File Operations On the Remote Hosts

All libraries supports more or less ideal file operations over SFTP on remote machines. In SSHJ SFTPClient has also methods for file operations. The names of the methods are the same as the file operations on a Linux system. The following code snippet shows how to delete a file.


//here: creation and configuration of sshClient

try (SFTPClient sftpClient = sshClient.newSFTPClient()) {
  sftpClient.rm(remotePath);
}

Commons VFS’s core functionality is file operations. The usage takes getting used to. A file object has to be resolve and the file operations can be done on it.


// here: creation and configuration of remoteRootDirectory

FileObject remoteFileObject = remoteRootDirectory.resolveFile(remotePath);
try {
  remoteFileObject.delete();
} finally {
  remoteFileObject.close();
}

JSch’s SFTPClient ChannelSFTP has also method for file operations. The mostly file operations are supported by this channel. For e.g. the file copy operation on the remote machine has to be done by plain shell commands over the ChannelExec.

// here: creation and configuration of session
ChannelSftp sftpChannel = null;
try {
  sftpChannel = (ChannelSftp) session.openChannel("sftp");
  sftpChannel.connect();
  sftpChannel.rm(remotePath);
} catch (SftpException | JSchException ex) {
  throw new IOException(ex);
} finally {
  if (sftpChannel != null) {
    sftpChannel.disconnect();
  }
}

Conclusion

After this comparison I have two favourites, SSHJ and Commons VFS. SSHJ has a very clear API and I would choose it if I need a common SSH client or file operation support over SFTP is sufficient. I would choose Commons VFS if I have file operation over many file system protocols or a common SSH client is not needed. For the case, that I need both, I could use JSch directly to execute commands over SSH. The API of Commons VFS takes getting used to. But after understanding the concept behind, the usage of the API is straightforward.

The whole source code examples of this comparison are hosted on Github.

Useful Links

  1. SSHJ homepage
  2. JSch homepage
  3. Commons-vfs homepage
  4. Wikipedia page about SFTP
  5. Source Code of this comparison on Github