SKM IT World

Just another blog about IT


Leave a comment

Apache2 as Reverse Proxy for NPM Registry Proxies in Sonatype Nexus 3

I use a NPM registry proxy in Sonatype Nexus 3 behind an Apache2 as reverse proxy. With the “standard” Apache2 VirtualHost configuration


<VirtualHost:80>

  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  ProxyPass / http://localhost:8081/
  ProxyPassReverse / http://localhost:8081/

</VirtualHost:80>

I got following failure when I tried to install the dependency @sinonjs/formatio.

$ yarn add @sinonjs/formatio --verbose
yarn add v1.3.2
warning package.json: No license field
verbose 0.337 Checking for configuration file "/home/sparsick/dev/workspace/yarn-test-module/.npmrc".
verbose 0.337 Checking for configuration file "/home/sparsick/.npmrc".
verbose 0.337 Found configuration file "/home/sparsick/.npmrc".
verbose 0.337 Checking for configuration file "/usr/etc/npmrc".
verbose 0.338 Found configuration file "/usr/etc/npmrc".
verbose 0.338 Checking for configuration file "/home/sparsick/dev/workspace/yarn-test-module/.npmrc".
verbose 0.338 Checking for configuration file "/home/sparsick/dev/workspace/.npmrc".
verbose 0.338 Checking for configuration file "/home/sparsick/dev/.npmrc".
verbose 0.338 Checking for configuration file "/home/sparsick/.npmrc".
verbose 0.338 Found configuration file "/home/sparsick/.npmrc".
verbose 0.338 Checking for configuration file "/home/.npmrc".
verbose 0.341 Checking for configuration file "/home/sparsick/dev/workspace/yarn-test-module/.yarnrc".
verbose 0.342 Found configuration file "/home/sparsick/dev/workspace/yarn-test-module/.yarnrc".
verbose 0.343 Checking for configuration file "/home/sparsick/.yarnrc".
verbose 0.344 Found configuration file "/home/sparsick/.yarnrc".
verbose 0.344 Checking for configuration file "/usr/etc/yarnrc".
verbose 0.344 Checking for configuration file "/home/sparsick/dev/workspace/yarn-test-module/.yarnrc".
verbose 0.345 Found configuration file "/home/sparsick/dev/workspace/yarn-test-module/.yarnrc".
verbose 0.345 Checking for configuration file "/home/sparsick/dev/workspace/.yarnrc".
verbose 0.345 Checking for configuration file "/home/sparsick/dev/.yarnrc".
verbose 0.345 Checking for configuration file "/home/sparsick/.yarnrc".
verbose 0.345 Found configuration file "/home/sparsick/.yarnrc".
verbose 0.345 Checking for configuration file "/home/.yarnrc".
verbose 0.347 current time: 2018-02-27T08:04:43.357Z
warning yarn-test-module: No license field
[1/4] Resolving packages...
verbose 0.45 Performing "GET" request to "http://mycompany/repository/npm-public/@sinonjs%2fformatio".
verbose 0.55 Request "http://mycompany/repository/npm-public/@sinonjs%2fformatio" finished with status code 404.
verbose 0.551 Error: Couldn't find package "@sinonjs/formatio" on the "npm" registry.
at /usr/lib/node_modules/yarn/lib/cli.js:49061:15
at Generator.next (<anonymous>)
at step (/usr/lib/node_modules/yarn/lib/cli.js:92:30)
at /usr/lib/node_modules/yarn/lib/cli.js:103:13
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
error Couldn't find package "@sinonjs/formatio" on the "npm" registry.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.</span>

The problem is that Apache2 canonicalizes URLs as default. So I have to configure Apache2 to not canonicalize URLs and additionally, I have to allow encoded slashes:


<VirtualHost:80>

  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  ProxyPass / http://localhost:8081/ nocanon
  ProxyPassReverse / http://localhost:8081/

  AllowEncodedSlashes NoDecode

</VirtualHost:80>

With the above Apache2 Virtualhost configuration I could install my dependency via the NPM registry proxy.


$ yarn add @sinonjs/formatio

yarn add v1.3.2
warning package.json: No license field
info No lockfile found.
warning yarn-test-module: No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 2 new dependencies.
├─ @sinonjs/formatio@2.0.0
└─ samsam@1.3.0
warning yarn-test-module: No license field
Done in 0.60s.

Big thanks to Sonatype support team, that gave me this advice.

Advertisements


Leave a comment

Mocking SecurityContext in Jersey Tests

Jersey has a great possibility to write integration test for REST-APIs, written with Jersey. Just extend the class JerseyTest and go for it.

I ran in an issue, where I had to mock a SecurityContext, so that the SecurityContext includes a special UserPrincipal. The challenge is that Jersey wraps the SecurityContext in an own class SecurityContextInjectee in tests. So I have to add my SecurityContext Mock to this Jersey’s wrapper class. Let me demonstrate it in an example.

Let say I have the following Jersey Resource:


@Path("hello/world")
public class MyJerseyResource {

@GET
public Response helloWorld(@Context final SecurityContext context) {
String name = context.getUserPrincipal().getName();
return Response.ok("Hello " + name, MediaType.TEXT_PLAIN).build();
}

}

In my test, I have to mock the SecurityContext, so that a predefined user principal can be used during the tests. I use Mockito as mocking framework. My mock looks like the following one

final SecurityContext securityContextMock = mock(SecurityContext.class);
when(securityContextMock.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
return "Alice";
}
});

For adding this mocked SecurityContext to the wrapper class SecurityContextInjectee, I have to configure a ResourceConfig with a modified ContainerRequestContext in my Jersey Test. The mocked SecurityContext can be set in this modified ContainerRequestContext and then it will be used in the wrapper class:

@Override
public Application configure() {
final SecurityContext securityContextMock = mock(SecurityContext.class);
when(securityContextMock.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
return "Alice";
}
});

ResourceConfig config = new ResourceConfig();
config.register(new ContainerRequestFilter(){
@Override
public void filter(final ContainerRequestContext containerRequestContext) throws IOException {
containerRequestContext.setSecurityContext(securityContextMock);
}
});
return config;
}

Then, the whole test for my resource looks like the following one:

public class MyJerseyResourceTest extends JerseyTest {

@Test
public void helloWorld() throws Exception {
Response response = target("hello/world").request().get();

assertThat(response.getStatus()).isEqualTo(HttpStatus.SC_OK);
assertThat(response.getEntity()),isEqualTo("Hello Alice");
}

@Override
public Application configure() {
final SecurityContext securityContextMock = mock(SecurityContext.class);
when(securityContextMock.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
return "Alice";
}
});

ResourceConfig config = new ResourceConfig();
config.register(new ContainerRequestFilter(){
@Override
public void filter(final ContainerRequestContext containerRequestContext) throws IOException {
containerRequestContext.setSecurityContext(securityContextMock);
}
});
return config;
}

Do you have a smarter solution for this problem? Let me know it and write a comment below.


Leave a comment

Pimp My Git – Generate Content for .gitignore From the Scratch

When I start a new Git repository, I lose a lot of time to set up my .gitignore file and normally, I don’t match everything on the first shoot. Fortunately, there exists some tools, that help to bootstrapping it. I’d like to show two of them. One is a website that can be used on the command line and the another is a plugin for the IDE IntelliJ IDEA.

Website gitignore.io

There is a website http://gitignore.io that lists the common ignore pattern for you specific programming language, tool, IDE etc.
The usage is very simple: Fill the search with names of  tools, framework, programming language etc, which you want to use in your Git project, and the website generates the content for your .gitignore file.

You can also run gitignore.io from your command line. Therefore, you need an active internet connection and an environment function. I’ll demonstrate the integration of gitignore.io in zsh. For the integration in other shells or clients, please look into the documentation.

Firstly, we have to create a function gi in our ~/.zshrc:


echo "function gi() { curl -L -s https://www.gitignore.io/api/\$@ ;}" >> ~/.zshrc && source ~/.zshrc

Now, we can use it on the command line.


$ gi java,maven # Preview of the content for .gitignore

# Created by https://www.gitignore.io/api/java,maven

### Java ###
# Compiled class file
*.class

# Log file
*.log

# BlueJ files
*.ctxt

# Mobile Tools for Java (J2ME)
.mtj.tmp/

# Package Files #
*.jar
*.war
*.ear
*.zip
*.tar.gz
*.rar

# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*

### Maven ###
target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
pom.xml.next
release.properties
dependency-reduced-pom.xml
buildNumber.properties
.mvn/timing.properties

# Avoid ignoring Maven wrapper jar file (.jar files are usually ignored)
!/.mvn/wrapper/maven-wrapper.jar

# End of https://www.gitignore.io/api/java,maven

$ gi list # list currently available templates
1c-bitrix,a-frame,actionscript,ada,adobe
advancedinstaller,agda,alteraquartusii,altium,android
androidstudio,angular,anjuta,ansible,apachecordova
apachehadoop,appbuilder,appceleratortitanium,appcode,appcode+all
appcode+iml,appengine,aptanastudio,arcanist,archive
archives,archlinuxpackages,aspnetcore,assembler,atmelstudio
ats,audio,automationstudio,autotools,backup
basercms,basic,batch,bazaar,bazel
bitrix,bittorrent,blackbox,bluej,bower
bricxcc,buck,c,c++,cake
.... furthermore

$ gi java,maven >> .gitignore # append the content in your project's .gitignore

IntelliJ IDEA Plugin – .ignore

There is a plugin for IntelliJ IDEA that helps creating .gitignore file with content for your selected tool, programming language etc. . At first you have to install the plugin .ignore (Go to File -> Settings -> Plugins and search for .ignore).

You can now create .gitignore file via the .ignore plugin. By the way, the plugin can also create ignore files for other tools like Docker or Mercurial. Then a file generator is opened and you can choose templates of tools, programming language etc that you will use in the Git project.A preview shows you the possible content. A click on Generate and you are ready.

Do you have other tips and tricks to boost the initialization time of a Git project? Share them and write a comment below.

Links

  1. gitignore.io
  2. Website of .ignore


Leave a comment

How to Mark a Jenkins Job Red When Tests Fail In A Maven Build

The default setting in Jenkins is to mark a job yellow, when a Maven build fails because of failing tests. If you don’t want to have three status of your jobs, you can configure Jenkins so, that the jobs also mark red independent why a Maven build fails.

For this you will need administration rights on your Jenkins instance. Following steps have to be done:

  1. Go to Manage Jenkins -> Manage system.
  2. Add -Dmaven.test.failure.ignore=false to Maven Project Configuration -> Global Maven_OPTS.
  3. Save this change and that’s it.

Your next job run will consider this configuration. Unfortunately, this configuration has only effects for Maven jobs. Freestyle jobs ignore this configuration (see also this bug).

But a workaround exists:

  1. Install the TextFinder plugin via Manage Jenkins -> Manage Plugin.
  2. Open the Freestyle job’s configuration that should be marked red, when Maven tests fail.
  3. Click on Add a post-build action (in section Post-build Action) and select Jenkins Text Finder.
  4. Activate the check box Also search the console output.
  5. Add the value There are test failures to Regular expression.
  6. Save this change.

 

 

 


1 Comment

Generate P2 Repository From Maven Artifacts In 2017

Some years ago, I wrote a blog post about how to generate a P2 repository based on Maven artifacts. That described approach is obsolete nowadays and I’d like to show a new approach that is based on the p2-maven-plugin that was created to solve exactly this problem.

P2-Maven-Plugin Integration in Maven Build Life Cycle

First at all, we bind the p2-maven-plugin’s goal site to the Maven’s life cycle phase package. This goal is responsible for the generation of the P2 repository.

<plugin>
  <groupId>org.reficio</groupId>
  <artifactId>p2-maven-plugin</artifactId>
  <version>1.3.0</version>
  <executions>
    <execution>
      <id>default-cli</id>
      <phase>package</phase>
      <goals>
        <goal>site</goal>
      </goals>
      <!--... -->
    </execution>
  </executions>
</plugin>

Generating P2 Repository

Now, we can define which Maven artifacts should be a part of the new P2 repository. It is irrelevant for the p2-maven-pluging if the defined artifacts have already a OSGi manifest or not. If no OSGi manifest exists, the plugin will generate one.


<execution>
<!-- ... -->
<configuration>
  <artifacts>
    <!-- specify your dependencies here -->
    <!-- groupId:artifactId:version -->
    <artifact>
      <id>com.google.guava:guava:jar:23.0</id>
      <!-- Artifact with existing OSGi-Manifest-->
    </artifact>
    <artifact>
      <id>commons-io:commons-io:1.3</id>
      <!-- Artifact without existing OSGi-Manifest-->
    </artifact>
  </artifacts>
</configuration>
</execution>

The artifacts are specified by the pattern groupId:artifactId:version. If you want to save some typing, use the Buildr tab on MVN repository website for copying the right dependency declaration format.

This sample configuration creates a P2 repository that look like the following one:


target/repository
├── artifacts.jar
├── category.xml
├── content.jar
└── plugins
    ├── com.google.code.findbugs.jsr305_1.3.9.jar
    ├── com.google.errorprone.error_prone_annotations_2.0.18.jar
    ├── com.google.guava_23.0.0.jar
    ├── com.google.j2objc.annotations_1.1.0.jar
    ├── commons-io_1.3.0.jar
    └── org.codehaus.mojo.animal-sniffer-annotations_1.14.0.jar

1 directory, 9 files

 

The default behavior of the plugin is, that all transitive dependencies of the defined artifact are also downloaded and packed into the P2 repository. If you don’t want it, then you have to set the option transitive to false in the corresponded artifact declaration. If you need the sources (if they exist in the Maven repository) of the defined artifact in the P2 repository, then you have to set the option source to true in the corresponded artifact declaration.

<!-- ... -->
<artifact>
  <id>com.google.guava:guava:jar:23.0</id>
  <transitive>false</transitive>
  <source>true</source>
</artifact>
<!-- ... -->

Then the generated P2 repository looks like the following one:


target/repository
├── artifacts.jar
├── category.xml
├── content.jar
└── plugins
    ├── com.google.guava.source_23.0.0.jar
    ├── com.google.guava_23.0.0.jar
    └── commons-io_1.3.0.jar

1 directory, 6 files

Generating P2 Repository With Grouped Artifacts

In some situations, you want to group artifacts in so-called feature. p2-maven-plugin provides an option that allows to group the Maven artifact directly into features. The definition of the artifacts is the same like above. The difference is that it has to be inside the corresponded feature. Then, the feature definition needs some meta data information like feature ID, feature version, description etc.


<!-- ...-->
<configuration>
  <featureDefinitions>
    <feature>
      <!-- Generate a feature including artifacts that are listed below inside the feature element-->
      <id>spring.feature</id>
      <version>4.3.11</version>
      <label>Spring Framework 4.3.11 Feature</label>
      <providerName>A provider</providerName>
      <description>${project.description}</description>
      <copyright>A copyright</copyright>
      <license>A licence</license>
      <artifacts>
        <artifact>
          <id>org.springframework:spring-core:jar:4.3.11.RELEASE</id>id>
        </artifact>
        <artifact>
          <id>org.springframework:spring-context:jar:4.3.11.RELEASE</id>id>
          <source>true</source>
        </artifact>
      </artifacts>
    </feature>
    <!--...-->
  </featureDefinitions>
  <!-- ... -->
<configuration>

Then the generated P2 repository looks like the following one:


target/repository
├── artifacts.jar
├── category.xml
├── content.jar
├── features
│   └── spring.feature_4.3.11.jar
└── plugins
    ├── org.apache.commons.logging_1.2.0.jar
    ├── org.springframework.spring-aop.source_4.3.11.RELEASE.jar
    ├── org.springframework.spring-aop_4.3.11.RELEASE.jar
    ├── org.springframework.spring-beans.source_4.3.11.RELEASE.jar
    ├── org.springframework.spring-beans_4.3.11.RELEASE.jar
    ├── org.springframework.spring-context.source_4.3.11.RELEASE.jar
    ├── org.springframework.spring-context_4.3.11.RELEASE.jar
    ├── org.springframework.spring-core_4.3.11.RELEASE.jar
    ├── org.springframework.spring-expression.source_4.3.11.RELEASE.jar
    └── org.springframework.spring-expression_4.3.11.RELEASE.jar

2 directories, 14 files

Of course both options (generating p2 repository with feature and only with plugins) can be mixed.

p2-maven-plugin provides more options like excluding specific transitive dependencies, referencing to other eclipse features and so on. For more information, please look at the p2-maven-plugin homepage.

Now, we can generate P2 repositories from Maven artifacts. We lacks of how to deploy this P2 repository to a Repository manager like Artifactory or Sonatype Nexus. Both repository manager supports P2 repositories, Artifactory in the Professional variant (cost money) and Sonatype Nexus in OSS variant (free). For Nexus, it’s important that you use the version 2.x. The newest version, 3.x, doesn’t yet support P2 repositories.

Deploying P2 Repository to a Repository Manager

First at all, we want that our generated P2 repository is packed into a zip file. Therefore, we add the tycho-p2-repository-plugin to the Maven build life cycle:


<plugin>
  <groupId>org.eclipse.tycho</groupId>
  <artifactId>tycho-p2-repository-plugin</artifactId>
  <version>1.0.0</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>archive-repository</goal>
      </goals>
    </execution>
  </executions>
</plugin>


Then, we have to mark this zip file, so that Maven recognize that it has to deploy it during the deploy phase to a repository manager. For this, we add the build-helper-maven-plugin to the Maven build life cycle.

<!-- Attach zipped P2 repository to be installed and deployed in the Maven repository during the deploy phase. -->
<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <goals>
        <goal>attach-artifact</goal>
      </goals>
      <configuration>
        <artifacts>
          <artifact>
            <file>target/${project.artifactId}-${project.version}.zip</file>
            <type>zip</type>
          </artifact>
        </artifacts>
      </configuration>
    </execution>
  </executions>
</plugin>

Now, the generated P2 repository can be addressed by other projects. For more information about how to address the P2 repository, please have a look on the documentation of your repository manager.

A whole pom.xml sample can be found on Github.

Links


Leave a comment

How to Configure Apache2 as Forward and Reverse Proxy

This is a cook recipe to configure an Apache2 as a forward and reverse proxy on Debian-based Linux systems like Ubuntu or Debian itself.

Installation

It is assumed that the apache2 package is already installed on your system. For the proxy feature, we have to install the Apache2 module libapache2-mod-proxy-html on the system and activate theses Apache modules. At the end, Apache2 has to be restarted, so that the modules can be used.


sudo apt-get install libapache2-mod-proxy-html
sudo a2enmod proxy
sudo a2enmod proxy_html
sudo a2enmod proxy_http
sudo service apache2 restart

Configuration Forwarding and Rewarding

We want to forward the URL request http://jenkins.mycompany.com to http://jenkins.mycompany.com:8083 and rewarding http://jenkins.mycompany.com:8083 to http://jenkins.mycompany.com for every response.

For that, we have to create a so-called Virtual Host in Apache2. It is easiest to copy the configuration of the default one and adjust it.


cd /etc/apache2/sites-available
sudo cp 000-default.conf jenkins_ci.conf

It is best practice to create one conf file per Virtual Host. Adding and removing Virtual Host is easier with this approach.

The content of the Virtual Host configuration should look similar to the following one:


<VirtualHost jenkins.mycompany.com:80>
ServerName jenkins.mycompany.com

ServerAdmin webmaster@localhost

ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://localhost:8083/
ProxyPassReverse / http://localhost:8083/

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

We have to add the host name (in this case jenkins.my.company.com) to the line 1 and 2, so that Apache2 knows for which host name is this Virtual Host. In line 11 and 12 the mapping is configured (in this case, everything that call the host name directly should be forwarded to http://localhost:8083; the opposite for rewarding).

At the end this configuration has to be enabled.


sudo a2ensite jenksin_ci.conf
sudo service apache2 reload

 

Further Information

  1. https://wiki.ubuntuusers.de/Apache/mod_proxy_html/
  2. https://wiki.ubuntuusers.de/Apache/Virtual_Hosts/

 


Leave a comment

Pimp My Git – Manage Different Git Identities

I usually work on different Git projects that need different Git identities. My work flow for new repositories was

  1. Clone new repository.
  2. Go to cloned repository.
  3. If it is necessary to change the Git identity, call a shell script that runs `git config user.name “Sandra Parsick”; git config user.email sparsick@web.de`

I was never happy with this solution, but it works. Fortunately, a tweet of @BenediktRitter and one of @wosc suggest two alternatives to my method.

The first method bases on the Git feature “Conditional Includes” (required Git Version at least 2.13). The idea is that you define a default Git identity and separate Git identities per specific directory. That means, every repository, that is cloned beneath one of the specific directory, has automatically its specified Git identities.

The second method bases on a Python script, that is inspired by the Mercurial extension hg-persona. The idea is that you can individually set a Git identity per Git repository. It is an one command replacement for the git config user.* command serie.

In the next two sections I’d like to summarize how to set up and how to use these two methods. I have tested it on a Debian-based system. Let’s start with the first one.

Summarize Git identity for several Git repositories

As above described, this method bases on the Git feature “Conditional Includes”. Therefore, ensure you have installed your Git client in at least version 2.13 . Assume, we want to have two Git identities, one for Github and one for work. Therefore, create two .gitconfig files in your home folder.


touch ~/.gitconfig_github
touch ~/.gitconfig_work

Then add the specific Git identity in respective .gitconfig files.


~/.gitconfig_github

[user]
   name = YourNameForGithub
   email = name@forgithub.com

~/.gitconfig_work

[user]
   name = YourNameForWork
   email = name@forwork.com

The next step is to add these two .gitconfig files to our global one and to specify when to use them.

~/.gitconfig

[user]
   name = defaultName
   email = default@email.com

[includeIf "gitdir:~/workspace_work/"]
   path = .gitconfig_work

[includeIf "gitdir:~/workspace_github/"]
   path = .gitconfig_github

Now, every repository, that is cloned beneath ~/workspace_work/, has automatically the Git identity for Work (.gitconfig_work) and every repository, that is cloned beneath ~/workspace_github/, has automatically the Git identity for Github (.gitconfig_github). Otherwise, the default Git identity is used.

Setting Git identity individually per Git repository

For the second method, you have to install ws.git-persona from PyPI.


sudo apt-get install pip # if PyPI isn't install
pip install ws.git-persona

Then, open your global ~/.gitconfig and add your personas. In our cases, we add two personas, one for Github and one for work.


~/.gitconfig

[persona]
  github = YourNameForGithub &lt;name@forgithub.com&gt;
  work = YourNameForWork &lt;name@forwork.com&gt;

In the next step, we want to switch our Git identity in a Git repository. This is now possible with the command git-persona. In the following example we switch to the identity for Github and then to the identity for work.


> git-persona -n github
Setting user.name="YourNameForGithub", user.email="name@forgithub.com"
> git config user.name
YourNameForGithub
> git config user.email
name@forgithub.com
> git-persona -n work
Setting user.name="YourNameForWork", user.email="name@forwork.com"
> git config user.email
name@forwork.com
> git config user.name
YourNameForWork

If you have other methods to manage different Git identities, let me know it and write a comment.

 

Links

  1. Blog post about Git feature “Conditional Includes”.
  2. Github repository of git-personas.