SKM IT World

Just another blog about IT


Leave a comment

How to Format a Large Code Base Automatically

If you introduce code formatting rules retroactively, you have to solve the problem how to format existing code base according to the new formatting rules. You could checkout every code repository one by one in your IDE and click on Autoformat the whole project. But this is boring and waste of time. Fortunately, Intellij IDEA has a format CLI tool in its installation. You can locate it in the path <your Intellij IDEA installation>/bin. It’s called format.sh.

In the next section I’d like to show you how you can automate formatting big code base. First, I will show the preparation steps like exporting your code formatting rule setting from the IDE. Then, I will demonstrate how to use the CLI-Tool format.sh. At the end, I will show a small Groovy script that query all repositories (in this case they are Git repositories), formatting the code and push it back to the remote SCM.

Preparations

First at all, we need the code formatting rule setting exported from Intellij IDEA. In your Intellij IDEA follow the next step

  1. Open File -> Settings -Editor-> Code Style
  2. Click on Export…
  3. Choose a name for the XML file (for example, Default.xml) and a location where this file should be saved (for example, /home/foo ).

Then, checkout or clone your SCM repository and remember the location where you checkout/clone it (for example, /home/foo/myrepository).

Format Code Base Via format.sh  CLI Tool

Three parameters are important for format.sh:

  • -s : Set a path to Intellij IDEA code style settings .xml file (in our example: /home/foo/Default.xml).
  • -r : Set that directories should be scanned recursively.
  • path<n> : Set a path to a file or a directory that should be formatted (in our example: /home/foo/myrepository).

> ./format.sh
IntelliJ IDEA 2018.2.4, build IC-182.4505.22 Formatter
Usage: format [-h] [-r|-R] [-s|-settings settingsPath] path1 path2...
-h|-help Show a help message and exit.
-s|-settings A path to Intellij IDEA code style settings .xml file.
-r|-R Scan directories recursively.
-m|-mask A comma-separated list of file masks.
path<n> A path to a file or a directory.

> /format.sh -r -s ~/Default.xml ~/myrepository

It’s possible that the tool cancels scanning because of a java.lang.OutOfMemoryError: Java heap space. Then, you have to increase Java’s maximum memory size (-Xmx) in <your Intellij IDEA installation>/bin/idea64.vmoptions.


> nano idea64.vmoptions
-Xms128m
-Xmx750m // <- here increase the maximum memory size
-XX:ReservedCodeCacheSize=240m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-Djdk.http.auth.tunneling.disabledSchemes=""
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-Dawt.useSystemAAFontSettings=lcd
-Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine

Groovy Script For Formatting Many Repository In a Row

Now, we want to bring everything together. The script should do four things:

  1. Find all repository URLs whose code has to be formatted.
  2. Check out / Clone the repositories.
  3. Format the code in all branches of each repostory.
  4. Commit and push the change to the remote SCM.

I choose Git as SCM in my example. The finding of the repository URLs depends on the Git Management System (like BitBucket, Gitlab, SCM Manager etc.) that you use. But the approach is in all system the same:

  1. Call the RESTful API of your Git Management System.
  2. Parse the JSON object in the response after the URLs.

For example, in BitBucket it’s like that:



@Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1')
import groovyx.net.http.*

import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*

def cloneUrlsForProject() {

    def projectUrl = "https://scm/rest/api/1.0/projects/PROJECT_KEY/repos?limit=1000"
    def token = "BITBUCKET_TOKEN"

    def projects = []
    def cloneUrls = []

    def http = new HTTPBuilder(projectUrl)
    http.request(GET) {
        headers."Accept" = "application/json"
        headers."Authorization" = "Bearer ${token}"

        response.success = { resp -> projects = new JsonSlurper().parseText(resp.entity.content.text)}

        response.failure = { resp ->
            throw new RuntimeException("Error fetching clone urls for '${projectKey}': ${resp.statusLine}")
        }
    }

    projects.values.each { value ->
        def cloneLink = value.links.clone.find { it.name == "ssh" }
        cloneUrls.add(cloneLink.href)
    }

    return cloneUrls
}

Then, we have to clone the repositories and checkout each branch. In each branch, the format.sh has to be called. For the git operation, we use the jgit library and for the format.sh call we use a Groovy feature for process calling. In Groovy it’s possible to define the command as a String and then to call the method execute() on this String like “ls -l”.execute(). So the Groovy script for the last three tasks would be looked like that:

#!/usr/bin/env groovy
@Grab('org.eclipse.jgit:org.eclipse.jgit:5.1.2.201810061102-r')
import jgit.*
import org.eclipse.jgit.api.CreateBranchCommand
import org.eclipse.jgit.api.Git
import org.eclipse.jgit.api.ListBranchCommand
import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider

import java.nio.file.Files


def intellijHome = 'path to your idea home folder'
def codeFormatterSetting = 'path to your exported code formatter setting file'
def allRepositoriesUrls = ["http://scm/repo1","http://scm/repo2"] // for simplifying

allRepositoriesUrls.each { repository ->
    def repositoryName = repository.split('/').flatten().findAll { it != null }.last()
    File localPath = Files.createTempDirectory("${repositoryName}-").toFile()
    println "Clone ${repository} to ${localPath}"
    Git.cloneRepository()
       .setURI(repository)
       .setDirectory(localPath)
       .setNoCheckout(true)
       .setCredentialsProvider(new UsernamePasswordCredentialsProvider("user", "password")) // only needed when clone url is https / http
       .call()
       .withCloseable { git ->
        def remoteBranches = git.branchList().setListMode(ListBranchCommand.ListMode.REMOTE).call()
        def remoteBranchNames = remoteBranches.collect { it.name.replace('refs/remotes/origin/', '') }

        println "Found the following branches: ${remoteBranchNames}"

        remoteBranchNames.each { remoteBranch ->
            println "Checkout branch $remoteBranch"
            git.checkout()
               .setName(remoteBranch)
               .setCreateBranch(true)
               .setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK)
               .setStartPoint("origin/" + remoteBranch)
               .call()

            def formatCommand = "$intellijHome/bin/format.sh -r -s $codeFormatterSetting $localPath"

            println formatCommand.execute().text

            git.add()
               .addFilepattern('.')
               .call()
            git.commit()
               .setAuthor("Automator", "no-reply@yourcompany.com")
               .setMessage('Format code according to IntelliJ setting.')
               .call()

            println "Commit successful!"
        }

        git.push()
           .setCredentialsProvider(new UsernamePasswordCredentialsProvider("user", "password")) // only needed when clone url is https / http
           .setPushAll()
           .call()

        println "Push is done"

    }
}

Do you have another approach? Let me know and write a comment below.

Advertisements


Leave a comment

Pimp My Git – Generate Content for .gitignore From the Scratch

When I start a new Git repository, I lose a lot of time to set up my .gitignore file and normally, I don’t match everything on the first shoot. Fortunately, there exists some tools, that help to bootstrapping it. I’d like to show two of them. One is a website that can be used on the command line and the another is a plugin for the IDE IntelliJ IDEA.

Website gitignore.io

There is a website http://gitignore.io that lists the common ignore pattern for you specific programming language, tool, IDE etc.
The usage is very simple: Fill the search with names of  tools, framework, programming language etc, which you want to use in your Git project, and the website generates the content for your .gitignore file.

You can also run gitignore.io from your command line. Therefore, you need an active internet connection and an environment function. I’ll demonstrate the integration of gitignore.io in zsh. For the integration in other shells or clients, please look into the documentation.

Firstly, we have to create a function gi in our ~/.zshrc:


echo "function gi() { curl -L -s https://www.gitignore.io/api/\$@ ;}" >> ~/.zshrc && source ~/.zshrc

Now, we can use it on the command line.


$ gi java,maven # Preview of the content for .gitignore

# Created by https://www.gitignore.io/api/java,maven

### Java ###
# Compiled class file
*.class

# Log file
*.log

# BlueJ files
*.ctxt

# Mobile Tools for Java (J2ME)
.mtj.tmp/

# Package Files #
*.jar
*.war
*.ear
*.zip
*.tar.gz
*.rar

# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*

### Maven ###
target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
pom.xml.next
release.properties
dependency-reduced-pom.xml
buildNumber.properties
.mvn/timing.properties

# Avoid ignoring Maven wrapper jar file (.jar files are usually ignored)
!/.mvn/wrapper/maven-wrapper.jar

# End of https://www.gitignore.io/api/java,maven

$ gi list # list currently available templates
1c-bitrix,a-frame,actionscript,ada,adobe
advancedinstaller,agda,alteraquartusii,altium,android
androidstudio,angular,anjuta,ansible,apachecordova
apachehadoop,appbuilder,appceleratortitanium,appcode,appcode+all
appcode+iml,appengine,aptanastudio,arcanist,archive
archives,archlinuxpackages,aspnetcore,assembler,atmelstudio
ats,audio,automationstudio,autotools,backup
basercms,basic,batch,bazaar,bazel
bitrix,bittorrent,blackbox,bluej,bower
bricxcc,buck,c,c++,cake
.... furthermore

$ gi java,maven >> .gitignore # append the content in your project's .gitignore

IntelliJ IDEA Plugin – .ignore

There is a plugin for IntelliJ IDEA that helps creating .gitignore file with content for your selected tool, programming language etc. . At first you have to install the plugin .ignore (Go to File -> Settings -> Plugins and search for .ignore).

You can now create .gitignore file via the .ignore plugin. By the way, the plugin can also create ignore files for other tools like Docker or Mercurial. Then a file generator is opened and you can choose templates of tools, programming language etc that you will use in the Git project.A preview shows you the possible content. A click on Generate and you are ready.

Do you have other tips and tricks to boost the initialization time of a Git project? Share them and write a comment below.

Links

  1. gitignore.io
  2. Website of .ignore


Leave a comment

Pimp My Git – Manage Different Git Identities

I usually work on different Git projects that need different Git identities. My work flow for new repositories was

  1. Clone new repository.
  2. Go to cloned repository.
  3. If it is necessary to change the Git identity, call a shell script that runs `git config user.name “Sandra Parsick”; git config user.email sparsick@web.de`

I was never happy with this solution, but it works. Fortunately, a tweet of @BenediktRitter and one of @wosc suggest two alternatives to my method.

The first method bases on the Git feature “Conditional Includes” (required Git Version at least 2.13). The idea is that you define a default Git identity and separate Git identities per specific directory. That means, every repository, that is cloned beneath one of the specific directory, has automatically its specified Git identities.

The second method bases on a Python script, that is inspired by the Mercurial extension hg-persona. The idea is that you can individually set a Git identity per Git repository. It is an one command replacement for the git config user.* command serie.

In the next two sections I’d like to summarize how to set up and how to use these two methods. I have tested it on a Debian-based system. Let’s start with the first one.

Summarize Git identity for several Git repositories

As above described, this method bases on the Git feature “Conditional Includes”. Therefore, ensure you have installed your Git client in at least version 2.13 . Assume, we want to have two Git identities, one for Github and one for work. Therefore, create two .gitconfig files in your home folder.


touch ~/.gitconfig_github
touch ~/.gitconfig_work

Then add the specific Git identity in respective .gitconfig files.


~/.gitconfig_github

[user]
   name = YourNameForGithub
   email = name@forgithub.com

~/.gitconfig_work

[user]
   name = YourNameForWork
   email = name@forwork.com

The next step is to add these two .gitconfig files to our global one and to specify when to use them.

~/.gitconfig

[user]
   name = defaultName
   email = default@email.com

[includeIf "gitdir:~/workspace_work/"]
   path = .gitconfig_work

[includeIf "gitdir:~/workspace_github/"]
   path = .gitconfig_github

Now, every repository, that is cloned beneath ~/workspace_work/, has automatically the Git identity for Work (.gitconfig_work) and every repository, that is cloned beneath ~/workspace_github/, has automatically the Git identity for Github (.gitconfig_github). Otherwise, the default Git identity is used.

Setting Git identity individually per Git repository

For the second method, you have to install ws.git-persona from PyPI.


sudo apt-get install pip # if PyPI isn't install
pip install ws.git-persona

Then, open your global ~/.gitconfig and add your personas. In our cases, we add two personas, one for Github and one for work.


~/.gitconfig

[persona]
  github = YourNameForGithub &lt;name@forgithub.com&gt;
  work = YourNameForWork &lt;name@forwork.com&gt;

In the next step, we want to switch our Git identity in a Git repository. This is now possible with the command git-persona. In the following example we switch to the identity for Github and then to the identity for work.


> git-persona -n github
Setting user.name="YourNameForGithub", user.email="name@forgithub.com"
> git config user.name
YourNameForGithub
> git config user.email
name@forgithub.com
> git-persona -n work
Setting user.name="YourNameForWork", user.email="name@forwork.com"
> git config user.email
name@forwork.com
> git config user.name
YourNameForWork

If you have other methods to manage different Git identities, let me know it and write a comment.

 

Links

  1. Blog post about Git feature “Conditional Includes”.
  2. Github repository of git-personas.


1 Comment

Pimp My Git – Git Mergetool

I like to work with git on the command line. But in some cases I prefer UI support. For example, solving merge conflicts is such a case. Git has a command mergetool, which can open a graphical tool to solve merge conflicts. But before you can use this command, you had to configure it. In this blog post I’d like to show you how to configure mergetool and how to use it.

Configuration

First at all, open a shell on Linux. On Windows, open Git Bash. Then choose a graphic tool that should support you solving merge conflicts. git mergetool –tool-help shows a list which tools are supported on your machine


 sparsick@sparsick-ThinkPad-T430s > git mergetool --tool-help
'git mergetool --tool=<tool>' may be set to one of the following:
                araxis
                kdiff3
                meld

The following tools are valid, but not currently available:
                bc
                bc3
                codecompare
                deltawalker
                diffmerge
                diffuse
                ecmerge
                emerge
                gvimdiff
                gvimdiff2
                gvimdiff3
                opendiff
                p4merge
                tkdiff
                tortoisemerge
                vimdiff
                vimdiff2
                vimdiff3
                winmerge
                xxdiff

Some of the tools listed above only work in a windowed
environment. If run in a terminal-only session, they will fail.

This command shows two lists. The first list shows all tools that are supported by git and that are available on your machine (in sample, it is araxis, kdiff3 and meld). The second list shows that are also supported by git, but they aren’t install on your machine.

I use meld as graphical tool. It’s runnable on Windows and Linux. If you haven’t install meld on your machine, then it’s now the right time to do it or choose another tool.

We want to set mergetool globally for all our repositories.


sparsick@sparsick-ThinkPad-T430s > git config --global merge.tool meld
sparsick@sparsick-ThinkPad-T430s > git mergetool
No files need merging

If git mergetool returns more than No files need merging, then the path to your graphic tool isn’t set in your $PATH system variable (The normal case on Windows systems). It’s possible to set the path to the graphical tool directly in git.

sparsick@sparsick-ThinkPad-T430s > git config --global mergetool.meld.path /c/Program\ Files\ \(x86\)/Meld/Meld.exe</pre>

Bear two important things in mind: mergetool is written without a dot between merge and tool and meld is a placeholder for the name of the graphical tool in the above sample. If you use another tool such like vimdiff, then the config key is called mergetool.vimdiff.path .

Now git mergetool is ready to use.

Usage

Now I’d like to demonstrate how to use git mergetool. It is used in when we have merge conflicts during a merge action. Let’s say we want to merge branch branch1 into master and this merge will have some merge conflicts.


sparsick@sparsick-ThinkPad-T430s > git merge branch1

Auto-merging test
CONFLICT (content): Merge conflict in test
Automatic merge failed; fix conflicts and then commit the result.

Now, we want to solve these conflicts with a graphical tool (in the example, it’s meld). git mergetool on the command line open the graphical tool of our choice.

sparsick@sparsick-ThinkPad-T430s > git mergetool

Merging:
test

Normal merge conflict for 'test':
{local}: modified file
{remote}: modified file

After solving the merge conflicts, the change has to commit.

sparsick@sparsick-ThinkPad-T430s > git status

On branch master
All conflicts fixed but you are still merging.
(use "git commit" to conclude merge)

Changes to be committed:

modified:   test

Untracked files:
(use "git add <file>..." to include in what will be committed)

test.orig
sparsick@sparsick-ThinkPad-T430s > git commit

You can see that we have a new untracked file test.orig . This is a backup of the merged file created by mergetool. You can configure that this backup should be removed after a successful merge.

sparsick@sparsick-ThinkPad-T430s > git config --global mergetool.keepBackup false

Further files are created when using git mergetool:

sparsick@sparsick-ThinkPad-T430s > git status

On branch master
Untracked files:
(use "git add ..." to include in what will be committed)

test.BACKUP.7344
test.BASE.7344
test.LOCAL.7344
test.REMOTE.7344

If only these files are currently untracked, then a git clean can help. Otherwise they have to be removed manually.

sparsick@sparsick-ThinkPad-T430s > git clean -f

Removing test.BACKUP.7344
Removing test.BASE.7344
Removing test.LOCAL.7344
Removing test.REMOTE.734

Links

  1. Meld Homepage
  2. git mergetool Documentation


Leave a comment

Git Resources for Beginner

In this post I’d like to share resources that help me learning and understanding Git.

Links

Do you have some more resources that you can recommend? Let me know it and write a comment.