Configuration files examples

Linux OSes

Chef's Bento is a must, at least to get started : https://github.com/chef/bento/

Windows

Try Bento's like above, or the one adviced in Bento's README.md, or maybe https://github.com/joefitzgerald/packer-windows or the (more active, improved ?) fork https://github.com/StefanScherer/packer-windows

Build environment setup

For VirtualBox

On your workstation or a VM, Linux or Windows should be fine

  • Install Packer
  • Install Virtualbox (might have to reboot for kernel headers)
  • Install Git and clone Bento git clone https://github.com/chef/bento/
  • Move in the target OS's folder, like cd bento/centos
  • Build packer build -only=virtualbox -var "headless=true" centos-7.5-x86_64.json

For VMWare

There are two possibilities to build a box / template for VMWare. Either Packer uses a local VMWare Workstation or Player, or it uses a remote ESX(i).

Over ESX / ESXi

If you don't have an ESXi already available, here's the free ESXi 6 ISO which can be installed on a VirtualBox VM. The minimal setup is 2 CPU, 4Go RAM, 30Go HDD. Don't forget to set up a local network adapter with DHCP enabled. Then:

  • Install packer
  • Install VMWare Player and add ovftool directory to your PATH (C:\Program Files (x86)\VMware\VMware Player\OVFTool for Windows)
  • Enable SSH on the ESXi
  • Enable GuestIPHack on the ESXi esxcli system settings advanced set -o /Net/GuestIPHack -i 1
  • Get Bento or this repository
  • Edit the packer configuration json file in the vmware-iso bloc
    "floppy_files": [
      "../centos/http/{{user `ks_path`}}"
    ],
    "boot_command": "<tab> inst.text inst.ks=hd:fd0:/ks.cfg <enter><wait>",
    ...
    "vmx_data": {
      ...
      "ethernet0.networkName": "{{ user `esx_network` }}"
    },
    ...
    "remote_type": "esx5",
    "remote_host": "{{ user `esx_host` }}",
    "remote_username": "{{ user `esx_username` }}",
    "remote_password": "{{ user `esx_password` }}",
    "remote_datastore": "{{ user `esx_datastore` }}",
    "network": "{{ user `esx_network` }}",
    "network_adapter_type": "vmxnet3",
    "vnc_bind_address": "0.0.0.0",
    "vnc_disable_password": false
    

Note, the {{ user \xxx` }}is packer syntax for variables which can be specified on the command line with-var "var=value"`. So manually building CentOS 7 would look like

cd centos
packer build -var "cpus=2" -var "memory=2048" -var "headless=true" -var "http_proxy=http://10.50.51.31:8080" -var "https_proxy=http://10.50.51.31:8080" -only=vmware-iso -var "esx_host=10.35.0.246" -var "esx_username=root" -var "esx_password=XXXXXXXXXXX" -var "esx_datastore=datastore2" -var "esx_network=VM Network" centos-7.5-x86_64.json

Why oh why about

  • SSH not connecting, no IP Address on interface : open tty1, vagrant/vagrant, sudo su, nmtui, activate ens160.
  • Kickstart file : packer will create the VM to package as a template, which will need to access the machine running packer on a random port between 8000~9000 to grab ks.cfg for OS silent install. You'll have to enable packer.exe in Windows Firewall input rules, or iptables -A INPUT -p tcp –dport 8000:9000 -j ACCEPT for Linux's iptable. Another solution is to provide the kickstart in a floppy, be aware that inst.ks=hd:fd0:/ks.cfg is a RHEL7+ syntax, see this part of the doc
  • VNC : doc say you've to disable password for ESXi 6.5 and 6.7. Update "vnc_disable_password": false to true if needed.
  • Secrets & passwords shouldn't be hardcoded, especially with git or svn versionning. For the weekly build, passwords are stored in Gitlab (see CI/CD vars panel), passed down via the pipeline (.gitlab-ci.yml file) and the command like the aboxe example.

If something doesn't work, export PACKER_LOG=1 and run the command again to see what's wrong. And forget about -debug flag.

Disabling ESXi firewall might help : esxcli network firewall set --enabled false

Over VMWare Player

Couldn't get it to work... tried a lot of Workstation / PLayer / VIX combination w/o success

  • Install Packer
  • Install Git and clone Bento git clone https://github.com/chef/bento/
  • If you don't wan't to deal with VMWare tools ISO, update _common/vmware.sh to install open-vm-tools instead of building tools, then set tools_upload_flavor = "" in the json files. Warning, HGFS seems to be broken in open-vm-tools even if VMWare recommend it over tools from ISO.
  • Get VMWare Workstation Player https://my.vmware.com/fr/web/vmware/free#desktop_end_user_computing/vmware_workstation_player/
  • Get VMWare VIX API https://my.vmware.com/fr/web/vmware/free#desktop_end_user_computing/vmware_workstation_player/14_0|PLAYER1413|drivers_tools
  • Install them both bash VMware-*.bundle --eulas-agreed --required
  • Install VMWare requirements yum install -y qemu-img libX11.x86_64 libXext.x86_64 libXi.x86_64 libXinerama.x86_64 libXcursor.x86_64 libXtst.x86_64
  • Check VMWare network vmware-networks --status
    Bridge networking on vmnet0 is running
    DHCP service on vmnet1 is running
    Hostonly virtual adapter on vmnet1 is enabled
    DHCP service on vmnet8 is running
    NAT service on vmnet8 is running
    Hostonly virtual adapter on vmnet8 is enabled
    All the services configured on all the networks are running
    
  • Stop it vmware-networks --stop
  • Write file /etc/vmware/netmap.conf according to the previous status
    network0.name = "Bridged"
    network0.device = "vmnet0"
    network1.name = "HostOnly"
    network1.device = "vmnet1"  
    network8.name = "NAT"
    network8.device = "vmnet8"
    
  • Start it vmware-networks --start
  • Build /usr/bin/packer build -only=vmware-iso -var "headless=true" centos-7.5-x86_64.json

If there's an issue, export PACKER_LOG=1 and / or look at logs in /tmp/vmware-root

Some reading and download links :

Over Docker

Never tried, but saw https://communities.vmware.com/thread/563830

Automate build

Sonatype Nexus OSS 2

This is where the boxes are stored in my current job, no better choice available. Here's the configuration : add a site repository for the boxes

nexus-cfg-repo

Then create a packer read-only role

nexus-cfg-role

That you'll attach to the anonymous user, which need to be enabled

nexus-cfg-user-anonymous

And of course a packer user allowed to write in the packer repository, for the CI to upload the boxes, and specified in NEXUS_USER and NEXUS_PWD in .gitlab-ci.yml file

Gitlab CI pipeline

Using Bento basis, I just modified mirror and mirror_directory aiming at local ISO mirror. Plus a company specific script I added in _common/.

Gitlab's runner must have packer, unzip, curl and VirtualBox installed. Only the last one can be tricky, if you install the required kernel stuff you might have to reboot.

---
# Timeout updated to 1 hour * build & upload job in https://astgitlab.loc/.../packer/settings/ci_cd, general pipelines settings, timeout

stages:
  - build

variables:
  PACKER_VERSION: "1.2.5"
  NEXUS_URL: "https://nexus.loc/content/sites/packer/"
  NEXUS_LOGIN: "packer_rw"
  NEXUS_PWD: "verysecretpassworduwouldntbelieve_orjustsetitingitlabvar"
  PROXY: "http://proxy:8080"

before_script:
  - export HTTP_PROXY=${PROXY}
  - export HTTPS_PROXY=${PROXY}
  - |
    if [ ! -x ~/bin/packer ]
    then
      curl -O https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip
      mkdir -p ~/bin
      unzip -n -d ~/bin packer_${PACKER_VERSION}_linux_amd64.zip
      chmod +x ~/bin/packer
      rm packer_${PACKER_VERSION}_linux_amd64.zip
    fi
  - ~/bin/packer --version
  - |
    if [ ! $(which VBoxManage) ]
    then
      sudo curl -o /etc/yum.repos.d/virtualbox.repo http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
      sudo yum install -y VirtualBox-5.2 kernel kernel-devel kernel-headers
      sudo vboxconfig
      # reboot # if kernel got updated
    fi
  - VBoxManage --version
  - |
    if [ ! $(which unzip) ]
    then
      sudo yum install -y unzip
    fi

...

centos-7-x86_64:
  stage: build
  tags:
  - ansible
  - docker
  script:
    - export http_proxy=${PROXY}
    - export https_proxy=${PROXY}
    - cd centos && ~/bin/packer build -var "cpus=2" -var "memory=2048" -var "headless=true" -var "http_proxy=${PROXY}" -var "https_proxy=${PROXY}" -only=virtualbox-iso centos-7.5-x86_64.json
    - cd ../builds/ && curl -k -u ${NEXUS_LOGIN}:${NEXUS_PWD} --upload-file centos-7.5.virtualbox.box ${NEXUS_URL}/centos-7-x86_64.virtualbox.box
  retry: 2

Jenkinsfile

Same as for Gitlab's runner, slave must have packer, unzip, curl and VirtualBox installed.

pipeline {

  agent {
    label 'packer'
  }

  stages {

    stage ('Get latest code') {
      steps {
        checkout scm
      }
    }

    stage ('Packer setup') {
      steps {
        def packerVersion = "1.2.5"
        sh '''
          if [ ! -x ~/bin/packer ]
          then
            echo "Missing ~/bin/packer, installing version ${packerVersion}"
            export HTTP_PROXY=http://proxy:8080
            export HTTPS_PROXY=http://proxy:8080
            wget https://releases.hashicorp.com/packer/${packerVersion}/packer_${packerVersion}_linux_amd64.zip
            mkdir -p ~/bin
            unzip -d ~/bin packer_${packerVersion}_linux_amd64.zip
            chmod +x ~/bin/packer
            rm packer_${packerVersion}_linux_amd64.zip
          else
            echo "Packer already installed"
          fi
          packer --version
        '''
      }
    }

    ...

    stage ('Building CentOS 7.5 x86_64 for VirtualBox') {
      steps {
        sh '''
          cd centos
          packer build -only=virtualbox-iso centos-7.5-x86_64.json
        '''
        nexusArtifactUploader {
          nexusVersion('nexus2')
          protocol('http')
          nexusUrl('http://nexus.loc')
          repository('packer')
          credentialsId('ldap.nexus')
          artifact {
              artifactId('centos-7.5.virtualbox.box')
              file('builds/centos-7.5.virtualbox.box')
          }
        }
      }
    }
  }
}

Known issues

VirtualBox crashing with 0x80004005

It's most likely about VirtualBox's GUI. Something like

==> virtualbox-iso: Error starting VM: VBoxManage error: VBoxManage: error: The virtual machine 'centos-7.5-x86_64' has terminated unexpectedly during startup because of signal 6
==> virtualbox-iso: VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

Can be solved enforcing headless: true in the configuration file, which will remove --type gui from the VBoxManage command used by Packer.

{
  "builders": [
    {
      ...      
      "headless": "true",
      ...
    },
    {
      ...
      "headless": "true",      
      ...
    },
    ...

Or via the command line option -var "headless=true". Note, if using Jenkins there's a Xvfb plugin.

Accessing boxes via windows share doesn't work

It's more a Vagrant issue but let's write it down here. I tried accessing the box via samba. The Vagrantfile looked like

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  # config.vm.box = "centos/7"
  config.vm.box = "local/centos-7.5"  
  config.vm.box_url = 'file:////nexus.loc/packer$/centos-7.5.virtualbox.box'
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "1024"
    vb.cpus = "1"
  end
end

Which should work over Vagrant 2.1.1 but I ended up with another error message. For 2.1.0 it requires 6 slashes (file://////smb/your.box), and is broken on last version 2.1.2. So, just don't rely on samba share, it'll never work reliably. Otherwise the latest issue on the topic is here

The best solution is to use http, see the Nexus chapter for

VMWare build error on missing X11 libraries

Here's the example error log

vmware-iso output will be in this color.

==> vmware-iso: Retrieving ISO
    vmware-iso: Found already downloaded, initial checksum matched, no download needed: http://repo/centos75-x86_64/iso/CentOS-7-x86_64-DVD-1804.iso
==> vmware-iso: Creating required virtual machine disks
==> vmware-iso: Building and writing VMX file
==> vmware-iso: Starting HTTP server on port 8151
==> vmware-iso: Starting virtual machine...
    vmware-iso: The VM will be run headless, without a GUI. If you want to
    vmware-iso: view the screen of the VM, connect via VNC with the password "vqb52V90" to
    vmware-iso: vnc://127.0.0.1:5990
==> vmware-iso: Error starting VM: VMware error: /usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory
==> vmware-iso: Error starting VM: VMware error: /usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory
...

Run /usr/lib/vmware/bin/vmware-vmx for more info

I solved it installing the missing libraries specificaly for x86_64 : yum install -y libX11.x86_64 libXext.x86_64 libXi.x86_64 libXinerama.x86_64 libXcursor.x86_64 libXtst.x86_64