Be ready for tomorrow prez #hadoop #devops !

Minimum set of tools to do devOps with Hadoop !

DevOps is a way to do / frameworks to use to ease life of IT teams from developer to admin / prod in a complex, multi-parameter, collaborative environment.

• Continuous integration tool : Jenkins
• Build automation tool : Maven
• Team code collaboration tool : Gerrit Code Review
• Database : PostgreSQL
• Admin and application monitoring
• Visualisation tool : Zeppelin
• CEO / admin / dev dashboards
• A Versionning tool : Git
• code
• configuration
• template

List of Jenkins plugins and configuration for Hadoop automatic deployment !

Configure :

• JDK
• Maven
• Security
• Share ssh public key from the jenkins hosts

Plugins : 

• Locale Plugin (en_GB and Ignore browser preference and force this language to all users)
• GitHub Plugin (for Git interaction)
• Nested View (to allow grouping job views into multiple levels)
• SafeRestart (This plugin allows you to restart Jenkins safely)
• Conditional BuildStep (It will allow you to define a condition controling the execution of the step(s))
• Maven Integration plugin (Jenkins plugin for building Maven 2/3 jobs via a special project type)
• JobConfigHistory (Saves copies of all job and system configurations)
• Email-ext plugin (email notification functionality)
• PostgreSQL+Database+Plugin (This is a driver plugin for Database Plugin)
• thinBackup (simply backs up the global and job specific configurations)
• Dynamic Parameter Plug-in (dynamic generation of default build parameter values)
• Plot Plugin (This plugin provides generic plotting (or graphing) capabilities in Jenkins)
• Build Pipeline (Build Pipeline View of upstream and downstream connected jobs)
• View Job Filters (Create smart views with exactly the jobs you want)
• Folder Plugin
• xUnit Plugin
• jUnit Plugin
• R Plugin
• Ansible plugin
• Python Plugin
• Vagrant Plugin

Currently in testing :

• Job DSL
• Task Scanner

Jenkins, Maven, SVN and Hortonworks HDP2.3 sandbox !

If you are also an automation and Open Source fan and you are (or not) in the process to build Hadoop application, I strongly suggest to use (minimum) :

• Continuous integration tool (Jenkins, TeamCity, Travis CI)
• Build tool (Maven, Ant, Gradle)
• Provisionning tool (Chef, Ansible, shell script, Puppet)
• Versionning system (Git, SVN, CVS)

In order to improve overall quality project / to stop loosing time / to ease Hadoop migration and testing / to be more efficient (yes a lot of good reasons).

I have the pleasure to use SVN + Jenkins + Maven + few Shell script + HDP sandbox on my laptop and this is really awesome.

Thanks ;-)