22 days old

Data Engineer Elastic Engineer

Austin, TX 78701
  • Job Code
    270442BR

Introduction
At IBM, work is more than a job - it's a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you've never thought possible. Are you ready to lead in this new era of technology and solve some of the world's most challenging problems? If so, lets talk.


Your Role and Responsibilities

We are looking for a Data engineer to deploy complex big data and analytics solutions using Elastic stack. You should have strong experience in deploying and managing Elastic cluster on Kubernetes in multi-site, multi cluster environment in both on-Premise as well as Cloud platforms. You should have applied or expert knowledge in big data platforms. The main use case for such a platform for us would be Real-Time Anomaly Detection and Time Series models on IT operations data like logs, metrics, events, wired data, transaction flow, ITIL process related data, knowledge repositories, etc

Business Unit/ Team Overview

Global Technology Services (GTS) at IBM manages the IT infrastructure for some of the worlds leading corporations and with that comes the responsibility of managing enormous amounts of IT data and the opportunity for making better decisions using that data. In GTS analytics team at IBM, Data Scientists, Data Engineers, and BigData IT Architects are developing novel models, cutting edge algorithms, and custom analytics solutions to tackle BigData challenges in the IT Infrastructure space.

ITOA / AIops provides real time machine-data (log, events, performance, capacity, ITIL data, wire data, etc) analytics solutions that helps customers manage Business Services and manage the quality of the end-user experience

It can tell a client in real time What happened, Why did it happen, Will it happen again and What to do if it happens again? etc

Keeps everyone on the same page by looking at the same Business Transaction data and metrics.

Keeps the focus on operational data that translate to the business value the application delivers; dive in deeper when appropriate.

Identify resolution criteria, assign ownership

Take lessons learned to improve development, test, deployment, and production processes

Education & Experience

Minimum 4 years of relevant experience working on the Elastic based products & distributions specifically used in Real Time ITOA or AIops use-cases processing logs, metrics, events, etc

At least 4 years of experience in development & implementation of logging and metrics solutions in with TB+ / day ingestion per day

At least 5 years of hands-on experience in IT support (Infrastructure / Application) and IT monitoring tools

Overall 7+ years of core Big data / Analytics experience in various domains

Degree / Masters degree in computers or equivalent

Certifications:

Elastic certified engineer

Certifications showing proficiency in the Usage, design & deployment of ITOA / AIOps solutions like Elastic or Splunk

Specialized certifications on specific technologies like Hadoop, Cloudera, Spark, Kafka, etc

Job Responsibilities

Deploy Elastic stack cluster on native kubernetes or Kubernetes services and maintain the clusters efficiently

Leading end to end deployment of ITOA / AIOps solutions for enterprise customers

Provide engineering inputs to Architects and Data scientists on various stages of solution design

Perform Integration and deployment of ITOA solution as per design provided by Architects

Participate & be an active member of internal capability building projects

Train & support junior resources as needed

Provide resolution to customer queries and issues

Skills Required:

Excellent knowledge on log analytics, time series data anomaly detection and correlation of events

Hands-on experience with IT operational data like logs, metrics, events, RDBMS tables, etc and ingesting them into Elastic stack

Expert Knowledge on GO/grok/REGEX/Logstash/Fluentd to perform Extract, Transform and Load for IT operational data into big data repositories like Elasticsearch, Cassandra, Hadoop, etc

Expert level experience in managing large Elastic cluster and in-depth knowledge in Elastic features

Alerting

Security

Curator

Reporting

Monitoring

Backup and resiliency

Kubernetes cluster management

Python/R/Scala languages/Scripting Languages in context of Anomaly Detection & Time Series modelling

Working experience with ITIL Framework

Working knowledge on Apache Hadoop, Spark, Airflow, Cassandra and Kafka ecosystem

Prior experience in deploying Elastic solutions in production environments processing operational data in terms of at least 500 GB / day

Experience with SQL based tools & expertise on any one traditional RDBMS mySQl; MSSQL;Oracle;DB2 etc

Prior experience with DevOps projects, Github, Jira, Travis, etc

Working knowledge on Windows, Linux and AIX platforms

Working knowledge of Top commercial distributions of the above stack MapR; Cloudera; Hortonworks etc

Knowledge on shell scripting

Good knowledge on other Top level Apache Big Data technologies like Cassandra, NIFI, Fluentd, Drill, Sentry etc

Excellent understanding on HDFS & other similar Map/Reduce paradigms

Knowledge on 1-2 NoSQL databases Redis; MongoDB;Cassandra;Neo4j; VoltDB etc

Preferred:

Experience with Elastic ML and real-time operations analytics using Apache suit of products like Spark using Python or Scala

Experience with Kibana plug-in development and other UI development

Experience working with large data sets leveraging distributed systems e.g. Spark/Hadoop.

Tools & Methods (Experience in at least one in each category or similar if not listed below)

Log Analytics Elastic Search, Apache Solr

Data Pipelines: Logstash, Fluentd, Kafka, Nifi

Languages: Python, PySpark, Spark, Scala, R, Java, Java Script

Visualization : Kibana, Tableau, Cognos

Machine learning Elastic ML, Python, Spark, Tensorflow, H2O

Streaming: Spark; Storm;

Relational Database technologies: Oracle, Db2, SQL, MySQL,

NoSQl DBs: MongoDB, Cassandra, Neo4J,Redis, VoltDB, CouchDB

Apache Hadoop Distribution Apache, Hortonworks, Cloudera, MapR

ETL technologies: Datastage, Informatica, Pentaho DI, SAS DI, SSIS or R, Python based Data munging

Cloud technologies: AWS, Azure, IBM Softlayer

Soft Skills

Excellent Written & Verbal Communication

Excellent Analytical & Virtual troubleshooting skills

Skills to work in team and collaborative environment

Customer/Vendor interaction & co-ordinations




Required Technical and Professional Expertise
7+ years of professional hands on experience in IT operations
Excellent knowledge on log analytics, time series data anomaly detection and correlation of events
Hands-on experience with IT operational data like logs, metrics, events, RDBMS tables, etc and ingesting them into Elastic stack
Expert Knowledge on GO/grok/REGEX/Logstash/Fluentd to perform Extract, Transform and Load for IT operational data into big data repositories like Elasticsearch, Cassandra, Hadoop, etc
Expert level experience in managing large Elastic cluster and in-depth knowledge in Elastic features
Python/R/Scala languages/Scripting Languages in context of Anomaly Detection & Time Series modelling
Working experience with ITIL Framework
Working knowledge on Apache Hadoop, Spark, Airflow, Cassandra and Kafka ecosystem
Prior experience in deploying Elastic solutions in production environments processing operational data in terms of at least 500 GB / day



Preferred Technical and Professional Expertise
Experience with SQL based tools & expertise on any one traditional RDBMS mySQl; MSSQL;Oracle;DB2 etc
Prior experience with DevOps projects, Github, Jira, Travis, etc
Working knowledge on Windows, Linux and AIX platforms
Working knowledge of Top commercial distributions of the above stack MapR; Cloudera; Hortonworks etc
Knowledge on shell scripting
Good knowledge on other Top level Apache Big Data technologies like Cassandra, NIFI, Fluentd, Drill, Sentry etc
Excellent understanding on HDFS & other similar Map/Reduce paradigms
Knowledge on 1-2 NoSQL databases Redis; MongoDB;Cassandra;Neo4j; VoltDB etc



About Business Unit
At Global Technology Services (GTS), we help our clients envision the future by offering end-to-end IT and technology support services, supported by an unmatched global delivery network. It's a unique blend of bold new ideas and client-first thinking.If you can restlessly reinvent yourself and solve problems in new ways, work on both technology and business projects, and ask, "What else is possible?" GTS isthe place for you!


Your Life @ IBM
What matters to you when youre looking for your next career challenge?

Maybe you want to get involved in work that really changes the world? What about somewhere with incredible and diverse career and development opportunities where you can truly discover your passion? Are you looking for a culture of openness, collaboration and trust where everyone has a voice? What about all of these? If so, then IBM could be your next career challenge. Join us, not to do something better, but to attempt things you never thought possible.

Impact. Inclusion. Infinite Experiences. Do your best work ever.


About IBM
IBMs greatest invention is the IBMer. We believe that progress is made through progressive thinking, progressive leadership, progressive policy and progressive action. IBMers believe that the application of intelligence, reason and science can improve business, society and the human condition. Restlessly reinventing since 1911, we are the largest technology and consulting employer in the world, with more than 380,000 IBMers serving clients in 170 countries.


Location Statement
For additional information about location requirements, please discuss with the recruiter following submission of your application.


Being You @ IBM
IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.

Categories

Industry

  • Computers Software and Hardware
Posted: 2019-10-01 Expires: 2019-10-31

Before you go...

Our free job seeker tools include alerts for new jobs, saving your favorites, optimized job matching, and more! Just enter your email below.

Share this job:

Data Engineer Elastic Engineer

IBM
Austin, TX 78701

Join us to start saving your Favorite Jobs!

Sign In Create Account
Powered ByCareerCast