Multi Recruit

<ul><li>Proven experience as Web Scrapper/Crawler or similar role</li><li>Have a strong understanding and working knowledge of web crawlers, web scrapers, and other automation tools, to help browse the web content</li><li>Knowledge of web scraping and tools</li><li>Strong knowledge of any of multiple open-source and proprietary scraping frameworks available</li><li>Hands-on-experience with SQL/NO-SQL (MySQL/ Postgres/Cassandra /MongoDB)</li><li>Good knowledge and coding experience in one or more programming languages such as Python, Java, JavaScript</li><li>Experience in creating scrapy spiders for websites with Captcha, IP ban, geolocation ban, Cloudflare / Distil / Imperva firewalls, sites that required login to access data, Dynamic websites loading through JS / REST API / Graphql, etc.</li><li>Knowledge of Object-oriented programming</li><li>Experience with applications designed to display archived web content</li><li>Experience with AWS cloud services (EC2)</li><li>Python Tech stack (Python libraries - scrappy, requests, urllib, Beautiful soup, splash, Selenium, pandas)</li><li>2-4 years of experience with a bachelor’s degree in Computer Science, Engineering, Technology, or related field required</li></ul>Responsibilities<ul><li>Program and apply your knowledge set to fetch data from multiple online sources, cleanse it</li><li>Develop application frameworks for automating and maintaining a constant flow of data from multiple sources</li><li>Design, and build web crawlers to scrape data and URLs by using Python modules [scrapy, selenium, requests, Beautiful Soup, splash, etc.]</li><li>Create crawlers for all types of websites irrespective of the technical roadblocks.</li><li>Manage the crawlers to overcome technical challenges like IP ban, geolocation ban, captcha and bot blocking services</li><li>Design scrappy pipelines to connect the crawler output to the MySQL database</li><li>Integrate the data crawled and scraped into our databases</li><li>Build and maintain high-quality reusable code</li><li>Automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability</li></ul>

Job Details

Skills Required

More Details

Additional Information

Job Closed