Ofure Ikheloa
c370de83d5
- Introduced RedisManager class in scraper.py for centralized Redis operations including job tracking and caching. - Enhanced job scraping logic in MultiPlatformJobScraper to support multiple platforms (Ashby, Lever, Greenhouse). - Updated browser initialization and context management to ensure better resource handling. - Improved error handling and logging throughout the scraping process. - Added SSL connection parameters management in a new ssl_connection.py module for RabbitMQ connections. - Refactored sender.py to utilize RedisManager for job deduplication and improved logging mechanisms. - Enhanced CSV processing logic in sender.py with better validation and error handling. - Updated connection parameters for RabbitMQ to support SSL configurations based on environment variables.
10 lines
178 B
INI
10 lines
178 B
INI
|
|
[rabbitmq]
|
|
queue_name = job_queue
|
|
|
|
[files]
|
|
directory = C:\Users\OfuRich\jobs\csv
|
|
|
|
[logging]
|
|
log_file = C:\Users\OfuRich\Documents\ai jobhub\Web_scraping_project\logs\sender.log
|